FRAMEWORK FOR DATASTAGE
Standardization of the conception and realization of integration processes
- Project teams receive a modular best-practice solution for loading processes in DWH and data integration platforms. With reusable components, the process steps for loading data from source to target are carried out in a uniform and tested quality.
- The standardization increases the comprehensibility and maintainability of the ETL processes. The generation of technical and business metadata ensures the traceability of all job processes.
including documentation of the concept and implementation
All components of the framework are described in general terms. The exemplary implementation of functions or shared containers with IBM DataStage is explicitly documented. Flexible adaptation or extension to customer-specific requirements is thus easily possible.
Ready basis - delivery on the basis of an example project
- The components of the framework are delivered on the basis of a sample project.
- The framework can be individually adapted to the naming conventions of the project. These include:
- In DataStage: Jobs, Sequences, ParameterSets, Paramater
- In DB: schema, tables, names of technical fields
- Loading Staging
- Typed and non-typed data transfer in staging area
- Syntactic harmonization using mapping table and reusable components (shared container)
- Surrogate key
- Generate: Allocation of new Surrogate Keys
- Read: Assignment of an existing Surrogate key to a source system key
- Delta Determination and Technical Historization
- Change detection of delivered data against the target table; determination of the basic operation Insert, Update, Delete with subsequent technical historization
- Technical logging
- Logging of harmonization errors, reject records in provided table and using two shared containers
- Configuration of e-mail notification
- Technical Metadata
- Metatables for managing deliveries and production runs
- Technical logging of the runs for documentation of important parameters
- Supplying the records inserted or changed in the target table with relevant technical metadata
- ETL Standards
- Recommendations for naming conventions and other standard settings
better development speed
The framework provides the starting point for the realization of an ETL architecture. The framework is adapted to customer requirements. It lays the foundation for ensuring that recurring tasks in the data integration process are not redesigned and implemented every time. A wheel is invented only once..
The concept of the framework is based on years of experience from many projects and takes relevant aspects of functional and non-functional requirements into account. The implemented components are quality-tested and guarantee correct functionality.
The modular structure of the framework allows the targeted use of the components in existing projects. If required, the framework can be adapted to the requirements of the project.
- The application of the framework and its components reduces the design and development costs in a project while simultaneously reducing risk.
- Training times and maintenance tasks are carried out more cost-effectively.