The aim of the project is to monitor all ETL or ELT processes within the framework of data management of an Enterprise Data Warehouse (EDWH). In terms of quality assurance for both the development and the operation of an EDWH, the integration processes are to be made transparent in order to identify sources of error at an early stage. The PTA designs and implements dashboards in order to visualise selected metrics across components in a clear form according to the drill-down principle.
PTA identifies relevant or appropriate metrics to monitor the ETL or ELT processes. For this purpose, the metadata is extracted from the Oracle Repository Database of the ETL/ELT tool Oracle Data Integrator (12c) at regular intervals and imported as metrics into the designated timeseries database InfluxDB. Analogous to the existing data warehouse architecture in the form of a Business Object Model (BOM), the PTA creates dedicated Grafana dashboards based on the metrics for all components. In addition, each component is visualised as a tile in an overview dashboard, which shows the current status based on the metrics and defined threshold values in green, yellow or red. The detailed dashboard of the respective component or sub-components can be called up at any time via drill-down.
The monitoring stack used, based on Grafana and InfluxDB, enables a clear presentation of all components and ETL/ETL processes in accordance with the specified architecture of the data warehouse (DWH). In addition, the metrics can be analysed at different levels of detail (component/subcomponent) over time in order to identify anomalies.