This IT project is part of our digitalization and optimization of our customers’ IT landscape. Through targeted measures, we promote technological progress, optimize cross-system processes and create a sustainable basis for future developments. Our IT reference projects serve as a basis for orientation. They support the reusability of tried and tested concepts as part of project implementation.
This is one of 4879 IT projects that we have successfully completed with our customers.How can we support you?
AI-assisted development of a data warehouse with a BI frontend
Brief description
A company-wide data warehouse and business intelligence platform is being rebuilt for a client. Data from multiple source systems (TMS) is regularly imported as file exports, harmonized within a multi-tier data model, and made available as key metrics in a BI front end. The platform runs entirely on an open-source stack: PostgreSQL as the database, dbt Core for transformations, Apache Airflow for orchestration, and Apache Superset as the BI front end, operated as a containerized Docker Compose stack. In addition to technical implementation, the scope of work includes gathering requirements from the business units and defining the key metrics. The implementation is being carried out as an MVP with a clearly defined initial set of key metrics.
Supplement
The data model is implemented using a Medallion architecture: a raw data layer (Bronze), a layer for harmonizing and cleansing data from the various source systems (Silver), and an analytics layer (Gold) that provides the key metrics for reporting. Transformations, historical data management (SCD Type 2), and data integrity tests are implemented using dbt. Apache Airflow is used for the scheduled, automated loading of the layers and the execution of dbt tests. In the Apache Superset BI front end, dashboards, visualizations, and KPIs are implemented using views, among other methods. Access control is implemented using RLS on two levels: first, in the PostgreSQL database, and second, in Apache Superset. All implementations and configurations of the tools and services are fully versioned using Git. Documentation of data provenance is generated automatically. The entire platform is containerized and can be set up reproducibly in just a few steps.
Subject description
The reports are currently based on separate source systems and manually maintained tables. The data warehouse creates a unified basis for key performance indicators: Every figure can be traced back to its source field, and the calculation rules are available as code and are automatically verified. The separation into raw data, harmonization, and analysis layers makes it possible to integrate additional source systems and metrics without altering existing analyses. Since the corporate group consists of several independent companies, the solution provides not only company-specific access control but also a consolidated view at the holding company level, enabling a joint analysis of all companies. Operating the system in the company´s own data center using open-source components avoids licensing costs and reduces vendor lock-in. Since the platform is available entirely as code, it is reproducible and can be maintained and expanded after handover.
IT project data
- Industries Logistics
- Specialist tasks artificial intelligence, business intelligence, data analysis, data warehouse, transportation
- DP tasks Data Warehouse
- System environments Artificial Intelligence (AI), Business Intelligence, host systems
- Procedure model Kanban
- Programming languages Bash, Python, SQL
- Databases PostgreSQL
- Development and test tool DBeaver, Git
- Operating systems Linux
- Standard software and BI Apache Superset, MS PowerBI
- System software Docker
- Middleware and frameworks dbt (data build tool)
Project start01.04.2026
Customers who trust us
Have we sparked your interest ?
