This IT project is part of our digitalization and optimization of our customers’ IT landscape. Through targeted measures, we promote technological progress, optimize cross-system processes and create a sustainable basis for future developments. Our IT reference projects serve as a basis for orientation. They support the reusability of tried and tested concepts as part of project implementation.
This is one of 4822 IT projects that we have successfully completed with our customers.How can we support you?
Data integration from web sources: scraping, storage in Snowflake, and visualization with Streamlit
Project duration: 3 months
Brief description
This project implements an automated data pipeline for extracting, storing, and visualizing web data. Open and closed source data is scraped, stored in Snowflake, and interactively visualized. The solution enables scalable and efficient analysis of web-based data sources on short notice.
Supplement
Data collection is performed using web scraping in Python, with Databricks serving as the computing environment. The collected data is stored in a Snowflake database in the curated layer of a datalake solution. The solution enables efficient further processing and analysis. Streamlit is used as a low code solution for interactive visualization of the data and it is hosted in Azure.
Subject description
Data from different sources is bundled and presented in a clear manner. This facilitates flexible data analysis and is used to make forecasts.
IT project data
- Industries Energy trading
- Specialist tasks energy trade, forecast, Forecasting
- DP tasks customised software further development
- Procedure model customer standard
- Programming languages Python
- Databases Snowflake
- Development and test tool Azure DevOps
- Standard software and BI Databricks, Streamlit
- Middleware and frameworks Azure App Service
Project period03.03.2025 - 30.05.2025
Customers who trust us
Have we sparked your interest ?
