This is one of 4713 IT projects that we have successfully completed with our customers.

How can we support you?

Weißes Quadrat mit umrandeten Seiten rechts oben

Data integration from web sources: scraping, storage in Snowflake, and visualization with Streamlit

Project duration: 3 months

Brief description

This project implements an automated data pipeline for extracting, storing, and visualizing web data. Open and closed source data is scraped, stored in Snowflake, and interactively visualized. The solution enables scalable and efficient analysis of web-based data sources on short notice.

Supplement

Data collection is performed using web scraping in Python, with Databricks serving as the computing environment. The collected data is stored in a Snowflake database in the curated layer of a datalake solution. The solution enables efficient further processing and analysis. Streamlit is used as a low code solution for interactive visualization of the data and it is hosted in Azure.

Subject description

Data from different sources is bundled and presented in a clear manner. This facilitates flexible data analysis and is used to make forecasts.

Overview

Project period03.03.2025 - 30.05.2025

Have we sparked your interest?

Dr. Andreas Schneider, grauhaariger Mann mit Brille

Dr. Andreas Schneider

Head of Energy

Jetzt Kontakt aufnehmen

Zum Umgang mit den hier erhobenen Daten informieren wir in unserer Datenschutzerklärung.

Contact now

We provide information on the handling of the data collected here in our privacy policy.

Download file

We provide information on the handling of the data collected here in our privacy policy.