This is one of 4713 IT projects that we have successfully completed with our customers.How can we support you?
Data integration from web sources: scraping, storage in Snowflake, and visualization with Streamlit
Project duration: 3 months
Brief description
This project implements an automated data pipeline for extracting, storing, and visualizing web data. Open and closed source data is scraped, stored in Snowflake, and interactively visualized. The solution enables scalable and efficient analysis of web-based data sources on short notice.
Supplement
Data collection is performed using web scraping in Python, with Databricks serving as the computing environment. The collected data is stored in a Snowflake database in the curated layer of a datalake solution. The solution enables efficient further processing and analysis. Streamlit is used as a low code solution for interactive visualization of the data and it is hosted in Azure.
Subject description
Data from different sources is bundled and presented in a clear manner. This facilitates flexible data analysis and is used to make forecasts.
Overview
- Industries Energy trading
- Specialist tasks energy trade, forecast, Forecasting
- DV tasks customised software further development
- Procedure model customer standard
- Programming language Python
- Databases Snowflake
- Development/ test tools Azure DevOps
- Standard software Databricks, Streamlit
Project period03.03.2025 - 30.05.2025
Have we sparked your interest?
Contact
PTA GmbH Head Office
Weberstraße 2-4
D-68165 Mannheim
Service
Industries
© Copyright 2025 PTA GmbH | All rights reserved | Imprint | Privacy | Legal notice | Values & Code of Conduct
