Building a Modern Financial Data Pipeline: My Journey from Excel to dbt

Have you ever worked with financial data trapped in Excel files? After facing these challenges in my professional roles at MediaMarktSaturn and Flix SE, I decided to build a solution – and that's how this project was born.

My Approach: Building a Modern Data Stack

1. Data Generation: Python script to generate realistic synthetic financial transaction data.
2. Database Setup: PostgreSQL (Docker) for raw data storage.
3. Data Transformation: dbt to transform raw data into clean, analysis-ready models.
4. Testing & Documentation: Data quality tests and lineage documentation.
5. Visualization: Streamlit and Plotly for interactive financial dashboards.

Key Technologies Used

Python | Pandas | SQLAlchemy | PostgreSQL | Docker | dbt | Streamlit | Plotly

View GitHub Repository →