Project Overview
A non-profit organization sought to reduce its technical debt, better understand the organization's KPIs, and migrate end users away from a legacy Microsoft Access database platform that was acting as double data entry with a SaaS product for dozens of staff members.
While on staff at the organization, and afterwards as a contractor, John architected and developed a new data platform that restored confidence in the organization's KPIs and eliminated the double data entry.
Project details
- Project type: Data Engineering, Data analysis, Web Development
- Technology: Python, Flask, Apache Superset, Dagster, Data Build Tool (DBT), PostgreSQL, SQL, Pandas, JavaScript, HTML, SASS/CSS
The Challenge
The Nonprofit's operations team was stuck in a loop of duplicative data entry and unclear reporting. Their legacy Microsoft Access database no longer met their needs - but it remained embedded in internal workflows due to the difficulty of transitioning away. Meanwhile, their primary SaaS tool did not offer the kind of flexible reporting or data exports they needed to confidently track KPIs or report to stakeholders.
Further, the unmaintainable legacy reporting system and double data entry dynamics had led to a loss of accuracy in reports and confidence in organizational KPIs, requiring painstaking manual intervention by management staff to report accurate KPIs.
The organization needed a solution that would respect existing workflows while steadily modernizing their data infrastructure. They also needed reporting tools that non-technical staff could easily use.
The Solution
We delivered a lightweight, user-friendly data platform tailored to the organization's needs. The core components were:
- A custom upload portal built with Flask, where end users could submit Excel exports from their SaaS platform without needing technical help.
- A modern data pipeline using Dagster, Pandas, and DBT to clean, validate, and transform incoming data into a centralized warehouse.
- Flexible, visual reporting via Apache Superset dashboards and pre-defined Excel reports, giving the team access to the KPIs they cared about most.
This approach allowed the organization to phase out the legacy Access system without disrupting daily operations. Instead of enforcing a sudden shift, we integrated with their existing processes and quietly moved them toward a more reliable and scalable backend.
Results
- Time savings: End users no longer needed to perform manual double-entry into two systems.
- Improved visibility: Leadership gained access to reliable, dynamic dashboards and KPI reports.
- Reduced risk: The organization began retiring its fragile Access database in favor of a modern and maintainable infrastructure.
- Scalable foundation: The platform was built to grow - capable of ingesting new data sources or supporting automation as their needs evolve.
Tech solution snapshot
- Python & Flask: Built a simple web portal for uploading and downloading data exports.
- Dagster: Automated and orchestrated data workflows to ensure clean, reliable data pipelines.
- DBT (Data Build Tool): Applied tested transformations to create a trusted, consistent reporting layer with clear documentation for data warehouse users.
- Pandas: Processed and validated data for accuracy during the pipeline.
- Apache Superset: Delivered interactive dashboards and visual reports for business users.
This modern stack replaced fragile manual processes with automated, transparent, and easy-to-use tools for leaders and staff members alike.