The workflow before#
TEAM Software builds workforce and operations software, and its Power BI reports drew on data in an AWS RDS application database. Getting that data into reporting was a manual job. On every refresh someone exported tables to CSV, checked them and imported them into the reporting tools by hand. Every step was a chance for a missed file, a renamed column or a half-finished load, and when a number looked wrong the first suspect was the export. The process kept internal reports running, but TEAM wanted to offer analytics to its own customers, and nobody was going to build a customer-facing service on a person copying files.
What we built#
We built TEAM Software an automated pipeline in Microsoft Fabric that moves reporting data from the application database on a schedule, with no one exporting files. Data lands raw, gets cleaned, and ends up as the aggregates the reports read. The reports look exactly the same to the people using them. What changed is that nobody has to feed them.
Incremental loads from the app database
The pipeline reads about 20 high-value tables from AWS RDS on a schedule and moves only the rows that changed since the last run. We started with the tables the reports depend on most rather than copying the whole database, so the first cutover covered what mattered and stayed small enough to test properly.
Three layers with one job each
Raw data lands in a Bronze layer exactly as it came out of the source. Silver cleans it and standardizes types. Gold holds the aggregates Power BI reads. Keeping raw data untouched means that when a figure looks off, an engineer can trace it back to the source row instead of guessing which export went wrong.
Cutover with matching numbers
Before retiring the manual process we ran the automated feed alongside it and compared outputs with historical reporting data until the numbers matched exactly. Existing Power BI reports kept working with no change for users. We set exact parity as the bar because a report that's close but different would have cost more trust than the manual process ever did.
How we delivered it
We worked with TEAM's engineering and reporting staff to pick the tables, build the layers and run the parallel tests. Once parity held, the CSV process was switched off. TEAM's team owns the Fabric workspace and can add tables to the same pattern.
Before and after#
| Measure | Before | After |
|---|---|---|
| Data movement to reporting | Manual CSV export and import | Scheduled incremental pipeline |
| Tables automated | 0 | About 20 |
| Refresh effort | Hands-on work every refresh | None |
| Report accuracy at cutover | Dependent on manual steps | Exact parity with historical reports |
What's next#
The Gold layer is the base for the analytics TEAM wants to offer its own customers, built on data that refreshes itself. It's also the clean, consistent data a reporting assistant would need to answer customers' questions about their operations in plain language.
