The workflow before
Suvretta's daily data runs were built on Python scripts and dbt models orchestrated by Airflow into Snowflake. Many steps assumed someone would notice a problem and fix it by hand: rerunning a job, patching a load, or correcting an input. That made the pipeline fragile and tied up the data team every morning.
What we built
We audited the Python and dbt codebases end to end and mapped every point where a person had to intervene. We then rewrote the code to remove those dependencies: making loads safe to rerun, adding tests and checks in dbt so bad inputs stop the run with a clear error, and letting Airflow retry and recover on its own.
The result is a codebase the fund's team can run unattended on most days, with exceptions surfaced clearly instead of discovered late.
Before and after
| Measure | Before | After |
|---|---|---|
| Manual interventions in daily runs | Routine, several per week | About half as many |
| Failure handling | Found and rerun by hand | Automatic retries, clear alerts on the rest |
What's next
A dependable daily pipeline into Snowflake is the base for Level 2 work, such as giving investment and operations staff reliable, queryable data they can ask questions of directly.
