The workflow before#
Betterworks sells OKR and performance management software, and its larger customers wanted real-time analytics inside the product. The data stack was custom-built on AWS: every new source meant weeks of pipeline code, queries slowed as customer data grew, and engineers spent a large share of their time keeping pipelines alive instead of shipping features. Adding a data source took about 2 weeks; a new pipeline took about 3.
What we built#
We moved Betterworks from building data infrastructure to buying it. Fivetran now handles ingestion with managed connectors and schema drift, Snowflake is the warehouse, and dbt builds the analytics models inside Snowflake.
Looker sits on top with a LookML semantic layer, so every metric has one definition, and its embedded analytics power the customer-facing dashboards in the Betterworks app. Airflow orchestrates the jobs, Terraform defines the infrastructure, and GitHub Actions runs CI/CD.
We also stood up an Azure Machine Learning workspace connected to Snowflake, with a feature store and deployment pipeline, ready for predictive goal and customer success models. The engagement ran 24 weeks.
Before and after#
| Measure | Before | After |
|---|---|---|
| Add a new data source | 2 weeks | 1 day |
| Build a new pipeline | 3 weeks | 5 days |
| Analytics feature delivery | 8 weeks | 3 weeks |
| Engineering time on data maintenance | Baseline | 50% less |
What's next#
Customer data is now modeled in one Snowflake warehouse with an ML workspace attached. The next step is putting L2 features in front of customers: predictions of which goals are at risk, surfaced inside the product.
