The workflow before#
Booking.com's operational data lived in MySQL and moved downstream in batch jobs. When a property changed a price or closed a room, analytics and downstream systems didn't see it until the next run. MySQL write throughput was the ceiling on how much could move, and scaling meant buying more hardware. Engineers spent a large share of their week keeping the pipelines alive: rerunning failed jobs, chasing lag, patching scripts. Peak booking seasons made it worse. The company was building its Big Data Exchange (BDX) on Snowflake and its own governance tooling, and it needed a way to feed it that could keep up.
What we built#
We replaced the batch hand-off with a change data capture pipeline. Every change in MySQL is read as it happens and streamed into BDX and Snowflake, so a price or availability update reaches analytics in under a second. The pipeline scales with volume on its own, and engineers stopped spending their weeks keeping it running.
Every change, as it happens
Debezium reads the MySQL change log and publishes each insert, update and delete to Kafka. We chose change data capture over faster batch jobs because it takes the load off the source databases and gives every downstream team the same ordered stream of changes.
Scaling that follows the traffic
Airflow orchestrates the pipelines and Spark handles the large batch analytics, all on Kubernetes with autoscaling tied to data volume. The system absorbs 3x traffic spikes without anyone stepping in, so a holiday rush is no longer an on-call event.
Data that lands where teams work
Changes land in BDX under Booking.com's own governance rules, in Snowflake for analysts and in DynamoDB for fast lookups. Each team reads from the store that fits its job instead of building another copy.
How we delivered it
A team of 8 worked embedded with Booking.com's data engineers over a 40 week engagement. We proved the architecture with a proof of concept first, load tested at 2x projected peak, then rolled out in phases. Old and new paths ran side by side with traffic shifting gradually until the numbers matched. We paired with Booking.com's engineers throughout so they owned the system at handover.
Before and after#
| Measure | Before | After |
|---|---|---|
| Price and availability propagation | Next batch run | Under 1 second |
| Pipeline throughput | Capped by MySQL writes | 10x higher |
| Engineer time on pipeline upkeep | Baseline | 70% less |
| Infrastructure cost for the pipelines | Baseline | 30% lower |
| Traffic spikes | Manual intervention | 3x absorbed automatically |
What's next#
The streaming layer gives Booking.com clean, current event data in one place. That's the footing for models and copilots that answer questions on live pricing and inventory instead of yesterday's extract, and for new pipelines built on the same pattern without starting over.
