The workflow before
Booking.com's operational data lived in MySQL and moved downstream in batch jobs. Write throughput was the ceiling, so pricing and availability data reached analytics and downstream systems late. Scaling meant buying more hardware, and engineers spent a large share of their week keeping the pipelines running instead of shipping features.
What we built
We built a change data capture pipeline that reads every MySQL change with Debezium and streams it through Kafka into Booking.com's Big Data Exchange (BDX) and Snowflake. Price and availability changes now propagate in under a second instead of waiting for the next batch.
Airflow orchestrates the pipelines and Spark handles the large batch analytics, all running on Kubernetes (Amazon EKS) with autoscaling tied to data volume. The system absorbs 3x traffic spikes without anyone stepping in.
We ran a 40 week engagement with a team of 8, load tested at 2x projected peak, and rolled out in phases with traffic shifting so the old and new paths ran side by side until the numbers matched.
Before and after
| Measure | Before | After |
|---|---|---|
| Price and availability propagation | Next batch run | Under 1 second |
| Pipeline throughput | Capped by MySQL writes | 10x higher |
| Engineer time on pipeline upkeep | Baseline | 70% less |
| Infrastructure cost for the pipelines | Baseline | 30% lower |
What's next
The streaming layer gives Booking.com clean, current event data in one place. That is the foundation for L2 work: models and copilots that answer questions on live pricing and inventory instead of yesterday's extract.
