Why Consider a Hadoop to Databricks Lakehouse Migration?
Hadoop offers the option to maintain huge on-prem workloads but enterprises need to migrate this data into cloud-based managed services...
The client is a leader in the commercial real estate industry that creates economic, social, and environmental value as a global real estate advisor. Their team provides global insight, local market expertise, and access to some of the smartest technology in the industry.
The client's data architecture team was in the early stages of developing a platform to support the data needs of multiple teams within the company. However, there were concerns that as its user base continued to grow, the platform would be stretched beyond its operational capacity.
Blue Orange was brought in to outline a set of recommendations for architectural and procedural improvements that would help the data architecture team provide best-of-breed data services and support a wide array of use cases across all of the client's departments. This solution architecture ended up including the Astronomer Platform for managed Airflow, containerized workloads, Azure Resource Group, and Terraform.
The architecture and usage patterns of the client’s existing system were designed quickly in order to achieve an operational state as soon as possible. Blue Orange determined that the system would benefit from a reevaluation in order to bring the data platform architecture and utilization in line with industry best practices.
After reviewing the existing system’s architecture, the following issues were identified:
In order to address the challenges identified in the client’s data environment, Blue Orange would implement a scalable and reliable multi-tenant Airflow environment to provide flexible, general-purpose workflow orchestration services across all of their departments. This would also involve designing a set of task-specific Docker images to be hosted on Azure Container Registry and to be executed in Azure Container Instances.
To support long-term growth for new Airflow users and business units, Blue Orange selected the Astronomer Platform to manage the Airflow deployments, using Astronomer Workspace’s fine-grained access controls to secure each deployment.
Finally, Blue Orange would document and adopt software development best-practices for data engineering to upskill the client’s internal teams, including local development with the Astronomer CLI and out-of-the-box Astronomer CI/CD workflows for both Airflow configuration and data pipelines.
The client was in the early stages of developing a platform to support the data needs of multiple teams within the company. However, as its user base continued to grow, the platform was going to be stretched beyond its operational capacity.
The Blue Orange team was brought in to outline a set of recommendations for architectural and procedural improvements in a solution architecture that ended up including the Astronomer Platform for managed Airflow, containerized Dockerized workload file, Azure Resource Group, and Terraform. Using Astronomer Workspaces, the Blue Orange team was able to design a serverless, containerized approach to Airflow task execution.
If you are interested in learning more about how Blue Orange can develop a tailored solution architecture for your business’ needs, please contact our team today and schedule your free consultation. You can also request a demo of Astronomer’s data orchestration platform customized around your organization’s needs.