Blue Orange + Astronomer Drive Better Outcomes for Operational Capacity

author Josh Miramant June 7, 2022

Introduction

The client is a leader in the commercial real estate industry that creates economic, social, and environmental value as a global real estate advisor. Their team provides global insight, local market expertise, and access to some of the smartest technology in the industry. 

The client's data architecture team was in the early stages of developing a platform to support the data needs of multiple teams within the company. However, there were concerns that as its user base continued to grow, the platform would be stretched beyond its operational capacity. 

Blue Orange was brought in to outline a set of recommendations for architectural and procedural improvements that would help the data architecture team provide best-of-breed data services and support a wide array of use cases across all of the client's departments. This solution architecture ended up including the Astronomer Platform for managed Airflow, containerized workloads, Azure Resource Group, and Terraform.

The Challenge

The architecture and usage patterns of the client’s existing system were designed quickly in order to achieve an operational state as soon as possible. Blue Orange determined that the system would benefit from a reevaluation in order to bring the data platform architecture and utilization in line with industry best practices.

After reviewing the existing system’s architecture, the following issues were identified:

  1. The Airflow environment was nearing its operational capacity, and resource limitations were causing performance issues.
  2. The Airflow solution didn’t have support for advanced user management, governance, or multi-tenancy. This would become an issue as the data architecture team grew and needed to support multiple independent client groups.
  3. The data architecture team did not have a documented or established approach for implementing software-development best practices for data engineering. This would also become a significant issue as the number of supported workloads increased, and would negatively impact the performance and maintainability of the environment.

The Solution

In order to address the challenges identified in the client’s data environment, Blue Orange would implement a scalable and reliable multi-tenant Airflow environment to provide flexible, general-purpose workflow orchestration services across all of their departments. This would also involve designing a set of task-specific Docker images to be hosted on Azure Container Registry and to be executed in Azure Container Instances. 

To support long-term growth for new Airflow users and business units, Blue Orange selected the Astronomer Platform to manage the Airflow deployments, using Astronomer Workspace’s fine-grained access controls to secure each deployment.

Finally, Blue Orange would document and adopt software development best-practices for data engineering to upskill the client’s internal teams, including local development with the Astronomer CLI and out-of-the-box Astronomer CI/CD workflows for both Airflow configuration and data pipelines.

Solution Architecture Diagram

Conclusion

The client was in the early stages of developing a platform to support the data needs of multiple teams within the company. However, as its user base continued to grow, the platform was going to be stretched beyond its operational capacity. 

The Blue Orange team was brought in to outline a set of recommendations for architectural and procedural improvements in a solution architecture that ended up including the Astronomer Platform for managed Airflow, containerized Dockerized workload file, Azure Resource Group, and Terraform. Using Astronomer Workspaces, the Blue Orange team was able to design a serverless, containerized approach to Airflow task execution.

If you are interested in learning more about how Blue Orange can develop a tailored solution architecture for your business’ needs, please contact our team today and schedule your free consultation. You can also request a demo of Astronomer’s data orchestration platform customized around your organization’s needs.  



logo-colored

Full-service data transformation to make it easy to get from raw data to insights.


Recent posts

Contact