Onboarding Phase: T&M Development & Modification of Data Pipelines for Enhanced Data Flows

Josh Miramant

Posted On:
May 23, 2022

Introduction

The client has been developing communities since 2012 and focuses on designing intuitive, easy to install, and powerful technologies that help build safer, happier communities.  They utilize innovative technologies to ensure apartment communities can thrive in the long term.

The company employed Blue Orange Digital to support its business needs in data governance, developing and modifying data pipelines for data transformation, and designing data models for ad hoc analysis. Additionally, they were considering architectural modifications to enhance data flows and allow for scalability of use and training for their team. 

The Challenge

The client was utilizing a legacy Apartment Management System along with building a new more integrated platform that would replace the existing one. They will work on the development of the new modules (micro-services) in order to migrate the legacy system, the migration of the data itself from the old data model to the new one, and the creation of a new reporting platform that should allow them to generate tabular reports to integrate into the platform.  Additionally, the creation of integrated dashboards and specific KPIs would be explored and planned.

The client’s needs for the Blue Orange Team were to provide support with analysis, development, and deployment of all the necessary tools, infrastructure, and code.  Furthermore, the Blue Orange team would support activities with: 

  • The transition from SQL to JSON reports. 
  • Building additional pipeline transformations.
  • Designing data models for ad hoc analysis. 
  • Providing best practices for architectural improvements. 
  • Data standardization and naming conventions.
  • Data governance methods based on business needs.
  • Capturing business and technical requirements for data infrastructure strategy 

The Solution

During the Onboarding Phase, Blue Orange reviewed the client’s current platform, data architecture, data sources, data structures, pipelines, and reporting systems as well as insights regarding the new platform in development. The main focus was to understand the usage and features of the legacy system, to have a better understanding of the client’s modeling, reporting, and dashboard needs.  Then, the team would be able to detail an initial backlog of reports that would be delivered and re-built for consumption into the new platform.

The solution would utilize some of the existing services (such as Amorphic and Redshift) and would propose dbt as a development framework to create and update the new data model for reporting and dashboards. The replication process would be evaluated in order to define the best approach to achieve a real-time replication.

The proposed elements for the project architecture would include:

  • PostgreSQL as the main transactional data source with information being replicated into Redshift. 
  • Data extraction and loading from PostgreSQL into the Redshift Datawarehouse would be performed by Amorphic.
  • Modeling within the data warehouse would be orchestrated by dbt, which would be managed by Amorphic.
  • Mapping and documenting tables, columns, and reports from the legacy system would be done to design and develop the new data models or data cubes using dbt.
  • Reports will consume data warehouse information through views, stored procedures, or direct query.

Conclusion

The client sought support from Blue Orange Digital for their business needs in data governance, developing and modifying data pipelines for data transformation, and designing data models for ad hoc analysis. 

Utilizing the structured Onboarding Phase, Blue Orange would be able to review the client’s current platform, data architecture, data sources, data structures, pipelines, and reporting systems as well as insights regarding the new platform in development. This would enable the client to make architectural modifications to enhance data flows, allow for scalability of use, and bolstered knowledge and training for their team. 

Blue Orange’s phased Onboarding approach ensures that the client’s team would be  provided with a defined scope and path forward for migrating data from their legacy system to their new data environment. This highly curated process is applicable across industries and sectors and if your business requires these consulting services contact our team today to discuss your project!  If you are interested in reading further about our additional client success stories, you can view our full Case Studies listing here.

See where smart data management can take your business