data pipeline enhancement

Data Pipeline Enhancement for a Multi-Billion Dollar Hedge Fund

Josh Miramant

Posted On:
May 23, 2022

Introduction

The client invests in small-and mid-cap companies in both public and private markets. They reached out to Blue Orange to migrate their data and enhance their data pipeline to support the investment team  with advanced analytics and dashboards.

Their legacy data infrastructure lacked cohesive integration with their cloud environment so Blue Orange created an 8-week implementation to add multiple data streams to the existing cloud environment in a Snowflake warehouse using AWS where appropriate.  The resulting end-to-end data pipeline flowed into more comprehensive dashboards that were informative, comparative, and devoid of bias so that the business could easily derive pressing information from the data.

The Challenge

The client was using Revelio Labs and Sensor Tower in conjunction with Tableau as their data infrastructure; however, this environment was hindering the addition of other data streams to create multiple reports in their BI tool.  Blue Orange launched a two-week Sprint review to identify tables for ingestion, transformation, and connection to Tableau.

After the Blue Orange team was given access to all data sources and tools, they were able to create a list of recommendations for the project implementation which included:

  • Mirror/migrate Revelio Labs and Sensor Tower data
  • Solutions Architect would design a solution to meet all technical and business requirements
  • Project Manager and Data Engineer would implement design solution
  • Connect SnowFlake to Tableau for dashboarding
  • Develop Initial Proof-of-Concept dashboards in Tableau
  • Ongoing monitoring and support to meet SLA

The Solution

The Blue Orange team designed a solution that would incorporate existing Revelio Labs and Sensor Tower data into a Snowflake data warehouse that would yield business intelligence dashboards in Tableau.  To accomplish this, the Blue Orange team took the following actions:

  • Create ingestion pipeline for Consumer Edge
  • Provision and integrate Alteryx, Snowflake, & AWS to create an end-to-end data pipeline.
  • Model a utilized Consumer Edge data in Snowflake data warehouse
  • Connect SnowFlake to Tableau for dashboarding
  • Develop Initial Proof-of-Concept dashboards in Tableau

The resulting integration between AWS, Snowflake, and Tableau was able to demonstrate consistent reliability so that the client’s Analytics Team would be able to depend on the system’s architecture.  Throughout the effort, Blue Orange’s data engineers, solutions architects, quality assurance, and project management professionals maintained consistent workflow with the existing data team to keep processes smooth and uninterrupted; thereby, maintaining a track record of accomplishments and facilitation of decisions for upcoming requirements.

Conclusion

The client’s legacy data infrastructure lacked cohesive integration with their cloud environment and data streams. Blue Orange experts were able to design an end-to-end data pipeline for more comprehensive dashboards that were informative, comparative, and devoid of bias so that business intelligence could easily be derived from the data. 

The client’s internal data team was able to derive enhanced insights on desirable capital fund opportunities for their small and mid-cap targets that range in value from $100M to $1B.  Additionally, their internal data team was able to visualize and report on the status of current investments in their portfolio. If your hedge fund or private equity firm is lacking effective data transformation, contact our experts today to leverage Modern Data Stack solutions for your team!

See where smart data management can take your business