From Cron to Modern Data Stack (MDS): Dataflow Automation and Its Current State
The concept that makes the technological miracles of today possible are defined by data. Enormous amounts of data are collected...
A leading Private Equity and Venture Capital firm, with over $30B under management, requested an end-to-end data audit of their deal platform. They were looking to evolve the platform to make a scalable and unified sourcing tool with consistent architecture and infrastructure. They wanted an independent third-party to assess the technical decisions made to date in the development.
The firm needed help integrating 4 newly acquired CRM/ERP companies. Each acquired company had its own databases, in its own format. A lack of visibility into the sales process of these siloed data systems hindered coordination, planning, and tracking. The blind sales department had no concise data to direct their time and resources. Due to disparate data sets, the company had no insight into the efficacy of their upper funnel engagement or attribution across their sales cycle. As a result, they had a low conversion rate on sales efforts.
The company’s existing development team had no resources for an internally focused, stand-alone project so they hired Blue Orange. The goal was to provide architectural guidance on their data infrastructure to support unified data and sales optimization.
A complete project assessment was provided with a budget for the AWS data lake transition and the cost-benefit analysis of these improvements to the sales projections with the new system.
Vertical: Big Data Processing, Sales Optimization
Stage: AWS Data Lake
PHASE 1: Data Transformation: Created a scalable architecture that confidently could handle all data-driven operational growth and would not be outpaced by all the input.
PHASE 2: Data Visualization and Integration: Improved sales modeling and oversight with real-time, full-funnel dashboards.
PHASE 3: Predictive Analytics and Automation: Increased top of funnel conversion using ML prediction to improve lead segmentation. Quick results. They needed to solve their critical problems quickly and then add complexity later.
PHASE 4: Improve Portfolio Company Performance: As an investor, offering real-time data science as a service to your portfolio companies, you will improve their performance and thus your investment.
After Blue Orange concluded the audit of the previous system and presented a budget plan for a complete revamp, we secured a development contract. Implementation began by partnering with the existing development team to improve all the identified flaws in the old process. The goal was to develop a best practice, production application with a modern AWS data architecture that will augment decision making for all levels of users in the system.
Blue Orange designed a secure, high-performing, resilient, and efficient infrastructure for the new Deal Sourcing Platform. Based on five pillars of the AWS Well-Architected Framework — operational excellence, security, reliability, performance efficiency, and cost optimization.
This newly implemented architecture will scale over time with a pay-as-you-go payment model. AWS’s unique, auto-scaling consumption model enables you to pay only for the computing resources you consume and increase or decrease usage depending on business requirements without elaborate forecasting.
The most important improvement of Phase 1 is the creation of a unified source. As opposed to siloed data, unified data means storing all of an organization’s data in its raw form, regardless of the data source. Both structured and unstructured data are stored in a single location and become available for predictive analytics. Compiling all the data into a single unified source erases the limits of rows and columns to search for trends and associations with real-time data.
The implications for sales and marketing teams are non-negligible: unified data provides a single access point to ever-increasing amounts of customer information. Marketing and sales automation tools, CRM systems, and campaign monitoring platforms all become connected at the data layer. When user-generated information (web & mobile activity, reviews, social media feeds, sensor data, etc.) is added to the mix, we can say that unified data provides the full 360-degree view.
There are two pathways to bring data science and analytics capabilities to your firm. First, you can adopt the “build” approach – hire a whole department of specialists in data. This approach can work! However, it is slow and expensive to build such a department, especially if it is outside of your firm’s core competency.
The second choice: partner with a data science firm like Blue Orange. With this approach, you get data science expertise for your portfolio firms as needed. Contact us to take a closer look at how Blue Orange makes both of these wins possible.
See the next phases of this Private Equity Firm's data transformation journey.
Schedule 15-min with Blue Orange Digital to discuss which option is right for your data sources and future goals.
Josh Miramant is the CEO and founder of Blue Orange Digital, a data science and machine learning agency with offices in New York City and Washington DC.
Miramant is a popular speaker, futurist, and a strategic business & technology advisor to enterprise companies and startups. As an example of thought leadership, Miramant has been featured in IBM ThinkLeaders, Dell Technologies, Global Banking & Finance Review, the IoT Council of Europe, among others. He can be reached at firstname.lastname@example.org.
Blue Orange Digital is recognized as a “Top AI Development and Consultant Agency,” by Clutch and YahooFinance, for innovations in predictive analytics, automation, and optimization with machine learning in NYC.
They help organizations optimize and automate their businesses, implement data-driven analytic techniques, and understand the implications of new technologies such as artificial intelligence, big data, and the Internet of Things.