Introduction
The client is an OKR (Objective and Key Results) software startup that improves employee engagement and project success by aligning people, strategy, and results. The company was looking to re-architect their data platform to meet the requirements of a new segment of large enterprise customers. The client planned to expand their platform capabilities to compete with other vendors offering continuous performance management.
Their technical goals included improved backend performance as well as the ability to provide additional insights via client dashboards and advanced analytics. To achieve their goals, they developed a multi-year roadmap that detailed their goals for building a new data platform in order to enable robust analytics and establish the foundation for data science and machine learning initiatives. Blue Orange was brought in for a short-term consulting engagement to provide an assessment of the current roadmap and guidance around architectural improvements to reduce cost and time to market.
The Challenge
The client had planned a future architecture through an AWS centric data platform. They also required a Machine Learning Architecture to help their Data Science Team in building, iterating, and deploying models quickly with little to no Devops support. In addition to providing NLP based recommendations to their clients, the company assumed that these projects would be zero maintenance and would require no support post-completion within the aggresive timelines.
This planned future architecture would work well to support the product roadmap; however, Blue Orange’s assessment identified potential risks to the timeline:
- Build Time: The client’s complex services and data flows could overload the development environment as the number of users increased over time.
- Serviceability: The suggested service level agreement called for less downtime than the cloud service provider. To meet this requirement, the client would have to add expensive redundancy that would outweigh the benefits.
- Data Governance: There was a lack of governance structure that would have been a significant blocker for an aggressive development timeframe.
The Solution
Based on the potential risks the Blue Orange Audit Team identified with the timeline, they were able to create recommendations that would address the data flows, service time, and governance structure. To accelerate the development roadmap, Blue Orange proposed adopting a buy (not build) strategy, using SaaS products.
The solution was designed around the following considerations:
- Prioritizing Time-to-Market and Quality of Components to provide agility.
- Minimal setup,configuration, and ongoing maintenance for serviceability.
- Implementing a self-log for user activities in every component.
- Coordinating tradeoffs between cheaper PaaS with more setup time to start running and slightly more expensive SaaS with shorter setup, configuration.
- No migration from legacy systems.
In order to implement these considerations, the Blue Orange Audit Team created a listing of suggested data analytics and cloud tools that would be the best fit for their company goals. These tools included:
- Fivetran as the Data Migration Platform.
- This platform would allow 1-click configuration to ingest several sources to a central database and also transform the data in parallel with the data integration pipelines.
- Snowflake as the Database.
- This database would provide a zero maintenance structure that would store the transformed data from Fivetran with an unlimited capacity in space and processing power. Furthermore, it allows for control over accessing processing units separately to prevent performance impacts. Snowflake’s robust data sharing capabilities could also be used in the future to give customers access to their data directly.
- Looker for Data Visualization.
- This tool would enable embedded analytics for the client allowing users to create business models, reports, and dashboards. Then users can interact with the models manually by dragging and dropping both dimensions and measures to build their own reports and dashboards. Looker also enables embedded reports and filters them using URL configuration. Additionally, it can be used as a framework for interactively working with the data, creating formularies, and applying data dictionary features to help understand the data model.
- Azure as the Machine Learning Platform.
- This platform provides a one-click zero maintenance data science and machine learning solution to allow users to create, train, test, and deploy models online behind a REST API. It automatically documents the service and model to generate code samples ready to use for consumption. Users can then employ both the designer (code-free machine learning development) or notebooks (Python or R).
Conclusion
The modified architecture proposed by the Blue Orange Audit Team was aimed at reducing the time to market while taking into consideration team capabilities and proposing a buy (not build) strategy using SaaS products. We recommended sprinting to market on the backs of best-in-class SaaS products to allow the features and capability requirements to be validated quickly.
These capabilities and the increased speed to market would provide a distinct competitive advantage for the client, while reducing configuration and maintenance overhead for the company. After validation, the client’s Data Science Team can focus on optimizing the performance and cost of these solutions to support scaling over the coming years.
To learn more about Blue Orange’s Machine Learning and Data Transformation capabilities, please contact our team. If you are interested in reading more about our custom client solutions, you can view our full Case Studies listing here.
