When is building better than buying an off-the-shelf solution?
Companies can engage in different approaches to model development. From fully managed ML services, all the way to custom models. Depending on business requirements, available expertise, and planning constraints, they must make a choice: should they develop custom solutions from scratch? Or should they choose an off-the-shelf service?
For all stages of ML workloads, a decision must be met concerning how the different puzzle pieces will fit together. From data collection, preparation, and visualization, all the way to feature engineering, model training, and evaluation, machine learning engineers repeatedly ask themselves the same question: Will it be a custom implemented solution, written and developed from scratch? Or will it be an off-the-shelf service?
But when is building better than buying an off-the-shelf solution? The main differentiating factors between the two approaches: preprocessing efforts, development speed, and the required expertise.
Things to consider when determining to use off-the-shelf or custom machine learning models?
Preprocessing efforts
ML projects are facing all kinds of challenges, but perhaps the biggest challenge is the availability of training data. The lack of training data can stop a project before it even starts. Before a project even starts, it can face significant preprocessing costs from collecting data, labeling data, cleaning, and preprocessing efforts. This is the well-known trap in which many ML projects fail: preprocessing ends up taking 80% of the resources allocated, whereas few resources are left for the actual model training and evaluation.
Off-the-shelf solutions alleviate the strains and pains of preprocessing efforts. They are built to perform the most common operations with only a little configuration required. The best thing about them is: off-the-shelf solutions exist for all stages of ML workloads.
On the other hand, custom-made implementations usually require more preprocessing efforts. That does not mean that they need to be dismissed entirely: they are still required to finetune a certain ML stage to the specifics of the problem being solved. An especially dirty dataset may require some special kind of cleanup rules. At the same time, a specific feature set may require custom feature engineering, just as neural architectures may require slight adjustments. In this case, custom solutions built from scratch are likely to cover all needs.
Development speed
Off-the-shelf solutions focus on configuration rather than implementation. Instead of allocating resources to figuring out what should be done, ML teams will focus on how the different puzzle pieces will fit together. This approach allows companies, researchers, and engineers to quickly implement prototypes and proofs of concept. Instead of reinventing the wheel, off-the-shelf solutions make it possible to leverage existing knowledge, thus saving development time.
Custom-made solutions implemented from scratch are known to be much slower wrt development speed. This is due to their increased maintenance needs: engineers must figure out both the what and the how of the solution. Likewise, the more complex the solution, the more time resources are required to ensure its scalability and availability whilst in production. From this perspective, custom-made solutions and time efforts are directly proportional: the more complex a solution, the more time it will require.
Usually, however, the truth is somewhere in the middle: an existing codebase will be refactored and adapted to the current project’s needs. Such is the case of the well-known transfer learning approach to model training.
Expertise
Just as there are multiple layers at which Machine Learning is done, there are multiple levels of expertise at which ML models can be developed, ranging from code-free interfaces all the way to building models from scratch.
Off-the-shelf solutions exist for which very little machine learning expertise is required. By utilizing intuitive interfaces and even drag and drop approaches, it has become extremely simple for anyone (from business analysts to software engineers) to build and deploy some kind of machine learning model. While this simple approach to model development may work for prototyping purposes, it is unlikely to meet the requirements of production systems.
Expertise is still required to properly configure, set up, and maintain off-the-shelf solutions in production. Workarounds, code-patches, connecting to different API interfaces, and dealing with deployment issues are common tasks required to ensure models’ performance in production environments.
Custom-made solutions are usually implemented at an infrastructural level and there is no way around it: expertise is definitely required. Depending on the company size and project goals, multidisciplinary teams may be required for maintaining production systems. Data scientists, ML engineers, and business analysts come together to make sense of inference results and maintain production models.
What should you use: an off-the-shelf or custom machine learning model?
An ML solution will be built of many individual components and services which need to come together as a cohesive solution. It is never about going 100% custom or going 100% off-the-shelf since different business problems require different solutions. More often than not, ML-based solutions are built by a mix of the two: off-the-shelf services to extract general insights, combined with custom models for increased accuracy and modeling domain-specific knowledge.
The trick is knowing when to implement custom solutions from scratch and which parts of the project can leverage the benefits of off-the-shelf services. This vastly depends on the type of problem being tackled, the business requirements, the data available, and the overall constraints of the development environment.
For more on AI and technology trends, see Josh Miramant, CEO of Blue Orange Digital’s data-driven solutions for Supply Chain, Healthcare Document Automation, and more.
You also might like:
Use NLP to classify comments on social Media
How Language Processing is Being Enhanced Through Google’s Open Source BERT Model
Originally Posted: https://www.unite.ai/off-the-shelf-vs-custom-machine-learning-models/
