From Cron to Modern Data Stack (MDS): Dataflow Automation and Its Current State
The concept that makes the technological miracles of today possible are defined by data. Enormous amounts of data are collected...
In business, the needle in a haystack problem is a constant challenge. Recommendation Engines are here to help tackle that challenege.
In e-commerce and retail, you offer hundreds or thousands of products. Which is the right product for your customers?
In sales and marketing, you have a large number of prospects in your pipeline. Yet, you only have so many hours in the day. So, you face the challenge of deciding where precisely to focus your effort.
There is a specialized technology powered by AI and Big Data, which makes these challenges much easier to manage, recommendation engines.
In its simplest terms, a recommendation engine sorts through many items and predicts the selection most relevant to the user. For consumers, Amazon’s product recommendation engine is a familiar example. In the entertainment world, Netflix has worked hard to develop their engine. Netflix’s recommendation engine has delivered bottom-line benefits:
“[Netflix’s] sophisticated recommendation system and personalized user experience, it has allowed them to save $1 billion per year from service cancellations.” - The ROI of recommendation engines for marketing
From the end user’s perspective, it is often unclear how recommendation engines work. We’re going to pull the curtain back and explain how they work, starting with the key ingredient: data.
The data you need for a recommendation engine depends on your goal. Assume your goal is to increase sales in an e-commerce company. In that case, the bare minimum required data would fall into two categories: a product database and end-user behavior. To illustrate how this works, look at this simple example.
In practice, the best recommendation engines use much more data. As a general rule, recommendation engines produce better business results when they have a large volume of data to use.
Many recommendation engines use a handful of techniques to process your data.
This type of recommendation algorithm combines user preferences and attempts to recommend similar items. In this case, the engine is focused on the product and highlighting related items. This type of recommendation engine is relatively simple to build. It is a good starting point for companies with limited data.
Have you asked somebody else for a recommendation before making a purchase? Or considered online reviews in your buying process? If so, you have experienced collaborative filtering. More advanced recommendation engines analyze user reviews, ratings, and other user-generated content to produce relevant suggestions. This type of recommendation engine strategy is powerful because it leverages social proof.
Hybrid recommendation engines combine two or more recommendation methods to produce better results. Returning to the e-commerce example outlined above, let’s say you have acquired user reviews and ratings (e.g., 1 to 5 stars) over the past year. Now, you can use both content-based filtering and collaborative filtering to present recommendations. Combining multiple recommendation engines or algorithms successfully usually takes experimentation. For that reason, it is best considered a relatively advanced strategy.
A recommendation engine is only successful if you feed it with high-quality data. It also cannot perform effectively if you have errors or out of date information in your company database. That’s why you need to invest resources in data quality continuously.
There are more than 50 applicants on average per job posting, according to Jobvite research. For human resources departments and managers, that applicant volume creates a tremendous amount of work. To simplify the process, Blue Orange implemented a recommendation engine for a fortune 500 hedge fund. This HR automation project helped the company to rank candidates in a standardized way. Using ten years' worth of applicant data and resumes, the firm now has a sophisticated scoring model to find good-fit candidates.
A Hedge Fund in New York City needed to parse resumes that were inconsistent and required OCR to improve their hiring process. Even the best OCR parsing leaves you with messy and unstructured data. Then, as a candidate moves through the application process, humans get involved. Add to the data set free form text reviews of the applicant and both linguistic and personal biases. In addition, each data source is siloed providing limited analytical opportunity.
Approach: After assessing multiple companies hiring processes, we have found three consistent opportunities to systematically improve hiring outcomes using NLP machine learning. The problem areas are: correctly structuring candidate resume data, assessing job fit, and reducing human hiring bias. With a cleaned and structured data set, we were able to perform both sentiment analysis on the text and subjectivity detection to reduce candidate bias in human assessment.
Results: Using keyword detection classifiers, optical character recognition, and cloud-based NLP engines, we were able to scrub string text and turn it into relational data. With structured data, we provided a fast, interactive, and searchable Business Analytics dashboard in AWS QuickSight.
Another example of recommendation engines being implemented in the real-world comes from Zageno. Zageno is an e-commerce company that does for lab scientists what Amazon does for the rest of us. The caveat is that the needs of lab scientists are exact so the supplies procured for their research must be, as well. The quotes below from our interview with Zageno highlight how they use recommendation engines to deliver the most accurate supplies to lab scientists.
How has your company used a recommendation engine and what sort of results did you see?
There are two examples of the recommendation engines that ZAGENO employs for its scientific customers. To explain these we felt it best to bullet point them.
Do you recommend recommendation systems?
“Yes, but make sure you are using the right data to base your recommendation on both the quality and quantity reflecting true user expectations. Create transparency because nobody, particularly scientists, will trust or rely on a black box. Share with your users which information is used, how it is weighted, and keep on learning so as to continually improve. Finally, complete the cycle by taking the user feedback that you've collected and bring it back into the system.”- Zageno
The power of recommendation engines has never been greater. As shown by giants like Amazon and Netflix, recommenders can be directly responsible for increases in revenue and customer retention rates. Companies such as Zageno show that you do not need to be a massive company to leverage the power of recommenders. The benefits of recommendation engines span across many industries such as e-commerce to human resources.
Developing a recommendation engine takes data expertise. Your internal IT team may not have the capacity to build this out. If you want to get the customer retention and efficiency benefits of recommendation engines, you don’t have to wait for IT to become less busy. Drop us a line and let us know. The Blue Orange Digital data science team is happy to make recommenders work for your benefit too!
Josh Miramant is the CEO and founder of Blue Orange Digital, a data science and machine learning agency with offices in New York City and Washington DC. Miramant is a popular speaker, futurist, and a strategic business & technology advisor to enterprise companies and startups. He is a serial entrepreneur and software engineer that has built and scaled 3 startups. He helps organizations optimize and automate their businesses, implement data-driven analytic techniques, and understand the implications of new technologies such as artificial intelligence, big data, and the Internet of Things.
Featured on IBM ThinkLeaders, Dell Technologies, and NYC’s Top 10 AI Development and Custom Software Development Agencies as reviewed on Clutch and YahooFinance for his contributions to NLP, AI, and Machine Learning. Specializing in predictive maintenance, unified data lakes, supply chain/grid/marketing/sales optimization, anomaly detection, recommendation systems, among other ML solutions for a multitude of industries.
Image source: Canva
Originally published: Unite.AI