In business, the needle in a haystack problem is a constant challenge. Recommendation Engines are here to help tackle that challenege. 

In e-commerce and retail, you offer hundreds or thousands of products. Which is the right product for your customers?

In sales and marketing, you have a large number of prospects in your pipeline. Yet, you only have so many hours in the day. So, you face the challenge of deciding where precisely to focus your effort.

There is a specialized technology powered by AI and Big Data, which makes these challenges much easier to manage, recommendation engines.

What are recommender systems?

In its simplest terms, a recommendation engine sorts through many items and predicts the selection most relevant to the user. For consumers, Amazon’s product recommendation engine is a familiar example. In the entertainment world, Netflix has worked hard to develop their engine. Netflix’s recommendation engine has delivered bottom-line benefits:

“[Netflix’s] sophisticated recommendation system and personalized user experience, it has allowed them to save $1 billion per year from service cancellations.” - The ROI of recommendation engines for marketing

From the end user’s perspective, it is often unclear how recommendation engines work. We’re going to pull the curtain back and explain how they work, starting with the key ingredient: data.

Recommendation Engines: What data do they use?

The data you need for a recommendation engine depends on your goal. Assume your goal is to increase sales in an e-commerce company. In that case, the bare minimum required data would fall into two categories: a product database and end-user behavior. To illustrate how this works, look at this simple example.

• Company: USB Accessories, Inc. The company specializes in selling USB accessories and products like cables, thumb drives, and hubs to consumers and businesses.
• Product Data. To keep the initial recommendation engine simple, the company limits it to 100 products.
• User Data. In the case of an online store, user data will include website analytics information, email marketing, and other sources. For instance, you may find that 50% of customers who buy an external hard drive also buy USB cables.
• Recommendation Output. In this case, your recommendation engine may generate a recommendation (or a discount code) to hard drive buyers to encourage them to buy USB cables.

In practice, the best recommendation engines use much more data. As a general rule, recommendation engines produce better business results when they have a large volume of data to use.

How do recommendation engines use your data?

Many recommendation engines use a handful of techniques to process your data.

Content-based filtering

This type of recommendation algorithm combines user preferences and attempts to recommend similar items. In this case, the engine is focused on the product and highlighting related items. This type of recommendation engine is relatively simple to build. It is a good starting point for companies with limited data.

Collaborative filtering

Have you asked somebody else for a recommendation before making a purchase? Or considered online reviews in your buying process? If so, you have experienced collaborative filtering. More advanced recommendation engines analyze user reviews, ratings, and other user-generated content to produce relevant suggestions. This type of recommendation engine strategy is powerful because it leverages social proof.

Hybrid recommenders

Hybrid recommendation engines combine two or more recommendation methods to produce better results. Returning to the e-commerce example outlined above, let’s say you have acquired user reviews and ratings (e.g., 1 to 5 stars) over the past year. Now, you can use both content-based filtering and collaborative filtering to present recommendations. Combining multiple recommendation engines or algorithms successfully usually takes experimentation. For that reason, it is best considered a relatively advanced strategy.

A recommendation engine is only successful if you feed it with high-quality data. It also cannot perform effectively if you have errors or out of date information in your company database. That’s why you need to invest resources in data quality continuously.

Case Studies: 

Hiring Automated: Recommended Candidates

There are more than 50 applicants on average per job posting, according to Jobvite research. For human resources departments and managers, that applicant volume creates a tremendous amount of work. To simplify the process, Blue Orange implemented a recommendation engine for a fortune 500 hedge fund. This HR automation project helped the company to rank candidates in a standardized way. Using ten years' worth of applicant data and resumes, the firm now has a sophisticated scoring model to find good-fit candidates.

A Hedge Fund in New York City needed to parse resumes that were inconsistent and required OCR to improve their hiring process. Even the best OCR parsing leaves you with messy and unstructured data. Then, as a candidate moves through the application process, humans get involved. Add to the data set free form text reviews of the applicant and both linguistic and personal biases. In addition, each data source is siloed providing limited analytical opportunity.

Approach: After assessing multiple companies hiring processes, we have found three consistent opportunities to systematically improve hiring outcomes using NLP machine learning. The problem areas are: correctly structuring candidate resume data, assessing job fit, and reducing human hiring bias. With a cleaned and structured data set, we were able to perform both sentiment analysis on the text and subjectivity detection to reduce candidate bias in human assessment.

Results: Using keyword detection classifiers, optical character recognition, and cloud-based NLP engines, we were able to scrub string text and turn it into relational data. With structured data, we provided a fast, interactive, and searchable Business Analytics dashboard in AWS QuickSight.

E-Commerce: Zageno Medical Supplies

Another example of recommendation engines being implemented in the real-world comes from Zageno. Zageno is an e-commerce company that does for lab scientists what Amazon does for the rest of us. The caveat is that the needs of lab scientists are exact so the supplies procured for their research must be, as well. The quotes below from our interview with Zageno highlight how they use recommendation engines to deliver the most accurate supplies to lab scientists. 

Q&A: Blue Orange Digital interviews Zageno

Question:

How has your company used a recommendation engine and what sort of results did you see?

Answer:

There are two examples of the recommendation engines that ZAGENO employs for its scientific customers. To explain these we felt it best to bullet point them.

• ZAGENO's Scientific Score:
  • ZAGENO’s Scientific Score is a comprehensive product rating system, specifically developed for evaluating research products. It incorporates several aspects of product data, from multiple sources, to equip scientists with a sophisticated and unbiased product rating for making accurate purchasing decisions.
  • We apply sophisticated machine-learning algorithms to accurately match, group, and categorize millions of products. The Scientific Score accounts for these categorizations, as each product’s score is calculated relative to those in the same category. The result is a rating system that scientists can trust — one that is specific to both product application and product type.
  • Standard product ratings are useful to assess products quickly, but are often biased and unreliable, due to their reliance on unknown reviews or a single metric (e.g. publications). They also provide little detail on experimental context or application. The Scientific Score utilizes a scientific methodology to objectively and comprehensively evaluate research products. It combines all necessary and relevant product information into a single 0—10 rating to support our customers in deciding which product to buy and use for their application — saving hours of product research.
  • To ensure no single factor dominates, we add cut-off points and give more weight to recent contributions. The sheer number of factors we take into account virtually eliminates any opportunity for manipulation. As a result, our score is an objective measure of the quality and quantity of available product information, which supports our customers’ purchasing decisions.
• Alternative Products:
  • Alternative products are defined by the same values for key attributes; key attributes are defined for each category to account for specific product characteristics.
  • We are working on increasing underlying data and attributes and improving the algorithm to improve the suggestions
  • Alternatives product suggestions are intended to help both, scientist and procurement to consider and evaluate potential products, they might not have considered/known otherwise
  • Alternative products are solely defined by product characteristics  and independent of suppliers, brand or other commercial data

Do you recommend recommendation systems? 

“Yes, but make sure you are using the right data to base your recommendation on both the quality and quantity reflecting true user expectations. Create transparency because nobody, particularly scientists, will trust or rely on a black box. Share with your users which information is used, how it is weighted, and keep on learning so as to continually improve. Finally, complete the cycle by taking the user feedback that you've collected and bring it back into the system.” 

- Zageno

The power of recommendation engines has never been greater. As shown by giants like Amazon and Netflix, recommenders can be directly responsible for increases in revenue and customer retention rates. Companies such as Zageno show that you do not need to be a massive company to leverage the power of recommenders. The benefits of recommendation engines span across many industries such as e-commerce to human resources. 

The Fast Way To Bring Recommendation Engines To Your Company

Developing a recommendation engine takes data expertise. Your internal IT team may not have the capacity to build this out. If you want to get the customer retention and efficiency benefits of recommendation engines, you don’t have to wait for IT to become less busy. Drop us a line and let us know. The Blue Orange Digital data science team is happy to make recommenders work for your benefit too!

For more on AI and technology trends, see Josh Miramant, CEO of Blue Orange Digital’s data-driven solutions for Supply Chain, Healthcare Document Automation, and more.

Follow me on Twitter or LinkedIn. Check out my website. 

Josh Miramant is the CEO and founder of Blue Orange Digital, a data science and machine learning agency with offices in New York City and Washington DC. Miramant is a popular speaker, futurist, and a strategic business & technology advisor to enterprise companies and startups. He is a serial entrepreneur and software engineer that has built and scaled 3 startups. He helps organizations optimize and automate their businesses, implement data-driven analytic techniques, and understand the implications of new technologies such as artificial intelligence, big data, and the Internet of Things. 

Featured on IBM ThinkLeaders, Dell Technologies, and NYC’s Top 10 AI Development and Custom Software Development Agencies as reviewed on Clutch and YahooFinance for his contributions to NLP, AI, and Machine Learning. Specializing in predictive maintenance, unified data lakes, supply chain/grid/marketing/sales optimization, anomaly detection, recommendation systems, among other ML solutions for a multitude of industries.  

Visit BlueOrange.digital for more information and Case Studies.

Image source: Canva

Originally published: Unite.AI