From Cron to Modern Data Stack (MDS): Dataflow Automation and Its Current State
The concept that makes the technological miracles of today possible are defined by data. Enormous amounts of data are collected...
Sentiment analysis and natural language processing (NLP) of social media is a proven way to draw insight from people and society. Instead of asking an analyst to spend weeks reading social media comments and providing a report, sentiment analysis can give you a quick summary. That means you can make decisions faster.
You’re living in the age of big data. Take social media users as an example. In 2019, there were 3.4 billion active social media users in the world. On YouTube alone, one billion hours of video content are watched daily. Every indicator suggests that we will see more data produced over time, not less.
There is simply too much data for you to review manually. Even organizations with large budgets like national governments and global corporations are using data analysis tools, algorithms, and natural language processing.
By using these techniques, you can understand what people are saying about your brand right now. The ability to minimize selection bias and avoid relying on anecdotes mean your decisions will have a firm foundation. That means you will make fewer mistakes as you react to a rapidly changing world.
You might be wondering if these data analysis tools are useful in the real world or if they are reliable to use. These tools have been around for over a decade, and they are getting better every year. With NLP and sentiment analysis, you can solve problems faster.
In hiring, finding quality candidates is tough. Workopolis estimates that “as many as 75% of applicants for a given role aren’t actually qualified to do it.” Spending time on those candidates is not productive. Fortunately, natural language processing and analytics can help you identify good-fit candidates so that you can use time productively. That’s why Blue Orange Digital worked with a hedge fund to optimize their human resources process. Using ten years’ worth of applicant data and resumes, the firm now has a sophisticated scoring model to find good-fit candidates.
In 2020, we’ve all started to learn the value of large scale public health data analysis due to the rapid spread of COVID. In these crises, detecting changes in social behavior quickly is essential. With NLP, you can analyze social media to evaluate sentiment. For example, a recent project analyzed over 1,000 tweets using the keyword masks to understand how people are thinking and feeling about masks.
In marketing, you need to stay informed about how your target market thinks and feels. A 2019 study used Twitter sentiment analysis to understand clothing brands: Nike and Adidas better. Analyzing 30,895 English language tweets, the researchers found, “Adidas has more positive sentiment than Nike.” However, over 50% of tweets had a neutral sentiment. That means there is still a significant opportunity to earn more positive mentions from the marketplace.
Likes are the new currency, NLP in social media
For sentiment analysis to work effectively, there are a few essential technical points to keep in mind.
Decide what questions you want to answer and whether these data techniques are a good fit for those questions. Let’s consider two marketing questions
The first question concerns strategy and future possibilities, so there will not be much data to analyze. Therefore, we would suggest not attempting to answer this question with sentiment analysis. In contrast, question two is more promising for natural language processing. It still requires further refinement, but you have the start of an appropriate question.
Your next step is to find a relevant data source to analyze. Ideally, look for data sources that you already have rather than creating something new. For hiring, you probably have a database of applicants and successful hires in your applicant tracking system. In marketing, you can download data from social media platforms using APIs.
Tip: Data volume is vital for sentiment analysis to work. As a rule of thumb, your data set should have at least 1,000 examples (e.g., 1,000 tweets or 1,000 applicant profiles). Anything less than that, and you are less likely to obtain statistically meaningful results.
Most data sources, especially social media, and user-generated content, require pre-processing before you can work with it. Assuming you are analyzing a text resource, start by removing unnecessary punctuation, characters, and other cleaning text. Spending time on this step will improve the quality of the resulting analysis.
Since more extensive data sets tend to produce better results, use tools to clean the data further. For example, the Porter Stemmer Algorithm is a helpful way to clean up text data. This algorithm helps to identify root words and cut down on noise in your data.
Depending on your goals, there are different software tools and algorithms available to analyze the data. Assuming you are analyzing text, the Naïve Bayes algorithm is the right choice to conduct sentiment analysis.
You cannot merely accept the data analysis generated by machines uncritically. Researchers have found that machine learning tools tend to reflect human bias. For example, Amazon scrapped a human resources algorithm because it discriminated against female candidates. After all, historical data, in this case, was mainly based on men. That’s where your values – like a commitment to inclusion and diversity – need to balance data-driven insights.
This also applies to the outputs yielded by search engines. KISSPatent CEO D’vorah Graeser provides an example of how NLP is improving their search engines results when analyzing information from the World Intellectual Property Organization
“Using NLP is especially relevant and useful when trying to look for patents for new technologies such as blockchain or Artificial Intelligence, which don’t have defined categories in the World Intellectual Property Organization, for example. Being able to search and find patents is important to all innovators because that way they can know who’s working on certain innovations and if their innovations are as unique and new as they think.”
KISSPatent CEO, D’vorah Graeser
On its own, sentiment analysis will not change your business. You need to review those insights and make a decision. For example, you may find that you have a growing amount of negative sentiment about your brand online. In that case, you might start a research project to identify customer concerns and then release an improved version of your product.
Finding the right data, applying algorithms to that data, and getting usable business insights isn’t easy. After all, large companies with deep resources have made mistakes in their natural language processing projects. That’s why it pays to get an outside perspective on your data. Contact Blue Orange Digital today to find out how you can get faster insights from social media and other data in your organization.