Rule-Based Sentiment Analysis in Python

Introduction

According to experts, 80% of the world’s data is unstructured (images, videos, text, etc). This could be generated by Social media tweets/posts, call transcripts, survey or interview reviews, text across blogs, forums, news, etc.

Reading all the text across the web and finding patterns is humanly impossible. Yet, there is a need for the business to analyze this data for better actions.

One such process of drawing insights from textual data is Sentiment Analysis. To obtain the data for sentiment analysis, one can directly scrape the content from the web pages using different web scraping techniques.

Learning Objectives

1. Understand the importance of sentiment analysis in processing unstructured textual data and its applications in various fields such as market analysis, social media monitoring, customer feedback analysis, and market research.
2. Differentiate between rule-based and machine learning-based approaches in sentiment analysis, focusing on the rule-based (lexicon-based) approach.
3. Gain practical knowledge of the rule-based approach by implementing TextBlob, VADER, and SentiWordNet for sentiment analysis in Python.
4. Learn essential data preprocessing steps for text analysis, including cleaning text, tokenization, enrichment (POS tagging), stopwords removal, and obtaining stem words.
5. Explore and compare sentiment analysis results using different lexicon-based techniques (TextBlob, VADER, SentiWordNet) through visualizations and understand the significance of choosing the right approach for specific applications.

This article was published as a part of the Data Science Blogathon.

What is Sentiment Analysis?

Sentiment Analysis (also known as opinion mining or emotion AI) is a sub-field of NLP that measures the inclination of people’s opinions (Positive/Negative/Neutral) within the unstructured text.

Sentiment Analysis can be performed using two approaches: Rule-based and Machine Learning based.

Few applications of Sentiment Analysis Algorithms

• Market analysis
• Social media monitoring
• Customer feedback analysis – Brand sentiment or reputation analysis
• Market research

What is Natural Language Processing(NLP)?

Natural language is the way humans communicate with each other. It could be Speech or Text. NLP is the automatic manipulation of the natural language by software. NLP is a higher-level term combining Natural Language Understanding (NLU) and Natural Language Generation  (NLG).

NLP = NLU + NLG

Some of the Python Natural Language Processing (NLP) libraries are:

• Natural Language Toolkit (NLTK)
• TextBlob
• SpaCy
• Gensim
• CoreNLP

I hope we have a basic understanding of the terms Sentiment Analysis and NLP.

This article focuses on the Rule-based approach of Sentiment Analysis

Rule-based approach

This is a practical approach to analyzing text without training or using machine learning models. This approach results in rules based on which the text is labeled as positive/negative/neutral. These rules are also known as lexicons. Hence, the Rule-based approach is called the Lexicon-based approach.

Widely used lexicon-based approaches are TextBlob, VADER, and SentiWordNet.

Data preprocessing steps:

1. Cleaning the text
2. Tokenization
3. Enrichment – POS tagging
4. Stopwords removal
5. Obtaining the stem words

Before deep-diving into the above steps, lemme import the text data from a txt file.

Importing a text file using Pandas read CSV function.

Python Code

# install and import pandas library
import pandas as pd
# Creating a pandas dataframe from reviews.txt file
data = pd.read_csv('reviews.txt', sep='\t')
print(data.head())

# Data can be different as it is extracted from the website.

This doesn’t look cool. So, we will now drop the “Unnamed: 0″ column using the df.drop function.

mydata = data.drop('Unnamed: 0', axis=1)
mydata.head()

Our dataset has a total of 240 observations(reviews). 

Step 1: Cleaning the text

In this step, we need to remove the special characters, and numbers from the text. We can use the regular expression operations library of Python.

# Define a function to clean the text
def clean(text):
# Removes all special characters and numericals leaving the alphabets
    text = re.sub('[^A-Za-z]+', ' ', text)
    return text

# Cleaning the text in the review column
mydata['Cleaned Reviews'] = mydata['review'].apply(clean)
mydata.head()

Explanation:  “clean” is the function that takes text as input and returns the text without any punctuation marks or numbers in it. We applied it to the ‘review’ column and created a new column, ‘Cleaned Reviews,’ with the cleaned text. 

Great, look at the above image; all the special characters and the numbers are removed.

Step 2: Tokenization

Tokenization is the process of breaking the text into smaller pieces called Tokens. It can be performed at sentences(sentence tokenization) or word level(word tokenization).

I will be performing word-level tokenization using nltk tokenize function word_tokenize().

Note: As our text data is a little large, first I will illustrate steps 2-5 with small example sentences.

Let’s say we have a sentence: “This is an article on Sentiment Analysis“. It can be broken down into small pieces(tokens), as shown below.

Step 3: Enrichment – POS tagging

Parts of Speech (POS) tagging converts each token into a tuple with the form (word, tag). POS tagging is essential to preserve the word’s context and Lemmatization.

This can be achieved by using the nltk pos_tag function. 

Below shown are the POS tags of the example sentence “This is an article on Sentiment Analysis”.

Check out the list of possible POS tags from here.

Step 4: Stopwords removal

Stopwords in English are words that carry very little useful information. We need to remove them as part of text preprocessing. nltk has a list of stopwords of every language. 

See the stopwords in the English language.

Example of removing stopwords:

The stopwords This, is, an, on are removed and the output sentence is ‘article Sentiment Analysis’.

Step 5: Obtaining the stem words

A stem is a part of a word responsible for its lexical meaning. The two popular techniques of obtaining the root/stem words are Stemming and lematization.

The key difference is Stemming often gives some meaningless root words as it simply chops off some characters in the end. Lemmatization gives meaningful root words. However, it requires POS tags for the words.

Example to illustrate the difference between Stemming and Lemmatization: Click here for code.

If we look at the above example, the output from Stemming is Stem, and the output from Lemmatization is Lemma.

For the word glanced, the stem glanc is meaningless. Whereas the Lemma glance is perfect.

We now understood steps 2-5 by taking simple examples. Without any further delay, let us bounce back to our actual problem.

Code for Steps 2-4: Tokenization, POS tagging, Stopwords removal

import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize
from nltk import pos_tag
nltk.download('stopwords')
from nltk.corpus import stopwords
nltk.download('wordnet')
from nltk.corpus import wordnet

# POS tagger dictionary
pos_dict = {'J':wordnet.ADJ, 'V':wordnet.VERB, 'N':wordnet.NOUN, 'R':wordnet.ADV}Explore the significance of Sentiment Analysis in extracting insights from diverse textual sources for informed business decisions.





def token_stop_pos(text):
    tags = pos_tag(word_tokenize(text))
    newlist = []
    for word, tag in tags:
        if word.lower() not in set(stopwords.words('english')):
        newlist.append(tuple([word, pos_dict.get(tag[0])]))
    return newlist

mydata['POS tagged'] = mydata['Cleaned Reviews'].apply(token_stop_pos)
mydata.head()

Explanation: token_stop_pos is the function that takes the text and performs tokenization, removes stopwords, and tags the words to their POS. We applied it to the ‘Cleaned Reviews’ column and created a new column for ‘POS tagged’ data.

As mentioned earlier, to obtain the accurate Lemma the WordNetLemmatizer requires POS tags in the form of ‘n’, ‘a’, etc. But the POS tags obtained from pos_tag are in the form of ‘NN’, ‘ADJ’, etc.

To map pos_tag to wordnet tags,  we created a dictionary pos_dict. Any pos_tag that starts with J is mapped to wordnet.ADJ, any pos_tag that starts with R is mapped to wordnet.ADV, and so on.

Our tags of interest are Noun, Adjective, Adverb, Verb. Anything out of these four is mapped to None.

In the above fig, we can observe that each word of column ‘POS tagged’ is mapped to its POS from pos_dict.

Code for Step 5: Obtaining the stem words – Lemmatization

from nltk.stem import WordNetLemmatizer
wordnet_lemmatizer = WordNetLemmatizer()
def lemmatize(pos_data):
    lemma_rew = " "
    for word, pos in pos_data:
    if not pos:
        lemma = word
        lemma_rew = lemma_rew + " " + lemma
    else:
        lemma = wordnet_lemmatizer.lemmatize(word, pos=pos)
        lemma_rew = lemma_rew + " " + lemma
    return lemma_rew

mydata['Lemma'] = mydata['POS tagged'].apply(lemmatize)
mydata.head()

Explanation: lemmatize is a function that takes pos_tag tuples, and gives the Lemma for each word in pos_tag based on the pos of that word. We applied it to the ‘POS tagged’ column and created a column ‘Lemma’ to store the output.

Yay, after a long journey, we are done with the text preprocessing.

Now, take a minute to look at the ‘review’, ‘Lemma’ columns and observe how the text is processed.

Our final data looks clean as we are done with the data preprocessing. Take a short break and return to continue with the real task.

Also read: Starters Guide to Sentiment Analysis using Natural Language Processing

Sentiment Analysis using TextBlob

TextBlob is a Python library for processing textual data. It provides a consistent API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, and more.

The two measures that are used to analyze the sentiment are:

• Polarity – talks about how positive or negative the opinion is
• Subjectivity – talks about how subjective the opinion is

TextBlob(text).sentiment gives us the Polarity, Subjectivity values.
Polarity ranges from -1 to 1 (1 is more positive, 0 is neutral, -1 is more negative)
Subjectivity ranges from 0 to 1(0 being very objective and 1 being very subjective)

Example of TextBlob sentiment

Python Code

from textblob import TextBlob
# function to calculate subjectivity
def getSubjectivity(review):
    return TextBlob(review).sentiment.subjectivity
    # function to calculate polarity
    def getPolarity(review):
        return TextBlob(review).sentiment.polarity

# function to analyze the reviews
def analysis(score):
    if score < 0:
        return 'Negative'
    elif score == 0:
        return 'Neutral'
    else:
        return 'Positive'

Explanation: Functions were created to obtain polarity and subjectivity values and to label the review based on the polarity score.

Creating a new data frame with the review and Lemma columns and applying the above functions

fin_data = pd.DataFrame(mydata[['review', 'Lemma']])
# fin_data['Subjectivity'] = fin_data['Lemma'].apply(getSubjectivity) 
fin_data['Polarity'] = fin_data['Lemma'].apply(getPolarity) 
fin_data['Analysis'] = fin_data['Polarity'].apply(analysis)
fin_data.head()

Count the number of positive, negative, neutral reviews.

tb_counts = fin_data.Analysis.value_counts()
tb_counts

Count of positive, negative, neutral reviews

Sentiment Analysis using VADER

VADER stands for Valence Aware Dictionary and Sentiment Reasoner. Vader sentiment not only tells if the statement is positive or negative along with the intensity of emotion.

The sum of pos, neg, neu intensities give 1. Compound ranges from -1 to 1 and is the metric used to draw the overall sentiment.
positive if compound >= 0.5
neutral if -0.5 < compound < 0.5
negative if -0.5 >= compound

Python Code

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()
# function to calculate vader sentiment
def vadersentimentanalysis(review):
    vs = analyzer.polarity_scores(review)
    return vs['compound']
    fin_data['Vader Sentiment'] = fin_data['Lemma'].apply(vadersentimentanalysis)
# function to analyse
def vader_analysis(compound):
    if compound >= 0.5:
        return 'Positive'
    elif compound <= -0.5 :
        return 'Negative'
    else:
        return 'Neutral'
fin_data['Vader Analysis'] = fin_data['Vader Sentiment'].apply(vader_analysis)
fin_data.head()

Explanation: Created functions to obtain the Vader scores and to label the reviews based on compound scores

Count the number of positive, negative, neutral reviews.

vader_counts = fin_data['Vader Analysis'].value_counts()
vader_counts

Sentiment Analysis using SentiWordNet

SentiWordNet uses the WordNet database. It is important to obtain the POS, lemma of each word. We will then use the lemma, POS to obtain the synonym sets(synsets). We then obtain the positive, negative, and objective scores for all the possible synsets or the very first synset and label the text.

if positive score > negative score, the sentiment is positive
if positive score < negative score, the sentiment is negative
if positive score = negative score, the sentiment is neutral

Python Code

nltk.download('sentiwordnet')
from nltk.corpus import sentiwordnet as swn
def sentiwordnetanalysis(pos_data):
    sentiment = 0
    tokens_count = 0
    for word, pos in pos_data:
        if not pos:
            continue
            lemma = wordnet_lemmatizer.lemmatize(word, pos=pos)
        if not lemma:
            continue
            synsets = wordnet.synsets(lemma, pos=pos)
        if not synsets:
            continue
            # Take the first sense, the most common
            synset = synsets[0]
            swn_synset = swn.senti_synset(synset.name())
            sentiment += swn_synset.pos_score() - swn_synset.neg_score()
            tokens_count += 1
            # print(swn_synset.pos_score(),swn_synset.neg_score(),swn_synset.obj_score())
        if not tokens_count:
            return 0
        if sentiment>0:
            return "Positive"
        if sentiment==0:
            return "Neutral"
        else:
            return "Negative"

fin_data['SWN analysis'] = mydata['POS tagged'].apply(sentiwordnetanalysis)
fin_data.head()

Explanation: We created a function to obtain the positive and negative scores for the first word of the synset then label the text by calculating the sentiment as the difference of positive and negative scores.

Count the number of positive, negative, neutral reviews.

swn_counts= fin_data['SWN analysis'].value_counts()
swn_counts

We have seen the implementation of sentiment analysis using some of the popular lexicon-based techniques. Now, quickly do some visualization and compare the results.

Visual representation of TextBlob, VADER, SentiWordNet results.

We will plot the count of positive, negative, and neutral reviews for all three techniques.

import matplotlib.pyplot as plt
%matplotlib inline
plt.figure(figsize=(15,7))
plt.subplot(1,3,1)
plt.title("TextBlob results")
plt.pie(tb_counts.values, labels = tb_counts.index, explode = (0, 0, 0.25), autopct='%1.1f%%', shadow=False)
plt.subplot(1,3,2)
plt.title("VADER results")
plt.pie(vader_counts.values, labels = vader_counts.index, explode = (0, 0, 0.25), autopct='%1.1f%%', shadow=False)
plt.subplot(1,3,3)
plt.title("SentiWordNet results")
plt.pie(swn_counts.values, labels = swn_counts.index, explode = (0, 0, 0.25), autopct='%1.1f%%', shadow=False)

If we observe the above image, TextBlob and SentiWordNet results look slightly close, while the VADER results show a large variation.

Conclusion

In conclusion, rule-based sentiment analysis (text analysis) exemplifies the convergence of algorithms and machine learning techniques in deciphering sentiments from text data, which is crucial in data science and artificial intelligence. By analyzing product reviews on Amazon or customer feedback, these techniques highlight the significance of understanding positive and negative sentiments for data-driven decision-making.

Using classifiers like Naive Bayes and neural networks through deep learning demonstrates how machine learning algorithms efficiently tackle sentiment classification, turning unstructured text into meaningful sentiment scores. This process showcases the capability of sentiment analysis models to process online reviews and news articles and emphasizes the importance of machine learning in extracting nuanced insights from text.

Ultimately, sentiment analysis is a powerful tool in the arsenal of data science, enabling a deeper understanding of public opinion across various domains. Integrating machine learning with natural language processing will enhance our ability to analyze and interpret vast amounts of daily text data as technology evolves.

Key Takeaways

1. Sentiment Analysis Overview: The blog provides an overview of sentiment analysis, emphasizing its importance in analyzing unstructured data such as social media posts, call transcripts, and reviews for applications like market analysis and customer feedback analysis.
2. NLP and Sentiment Analysis Libraries: It introduces Natural Language Processing (NLP) and popular Python libraries like NLTK, TextBlob, SpaCy, Gensim, and CoreNLP for sentiment analysis, highlighting their role in understanding and processing natural language.
3. Rule-Based Sentiment Analysis: The focus is on the Rule-Based Sentiment Analysis, specifically the lexicon-based approach, using tools like TextBlob, VADER, and SentiWordNet. The blog details the data preprocessing steps, including cleaning text, tokenization, POS tagging, stopword removal, and obtaining stem words.
4. Implementation of Sentiment Analysis: It walks through the implementation of sentiment analysis using TextBlob, VADER, and SentiWordNet. Each method shows the steps of analysis, labeling reviews as positive, negative, or neutral, and counting the occurrences of each sentiment.
5. Comparison and Visualization: The blog concludes by comparing results from TextBlob, VADER, and SentiWordNet. Visual representations, like pie charts, are used to compare the count of positive, negative, and neutral reviews for each sentiment analysis technique. The conclusion emphasizes the role of sentiment analysis in data science and artificial intelligence.

Frequently Asked Questions

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Harika

Hi, my name is Harika. I am a Data Engineer and I thrive on creating innovative solutions and improving user experiences. My passion lies in leveraging data to drive innovation and create meaningful impact.