Natural Language Processing for Beginners: Using TextBlob

Introduction

Natural Language Processing (NLP) is an area of growing attention due to increasing number of applications like chatbots, machine translation etc. In some ways, the entire revolution of intelligent machines in based on the ability to understand and interact with humans.

I have been exploring NLP for some time now.  My journey started with NLTK library in Python, which was the recommended library to get started at that time. NLTK is a perfect library for education and research, it becomes very heavy and tedious for completing even the simple tasks.

Later, I got introduced to TextBlob, which is built on the shoulders of NLTK and Pattern. A big advantage of this is, it is easy to learn and offers a lot of features like sentiment analysis, pos-tagging, noun phrase extraction, etc. It has now become my go-to library for performing NLP tasks.

On a side note, there is spacy, which is widely recognized as one of the powerful and advanced library used to implement NLP tasks. But having encountered both spacy and TextBlob, I would still suggest TextBlob to a beginner due to its simple interface.

If it is your first step in NLP, TextBlob is the perfect library for you to get hands-on with. The best way to go through this article is to follow along with the code and perform the tasks yourself. So let’s get started!

Note : This article does not narrate NLP tasks in depth. If you want to revise the basics and come back here, you can always go through this article.

Table of Contents

1. About TextBlob?
2. Setting up the System
3. Having a go at NLP tasks using TextBlob
   1. Tokenization
   2. Noun phrase extraction
   3. POS-Tagging
   4. Words inflection and lemmatization
   5. N-grams
   6. Sentiment Analysis
4. Other cool things to do with TextBlob
   1. Spelling correction
   2. Creating a short summary of a text
   3. Translation and language detection
5. Text classification using TextBlob
6. Pros and Cons
7. End notes

1. About TextBlob?

TextBlob is a python library and offers a simple API to access its methods and perform basic NLP tasks. 

A good thing about TextBlob is that they are just like python strings. So, you can transform and play with it same like we did in python. Below, I have shown you below some basic tasks. Don’t worry about the syntax, it is just to give you an intuition about how much-related TextBlob is to Python strings.

So, to perform these things on your own let’s quickly install and start coding.

2. Setting up the System

Installation of TextBlob in your system in a simple task, all you need to do is open anaconda prompt ( or terminal if using Mac OS or Ubuntu) and enter the following commands:

pip install -U textblob

This will install TextBlob. For the uninitiated – practical work in Natural Language Processing typically uses large bodies of linguistic data, or corpora. To download the necessary corpora, you can run the following command

python -m textblob.download_corpora

3. NLP tasks using TextBlob

3.1 Tokenization

Tokenization refers to dividing text or a sentence into a sequence of tokens, which roughly correspond to “words”. This is one of the basic tasks of NLP. To do this using TextBlob, follow the two steps:

1. Create a textblob object and pass a string with it.
2. Call functions of textblob in order to do a specific task.

So, let’s quickly create a textblob object to play with.

from textblob import TextBlob

blob = TextBlob("Analytics Vidhya is a great platform to learn data science. \n It helps community through blogs, hackathons, discussions,etc.")

3.2 Noun Phrase Extraction

Since we extracted the words in the previous section, instead of that we can just extract out the noun phrases from the textblob. Noun Phrase extraction is particularly important when you want to analyze the “who” in a sentence. Lets see an example below.

blob = TextBlob("Analytics Vidhya is a great platform to learn data science.")
for np in blob.noun_phrases:
 print (np)
>> analytics vidhya
great platform
data science

As we can see that the results aren’t perfectly correct, but we should be aware that we are working with machines.

3.3 Part-of-speech Tagging

Part-of-speech tagging or grammatical tagging is a method to mark words present in a text on the basis of its definition and context. In simple words, it tells whether a word is a noun, or an adjective, or a verb, etc. This is just a complete version of noun phrase extraction, where we want to find all the the parts of speech in a sentence.

Let’s check the tags of our textblob.

for words, tag in blob.tags:
 print (words, tag)
>> Analytics NNS
Vidhya NNP
is VBZ
a DT
great JJ
platform NN
to TO
learn VB
data NNS
science NN

Here, NN represents a noun, DT represents as a determiner, etc. You can check the full list of tags from here to know more.

3.4 Words Inflection and Lemmatization

Inflection is a process of word formation in which characters are added to the base form of a word to express grammatical meanings. Word inflection in TextBlob is very simple, i.e., the words we tokenized from a textblob can be easily changed into singular or plural.

blob = TextBlob("Analytics Vidhya is a great platform to learn data science. \n It helps community through blogs, hackathons, discussions,etc.")
print (blob.sentences[1].words[1])
print (blob.sentences[1].words[1].singularize())

>> helps
help

TextBlob library also offers an in-build object known as Word. We just need to create a word object and then apply a function directly to it as shown below.

from textblob import Word
w = Word('Platform')
w.pluralize()
>>'Platforms'

We can also use the tags to inflect a particular type of words as shown below.

## using tags
for word,pos in blob.tags:
 if pos == 'NN':
 print (word.pluralize())
>> platforms
sciences

Words can be lemmatized using the lemmatize function.

## lemmatization
w = Word('running')
w.lemmatize("v") ## v here represents verb
>> 'run'

3.5 N-grams

A combination of multiple words together are called N-Grams. N grams (N > 1) are generally more informative as compared to words, and can be used as features for language modelling.  N-grams can be easily accessed in TextBlob using the ngrams function, which returns a tuple of n successive words.

for ngram in blob.ngrams(2):
print (ngram)
>> ['Analytics', 'Vidhya']
['Vidhya', 'is']
['is', 'a']
['a', 'great']
['great', 'platform']
['platform', 'to']
['to', 'learn']
['learn', 'data']
['data', 'science']

3.6 Sentiment Analysis

Sentiment analysis is basically the process of determining the attitude or the emotion of the writer, i.e., whether it is positive or negative or neutral.

The sentiment function of textblob returns two properties, polarity, and subjectivity.

Polarity is float which lies in the range of [-1,1] where 1 means positive statement and -1 means a negative statement. Subjective sentences generally refer to personal opinion, emotion or judgment whereas objective refers to factual information. Subjectivity is also a float which lies in the range of [0,1].

Let’s check the sentiment of our blob.

print (blob)
blob.sentiment
>> Analytics Vidhya is a great platform to learn data science.
Sentiment(polarity=0.8, subjectivity=0.75)

We can see that polarity is 0.8, which means that the statement is positive and 0.75 subjectivity refers that mostly it is a public opinion and not a factual information.

4. Other cool things to do

4.1 Spelling Correction

Spelling correction is a cool feature which TextBlob offers, we can be accessed using the correct function as shown below.

blob = TextBlob('Analytics Vidhya is a gret platfrm to learn data scence')
blob.correct()
>> TextBlob("Analytics Vidhya is a great platform to learn data science")

We can also check the list of suggested word and its confidence using the spellcheck function.

blob.words[4].spellcheck()
>> [('great', 0.5351351351351351),
 ('get', 0.3162162162162162),
 ('grew', 0.11216216216216217),
 ('grey', 0.026351351351351353),
 ('greet', 0.006081081081081081),
 ('fret', 0.002702702702702703),
 ('grit', 0.0006756756756756757),
 ('cret', 0.0006756756756756757)]

4.2 Creating a short summary of a text

This is a simple trick which we will be using the things we learned above. First, take a look at the code shown below and to understand yourself.

import random

blob = TextBlob('Analytics Vidhya is a thriving community for data driven industry. This platform allows \
people to know more about analytics from its articles, Q&A forum, and learning paths. Also, we help \
professionals & amateurs to sharpen their skillsets by providing a platform to participate in Hackathons.')

nouns = list()
for word, tag in blob.tags:
if tag == 'NN':
nouns.append(word.lemmatize())

print ("This text is about...")
for item in random.sample(nouns, 5):
word = Word(item)
print (word.pluralize())

>> This text is about...
communities
platforms
forums
platforms
industries

Simple, Ain’t it? What we did above that we extracted out a list of nouns from the text to give a general idea to the reader about the things the text is related to.

4.3 Translation and Language Detection

Can you guess what is written in the next line?

Haha! Can you guess which language is this? Don’t worry, let’s detect it using textblob…

blob.detect_language()
>> 'ar'

So, it is Arabic. Now, let’s find translate it into English so that we can know what is written using TextBlob.

blob.translate(from_lang='ar', to ='en')
>> TextBlob("that's cool")

Even if you don’t explicitly define the source language, TextBlob will automatically detect the language and translate into the desired language.

blob.translate(to= 'en') ## or you can directly do like this
>> TextBlob("that's cool")

This is seriously so cool!!! 😀

5. Text classification using TextBlob

Let’s build a simple text classification model using TextBlob. For this, first, we need to prepare a training and testing data.

training = [
('Tom Holland is a terrible spiderman.','pos'),
('a terrible Javert (Russell Crowe) ruined Les Miserables for me...','pos'),
('The Dark Knight Rises is the greatest superhero movie ever!','neg'),
('Fantastic Four should have never been made.','pos'),
('Wes Anderson is my favorite director!','neg'),
('Captain America 2 is pretty awesome.','neg'),
('Let\s pretend "Batman and Robin" never happened..','pos'),
]
testing = [
('Superman was never an interesting character.','pos'),
('Fantastic Mr Fox is an awesome film!','neg'),
('Dragonball Evolution is simply terrible!!','pos')
]

Textblob provides in-build classifiers module to create a custom classifier. So, let’s quickly import it and create a basic classifier.

from textblob import classifiers
classifier = classifiers.NaiveBayesClassifier(training)

As you can see above, we have passed the training data into the classifier.

Note that here we have used Naive Bayes classifier, but TextBlob also offers Decision tree classifier which is as shown below.

## decision tree classifier
dt_classifier = classifiers.DecisionTreeClassifier(training)

Now, let’s check the accuracy of this classifier on the testing dataset and also TextBlob provides us to check the most informative features.

print (classifier.accuracy(testing))
classifier.show_informative_features(3)
>> 1.0
Most Informative Features
            contains(is) = True              neg : pos    =      2.9 : 1.0
      contains(terrible) = False             neg : pos    =      1.8 : 1.0
         contains(never) = False             neg : pos    =      1.8 : 1.0

As, we can see that if the text contains “is”, then there is a high probability that the statement will be negative.

In order to give a little more idea, let’s check our classifier on a random text.

blob = TextBlob('the weather is terrible!', classifier=classifier)
print (blob.classify())
>> neg

So, based on the training on the above dataset, our classifier has provided us the right result.

Note that here we could have done some preprocessing and data cleaning but here my aim was to give you an intuition that how we can do text classification using TextBlob.

6. Pros and Cons

Pros:

1. Since, it is built on the shoulders of NLTK and Pattern, therefore making it simple for beginners by providing an intuitive interface to NLTK.
2. It provides language translation and detection which is powered by Google Translate ( not provided with Spacy).

Cons:

1. It is little slower in the comparison to spacy but faster than NLTK. (Spacy > TextBlob > NLTK)
2. It does not provide features like dependency parsing, word vectors etc. which is provided by spacy.

7. End Notes

I hope that you that a fun time learning about this library. TextBlob, actually provided a very easy interface for beginners to learn basic NLP tasks.

I would recommend every beginner to start with this library and then in order to do advance work you can learn spacy as well. We will still be using TextBlob for initial prototyping in the almost every NLP project.

You can find the full code of this article from my github repository.

Also, did you find this article helpful? Please share your opinions/thoughts in the comments section below.

Learn, compete, hack and get hired!

shubham.jain