Text Preprocessing in NLP with Python Codes

Introduction

Text preprocessing is an essential step in natural language processing (NLP) that involves cleaning and transforming unstructured text data to prepare it for analysis. It includes tokenization, stemming, lemmatization, stop-word removal, and part-of-speech tagging. In this article, we will introduce the basics of text preprocessing and provide Python code examples to illustrate how to implement these tasks using the NLTK library. By the end of the article, readers will better understand how to prepare text data for NLP tasks.

Learning Outcomes

• Learn about the essential steps in text preprocessing using Python, including tokenization, stemming, lemmatization, and stop-word removal.
• Discover the importance of text preprocessing in improving data quality and reducing noise for effective NLP analysis.
• With practical code examples, you can learn how to clean and prepare text data using Python and the NLTK library.
• Explore the differences between stemming and lemmatization and their impact on word meaning and context.
• Understand the application of preprocessing techniques on SMS spam data to prepare it for model building.

This article was published as a part of the Data Science Blogathon

What is Text Preprocessing in NLP?

Natural Language Processing (NLP) is a branch of Data Science that deals with text data. Apart from numerical data, text data is available to a great extent and is used to analyze and solve business problems. However, before using the data for analysis or prediction, processing the data is important.

We perform text preprocessing to prepare the text data for the model building. It is the very first step of NLP projects. Some of the preprocessing steps are:

• Removing punctuations like . , ! $( ) * % @
• Removing URLs
• Removing Stop words
• Lower casing
• Tokenization
• Stemming
• Lemmatization

Why is Text Preprocessing Important?

Text preprocessing is crucial in natural language processing (NLP) for several reasons:

Text preprocessing is crucial in preparing text data for NLP tasks. It improves data quality, reduces noise, and facilitates effective analysis and modelling.

SMS Spam Data for Text Preprocessing

We need to use the required steps based on our dataset. This article will use SMS spam data to understand the steps in text preprocessing in NLP using Python’s Pandas library.

Let’s start by importing the Pandas library and reading the data for text processing in NLP.

import pandas as pd
data = pd.read_csv("spam.csv",encoding="ISO-8859-1")
print(data.head())

#expanding the dispay of text sms column
pd.set_option('display.max_colwidth', -1)
#using only v1 and v2 column
data= data [['v1','v2']]
data.head()

The data has 5572 rows and 2 columns. You can check the shape of data using data.shape function. Let’s check the dependent variable distribution between spam and ham.

#checking the count of the dependent variable
data['v1'].value_counts()

Steps to Clean the Data

Now, let us learn the steps for cleaning the data.

Punctuation Removal

This step in text processing in NLP involves removing all the punctuation from the text. String library of Python contains some pre-defined list of punctuations such as ‘!”#$%&'()*+,-./:;?@[\]^_`{|}~’

#library that contains punctuation
import string
string.punctuation

#defining the function to remove punctuation
def remove_punctuation(text):
    punctuationfree="".join([i for i in text if i not in string.punctuation])
    return punctuationfree
#storing the puntuation free text
data['clean_msg']= data['v2'].apply(lambda x:remove_punctuation(x))
data.head()

We remove all the punctuations from v2 and store them in the clean_msg column, as shown in the above output.

Lowering the Text

Converting the text into the same case, preferably lowercase, is one of Python’s most common text preprocessing steps. However, doing this step for text processing every time you work on an NLP problem is unnecessary, as lower casing can lead to a loss of information for some problems.

For example, when dealing with a person’s emotions in any project, words written in upper case can signify frustration or excitement.

data['msg_lower']= data['clean_msg'].apply(lambda x: x.lower())

Output: All the text of clean_msg column is converted into lowercase and stored in the msg_lower column

Tokenization

In this step, the text is split into smaller units. Based on our problem statement, we can use sentence or word tokenization.

#defining function for tokenization
import re
def tokenization(text):
    tokens = re.split('W+',text)
    return tokens
#applying function to the column
data['msg_tokenied']= data['msg_lower'].apply(lambda x: tokenization(x))

Output: Sentences are tokenized into words.

Stop Word Removal

We remove commonly used stopwords from the text because they do not add value to the analysis and carry little or no meaning.

NLTK library consists of a list of stopwords considered stopwords in the English language. Some of them are : [i, me, my, myself, we, our, ours, ourselves, you, you’re, you’ve, you’ll, you’d, your, yours, yourself, yourselves, he, most, other, some, such, no, nor, not, only, own, same, so, then, too, very, s, t, can, will, just, don, don’t, should, should’ve, now, d, ll, m, o, re, ve, y, ain, aren’t, could, couldn’t, didn’t, didn’t]

However, using the provided list of stopwords is unnecessary, as they should be chosen wisely based on the project. For example, ‘How’ can be a stopword for a model but can be important for some other problem where we are working on customers’ queries. We can create a customized list of stopwords for different problems.

#importing nlp library
import nltk
#Stop words present in the library
stopwords = nltk.corpus.stopwords.words('english')
stopwords[0:10]
['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're"]

#defining the function to remove stopwords from tokenized text
def remove_stopwords(text):
    output= [i for i in text if i not in stopwords]
    return output

#applying the function
data['no_stopwords']= data['msg_tokenied'].apply(lambda x:remove_stopwords(x))

Output: Stop words in the nltk library, such as in, until, to, I, and here, are removed from the tokenized text, and the rest are stored in the no_stopwords column.

Stemming

This step, known as text standardization, stems or reduces words to their root or base form. For example, we stem words like ‘programmer,’ ‘programming,’ and ‘program’ to ‘program.’

However, stemming can cause the root form to lose its meaning or not be reduced to a proper English word. We will see this in the steps below.

#importing the Stemming function from nltk library
from nltk.stem.porter import PorterStemmer
#defining the object for stemming
porter_stemmer = PorterStemmer()

#defining a function for stemming
def stemming(text):
stem_text = [porter_stemmer.stem(word) for word in text]
    return stem_text
data['msg_stemmed']=data['no_sw_msg'].apply(lambda x: stemming(x))

Output: In the below image, we can see how some words stem from their base.

crazy-> crazi

available-> avail

entry-> entri

early-> earli

Now, let’s see how Lemmatization is different from Stemming.

Also Read: Stemming vs Lemmatization in NLP: Must-Know Differences

Lemmatization

It stems from the word lemmatize, which means reducing the different forms of a word to one single form. However, one must ensure that it does not lose meaning.  Lemmatization has a pre-defined dictionary that stores the context of words and checks the word in the dictionary while diminishing.

Let us now understand the difference between after stemming and after lemmatization:

from nltk.stem import WordNetLemmatizer
#defining the object for Lemmatization
wordnet_lemmatizer = WordNetLemmatizer()

#defining the function for lemmatization
def lemmatizer(text):
lemm_text = [wordnet_lemmatizer.lemmatize(word) for word in text]
    return lemm_text
data['msg_lemmatized']=data['no_stopwords'].apply(lambda x:lemmatizer(x))

Output: The difference between stemming and lemmatization can be seen in the output below.

In the first row- crazy has been changed to crazi which has no meaning, but for lemmatization, it remained the same, i.e. crazy.

In the last row- goes has changed to goe while stemming, but for lemmatization, it has converted into go, which is meaningful.

After performing all the text processing steps, we convert the final acquired data into numeric form using Bag of Words or TF-IDF.

Conclusion

Apart from the steps in this article, many other steps are a part of text processing NLP . Some of them are URL removal, HTML tags removal, Rare words removal, Frequent words removal, Spelling checking, and many more. You must choose the steps based on the dataset you are working on and what is necessary for the project.

Also Read: Text Cleaning Methods in NLP

Frequently Asked Questions

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Deepanshi