Build Your First Text Classification model using PyTorch

Overview

• Learn how to perform text classification using PyTorch
• Grasp the importance of Pack Padding feature
• Understand the key points involved while solving text classification

Introduction

I always turn to State of the Art architectures to make my first submission in data science hackathons. Implementing the State of the Art architectures has become quite easy thanks to deep learning frameworks such as PyTorch, Keras, and TensorFlow. These frameworks provide an easy way to implement complex model architectures and algorithms with least knowledge of concepts and coding skills. In short, it’s a goldmine for the data science community!

In this article, we will use PyTorch, which is well known for its fast computational power. So in this article, we will walk through the key points for solving a text classification problem. And then we will implement our first text classifier in PyTorch!

Note: I highly recommend to go through the below article before moving forward with this article.

• A Beginner-Friendly Guide to PyTorch and How it Works from Scratch

Table of Contents

1.Why PyTorch for Text Classification?

• Dealing with Out of Vocabulary words
• Handling Variable Length sequences
• Wrappers and Pre-trained models

2.Understanding the Problem Statement
3.Implementation – Text Classification in PyTorch

Why PyTorch for Text Classification?

Before we dive deeper into the technical concepts, let us quickly familiarize ourselves with the framework that we are going to use – PyTorch. The basic unit of PyTorch is Tensor, similar to the “numpy” array in python. There are a number of benefits for using PyTorch but the two most important are:

• Dynamic networks – Change in the architecture during the run time
• Distributed training across GPUs

I am sure you are wondering – why should we use PyTorch for working with text data? Let us discuss some incredible features of PyTorch that makes it different from other frameworks, especially while working with text data.

1. Dealing with Out of Vocabulary words

A text classification model is trained on fixed vocabulary size. But during inference, we might come across some words which are not present in the vocabulary. These words are known as Out of Vocabulary words. Skipping Out of Vocabulary words can be a critical issue as this results in the loss of information.

In order to handle the Out Of Vocabulary words, PyTorch supports a cool feature that replaces the rare words in our training data with Unknown token. This, in turn, helps us in tackling the problem of Out of Vocabulary words.

Apart from handling Out Of Vocabulary words, PyTorch also has a feature that can handle sequences of variable length!

2. Handling Variable Length sequences

Have you heard of how Recurrent Neural Network is capable of handling variable-length sequences? Ever wondered how to implement it? PyTorch comes with a useful feature  ‘Packed Padding sequence‘ that implements Dynamic Recurrent Neural Network.

Padding is a process of adding an extra token called padding token at the beginning or end of the sentence. As the number of the words in each sentence varies, we convert the variable length input sentences into sentences with the same length by adding padding tokens.

Padding is required since most of the frameworks support static networks, i.e. the architecture remains the same throughout the model training. Although padding solves the issue of variable length sequences, there is another problem with this idea – the architectures now process these padding token like any other information/data. Let me explain this through a simple diagram-

As you can see in the diagram (below), the last element, which is a padding token is also used while generating the output. This is taken care of by the Packed Padding sequence in PyTorch.

Packed padding ignores the input timesteps with padding token. These values are never shown to the Recurrent Neural Network which helps us in building a dynamic Recurrent Neural Network.

3. Wrappers and Pretrained models

The state of the art architectures are being launched for PyTorch framework. Hugging Face released Transformers which provides more than 32 state of the art architectures for the Natural Language Understanding Generation!

Not only this, PyTorch also provides pretrained models for several tasks like Text to Speech, Object Detection and so on, which can be executed within few lines of code.

Incredible, isn’t it? These are some really useful features of PyTorch among many others. Let us now use PyTorch for a text classification problem.

Understanding the problem statement

As a part of this article, we are going to work on a really interesting problem.

Implementation – Text Classification in PyTorch

Let us first import all the necessary libraries required to build a model. Here is a brief overview of the packages/libraries we are going to use-

Python Code:

#deal with tensors
import torch   

#handling text data
from torchtext import data

Pre-processsing Data:

Now, let us see how to preprocess the text using field objects. There are 2 different types of field objects – Field and LabelField. Let us quickly understand the difference between the two-

1. Field: Field object from data module is used to specify preprocessing steps for each column in the dataset.
2. LabelField: LabelField object is a special case of Field object which is used only for the classification tasks. Its only use is to set the unk_token and sequential to None by default.

Before we use Field, let us look at the different parameters of Field and what are they used for.

• Tokenize: specifies the way of tokenizing the sentence i.e. converting sentence to words. I am using spacy tokenizer since it uses novel tokenization algorithm
• Lower: converts text to lowercase
• batch_first: The first dimension of input and output is always batch size

TEXT = data.Field(tokenize='spacy',batch_first=True,include_lengths=True)
LABEL = data.LabelField(dtype = torch.float,batch_first=True)

Next we are going to create a list of tuples where first value in every tuple contains a column name and second value is a field object defined above. Furthermore we will arrange each tuple in the order of the columns of csv, and also specify as (None,None) to ignore a column from a csv file.

fields = [(None, None), ('text',TEXT),('label', LABEL)]

In the following code block I have loaded the custom dataset by defining the field objects.

Preparing input and output sequences:

The next step is to build the vocabulary for the text and convert them into integer sequences. Vocabulary contains the unique words in the entire text. Each unique word is assigned an index. Below are the parameters listed for the same

Parameters:

• Unknown token is used to handle Out Of Vocabulary words
• Padding token is used to make input sequences of same length

Now we will prepare batches for training the model. BucketIterator forms the batches in such a way that a minimum amount of padding is required.

Model Architecture

It is now time to define the architecture to solve the binary classification problem. The nn module from torch is a base model for all the models. This means that every model must be a subclass of the nn module.

I have defined 2 functions here: init as well as forward. Let me explain the use case of both of these functions-

1. Init: Whenever an instance of a class is created, init function is automatically invoked. Hence, it is called as a constructor. The arguments passed to the class are initialized by the constructor.We will define all the layers that we will be using in the model

2. Forward: Forward function defines the forward pass of the inputs.

Finally, let’s understand in detail about the different layers used for building the architecture and their parameters-

1. num_embeddings: No. of unique words in dictionary
2. embedding_dim:  No. of dimensions for representing a word

Now that we have a good understanding of all the blocks of the architecture, let us go to the code!

I will start with defining all the layers of the architecture:

The next step would be to define the hyperparameters and instantiate the model. Here is the code block for the same:

Let us look at the model summary and initialize the embedding layer with the pretrained embeddings

Here I have defined the optimizer, loss and metric for the model:

1. Training phase: model.train() sets the model on the training phase and activates the dropout layers.
2. Inference phase: model.eval() sets the model on the evaluation phase and deactivates the dropout layers.

Here is the code block to define a function for training the model

So we have a function to train the model, but we will also need a function to evaluate the mode. Let’s do that

Finally we will train the model for a certain number of epochs and save the best model every epoch.

Let us load the best model and define the inference function  that accepts the user defined input and make predictions

Amazing! Let us use this model to make predictions for few questions:

End Notes

We have seen how to build our own text classification model in PyTorch and learnt the importance of pack padding. You can play around with the hyper-parameters of the Long Short Term Model such as number of hidden nodes, number of  hidden layers and so on to improve the performance even further.

If you have any queries/feedback, leave in the comments section. I will get back to you.

Aravind Pai

Aravind Pai is passionate about building data-driven products for the sports domain. He strongly believes that Sports Analytics is a Game Changer.