Statistical Analysis of text is one of the important steps of text pre-processing. It helps us understand our text data in a deep, mathematical way. This type of analysis can help us understand hidden patterns, and the weight of specific words in a sentence, and overall, helps in building good language models. The pyNLPL or as we call it Pineapple library, is one of the best Python libraries for textual statistical analysis. This library is also useful for other tasks such as cleaning and analyzing text, and it provides text pre-processing functions like tokenizers, n-gram extractors, and more. Additionally, pyNLPL can be used to build simple language models.
In this blog, you will understand how to perform text analysis using pyNLPL. We will first understand all the ways to install this library on our systems. Next, we will understand the Term Co-Occurrence matrix and its implementation using the pyNLPL library. After that, we will learn how to create a frequency list to identify the most repeated words. Next, we will perform text distribution analysis to measure the similarity between two text documents or strings. Finally, we will understand and calculate the Leveshtein’s distance using this library. You can either follow along and code by yourself, or you can just click on the ‘Copy & Edit’ button in this link to execute all programs.
This article was published as a part of the Data Science Blogathon.
We can install this library in two ways, first using PyPI, and second using GitHub.
To install it using PyPI paste the below command in your terminal.
pip install pynlplIf you are using a notebook like Jupyter Notebook, Kaggle Notebook, or Google Colab, then add ‘!’ before the above command.
To install this library using GitHub, clone the official pyNLPL repository into your system using the below command.
git clone https://github.com/proycon/pynlpl.gitThen change the directory of your terminal to this folder using ‘cd’ then paste this below command to install the library.
python3 setup.py installLet us now explore on how we can use pyNLPL for text analysis.
Term Co-Occurrence Matrix (TCM) is a statistical method to identify how often a word co-occurs with another specific word in a text. This matrix helps us understand the relationships between words and can reveal hidden patterns that are useful. It is commonly used in building text summaries, as it provides relationships between words that can help generate concise summaries. Now, let’s see how to build this matrix using the pyNLPL library.
We will first import the FrequencyList function from pynlpl.statistics, which is used to count how many times a word has been repeated in a text. We will explore this in more detail in a later section. Additionally, we will import the defaultdict method from the collections module. Next, we will create a function named create_cooccurrence_matrix, which takes a text input and a window size, and returns the matrix. In this function, we will first split the text into individual words and create a co-occurrence matrix using defaultdict. For every word in the text, we will identify its context words within the specified window size and update the co-occurrence matrix. Finally, we will print the matrix and display the frequency of each term.
from pynlpl.statistics import FrequencyList
from collections import defaultdict
def create_cooccurrence_matrix(text, window_size=2):
words = text.split()
cooccurrence_matrix = defaultdict(FrequencyList)
for i, word in enumerate(words):
start = max(i - window_size, 0)
end = min(i + window_size + 1, len(words))
context = words[start:i] + words[i+1:end]
for context_word in context:
cooccurrence_matrix[word.lower()].count(context_word.lower())
return cooccurrence_matrix
text = "Hello this is Analytics Vidhya and you are doing great so far exploring data science topics. Analytics Vidhya is a great platform for learning data science and machine learning."
# Creating term co-occurrence matrix
cooccurrence_matrix = create_cooccurrence_matrix(text)
# Printing the term co-occurrence matrix
print("Term Co-occurrence Matrix:")
for term, context_freq_list in cooccurrence_matrix.items():
print(f"{term}: {dict(context_freq_list)}")Output:
A frequency list will contain the number of times a specific word has been repeated in a document or a paragraph. This is a useful function to understand the main theme and context of the whole document. We usually use frequency lists in fields such as linguistics, information retrieval, and text mining. For example, search engines use frequency lists to rank web pages. We can also use this as a marketing strategy to analyze product reviews and understand the main public sentiment of the product.
Now, let’s see how to create this frequency list using the pyNLPL library. We will first import the FrequencyList function from pynlpl.statistics. Then, we will take a sample text into a variable and split the whole text into individual words. We will then pass this ‘words’ variable into the FrequencyList function. Finally, we will iterate through the items in the frequency list and print each word and its corresponding frequency.
from pynlpl.statistics import FrequencyList
text = "Hello this is Analytics Vidhya and you are doing great so far exploring data science topics. Analytics Vidhya is a great platform for learning data science and machine learning."
words = text.lower().split()
freq_list = FrequencyList(words)
for word, freq in freq_list.items():
print(f"{word}: {freq}")Output:
In Text distribution analysis, we calculate the frequency and probability distribution of words in a sentence, to understand which words make up the context of the sentence. By calculating this distribution of word frequencies, we can identify the most common words and their statistical properties, like entropy, perplexity, mode, and max entropy. Let’s understand these properties one by one:
We can also calculate the information content of a specific word, meaning we can calculate the amount of information provided by a word.
Now let’s see how to implement all these using pyNLPL.
We will import the Distribution and FrequencyList functions from the pynlpl.statistics module and the math module. Next, we will create a sample text and count the frequency of each word within that text. To do this, we will follow the same steps as above. Then, we will create an object of the Distribution function by passing the frequency list. We will then display the distribution of each word by looping through the items of the distribution variable. To calculate the entropy, we will call the distribution.entropy() function.
To calculate the perplexity, we will call distribution.perplexity(). For mode, we will call distribution.mode(). To calculate the maximum entropy, we will call distribution.maxentropy(). Finally, to get the information content of a specific word, we will call distribution.information(word). In the example below, we will pass the mode word as the parameter to this function.
import math
from pynlpl.statistics import Distribution, FrequencyList
text = "Hello this is Analytics Vidhya and you are doing great so far exploring data science topics. Analytics Vidhya is a great platform for learning data science and machine learning."
# Counting word frequencies
words = text.lower().split()
freq_list = FrequencyList(words)
word_counts = dict(freq_list.items())
# Creating a Distribution object from the word frequencies
distribution = Distribution(word_counts)
# Displaying the distribution
print("Distribution:")
for word, prob in distribution.items():
print(f"{word}: {prob:.4f}")
# Various statistics
print("\nStatistics:")
print(f"Entropy: {distribution.entropy():.4f}")
print(f"Perplexity: {distribution.perplexity():.4f}")
print(f"Mode: {distribution.mode()}")
print(f"Max Entropy: {distribution.maxentropy():.4f}")
# Information content of the 'Mode' word
word = distribution.mode()
information_content = distribution.information(word)
print(f"Information Content of '{word}': {information_content:.4f}")Output:
Levenshtein distance is the measure of the difference between two words. It calculates how many single-character changes are needed for two words to become the same. It calculates based on the insertion, deletion, or substitution of a character in a word. This distance metric is usually used for checking spellings, DNA sequence analysis, and natural language processing tasks such as text similarity which we will implement in the next section, and it can be used to build plagiarism detectors. By calculating Levenshtein’s distance we can understand the relationship between two words, we can tell if two words are similar or not. If the levenshtein’s distance is very less then those words could have the same meaning or context, and if it is very high then it means they are completely different words.
To calculate this distance, we will first import the levenshtein function from the pynlpl.statistics module. We will then define two words, ‘Analytics’ and ‘Analysis’. Next, we will pass these words into the levenshtein function, which will return the distance value. As you can see in the output, the Levenshtein distance between these two words is 2, meaning it takes only two single-character edits to convert ‘Analytics’ to ‘Analysis’. The first edit is substituting the character ‘t‘ with ‘s‘ in ‘Analytics’, and the second edit is deleting the character ‘c‘ at index 8 in ‘Analytics’.
from pynlpl.statistics import levenshtein
word1 = "Analytics"
word2 = "Analysis"
distance = levenshtein(word1, word2)
print(f"Levenshtein distance between '{word1}' and '{word2}': {distance}")Output:
Measuring how similar two documents or sentences are can be useful in many applications. It allows us to understand how closely related the two documents are. This technique is used in many applications such as plagiarism checkers, code difference checkers, and more. By analyzing how similar the two documents are we can identify the duplicate one. This can also be used in recommendation systems, where the search results shown to user A can be shown to user B who typed the same query.
Now to implement this, we will use the cosine similarity metric. First, we will import two functions: FrequencyList from the pyNLPL library and sqrt from the math module. Now we will add two strings to two variables, in place of just strings we can open two text documents also. Next, we will create frequency lists of these strings by passing them to the FrequencyList function we imported earlier. We will then write a function named cosine_similarity, in which we will pass those two frequency lists as inputs. In this function, we will first create vectors from the frequency lists, and then calculate the cosine of the angle between these vectors, providing a measure of their similarity. Finally, we will call the function and print the result.
from pynlpl.statistics import FrequencyList
from math import sqrt
doc1 = "Analytics Vidhya provides valuable insights and tutorials on data science and machine learning."
doc2 = "If you want tutorials on data science and machine learning, check out Analytics Vidhya."
# Creating FrequencyList objects for both documents
freq_list1 = FrequencyList(doc1.lower().split())
freq_list2 = FrequencyList(doc2.lower().split())
def cosine_similarity(freq_list1, freq_list2):
vec1 = {word: freq_list1[word] for word, _ in freq_list1}
vec2 = {word: freq_list2[word] for word, _ in freq_list2}
intersection = set(vec1.keys()) & set(vec2.keys())
numerator = sum(vec1[word] * vec2[word] for word in intersection)
sum1 = sum(vec1[word] ** 2 for word in vec1.keys())
sum2 = sum(vec2[word] ** 2 for word in vec2.keys())
denominator = sqrt(sum1) * sqrt(sum2)
if not denominator:
return 0.0
return float(numerator) / denominator
# Calculatinng cosine similarity
similarity = cosine_similarity(freq_list1, freq_list2)
print(f"Cosine Similarity: {similarity:.4f}")Output:
pyNLPL is a powerful library using which we can perform textual statistical analysis. Not just text analysis, we can also use this library for some text pre-processing techniques like tokenization, stemming, n-gram extraction, and even building some simple language models. In this blog, we first understood all the ways of installing this library, then we used this library to perform various tasks like implementing the Term Co-Occurrence Matrix, creating frequency lists to identify common words, performing text distribution analysis, and understanding how to calculate levenshtein distance, and calculated document similarity. Each of these techniques can be used to extract valuable insights from our textual data, making it a valuable library. Next time you are doing text analysis, consider trying the pyNLPL (Pineapple) library.
A. PyNLPL, also known as Pineapple, is a Python library used for textual statistical analysis and text pre-processing.
A. This technique allows us to measure how similar two documents or texts are and could be used in plagiarism checkers, code difference checkers, and more.
A. The Term Co-Occurrence Matrix can be used to identify how often two words co-occur in a document.
A. We can use Levenshtein distance to find the difference between two words, which can be useful in building spell checkers.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
