Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Data Extraction from Unstructured PDFs

A
Ashish Last Updated : 14 Aug, 2024
9 min read

Introduction

Data Extraction is the process of extracting data from various sources such as CSV files, web, PDF, etc. Although in some files, data can be extracted easily as in CSV, while in files like unstructured PDFs we have to perform additional tasks to extract data from PDF Python.

There are a couple of Python libraries using which you can extract data from PDFs. For example, you can use the PyPDF2 library for extracting text from PDFs where text is in a sequential or formatted manner i.e. in lines or forms. You can also extract tables in PDFs through the Camelot library. In all these cases data is in structured form i.e. sequential, forms or tables.

However, in the real world, most of the data is not present in any of the forms & there is no order of data. It is present in unstructured form. In this case, it is not feasible to use the above Python libraries since they will give ambiguous results. To analyze unstructured data, we need to convert it to a structured form using methods like “python extract data from pdf.”

As such, there is no specific technique or procedure for extracting data from unstructured PDFs since data is stored randomly & it depends on what type of data you want to extract from PDF.

Here, I will show you a most successful technique & a python library through which you can extract data from bounding boxes in unstructured PDFs and then performing data cleaning operation on extracted data and converting it to a structured form.

In this article you will understand how to extract data from pdf python, extract unstructured data from pdf, extract information from pdf python, and perform unstructured pdf text extraction. Python provides powerful tools to extract data, information and unstructured text from PDF files. Libraries like PyPDF2 and pdfplumber enable extracting structured data as well as parsing unstructured PDF content programmatically.

This article was published as a part of the Data Science Blogathon

Table of contents

PyMuPDF

I have used the PyMuPDF library for this purpose. This library provided many applications such as extracting images from PDF, extracting texts from different shapes, making annotations, draw a bounded box around the texts along with the features of libraries like PyPDF2.

Now, I will show you how I extracted data from the bounding boxes in a PDF with several pages.

Here are the PDF and the red bounding boxes from which we need to extract data.

Data extraction from pdf
Data extraction | image 2
Data extraction | image 3

I have tried many python libraries like PyPDF2, PDFMiner, pikepdf, Camelot, and tabulat. However, none of them worked except PyMuPDF to extract data extraction from PDF using Python.

Before going into the code it’s important to understand the meaning of 2 important terms which would help in understanding the code.

Word: Sequence of characters without space. Ex – ash, 23, 2, 3.

Annots: An annotation associates an object such as a note, image, or bounding box with a location on a page of a PDF document, or provides a way to interact with the user using the mouse and keyboard. The objects are called annots. These annotations can be crucial when you extract data from PDF with Python, as they help identify specific data locations.

Please note that in our case the bounding box, annots, and rectangles are the same thing. Therefore, these terms would be used interchangeably.

Also Read: A Comprehensive Guide to Data Exploration

First, we will extract text from one of the bounding boxes. Then we will use the same procedure to extract data from all the bounding boxes of pdf.

Code

import fitz
import pandas as pd 
doc = fitz.open('Mansfield--70-21009048 - ConvertToExcel.pdf')
page1 = doc[0]
words = page1.get_text("words")

Firstly, we import the fitz module of the PyMuPDF library and pandas library. Then the object of the PDF file is created and stored in doc and the 1st page of the PDF is stored on page1. Using the PyMuPDF library to extract data from PDF with Python, the page.get_text() method extracts all the words from page 1. Each word consists of a tuple with 8 elements.

In words variable, the First 4 elements represent the coordinates of the word, 5th element is the word itself, 6th,7th, 8th elements are block, line, word numbers respectively.

OUTPUT

output 1

Extract the coordinates of the first object 

first_annots=[]

rec=page1.first_annot.rect

rec

#Information of words in first object is stored in mywords

mywords = [w for w in words if fitz.Rect(w[:4]) in rec]

ann= make_text(mywords)

first_annots.append(ann)

This function selects the words contained in the box, sort the words and return in form of a string

def make_text(words):

    line_dict = {} 

    words.sort(key=lambda w: w[0])

    for w in words:  

        y1 = round(w[3], 1)  

        word = w[4] 

        line = line_dict.get(y1, [])  

        line.append(word)  

        line_dict[y1] = line  

    lines = list(line_dict.items())

    lines.sort()  

    return "n".join([" ".join(line[1]) for line in lines])

OUTPUT

output 2

page.first_annot() gives the first annot i.e. bounding box of the page.

.rect gives coordinates of a rectangle.

Now, we got the coordinates of the rectangle and all the words on the page. We then filter the words which are present in our bounding box and store them in mywords variable.

We have got all the words in the rectangle with their coordinates. However, these words are in random order. Since we need the text sequentially and that only makes sense, we used a function make_text() which first sorts the words from left to right and then from top to bottom. It returns the text in string format.

Hurrah! We have extracted data from one annot. Our next task is to extract data from all annots of the PDF which would be done in the same approach.

 Extracting each page of the document and all the annots/rectanges

for pageno in range(0,len(doc)-1):

    page = doc[pageno]

    words = page.get_text("words")

    for annot in page.annots():

        if annot!=None:

            rec=annot.rect

            mywords = [w for w in words if fitz.Rect(w[:4]) in rec]

            ann= make_text(mywords)

            all_annots.append(ann)

 all_annots, a list is initialized to store the text of all annots in the pdf.

The function of the outer loop in the above code is to go through each page of PDF, while that of the inner loop is to go through all annots of the page and performing the task of adding texts to all_annots list as discussed earlier.

Printing all_annots provides us the text of all annots of the pdf which you can see below.

OUTPUT

output 3 | Data extraction

Finally, we have extracted the texts from all the annots/ bounding boxes.

Its time to clean the data and bring it in an understandable form.

Data Cleaning and Data Processing

Splitting to form column name and its values :

cont=[]

for i in range(0,len(all_annots)):

    cont.append(all_annots[i].split('n',1))

Removing unnecessary symbols *,#,:

liss=[]

for i in range(0,len(cont)):

    lis=[]

    for j in cont[i]:

        j=j.replace('*','')

        j=j.replace('#','')

        j=j.replace(':','')

        j=j.strip()

        #print(j)

        lis.append(j)

    liss.append(lis)

 

Spliting into keys and values and removing spaces in the values which only contain digits

keys=[]

values=[]

for i in liss:

    keys.append(i[0])

    values.append(i[1])

for i in range(0, len(values)):

    for j in range(0,len(values[i])):

        if values[i][j]>='A' and values[i][j]<='Z':

            break            

    if j==len(values[i])-1:
       values[i]=values[i].replace(' ','')

We split each string based on a new line (n) character to separate the column name from its values. By further cleaning unnecessary symbols like (*, #, 🙂 are removed. Spaces between digits are removed.

With the key-value pairs, we create a dictionary which is shown below:

 Converting to dictionary

report=dict(zip(keys,values))
report['VEHICLE IDENTIFICATION']=report['VEHICLE IDENTIFICATION'].replace(' ','')
dic=[report['LOCALITY'],report['MANNER OF CRASH COLLISION/IMPACT'],report['CRASH SEVERITY']]

l=0

val_after=[]

for local in dic:

    li=[]

    lii=[]

    k=''

    extract=''

    l=0

    for i in range(0,len(local)-1):

        if local[i+1]>='0' and local[i+1]<='9':

            li.append(local[l:i+1])

            l=i+1

    li.append(local[l:])

    print(li)

    for i in li:

        if i[0] in lii:

            k=i[0]

            break

        lii.append(i[0])

    for i in li:

        if i[0]==k:

extract=i

            val_after.append(extract)
break
report['LOCALITY']=val_after[0]
report['MANNER OF CRASH COLLISION/IMPACT']=val_after[1]
report['CRASH SEVERITY']=val_after[2]

OUTPUT 

output 3 | Data extraction

 

 

 

Lastly, dictionary is converted to dataframe with the help of pandas.

Converting to DataFrame  and exporting to CSV

data=pd.DataFrame.from_dict(report)
data.to_csv('final.csv',index=False)

OUTPUT

output

Now, we can perform analysis on our structured data or export it to excel.

How to Extract Data from Unstructured PDF Files with Python?

To extract data from unstructured PDF files using Python, you can use a combination of libraries such as PyPDF2 and NLTK (Natural Language Toolkit). Here’s a general approach:

  1. Install the required libraries by running the following command

    pip install PyPDF2 nltk

  2. Import the necessary modules in your Python script

    import PyPDF2
    from nltk.tokenize import word_tokenize
    from nltk.corpus import stopwords

  3. Load the PDF file using PyPDF2

    def extract_text_from_pdf(pdf_path):
    pdf_file = open(pdf_path, ‘rb’)
    pdf_reader = PyPDF2.PdfFileReader(pdf_file)
    num_pages = pdf_reader.numPages

    text = “”
    for page_num in range(num_pages):
    page = pdf_reader.getPage(page_num)
    text += page.extract_text()

    pdf_file.close()
    return text

  4. Preprocess the extracted text by removing stopwords and non-alphanumeric characters

    def preprocess_text(text):
    tokens = word_tokenize(text.lower())
    stop_words = set(stopwords.words(‘english’))
    cleaned_tokens = [token for token in tokens if token.isalnum() and token not in stop_words]
    cleaned_text = ‘ ‘.join(cleaned_tokens)
    return cleaned_text

  5. Call the functions and extract the data from the PDF

    pdf_path = “path/to/your/pdf/file.pdf”
    extracted_text = extract_text_from_pdf(pdf_path)
    preprocessed_text = preprocess_text(extracted_text)

    # Process the preprocessed text further as per your specific requirements
    # such as information extraction, entity recognition, etc.

This is a basic example to get you started. Depending on the structure and content of your PDF files, you may need to apply additional techniques for more accurate and specific data extraction.

How to extract information from pdf python?

To extract information from PDF files using Python, several libraries can be utilized, each with its own strengths and methods. Below are some popular libraries and examples of how to use them effectively.

Libraries for PDF Extraction

  1. PyPDF2: A widely used library for reading and manipulating PDF files. It allows you to extract text and metadata from PDFs.
  2. PDFMiner: This library excels in extracting structured data from PDFs, making it suitable for complex layouts.
  3. PyMuPDF (fitz): A powerful library that supports various file formats and provides efficient text extraction and manipulation capabilities.
  4. PDFQuery: This library uses CSS-like selectors to extract data from PDFs, making it user-friendly for specific data retrieval.
  5. IronPDF: A commercial library that simplifies PDF data extraction and manipulation, including interactive forms.

Example Code Snippets

Using PyPDF2

from PyPDF2 import PdfReader

# Create a PDF reader object
reader = PdfReader('example.pdf')

# Print the number of pages
print(len(reader.pages))

# Extract text from the first page
text = reader.pages[0].extract_text()
print(text)

Conclusion

Data extraction from PDFs, especially unstructured ones, can be challenging, but Python offers powerful libraries to simplify this process. While libraries like PyPDF2 and Camelot work well for structured data, PyMuPDF excels in handling unstructured PDFs by extracting data extraction from bounding boxes. This approach, combined with effective data cleaning and processing techniques, enables the conversion of unstructured data into a structured form suitable for analysis. By leveraging these tools, you can efficiently extract, clean, and utilize data from a variety of PDF formats, enhancing your data science projects.

Hope you like the article on how to extract data from PDF using Python. These libraries allow you to extract unstructured data from PDF, making unstructured PDF text extraction a breeze and helping you extract information from PDF with ease.

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Frequently Asked Questions

Q1. How do I extract specific data from a PDF?

A. To extract specific data from a PDF, you can use Python libraries like PyMuPDF or PyPDF2. These libraries allow you to search for and extract text or other elements from specific locations within the PDF.

Q2. How do I extract form data from a PDF?

A. You can extract form data from a PDF using the PyPDF2 or pdfplumber libraries in Python. These libraries can read and extract values from form fields in a PDF.

Q3. Can I extract data from PDF to Excel?

A. Yes, you can extract data from a PDF to Excel using libraries like PyMuPDF, Camelot, or tabula-py to read the PDF and extract the data, then use the pandas library to organize and export the data to an Excel file.

Q4. How to automatically extract data from PDF?

A. To automatically extract data from a PDF, you can use a combination of Python libraries such as PyPDF2 for text extraction and pandas for data processing. You can write a script that reads the PDF, processes the data, and saves it in the desired format, such as CSV or Excel. This script can be scheduled to run automatically using tools like cron jobs or task schedulers.

blogathonData Extractionextract data from pdf pythonextract information from pdf pythonextract unstructured data from pdfpdfunstructured pdf text extraction
A
Ashish

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Mohammed Nawaz Ahmed
Mohammed Nawaz Ahmed

hey ashish excellent article. however i struct somewhere in my code. how could i contact u

123

Jackson
Jackson

thankyou for this post but I would appreciate it if you can share a link for the pdf used for the data extraction as that would make it easier to try this exercise.

123

D S BALA KRISHNA SARAN
D S BALA KRISHNA SARAN

I have tried to execute /practice the same below code : import fitz import pandas as pd doc = fitz.open('FlipKart_Invoice_2022_3.pdf') page1 = doc[0] words = page1.get_text("words") first_annots=[] rec=page1.first_annot.rect mywords = [w for w in words if fitz.Rect(w[:4]) in rec] ann= make_text(mywords) first_annots.append(ann) but, I am facing the below error : rec=page1.first_annot.rect AttributeError: 'NoneType' object has no attribute 'rect' Could you help me here ?

123

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag
Congratulations, You Did It!
Well Done on Completing Your Learning Journey. Stay curious and keep exploring!
PyPDF2 Library for Working with PDF Files in Python
PyPDF2 Library for Working with PDF Files in Python
CV Data Extraction: Essential Tools and Methods for Recruitment
CV Data Extraction: Essential Tools and Methods for Recruitment

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details