Extracting information from reports using Regular Expressions Library in Python

Introduction

Many times it is necessary to extract key information from reports, articles, papers, etc. For example names of companies – prices from financial reports, names of judges – jurisdiction from court judgments, account numbers from customer complaints, etc.

These extractions are part of Text Mining and are essential in converting unstructured data to a structured form which are later used for applying analytics/machine learning.

Such entity extraction uses approaches like ‘lookup’, ‘rules’ and ‘statistical/machine learning’. In ‘lookup’ based approaches, words from input documents are searched against pre-defined data dictionary. In ‘rules’ based approach, pattern searches are made to find key information. Whereas in ‘statistical’ approach supervised-unsupervised methods are used to extract the information.

‘Regular expression (RegEx)’ is one of the ‘rules’ based pattern search method.

Basic syntax

Python supports regular expressions by the library called “re”(though it’s not fully Perl-compatible). Instead of regular strings, search patterns are specified using raw strings “r”, so that backslashes and meta characters are not interpreted by python but sent to RegEx directly.

Go through the following table of basic syntax.

Go through the following Python sample code for usage of RegEx.

Python Code:

import re
regex = r"([a-zA-Z]+) (\d+)" 
text = "June 24" 
match = re.search(regex, text) 
if match:
    #   match.group() or match.group(0) always returns the fully matched string i.e. "June 24"     
  print ("Match: %s" % (match.group(0)))
    #   match.group(1) match.group(2), ... will return the capture groups in order
  print ("Month: %s" % (match.group(1))) 
  print ("Day: %s" % (match.group(2))) # "24" 
else:
    # If re.search() does not match, then None is returned     
  print ("The regex pattern does not match. :(")

Instead of “re.search”, which returns all the exact matches, “re.findall()” can be used to return all captured groups. “re.sub” is used to substitute another pattern as a replacement for the given search pattern. For performance reasons, it is recommended to compile the pattern first using “re.compile” and then use the RegEx object for searching, as shown below.

regex = re.compile(r"(\w+) Lamb") 
text = "Mary had a little Lamb" 
result = regex.search(text)

More information about RegEx usage in Python can be found at Regex One and in this AV article.

Use Cases

Imagine writing code for searching telephone numbers like +91-9890251406 in a document, with multiple variations in format. With validations, the code will typically be surely more than 10 lines (sample here). But with RegEx, it’s just about 2/3 of lines of code, and with high customizability.

Following are some of the frequently occurring scenarios where RegEx can offer substantial help. Please note that the examples shown could have alternate ways of getting same results, especially by using meta characters such as “/d” for “[0-9]” representing digits. In most of the examples, expressive and simplistic patterns are used here just for clarity and understandability.

Finding email

Finding telephone number

The first two cases differ only in “-“or “.”  and thus can combined using “(-|\.)”

A sample code for more elaborate phone number is as follows:

phoneRegex = re.compile(r'''(
    (\d{3}|\(\d{3}\))?                       # area code
    (\s|-|\.)?                                     # separator
    (\d{3})                                           # first 3 digits
    (\s|-|\.)                                        # separator
    (\d{4})                                           # last 4 digits
    (\s*(ext|x|ext.)\s*(\d{2,5}))?  # extension
    )''', re.VERBOSE)

Finding date

The date pattern used above are only numeric. There are other usages such as ’27-Mar-1973′ or ’27 March 1973′. I would leave this as an open quiz and would want the panelist to think about their RegEx patterns!!

Finding account /credit card number

Adding linked Information

Citations in search papers or judgments have pre-defined formats and they refer to an external document. It is possible to append the hyperlink information by replacing the citation text.

For example, US legal judgments citation looks like “17 U.S.C. § 107”. The pattern is : text “U.S.C.”, another space, a § mark, another space, a set of numbers, and optionally, a year inside a parenthetical. It can be replaced with a “<a href …/a>” hyper link to actual judgment it refers to.

Tools for development, testing and debugging

Although RegEx is powerful but it can get complicated for non-trivial tasks. More challenging it would be if you must understand (and debug) RegEx by someone else!

There are quite a few friendly utilities which help in development and testing of RegEx. Try Regex 101. It gives facility to put your own text and try RegEx pattern.

Here is one sample text to try on Phone number, account number, date patterns, mentioned before.

Sites like RegExper give visual representation of the RegEx search pattern for better understanding. Refer to the below visualization for PhoneRegex search pattern mentioned earlier, used for phone numbers.

Once RegEx gives acceptable matches, the pattern can be used in programs. After good enough practice one can directly code the search patterns in the program itself.

All the RegEx patterns used here, with some minor modifications, can be used in programming language like Python, Perl, Java, etc. It can also be used in some of the popular text editors for Find-Replace functionality, like Microsoft Word (keep “Use Wildcard” option ON), OpenOffice and in IDEs like PyCharm. Read the comprehensive information about RegEx here.

End Note

RegEx is a versatile, portable and powerful way of extracting key information from textual data. Mastery over it can help automate many mundane tasks. Although, at times, it can get complicated and hard to develop-debug but owing to its immense capabilities it has become a must weapon in every programmer’s armour, especially for text analytics data scientists.

Let me conclude by giving a food for thought: Can RegEx be used to solve a crossword puzzle?

Drop your answers below. If you have questions feel free to post them in the comments section.

By Analytics Vidhya Team: This article was contributed by Yogesh Kulkarni who is the second rank holder of  Blogathon 3.

Learn, compete, hack and get hired!