Regular Expressions: How Can They Transform Your Coding Efficiency?

Introduction

Regular Expressions, also known as “regex in Python”, are used to match strings of text such as particular characters, words, or patterns of characters. It means that we can match and extract any string pattern from the text with the help of regular expressions. I have used two terms, match and extract and both the terms have a slightly different meaning. There may be cases when we want to match a specific pattern but extract a subset of it. For example, we want to extract the names of PhD scholars from a list of names of people in an organization.

This article was published as a part of the Data Science Blogathon

What is Regular Expressions?

Regular Expressions, often abbreviated as “regex”, are a powerful tool used in computing for matching patterns in text. Think of them as a kind of code that can be used to find, replace, or manipulate text based on specific conditions.

Imagine you’re reading a book and you want to find every instance where the word “magic” is used. You could go through the book page by page, which would be time-consuming. Or you could use a tool like a regular expression to find all instances of “magic” in no time at all!

A Simple Example

Let’s say we have a list of email addresses and we want to find all the Gmail addresses. Here’s how we could do it using a regular expression in Python:

import re


emails = ["[email protected]", "[email protected]", "[email protected]", "[email protected]"]


pattern = r'\b[A-Za-z0-9._%+-]+@gmail\.com\b'


gmail_addresses = [email for email in emails if re.search(pattern, email)]

print(gmail_addresses)

In this example, the regular expression r'\b[A-Za-z0-9._%+-]+@gmail\.com\b' is used to match any text that starts with one or more alphanumeric characters (or ., _, %, +, -), followed by “@gmail.com”. The re.search() function then checks if this pattern is found in each email address.

Regular expressions can be as simple or as complex as you need them to be, depending on what you’re trying to achieve. They’re a versatile tool that can save you a lot of time and effort when working with text!

Regular Expressions: A Powerful Tool

Imagine you’re a detective, and you’re given a huge book to find all instances of a specific word. You could go through the book page by page, which would be time-consuming and prone to errors. Now, imagine if you had a magic magnifying glass that could instantly highlight all instances of that word. That’s what Regular Expressions (RegEx) are in the world of programming!

Why Are Regular Expressions Important?

• Efficiency: Regular expressions can search through large amounts of text quickly and efficiently. They save programmers a lot of time and effort.
• Flexibility: Regular expressions are not just for finding exact matches. They can search for patterns, such as phone numbers, email addresses, and social security numbers. This makes them incredibly versatile.
• Cross-Platform: Regular expressions are supported in almost all programming languages, making them a universally applicable tool.
• Data Validation: Regular expressions are used to validate user input. For example, they can check if an email address is in the correct format.
• Text Manipulation: Regular expressions can also replace text. For example, they can change date formats or correct spelling mistakes.

Applications of Regular Expressions

Form Validation

The most common use of regular expressions is form validation, i.e. email validation, password validation, phone number validation, and many other fields of the form.

Validating Bank Account Details

You must have noticed that every bank has an IFSC code for its different branches that starts with the name of the bank. The credit card number consists of 16 digits and the first few digits represent whether the card is Master, Visa, or Rupay. In all these cases, regex is used.

Data Mining

How can we forget the importance of regex in data mining? When the data is present in unstructured form, i.e. in text form, it needs to be converted to numbers to train the model. Therefore, regex plays an important role in analyzing the data, find patterns in the data, and finally performing operations on the dataset.

Natural Language Processing (NLP)

NLP is a process through which a computer understands and generates human language. In NLP, regular expressions are used to remove unnecessary words, i.e., stop words, from the text, aiding in data cleaning. Regex also plays a role in analyzing texts, assisting in predicting the algorithm’s processing of the data.

Usage in Social Media Platforms

Social Media Platforms such as Google, Facebook, Twitter provide several techniques to search, which are different and efficient from a normal search. Believe me, if you know these techniques, you can explore much more. All these techs use regex in the backend to process these searches.

You can think of various other applications of regular expressions wherever pattern matching is required.

Common Patterns and Techniques

Wild Card Patterns

The smallest individual units through the regular expressions are formed are called wild-card patterns. The list of commonly used patterns in regular expressions-

Examples

• If you want to extract numbers from a document, the regex will be: [0-9]+
• If you want to extract all characters other than numbers regex will be: [^0-9]+
• To extract pattern such that a name starts with “A” and ends with “h” the regex will be: ^A[a-zA-Z]+h$
• A more complex regex if you want to extract email address be: ^[a-zA-Z][a-zA-Z0-9._+-]+@[A-Za-z]+.[A-Za-z] 

Lookahead and Lookbehind Techniques

Imagine you’re on a treasure hunt. You have a map that says, “Walk 10 steps forward from the big oak tree, then look ahead. If you see a blue flag, dig there.” This is similar to what Lookahead does in Regular Expressions.

What is Lookahead?

Lookahead is a technique in Regular Expressions that allows you to match a pattern only if it’s followed by another specific pattern. It’s like saying, “Find ‘cat’, but only if it’s followed by ‘nap’.” However, the ‘nap’ is not part of the match. It’s just a condition.

There are two types of Lookahead:

1. Positive Lookahead (?=pattern): Matches if the given pattern is ahead.
2. Negative Lookahead (?!pattern): Matches if the given pattern is not ahead.

What is Lookbehind?

Now, imagine the treasure map says, “Walk until you see a red flag, then look behind. If there’s a big oak tree, dig there.” This is what Lookbehind does.

Lookbehind is a technique that allows you to match a pattern only if it’s preceded by another specific pattern. It’s like saying, “Find ‘nap’, but only if it’s preceded by ‘cat’.” Again, ‘cat’ is not part of the match. It’s just a condition.

There are two types of Lookbehind:

• Positive Lookbehind (?<=pattern): Matches if the given pattern is behind.
• Negative Lookbehind (?<!pattern): Matches if the given pattern is not behind.

Examples and Use Cases in Regular Expressions

Validating Email Addresses

Email validation is crucial to ensure that communication with users is possible. A simple RegEx for email validation could be ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$. This checks if the email has the general format of [email protected].

Extracting URLs from Text

Extracting URLs from a block of text is a common task, especially in web scraping or reading through logs. A basic RegEx for this could be (http|https)://[a-zA-Z0-9./-]+.

Replacing Text Patterns

RegEx can be used to find and replace patterns in a text. For example, to replace all occurrences of “dog” with “cat” in a text, you could use the RegEx dog and the replacement string cat.

Parsing CSV Files

CSV files are commonly used for data storage and analysis. RegEx can be used to parse these files and extract data. A simple RegEx for this could be ([^,]+), which matches any character except a comma, until it hits a comma.

Cleaning Text Data

Cleaning text data is a common task in Natural Language Processing (NLP). RegEx can be used to remove unwanted characters, such as punctuation. A simple RegEx for this could be [^a-zA-Z ] which matches anything that is not a letter or a space.

Extracting Dates and Times

Dates and times often come in various formats. Regular expressions can be used to extract these from a text. A simple RegEx for a date could be \d{2}/\d{2}/\d{4} which matches a date in the format MM/DD/YYYY.

Validating Password Strength

RegEx can be used to enforce password policies. For example, a RegEx for a strong password could be ^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)[a-zA-Z\d]{8,}$ which requires at least one lowercase letter, one uppercase letter, one digit, and a minimum length of 8 characters.

Building Regex Patterns

Building Regex Patterns: The Basics

Regular Expressions are like a puzzle. You put together different pieces, each with its own meaning, to create a pattern that can match text. Here are some basic pieces:

• Literal Characters: The simplest piece is a literal character, like a or 5. It matches exactly that character.
• Dot (.): The dot is a wildcard. It matches any single character except a newline.
• Character Classes ([…]): A character class matches any one character enclosed in the brackets. For example, [abc] matches a, b, or c.
• Quantifiers: Quantifiers specify how many times a character or a group of characters can occur. Some examples are * (0 or more times), + (1 or more times), and ? (0 or 1 time).

Meta Characters and Their Usage

Meta characters are special characters that have a special meaning:

• Dot (.): As mentioned above, it’s a wildcard that matches any character except a newline.
• Caret (^): When used at the start of a RegEx, it matches the start of a line. For example, ^A matches any line that starts with A.
• **Dollar ()∗∗:WhenusedattheendofaRegEx,itmatchestheendofaline.Forexample,‘Amatches any line that ends withA`.
• Backslash (\): It’s used to escape a special character, turning it into a literal. For example, \\. matches a literal dot.
• Pipe (|): It acts like a logical OR. Matches the pattern before or the pattern after it.
• Parentheses ((…)): A quantifier can specify how many times parts of a RegEx, grouped together, should repeat.

Implementation in Python

Regular expressions is provided by many programming languages, such as python, java,  javascript, etc. Although the concept is the same everywhere yet, you may find some differences in different languages.

Now we will look into the various functions provided by python to implement regex along with its code.

Python does not provide an inbuilt regex module. You need to install it using the pip command and then import it into your Python IDE. Then we stored some text in a variable named string.

pip install regex
import regex as re
string = "Virat Kohli is one of the greatest players in the Indian cricket team.nHe was born on November 5, 1988, in Delhi.nHe has completed his education at Vishal Bharti School.nIn 2008, he won the World Cup for India on Omar’s children under 19 years. From 2011, he started Test cricket matches. nHe is currently the captain of all three formats of India.n In 2017, Virat Kohli got married to Hindi film actress Anushka Sharma.nVirat has won the Man of the Tour twice, in 2014 and 2016. nSince 2008, he has represented Delhi in-home teams. nHe has been awarded the Arjuna Award in recognition of the achievements of international cricket."

match method

This function searches for the RE pattern at the beginning of the string and returns the match object of the string. You can access the value in the object through the group() function. The syntax of the match function is

re.match(pattern, string, flags)

The pattern represents the regular expression, the string denotes the text searched to match the pattern, and the flags indicate the modifiers. If we want to apply any condition on the matching we use flags. This is an optional parameter.

python code for regular expressions

#pip install regex --> write in shell tab
import regex as re
string = "Virat Kohli is one of the greatest players in the Indian cricket team.nHe was born on November 5, 1988, in Delhi.nHe has completed his education at Vishal Bharti School.nIn 2008, he won the World Cup for India on Omar’s children under 19 years. From 2011, he started Test cricket matches. nHe is currently the captain of all three formats of India.n In 2017, Virat Kohli got married to Hindi film actress Anushka Sharma.nVirat has won the Man of the Tour twice, in 2014 and 2016. nSince 2008, he has represented Delhi in-home teams. nHe has been awarded the Arjuna Award in recognition of the achievements of international cricket."

pattern=r'(^[V].+?)s'
print(re.match(pattern,string))      # Returns the match object

print(re.match(pattern,string).group()) #Extracting value from the object

This function match whether the first string starts with V. 

search method

this function searches for the first occurrence of the RE pattern in the given string. This function also returns the match object if the pattern is found else returns none. The syntax is

re.search(pattern, string)

Please note that match checks for a match only at the beginning of the string, while search checks for a first match anywhere in the string.

python code

pattern=r'[0-9]+'
re.search(pattern,string)      # Returns the match object
print(re.search(pattern,string).group())

OUTPUT– 5

This function returns the first number present in the text.

findall method

This function will return all the occurrences of the RE pattern in the string. The syntax of findall is

re.findall(pattern, string)

python code

pattern=r'[0-9]+'
print(re.findall(pattern,string))

OUTPUT

['5', '1988', '2008', '19', '2011', '2017', '2014', '2016', '2008']

This function extracts all the numbers in the text.

sub method

This function is used to replace all the occurrences of the RE pattern with the new string/pattern. The syntax is:

re.sub(pattern, repl, string)

python code

repl=r’Chiku’
print(re.sub(pattern, repl, string))

OUTPUT

“Chiku Kohli is one of the greatest players in the Indian cricket team.nHe was born on November 5, 1988, in Delhi.nHe has completed his education at Vishal Bharti School.nIn 2008, he won the World Cup for India on Omar’s children under 19 years. From 2011, he started Test cricket matches. nHe is currently the captain of all three formats of India.n In 2017, Virat Kohli got married to Hindi film actress Anushka Sharma.nChiku has won the Man of the Tour twice, in 2014 and 2016. nSince 2008, he has represented Delhi in-home teams. nHe has been awarded the Arjuna Award in recognition of the achievements of international cricket.”

This function replaces Virat with Chiku i.e. Kohli’s nickname.

These are the most commonly used functions of “re” module. You can refer re documentation for more details.

Conclusion

In this blog, we’ve explored the fascinating world of Regular Expressions (regex in Python). From understanding what they are, their importance, to their diverse applications in form validation, data mining, NLP, and social media platforms. We looked into common patterns, techniques like Lookahead and Lookbehind, and various use cases.

We’ve seen how regex in Python can validate email addresses, extract URLs from text, replace text patterns, parse CSV files, clean text data, extract dates and times, and validate password strength. We’ve also learned how to build RegEx patterns and about the usage of meta characters.

Finally, we’ve seen how Python implements RegEx through methods like match, search, findall, and sub.

In essence, Regular Expressions are a powerful tool in the programmer’s toolkit. They are like a Swiss Army knife for handling text – versatile, efficient, and indispensable. Whether you’re a web developer, a data scientist, or just someone who deals with a lot of text data, mastering Regular Expressions will undoubtedly elevate your coding skills to the next level.