This article was published as a part of the Data Science Blogathon
Introduction
Regex is a shorthand for Regular Expression. It is a representation for a set, a set of strings. Say we have a list of emails and we want to check if they are in the correct format or not. One way is to check each and every mail manually but that’s not possible if the number of mails is quite high. So, regex here comes to your rescue. Whenever you want to check emails are valid or not, you can simply match them against the pattern. You’ll get true or false results whether the mail is in the correct format or not according to the regex pattern used.
What are Regular Expressions use for?
Some popular and very common use cases are:
- verify the structure of strings
- extract substrings from structured strings
- search / replace/rearrange parts of the string
- split a string into tokens
These are very normal tasks when working with text data or solving a Natural Language Processing (NLP) problem. So, we will look at some popular uses of regex in NLP problems. You can call it your cheat sheet for NLP tasks. But before moving forward let’s have a look at some major Regular Expression functions.
- re.findall – this module is used to search for “all” occurrences that match a given pattern.
- re.sub – it is used to substitute the matched RE pattern with given text
- re.match – The match function is used to match the RE pattern to string with some optional flags
- re.search – search method takes a regular expression pattern and a string and searches for that pattern with the string.
Out of all these, we will be using re.findall to detect patterns.
So, let’s get started!
Source: Google Images
1. URL
To find a URL in a sentence using the following regex code:
def find_url(string):
text = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*(),]|
(?:%[0-9a-fA-F][0-9a-fA-F]))+',string)
#convert return value from list to string
return "".join(text)
example="I love spending time at https://www.leetcode.com/" find_url(example)
Output:
'https://www.leetcode.com/'
2.Emoticons
To find emoticons in a sentence following code can be used:
def findEmoji(text):
emo_text=emoji.demojize(text)
line=re.findall(r':(.*?):',emo_text)
return line
example="I love ⚽ very much 😁"
findEmoji(example)
Output:
['soccer_ball', 'beaming_face_with_smiling_eyes']
3. Email
Get email id from the sentence by using this code:
def findEmail(text):
line = re.findall(r'[w.-]+@[w.-]+',str(text))
return ",".join(line)
example="Gaurav's gmail is [email protected]" findEmail(example)
Output:
'[email protected]'
4. Hash
You must be aware of Hash(#), it is used in Twitter to denote trends. So, to extract the hash words like #fifa or #cricket, use this code:
def findHash(text):
line=re.findall(r'(?<=#)w+',text)
return " ".join(line)
example="#Sushant is trending now in the world" findHash(example)
Output:
'Sushant'
5.Mentions
@ – This symbol is used to mention someone in tweets. So, to extract words associated with @ symbol use this code:
def findAt(text):
line=re.findall(r'(?<=@)w+',text)
return " ".join(line)
example="@Ajit, please help me" findAt(example)
Output:
'Ajit'
6. Number
Following regex, code is used to pick only numbers from a sentence.
def findNumber(text):
line=re.findall(r'[0-9]+',text)
return " ".join(line)
example="8853147 sq. km of area washed away in floods" findNumber(example)
Output:
'8853147'
7. Phone Number
Indian Mobile numbers have ten digits. So, the below regex is used to find an Indian phone number. You can modify the regex as per your need.
def findPhoneNumber(text):
line=re.findall(r"bd{10}b",text)
return "".join(line)
findPhoneNumber("9990001796 is a phone number of PMO office")
Output:
'9990001796'
8. Non-Alphanumeric
Non-Alphanumeric characters are characters other than alphanumeric. Example: space,
percent sign, underscore, pipe, colon, semicolon, etc are non-alphanumeric characters. So, to find these in-text use this regex code.
def findNonalp(text):
line = re.findall("[^A-Za-z0-9 ]",text)
return line
example="Twitter has lots of @ and # in posts.(2021 year is not good)" findNonalp(example)
Output:
['@', '#', '.', '(', ')']
9. Year
Following regex code can be used to extract years from 1940 till 2020.
def findYear(text):
line=re.findall(r"b(19[40][0-9]|20[0-1][0-9]|2020)b",text)
return line
example="My DOB year is 1998." findYear(example)
Output:
['1998']
10. Punctuations
- There are 14 punctuation marks that are used in the
English language. - Punctuation mark examples: period, question mark, exclamation
the point, comma, colon, semicolon, dash, hyphen, brackets, braces,
parentheses, quotation marks, apostrophes, and ellipsis.
To find these marks in your text use the below code:
def find_punct(text):
line = re.findall(r'[!"$%&'()*+,-./:;=#@?[\]^_`{|}~]*', text)
string="".join(line)
return list(string)
example="Corona virus kiled #24506 people. #Corona is un(tolerable)" print(find_punct(example))
Output:
['#', '.', '#', '(', ')']
11. Repetitive Character
Removes repetitive characters from words.
def rep(text):
grp = text.group(0)
if len(grp) > 1:
return grp[0:1]
# change the value here on repetition
def unique_char(rep,sentence):
convert = re.sub(r'(w)1+', rep, sentence)
return convert
example="heyyy this is a verrrry loong texttt" unique_char(rep,example)
Output:
'hey this is a very long text'
12. Number Greater
Find numbers that are greater than 930. You can modify it as per your use.
def num_great(text):
line=re.findall(r'9[3-9][0-9]|[1-9]d{3,}',text)
return " ".join(line)
example="Height of this bridge is 935m. Width of this bridge is 30 metre. It used 9274kg of steel." num_great(example)
Output:
'935 9274'
13. Number Lesser
Find numbers less than 930 in the input text.
def num_less(text):
only_num=[]
for i in text.split():
line=re.findall(r'^(9[0-2][0-0]|[1-8][0-9][0-9]|[1-9][0-9]|[0-9])$',i)
only_num.append(line)
all_num=[",".join(x) for x in only_num if x != []]
return " ".join(all_num)
example="There are some countries where less than 920 cases exist with 1100 observations" num_less(example)
Output:
'920'
14. Dates
Find the date in this mm-dd-yyyy format from the input text.
def findDates(text):
line=re.findall(r'b(1[0-2]|0[1-9])/(3[01]|[12][0-9]|0[1-9])/([0-9]{4})b',text)
return line
example="Todays date is 06/21/2021 for format mm/dd/yyyy, not 31/09/2020" findDates(example)
Output:
[('06', '21', '2021')]
15. Only Words
The below code only extracts the words, alphabets, and nothing else from a given piece of text.
def onlyWords(text):
line=re.findall(r'b[^dW]+b', text)
return " ".join(line)
example="Harish reduced his weight from 100 Kg to 75 kg." onlyWords(example)
Output:
'Harish reduced his weight from Kg to kg.'
16. Only Numbers
Extracts all the numbers and no other text.
def only_numbers(text):
line=re.findall(r'bd+b', text)
return " ".join(line)
example="Harish reduced his weight from 100 Kg to 75 kg." only_numbers(example)
Output:
'100 75'
17. Pick Sentences
If you want all sentences with a particular keyword. Use the below code:
def pick_only_key_sentence(text,keyword):
line=re.findall(r'([^.]*'+keyword+'[^.]*)', text)
return line
example="People are fighting with covid these days. Economy has fallen down. How will we survive covid" pick_only_key_sentence(example,'covid')
Output:
['People are fighting with covid these days', 'How will we survive covid']
18. Caps lock Words
Extract words starting with a capital letter from the input text.
def find_capital(text):
line=re.findall(r'b[A-Z]w+', text)
return line
example="Ajit Doval is the best National Security Advisor so far." find_capital(example)
Output:
['Ajit', 'Doval', 'National','Security','Advisor']
19. Tags
Most web scrapped data contain HTML tags. To remove that use below regex script:
def remove_tag(string):
text=re.sub('<.*?>','',string)
return text
example="Markdown sentences use <br> for breaks and <i> </i> for italics" remove_tag(example)
Output:
'Markdown sentences use for breaks and for italics'
20. IP Address
Extract IP address from the text.
def ip_add(string):
text=re.findall('d{1,3}.d{1,3}.d{1,3}.d{1,3}',string)
return text
example="My public IP address is 165.19.120.1" ip_add(example)
Output:
['165.19.120.1']
21. MAC Address
Extract Mac address from the text.
def mac_add(string):
text=re.findall('(?:[0-9a-fA-F]:?){12}',string)
return text
example="MAC ADDRESSES of this TOSHIBA laptop is 00:21:27:b1:aa:xx." mac_add(example)
Output:
['00:21:27:b1:aa:xx ']
22. PAN Validation
To Validate PAN Number use the below code.
Format: First 5 letters in CAPS+4 digits+Last letter in CAPS
def validPan(string):
text=re.findall(r'^([A-Z]){5}([0-9]){4}([A-Z]){1}$',string)
if text!=[]:
print("{} is valid PAN number".format(string))
else:
print("{} is not a valid PAN number".format(string))
validPan("ABCED3193P")
validPan("lEcGD012eg")
Output:
ABCED3193P is valid PAN number lEcGD012eg is not a valid PAN number
23. Percentage
Find the percentage numbers from the text. For example: if the sentence is “I scored 85% in 12th class.”. In that case, output will be “85”.
def find_percent(string):
text = re.findall(r'b(100|[1-9][0-9]|[0-9])%',string)
return text
example="COVID recovery rate is now 76%. But death rate is 4%" find_percent(example)
Output:
['76', '4']
24. File Format
This code will return you the filename with its file format like “abc.png”, “math.pdf”, etc.
def find_files(string):
text=re.findall(r'([a-zA-Z0-9_]+).(jpg|png|gif|jpeg|pdf|ipynb|py)',string)
# add any required file extensions
all_files=[]
for i in range(len(text)):
all_files.append('.'.join(text[i]))
return all_files
example="This image file name is cheatsheet.png . Titanic.py file is most common among beginners." find_files(example)
Output:
['cheatsheet.png', 'Titanic.py']
End Notes:
So, these are some regex codes that can be used in NLP tasks directly. I hope you find this article helpful. Let’s connect on Linkedin.
Thanks for reading if you reached here :).
Happy coding!
The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.
Love Programming, Blog writing and Poetry
Very useful and well explained, thanks a lot!!!