Data Science With Pandas: 2 Minutes Guide to Key Concepts

This article was published as a part of the Data Science Blogathon

Introduction

Pandas is one of the most popular and powerful data science libraries in Python. It can be considered as the stepping stone for any aspiring data scientist who prefers to code in Python. Even though the library is easy to get started, it can certainly do a wide variety of data manipulation. This makes Pandas one of the handiest data science libraries in the developer’s community. Pandas basically allow the manipulation of large datasets and data frames. It can also be considered as one of the most efficient statistical tools for mathematical computations of tabular data.

Today. we’ll cover some of the most important and recurring operations that we perform in Pandas. Make no mistake, there are tons of implementations and prospects of Pandas. Here we’ll try to cover some notable aspects only. We’ll use the analogy of Euro Cup 2020 in this tutorial. We’ll start off by creating our own minimal dataset.

Creating our small dataset

Let’s start off by creating a small sample dataset to try out various operations with Pandas. In this tutorial, we shall create a Football data frame that stores the record of 4 players each from Euro Cup 2020’s finalists – England and Italy.

import pandas as pd

# Create team data
data_england = {'Name': ['Kane', 'Sterling', 'Saka', 'Maguire'], 'Age': [27, 26, 19, 28]}
data_italy = {'Name': ['Immobile', 'Insigne', 'Chiellini', 'Chiesa'], 'Age': [31, 30, 36, 23]}

# Create Dataframe
df_england = pd.DataFrame(data_england)
df_italy = pd.DataFrame(data_italy)

The England data frame looks something like this

The Italy data frame looks something like this

The concat() function

Let’s start by concatenating our two data frames. The word “concatenate” means to “link together in series”. Now that we have created two data frames, let’s try and “concat” them.

We do this by implementing the concat() function.

frames = [df_england, df_italy]
both_teams = pd.concat(frames)
both_teams

The result looks something like this:

A similar operation could also be done using the append() function.

Try doing:

df_england.append(df_italy)

You’ll get the same result!

Now, imagine you wanted to label your original data frames with the associated countries of these players. You can do this by setting specific keys to your data frames.

Try doing:

pd.concat(frames, keys=["England", "Italy"])

And our result looks like this:

Setting conditions in Pandas

Conditional statements basically define conditions for data frame columns. There may be situations where you have to filter out various data by applying certain column conditions (numeric or non-numeric). For eg: In an Employee data frame, you might have to list out a bunch of people whose salary is more than Rs. 50000. Also, you might want to filter the people who live in New Delhi, or whose name starts with “A”. Let’s see a hands-on example.

Imagine we want to filter experienced players from our squad. Let’s say, we want to filter those players whose age is greater than or equal to 30. In such case, try doing:

both_teams[both_teams["Age"] >= 30]

Hmm! Looks like Italians are more experienced lads.

Now, let’s try to do some string filtration. We want to filter those players whose name starts with “S”. This implementation can be done by pandas’ startswith() function. Let’s try:

both_teams[both_teams["Name"].str.startswith('S')]

Impressive!

Adding a new column

Let’s try adding more data to our df_england data frame.

club = ['Tottenham', 'Man City', 'Arsenal', 'Man Utd']
# 'Associated Club' is our new column name
df_england['Associated Clubs'] = club
df_england

This will add a new column ‘Associated Club’ to England’s data frame.

Let’s try to repeat implementing the concat function after updating the data for England.

frames = [df_england, df_italy]
both_teams = pd.concat(frames)
both_teams

Now, this is interesting! Pandas seem to have automatically appended the NaN values in the rows where ‘Associated Clubs’ weren’t explicitly mentioned. In this case, we had only updated ‘Associated Clubs’ data on England. The corresponding values for Italy were set to NaN.

Filling NaN with string

Now, what if, instead of NaN, we want to include some other text? Let’s try adding “No record found” instead of NaN values.

both_teams['Associated Clubs'].fillna('No Data Found', inplace=True)
both_teams

Pretty cool!

Sorting based on column values

Sorting operation is straightforward in Pandas. Sorting basically allows the data frame to be ordered by numbers or alphabets (in either increasing or decreasing order). Let’s try and sort the players according to their names.

both_teams.sort_values('Name')

Fair enough, we sorted the data frame according to the names of the players. We did this by implementing the sort_values() function.

Let’s sort them by ages:

both_teams.sort_values('Age')

Ah, yes! Arsenal’s Bukayo Saka is the youngest lad out there!

Can we also sort by the oldest players? Absolutely!

both_teams.sort_values('Age', ascending=False)

Pandas “groupby”

Grouping is arguably the most important feature of Pandas. A groupby() function simply groups a particular column. Let’s see a simple example by creating a new data frame.

a = {
    'UserID': ['U1001', 'U1002', 'U1001', 'U1001', 'U1003'],
    'Transaction': [500, 300, 200, 300, 700]
}
df_a = pd.DataFrame(a)
df_a

Notice, we have two columns – UserID and Transaction. You can also see a repeating UserID (U1001). Let’s apply a groupby() function to it.

df_a.groupby('UserID').sum()

The function grouped the similar UserIDs and took the sum of those IDs.

If you want to unravel a particular UserID, just try mentioning the value name through get_group().

df_a.groupby('UserID').get_group('U1001')

And this is how we grouped our UserIDs and also checked for a particular ID name.

In the end

The overall content mentioned in this article is just the tip of the iceberg. Pandas, as I mentioned, is a powerful and comprehensive library with tons of functionalities. You can check out the important Pandas cheatsheet here or bump into a comprehensive 10 minutes to Pandas article here.

About the Author:

Hi there! My name is Akash and I’ve been working as a Python developer for over 4 years now. In the course of my career, I began as a Junior Python Developer at Nepal’s biggest Job portal site, Merojob. Later, I was involved in Data Science and research at Nepal’s first ride-sharing company, Tootle. Currently, I’ve been actively involved in some interesting Data Science as well as Web Development projects. As for web framework, I mostly work with Django these days.

You can find my other projects on:

https://github.com/akashadhikari

Connect me on LinkedIn

https://www.linkedin.com/in/akashadh/

Email: [email protected] | [email protected]

Website (Working on The Data Science Blog): https://akashadhikari.github.io/

End Notes:

Thanks for reading!

I hope enjoyed reading the article. If you found it useful, please share it among your friends on social media too. For any queries, suggestions, constructive criticisms, or any other discussion, please ping me here in the comments or you can directly reach me through email.

I am also planning to start The Data Science Blog on my Github page. I will try to include how real companies have been working in the field of Data Science, how to excel in Data Science and/or tech interviews, and other useful content related to Python and general programming. Feel free to check them once in a while.

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.