Exploratory Data Analysis (EDA) in Python

Introduction

Exploratory Data Analysis is a method of evaluating or comprehending data in order to derive insights or key characteristics. EDA can be divided into two categories: graphical analysis and non-graphical analysis.

EDA is a critical component of any data science or machine learning process. You must explore the data, understand the relationships between variables, and the underlying structure of the data in order to build a reliable and valuable output based on it.

The EDA stages will be carried out in this tutorial using the Python programming language.

The Dataset

For this article, we will be doing Customer Churn Prediction. When clients stop doing business with a company, this is known as customer churn or customer attrition.

Because the cost of getting a new customer is usually higher than keeping an existing one, understanding customer churn is critical to a company’s success. As a result, churn analysis is the first step in gaining a better understanding of your clients.

To gain a deeper grasp of our data, we will go deep into Exploratory Data Analysis (EDA). The dataset is available here.

Importing the Python Libraries

First of all, we need to import all the libraries that are required for the analysis, namely Pandas for handling data, Numpy for numerical calculations, Matplotlib and Seaborn for visualization.

.

Loading the Dataset in Python

Now, load the dataset into the pandas dataframe.

Structured Based Data Exploration

This is the first part of EDA where the data frame is evaluated for structure, columns and data types. The goal of this step is to get a general understanding of the dataset.

Display the first 5 Observations

We get the output as:

Display the Last 5 Observations

The output is:

Display the Number of Variables and Observations

This can be done with df.shape which gives the output as a tuple having 2 values. The first value counts the number of data points and the second value represents the number of features in the dataset.

In this dataframe, there are 7043 rows and 21 columns.

Display the Variable Names and their Data Types

Count the number of Non-Missing Values for each variable

df.count() counts the number of non-empty values. It gives the idea of missing values in our dataset.

Descriptive Statistics

Now to know more about the characteristics of the dataset we will use the df.describe() which by default gives statistical information of all numerical features in our data frame.

df.describe() gives some basic statistical details like count, percentile, mean, standard deviation, and the 5 point summary which includes minimum, first quartile, second quartile, third quartile and maximum of numerical features.

What about the categorical features?

By providing an include argument and assigning it the value ‘all’, we can get the summary of all the categorical features too.

Display the Complete Summary of the Dataset

df.info() gives the summary of the dataframe including data types, shape and memory storage.

Handling Missing Values

Missing values are the unknown values in the dataset. The concept of missing values is important to understand in order to successfully manage data. The first step is to detect the missing value in the dataset and then treat them using the appropriate method.

Detecting the Missing Values

• Using error = ‘coerce’ will replace all non-numeric values with NaN.

• isnull().sum() returns the number of missing values in the dataset.

We have 11 missing values in the ‘Total Charges’ column. Now, we will see different methods to deal with them.

Missing Value Treatment

To treat missing values we can use the following ways:

• Drop the variable

• Drop the observation(s)

• Mean imputation or median imputation or mode imputation

For variable ‘Total Charges’ only 11 values are missing. Since these data records are comparatively very low as compared to the total data set, we can drop them.

Done. Let’s check!

Analysis using Charts

Data Visualizations

Now, it’s time to visualize the data. We can see how the data appears and what sort of relation the properties of data hold with the help of data visualization. It’s the quickest approach to check if the features reflect the output.

Target Variable

Let’s visualize the target variable i.e. Churn. It has two categories- Yes or No.

Display a frequency distribution of churn

The plot shows a class imbalance of the data between churners and non-churners. To address this, resampling would be a suitable approach.

Categorical Variables

There are 17 Categorical features in the dataset. Let’s see their churning rate with respect to the target variable.

Note: I have only shown 5 graphs here which are more important according to me.

Relationship between Monthly Charges and Total Charges

Total charges are the sum total of monthly charges. So, let’s visualize their relationship.

• Here we can see that Total Charges and monthly charges are highly correlated.

Customer Contract Distribution

Here we are trying to visualize the churning rate with respect to Contract.

• About 75% of customers with Month-to-Month Contracts opted to move out as compared to 13% of customers with one-year contract and 3% with two-year contracts.

Payment Method Distribution

This is the visualization of the payment method. It has four categories.

•  The electronic check has the highest users.

Dependents distribution

This graph shows the churning rate with respect to Dependents.

• Customers without dependents are more likely to churn

Churn distribution w.r.t Partners

This graph shows the churning rate with respect to Partners.

• Customers that do not have partners are likely to churn more.

Conclusion

In this article, we tried to analyze customer behaviour. First, we explored the dataset on a basic level. We looked for missing values and treated them by dropping those values. Then we used the Pandas DataFrame to do Exploratory Data Analysis on sample data by plotting different graphs like Count plot, Pie Chart, Line Plot and Histplot. From this, we got some useful insights like: “Customers with month-to-month contracts churn the most”, “Total charges and monthly charges were highly correlated”, etc. This way, we perform EDA on the datasets to explore the data and extract all possible insights from it, which can help in model building and also better decision making.

However, this was only a basic overview of how EDA works; you can go deeper into it and attempt the stages on larger datasets.

You can reach out to me on LinkedIn.