Study of Regularization Techniques of Linear Models and Its Roles

This article was published as a part of the Data Science Blogathon

Introduction to Regularization

The below graph represents the entire parameters existing in the LR model and is very self-explanatory.

Significance Of Cost Function 

Cost function/Error function: Takes slope-intercept (m and c) values and returns the error value/cost value. It shows the error between predicted outcomes is compared with the actual outcomes. It explains how your model is inaccurate in its prediction.

It is used to estimate how badly models are performing for the given dataset and its dimensions.

Why is cost function important in machine learning? Yes, the cost function helps us reach the optimal solution, So how can we do this. will see all possible methods and simple steps using Python libraries.

This function helps us to a figure-out best straight line by minimizing the error

The best fit line is that line where the sum of squared errors around the line is minimized

Regularization Techniques

Let’s discuss the available Regularization techniques and followed by the implementation 

1. Ridge Regression (L2 Regularization):

Basically here, we’re going to minimize the sum of squared errors and sum of the squared coefficients (β). In the background,
the coefficients (β) with a large magnitude will generate the graph peak and
deep slope, to suppress this we’re using the lambda (λ) use to be called a
Penalty Factor and help us to get a smooth surface instead of an irregular-graph. Ridge Regression is used to push the coefficients(β) value nearing zero in terms of magnitude. This is L2 regularization, since its adding a penalty-equivalent to the Square-of-the Magnitude of coefficients.

Ridge Regression = Loss function + Regularized term

2. Lasso Regression (L1 Regularization):

This is very similar to Ridge Regression, with little difference in Penalty Factor that coefficient is magnitude instead of squared. In which there are possibilities of many coefficients becoming zero, so that corresponding attribute/features become zero and dropped from the list, this ultimately reduces the dimensions and supports for dimensionality reduction. So which deciding that those attributes/features are not suitable as predators for predicting target value. This is L1 regularization, because of adding the Absolute-Value as penalty-equivalent to the magnitude of coefficients.

           Lasso Regression = Loss function + Regularized term

3. Characteristics of Lambda

	λ = 0	λ => Minimal	λ =>High
Lambda or Penalty Factor (λ)	No impact on coefficients(β) and model would be Overfit. Not suitable for Production	Generalised model and acceptable accuracy and eligible for Test and Train. Fit for Production	Very high impact on coefficients (β) and leading to underfit. Ultimately not fit for Production.

Remember one thing that the Ridge never make coefficients into zero, Lasso will do. So, you can use the second one for feature selection.

Impact of Regularization

The below graphical representation clearly indicates the best fitment. 

4. Elastic-Net Regression Regularization:

Even though Python provides excellent libraries, we should understand the mathematics behind this. Here is the detailed derivation for your reference.

• Ridge: α=0
• Lasso: α=1

5. Pictorial representation of Regularization Techniques

Mathematical approach for L1 and L2

Even though Python provides excellent libraries and straightforward coding, we should understand the mathematics behind this. Here is the detailed derivation for your reference.

Let’s have below multi-linear regression dataset and its equation

As we know Multi-Linear-Regression

y=β₀+ β₁ x₁₊ β₂ x_{2+………………+} β_n x_n —————–1

y_i= β_{0+ Σ} β_i x_i —————–2

Σ y_i– β₀– _Σ β_i x_i

Cost/Loss function:  _Σ{ y_i– β₀– _Σ β_i x_ij}²—————–3

Regularized term: _λΣ β_i²—————-4

Ridge Regression = Loss function + Regularized term—————–5

Put 3 and 4 in 5

Ridge Regression = _{Σ {} y_i– β₀– _Σ β_i x_ij}²+ _{λ Σ} β_i²

Lasso Regression = _{Σ {} y_i– β₀– _Σ β_i x_ij}²+ _{λ Σ} |β_i|

• x ==> independent variables
• y ==> target variables
• β ==> coefficients
• λ ==> penalty-factor

How coefficients(β) are calculated internally

Code for Regularization

Let’s take Automobile – Predictive Analysis and apply the L1 and L2 and how it helps model score.

Objective: Predicting the Mileage/Miles Per Gallon (mpg) of a car using given features of the car.

print("*************************")

print("Import required libraries")

print("*************************")

%matplotlib inline

import numpy as np

import pandas as pd

import seaborn as sns

import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression

from sklearn.linear_model import Ridge

from sklearn.linear_model import Lasso

from sklearn.metrics import r2_score

Output

Python Code:

import pandas as pd

print("*************************")
print(" Using auto-mpg dataset ")
print("*************************")
df_cars = pd.read_csv("auto-mpg.csv")
print(df_cars.head(5))

*************************
 Using auto-mpg dataset 
*************************

EDA: Will do little EDA (Exploratory Data Analysis), to understand the dataset

print("############################################")
print("          Info Of the Data Set")
print("############################################")
df_cars.info()

Observation:
1. we could see that the features and its data type, along with Null constraints.
2. Horsepower and name features are objects in the given data set. have to take care of during the modelling.

Data Cleaning/Wrangling:

Is the process of cleaning and consolidating the complex data sets for easy access and analysis.

• Action:
  • replace(‘?’,’NaN’)
  • Converting “horsepower” Object type into int

df_cars.horsepower = df_cars.horsepower.str.replace('?','NaN').astype(float)
df_cars.horsepower.fillna(df_cars.horsepower.mean(),inplace=True)
df_cars.horsepower = df_cars.horsepower.astype(int)
print("######################################################################")
print("          After Cleaning and type covertion in the Data Set")
print("######################################################################")
df_cars.info()

Output

Scale all the columns successfully done

Train and Test Split

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y_scaled, test_size=0.25, random_state=1)

Observation:
1. We could see that the features/column/fields and its data type, along with Null count
2. horsepower is now int type and name still as an object type in the given data set, since this column not going to support either way as preditors.

# Statistics of the data
display(df_cars.describe().round(2))

# Skewness and kurtosis
print("Skewness: %f" %df_cars['mpg'].skew())
print("Kurtosis: %f" %df_cars['mpg'].kurt())

Output: Look at the curve and how it is distributed across and see the same.

Skewness: 0.457066
Kurtosis: -0.510781

sns_plot = sns.distplot(df_cars["mpg"])

plt.figure(figsize=(10,6))
sns.heatmap(df_cars.corr(),cmap=plt.cm.Reds,annot=True)
plt.title('Heatmap',
         fontsize=13)
plt.show()

Output: Look at the heatmap

There is a strong NEGATIVE correlation between mpg and below features

Feature Selection

print("Predictor variables") 
X = df_cars.drop('mpg', axis=1)
print(list(X.columns))
print("Dependent variable") 
y = df_cars[['mpg']]
print(list(y.columns))

Output: Here is the Feature Selection

Predictor variables
[‘cylinders’, ‘displacement’, ‘horsepower’, ‘weight’, ‘acceleration’, ‘model_year’, ‘origin_america’, ‘origin_asia’, ‘origin_europe’]
Dependent variable
[‘mpg’]

Scaling the feature to align the data

from sklearn import preprocessing
print("Scale all the columns successfully done") 
X_scaled = preprocessing.scale(X)
X_scaled = pd.DataFrame(X_scaled, columns=X.columns)
y_scaled = preprocessing.scale(y)
y_scaled = pd.DataFrame(y_scaled, columns=y.columns)

Output

Scale all the columns successfully done

Train and Test Split

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y_scaled, test_size=0.25, random_state=1)

LinearRegression Fit and finding the coefficient.

regression_model = LinearRegression()
regression_model.fit(X_train, y_train)
for idcoff, columnname in enumerate(X_train.columns):
    print("The coefficient for {} is {}".format(columnname, regression_model.coef_[0][idcoff]))

Output: Try to understand the coefficient (β_i)

The coefficient for cylinders is -0.08627732236942003
The coefficient for displacement is 0.385244857729236
The coefficient for horsepower is -0.10297215401481062
The coefficient for weight is -0.7987498466220165
The coefficient for acceleration is 0.023089636890550748
The coefficient for model_year is 0.3962256595226441
The coefficient for origin_america is 0.3761300367522465
The coefficient for origin_asia is 0.43102736614202025
The coefficient for origin_europe is 0.4412719522838424

intercept = regression_model.intercept_[0]
print("The intercept for our model is {}".format(intercept))
Output

The intercept for our model is 0.015545728908811594
Scores (LR)

print(regression_model.score(X_train, y_train))
print(regression_model.score(X_test, y_test))

Output

0.8140863295352218
0.843164735865974

Now, will apply regularization techniques and review the scores and impact of the techniques on the model.

Create a Regularized RIDGE Model and coefficients.

ridge = Ridge(alpha=.3)
ridge.fit(X_train,y_train)
print ("Ridge model:", (ridge.coef_))

Output: Compare with LR model coefficient

Ridge model: [[-0.07274955 0.3508473 -0.10462368 -0.78067332 0.01896661 0.39439233
0.29378926 0.36094062 0.37375046]]

Create a Regularized LASSO Model and coefficients

lasso = Lasso(alpha=0.1)
lasso.fit(X_train,y_train)
print ("Lasso model:", (lasso.coef_))

Output: Compare with LR model coefficient and RIDGE, Here you could see that the few coefficients and zeroed (0) and during the fitment, they are excluded from the feature list.

Lasso model: [-0. -0. -0.01262531 -0.6098498 0. 0.29478559
-0.03712132 0. 0. ]

Scores (RIDGE)

print(ridge.score(X_train, y_train))
print(ridge.score(X_test, y_test))

Output

0.8139778320249321
0.8438110638424217

Scores (LASSO)

print(lasso.score(X_train, y_train))
print(lasso.score(X_test, y_test))

Output

0.7866202435701324
0.8307559551832127

Certainly, there is an impact on the model due to the Regularization L2 and L1.

Compare L2 and L1 Regularization

Hope after seeing the code level implementation, you could able to relate the importance of regularization techniques and their influence on the model improvements. As a final touch let’s compare the L1 & L2.

Model fitment justification during training and testing

• Model is doing strongly at training set and poorly in test set means we’re at OVERFIT
• Model is doing poor at both (Training and Testing) means we’re at UNDERFIT
• Model is doing better and considers ways in both (Training and Test), means we’re at the RIGHT FIT

Conclusion

I hope, what we have discussed so for, would really help you all how and why regularization techniques are important and inescapable while building a model. Thanks for your valuable time in reading this article. Will get back to you with some interesting topics. Until then Bye! Cheers! Shanthababu.

The media shown in this article is not owned by Analytics Vidhya and are used at the Author’s discretion

Shanthababu Pandian

Shanthababu Pandian has over 23 years of IT experience, specializing in data architecting, engineering, analytics, DQ&G, data science, ML, and Gen AI. He holds a BE in electronics and communication engineering and three Master’s degrees (M.Tech, MBA, M.S.) from a prestigious Indian university. He has completed postgraduate programs in AIML from the University of Texas and data science from IIT Guwahati. He is a director of data and AI in London, UK, leading data-driven transformation programs focusing on team building and nurturing AIML and Gen AI. He helps global clients achieve business value through scalable data engineering and AI technologies. He is also a national and international speaker, author, technical reviewer, and blogger.