Mastering LeNet: Architectural Insights and Practical Implementation

Introduction

LeNet-5, a pioneering convolutional neural network (CNN) developed by Yann LeCun and his team in the 1990s, was a game-changer in computer vision and deep learning. This groundbreaking architecture was explicitly crafted to revolutionize the recognition of handwritten and machine-printed characters. Unlike traditional methods, LeNet-5 introduced a novel approach that eliminated the need for manual feature engineering, directly processing pixel images through convolutional layers, subsampling, and fully connected layers. Its success extended beyond character recognition, serving as a cornerstone for modern deep learning models and influencing subsequent architectures in computer vision, object recognition, and image classification.

Yann LeCun’s early application of backpropagation algorithms to practical problems laid the foundation for LeNet-5, designed to read handwritten characters and excelling in identifying zip code numbers provided by the US Postal Service. Its successive versions and applications, such as the ability to read millions of checks daily, triggered a surge of interest among researchers, shaping the landscape of neural networks and inspiring the evolution of deep learning.

The success of LeNet-5 and subsequent applications, such as systems capable of reading millions of checks per day, sparked widespread interest among researchers in neural networks. While today’s top-performing neural network architectures have evolved beyond LeNet-5, its groundbreaking design, and accomplishments laid the foundation for numerous subsequent models, significantly shaping and inspiring deep learning. LeNet-5 remains a testament to innovation and an enduring symbol of the evolution of machine learning and image recognition.

Learning Objectives

• Explore the historical significance and impact of LeNet-5 on the evolution of deep learning and computer vision.
• Compare LeNet-5 with contemporary neural network architectures, examining its foundational influence on current models in deep learning.
• Understand the architecture of LeNet-5, including its convolutional, subsampling, and fully connected layers.
• Analyze practical applications and case studies showcasing the effectiveness of LeNet-5 in image recognition tasks.

This article was published as a part of the Data Science Blogathon.

Understanding LeNet

LeNet, also known as LeNet-5, is a pioneering convolutional neural network (CNN) architecture developed by Yann LeCun and his team in the 1990s. It was designed explicitly for handwritten and machine-printed character recognition tasks.LeNet-5’s significance lies in its successful demonstration of hierarchical feature learning and its effectiveness in character recognition. Its impact extends beyond its original purpose, influencing the development of modern deep learning models and serving as a foundational architecture for subsequent advancements in computer vision, image recognition, and various machine learning applications.

The Architecture of LeNet

LeNet-5 is a Convolutional Neural Network (CNN) with a specific architecture employed in character recognition tasks. It consists of multiple layers, excluding the input layer, containing trainable parameters. Notably, it processes 32×32-pixel images, more significant than the characters in its database, focusing on potentially distinctive features’ centering. Input pixel values are normalized for better learning efficiency.

LeNet’s architecture combines convolutional, subsampling, and fully connected layers with specific connectivity patterns. It uses normalization for input pixels and a series of layers to extract distinctive features from the data for efficient learning. Additionally, it implements unique strategies to prevent saturation of activation functions and uses specific loss functions for efficient training.

Unique Strategies to Prevent Saturation

• Input Layer: LeNet processes 32×32-pixel images, more significant than the characters in the database, aiming to capture potential distinctive features at the center of the image.
• Convolutional and Subsampling Layers: Convolutional layers primarily extract features from the input data using learnable filters or kernels. Each layer comprises multiple filters that slide over the input data (image) and perform element-wise multiplications to produce feature maps. The initial layer contains 6 filters of size 5×5, activating with the tanh function, leading to feature maps of size 28x28x6. Subsequent layers utilize 16 filters of the same size, generating feature maps 10x10x16.
• Subsampling layers, also known as pooling layers, focus on reducing the dimensionality of the feature maps obtained from the convolutional layers. Pooling involves merging or downsampling the feature maps, typically by taking the maximum value (MaxPooling) or average value (AveragePooling) in defined regions. With filter sizes of 2×2 and stride variations, these layers result in feature map sizes of 14x14x6 and 5x5x16 successively.
• Fully Connected Layers: The architecture includes fully connected layers labeled Fx, which process the final classification based on the extracted features. A fully connected layer with 84 neurons and a final output layer with 10 neurons, employing the tanh activation function in the former and Softmax in the latter. The Softmax function assigns probabilities to each class, with the highest probability determining the prediction.
• Output Layer: LeNet uses Radial Basis Function units for classification, with distinct representations of characters for recognition and correction.

Step By Step workflow

[Input: 28x28x1]
    |
[Conv2D: 6 filters, 5x5, tanh]
    |
[Average Pooling: 2x2, stride 2]
    |
[Conv2D: 16 filters, 5x5, tanh]
    |
[Average Pooling: 2x2, stride 2]
    |
[Flatten]
    |
[Dense: 120, tanh]
    |
[Dense: 84, tanh]
    |
[Dense: 10, softmax (output)]

Convolutional Layer 1:

• Number of filters: 6
• Kernel size: 5×5
• Activation function: Tanh
• Input shape: 28x28x1

Average Pooling Layer 1:

• Pool size: 2×2
• Strides: 2

Convolutional Layer 2:

• Number of filters: 16
• Kernel size: 5×5
• Activation function: Tanh

Average Pooling Layer 2:

• Pool size: 2×2
• Strides: 2

Fully Connected Layers:

• Dense layer with 120 units and Tanh activation.
• Thick layer with 84 units and Tanh activation.
• Output layer with 10 units and Softmax activation for multi-class classification (MNIST dataset).

Key Features of LeNet

1. CNN Architecture: LeNet-5 was a pioneering Convolutional Neural Network featuring a structured architecture with convolutional and pooling layers.
2. Pattern Recognition in Handwritten Digits: Developed initially for handwritten digit recognition, showcasing high accuracy in identifying and classifying handwritten characters.
3. Convolutional and Pooling Layers: Introduction of convolutional layers for feature extraction and pooling layers for downsampling, allowing the network to learn hierarchical representations progressively.
4. Non-linearity Activation: Utilized hyperbolic tangent (tanh) activation functions, providing the network with non-linear capabilities essential for capturing complex relationships within data.
5. Influence on Deep Learning: LeNet-5’s success laid the groundwork for contemporary deep learning models and significantly influenced the development of neural networks for image recognition and classification.

Practical Implementation of LeNet:

Import Library

Start with the code to implement LeNet-5 in TensorFlow using the Keras API. It’s a good beginning to work with the MNIST dataset.

import tensorflow as tf
from tensorflow import keras
from keras.datasets import mnist
from keras.layers import Dense, Flatten, Conv2D, AveragePooling2D
from keras.models import Sequential
from tensorflow.keras.utils import plot_model

Load Dataset

Load the MNIST dataset for training and testing images. This function loads the dataset, which consists of handwritten digit images and their respective labels. The data is divided into training and testing sets.

(X_train, y_train), (X_test,y_test) = mnist.load_data()

Output:

Reshape

The reshape function in this context is adjusting the shape of the images to make them suitable for processing in a CNN. The shape (28, 28, 1) signifies that the images are 28×28 pixels and have a single channel (grayscale images). This transformation is necessary because most CNNs expect images to be in a specific shape, often represented as (width, height, channels).

#perfoming reshape
X_train = X_train.reshape(X_train.shape[0],28,28,1)
X_test = X_test.reshape(X_test.shape[0],28,28,1)
# Check the shape of data 
X_train.shape

Normalization

The code snippet you’ve provided normalizes the image pixel values in the training and testing datasets. Divining every pixel value by 255 ensures that the pixel values range from 0 to 1.

# Normalization ---> convert 0 to 1
X_train = X_train/255
X_test = X_test/255

One Hot Encoding

The classes for the MNIST dataset are transformed into categorical data with 10 classes. Each label is converted into a vector where each element represents a class, with 1 in the index corresponding to the class and 0 elsewhere.

# One hot encoding
y_train = keras.utils.to_categorical(y_train,10)
y_test = keras.utils.to_categorical(y_test,10)

Model Build

This code snippet demonstrates constructing the LeNet-5 model using the Keras Sequential API in TensorFlow. It defines the layers and their configurations and compiles the model with an optimizer, loss function, and metrics for evaluation.

model = Sequential()
# first layer
model.add(Conv2D(6, kernel_size=(5,5), padding="valid", activation="tanh", input_shape =(28,28,1)))
model.add(AveragePooling2D(pool_size=(2,2),strides=2, padding='valid'))

#second layer
model.add(Conv2D(16, kernel_size=(5,5), padding="valid", activation="tanh"))
model.add(AveragePooling2D(pool_size=(2,2),strides=2, padding='valid'))

# flatten layer
model.add(Flatten())

# ANN
model.add(Dense(120, activation='tanh'))
model.add(Dense(84, activation='tanh'))
model.add(Dense(10, activation='softmax'))

model.summary()

Output:

Model Compile

The “compile” method prepares the model for training by defining its optimization method, loss function, and the metrics to monitor.

model.compile(loss= keras.metrics.categorical_crossentropy, optimizer =keras.optimizers.Adam(),\
metrics= ['accuracy'])

Model training: The “fit ” function trains the model using the provided training data and validates it using the test data.

model.fit(X_train,y_train, batch_size=128,epochs=10 , verbose=1, validation_data=(X_test,y_test))

output:

Model Evaluation

The model “evaluate()” function is utilized to evaluate the model’s performance on a test dataset. The result provides the test loss and test accuracy.

score = model.evaluate(X_test,y_test)

print('Test loss', score[0])
print('Test Accuracy', score[1])

Visualization:

# Create a bar chart to visualize the comparison
import matplotlib.pyplot as plt
predicted_labels = np.argmax(predictions, axis=1)

# Compare predicted labels with true labels
correct_predictions = np.equal(predicted_labels, np.argmax(y_test, axis=1))

plt.figure(figsize=(12, 6))
plt.bar(range(len(y_test)), correct_predictions, \
color=['green' if c else 'red' for c in correct_predictions])
plt.title('Comparison of Predicted vs. True Labels')
plt.xlabel('Sample Index')
plt.ylabel('Correct Prediction (Green: Correct, Red: Incorrect)')
plt.show()

Output:

Impact and Significance of LeNet

LeNet’s influence extends far beyond its original task. Its success paved the way for deeper exploration into convolutional neural networks (CNNs). Its efficient design and performance on digit recognition tasks set the stage for advancements in various computer vision applications, including image classification, object detection, and facial recognition.

• Revolution in Handwritten Character Recognition: LeNet-5’s success in recognizing handwritten digits and characters led to a transformation in various practical applications, particularly in recognizing postal zip codes and checks. Its ability to recognize characters accurately contributed to these applications’ widespread adoption of neural networks.
• Influence on Future Architectures: LeNet’s architectural design principles laid the foundation for numerous subsequent CNN models. Its innovative use of convolution, subsampling, and fully connected layers inspired the development of more complex and sophisticated neural network architectures for various image-based tasks.
• Promoting Deep Learning: LeNet-5’s success demonstrated the potential of deep learning networks in image recognition, inspiring further research and development in the field. Its impact on the research community led to a paradigm shift towards using deep neural networks for various vision-based tasks and laid the groundwork for subsequent advancements in the domain.

Application of LeNet

The impact of LeNet extends to numerous real-world applications. From recognizing handwritten digits in postal services to revolutionizing healthcare by aiding in medical image analysis, the foundational concepts of LeNet have influenced a myriad of fields.

• Document Processing: LeNet’s capabilities have found usage in scanning and analyzing documents, parsing and processing different types of information, extracting data from documents, and automating data entry tasks in various industries.
• Handwriting Recognition: LeNet’s success in recognizing handwritten characters and digits remains fundamental in Optical Character Recognition (OCR) systems used in processing handwritten text in bank checks, postal services, and forms. It’s applicable in digitizing historical documents and recognizing hand-written information in various formats.
• Biometric Authentication: Handwriting recognition capabilities of LeNet have been applied to signature and fingerprint analysis, enabling biometric authentication methods and enhancing security systems.
• Real-time Video Analysis: The foundational concepts in LeNet serve as a basis for real-time video analysis, such as object tracking, surveillance systems, facial recognition, and autonomous vehicles.
• Image Classification: LeNet’s principles influence modern image classification systems. Applications include classifying and categorizing objects in images for numerous domains, such as identifying objects in photographs, quality control in manufacturing, medical imaging analysis, and security systems for object identification.

Challenges and Limitations of LeNet

• Feature Extraction Efficiency: With the evolution of neural network architectures, newer models have more efficient ways of feature extraction, making LeNet comparatively less efficient in identifying intricate patterns and features.
• Limited Adaptability: Its architecture, designed for specific tasks such as handwritten character recognition, might not be directly transferable to other domains without substantial modifications.
• Scalability: Although a pioneering model, LeNet might lack the scalability to adapt to modern data processing and deep learning demands.
• Overfitting: LeNet might suffer from overfitting when dealing with more complex datasets, necessitating additional regularization techniques to mitigate this issue.

Researchers have developed more complex CNN architectures to overcome these limitations, incorporating sophisticated techniques to address these challenges while improving performance on various tasks.

Conclusion

LeNet, as an early convolutional neural network, is a pivotal milestone in deep learning. Its inception by Yann LeCun and the team marked a breakthrough, particularly in handwritten character recognition and image analysis. LeNet faces challenges adapting to modern complex tasks and diverse datasets due to architectural simplicity and potential overfitting. Its legacy remains vital, inspiring more advanced architectures and playing a crucial role in developing deep learning models.

LeNet’s inception marked a pivotal moment in the history of deep learning. Its success in image recognition tasks and the principles has set the stage for the evolution of modern convolutional neural networks. Its enduring legacy continues to shape the landscape of computer vision and artificial intelligence.

Key Takeaways

• It introduced the concept of convolutional and subsampling layers, setting the foundation for modern deep-learning architectures.
• While LeNet made significant advancements in its time, its limitations in handling diverse and complex datasets have become apparent.
• Introduced convolutional and subsampling layers, revolutionizing deep learning.

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Neha Vishwakarma

I'm Neha Vishwakarma, a data science enthusiast with a background in information technology, dedicated to using data to inform decisions and tackle complex challenges. My passion lies in uncovering hidden insights through numbers.