Easy Hyperparameter Tuning in Neural Networks using Keras Tuner

This article was published as a part of the Data Science Blogathon

In the last few articles, we discussed Neural Networks, their work and, their practical implementation in Python on the MNIST dataset. Continuing to the same, in this article we will look at how to tune the parameters of Neural Network to achieve the appropriate parameters which provide the highest training and testing accuracy, we don’t want overfitting in our data, right.

I would highly suggest going through the Implementation of ANN on MNIST data blog to understand this one better.

What are Hyperparameters?

Hyperparameters are the values we provide to the model and are used to improve the performance of the model. They are not automatically learned during the training phase but have to be provided explicitly.

Hyperparameters play a major role in the performance of the model and should be chosen and set such that the model accuracy improves. In Neural Network some hyperparameters are the Number of Hidden layers, Number of neurons in each hidden layer, Activation functions, Learning rate, Drop out ratio, Number of epochs, and many more. In this article, We are going to use the simplest possible way for tuning hyperparameters using Keras Tuner. 

Using the Fashion MNIST Clothing Classification problem which is one of the most common datasets to learn about Neural Networks. But before moving on to the Implementation there are some prerequisites to use Keras tuner. The following is required:

• Python 3.6+
• Tensorflow 2.0+ (I had Tensorflow 2.1.0 in my system but still it didn’t work so had to upgrade it to 2.6.0)

Some Frequently asked questions(FAQs):

1. How to check the Tensorflow version:

#use this command
print(tensorflow.__version__)

2. How to upgrade Tensorflow?

#Use the following command 
pip install --upgrade tensorflow --user

3. What to do if it still does not work?
–> Use Google colab

Let’s move on to the problem statement now. In the Fashion MNIST dataset, we have images of clothing such as Tshirt, trousers, pullovers, dresses, coats, sandals,s and have a total of 10 labels.

#importing necessary libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Flatten
from tensorflow.keras.datasets import fashion_mnist

#loading the data
(X_train,y_train),(X_test,y_test)=fashion_mnist.load_data()

#visualizing the dataset
for i in range(25):
    # define subplot
    plt.subplot(5, 5, i+1)
    # plot raw pixel data
    plt.imshow(X_train[i], cmap=plt.get_cmap('gray'))
# show the figure
plt.show()

#normalizing the images
X_train=X_train/255
X_test=X_test/255

In the last MNIST digit classification example, we flattened the dataset before building the model, but here we will do it in the model building code itself. I have explained the model building code in detail in the last article, kindly refer to that for an explanation.

Model Building

model=Sequential([
    #flattening the images
    Flatten(input_shape=(28,28)),
    #adding first hidden layer
    Dense(256,activation='relu'),
    #adding second hidden layer
    Dense(128,activation='relu'),
    #adding third hidden layer
    Dense(64,activation='relu'),
    #adding output layer
    Dense(10,activation='softmax')
])

#compiling the model
model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
#fitting the model
model.fit(X_train,y_train,epochs=10)

#evaluating the model
model.evaluate(X_test,y_test)

We have built the basic ANN model and got the training and testing accuracy as shown in the above figures. We can see the difference in accuracies and losses of the training and test sets. The loss in the training data is less but increases for the test data which can lead to wrong predictions on the unseen data.

Now let’s tune the Hyperparameters to get the values that can help in improving the model. We will be optimizing the following Hyperparameters in the model:

• Number of hidden layers
• Number of neurons in each hidden layer
• Learning rate
• Activation Function

But first, we need to install the Keras Tuner.

#use this command to install Keras tuner
pip install keras-tuner

#installing the required libraries
from tensorflow import keras
from keras_tuner import RandomSearch

Defining the function to build an ANN model where the hyperparameters will be the Number of neurons in the hidden layer and Learning rate.

def build_model(hp):          #hp means hyper parameters
    model=Sequential()
    model.add(Flatten(input_shape=(28,28)))
    #providing range for number of neurons in a hidden layer
    model.add(Dense(units=hp.Int('num_of_neurons',min_value=32,max_value=512,step=32),
                                    activation='relu'))
    #output layer
    model.add(Dense(10,activation='softmax'))
    #compiling the model
    model.compile(optimizer=keras.optimizers.Adam(hp.Choice('learning_rate',values=[1e-2, 1e-3, 1e-4])),loss='sparse_categorical_crossentropy',metrics=['accuracy'])
    return model

In the above code, we have defined the function by the name build_model(hp) where hp stands for hyperparameter. While adding the hidden layer we use hp.Int( ) function which takes the Integer value and tests on the range specified in it for tuning. We have provided the range for neurons from 32 to 512 with a step size of 32 so the model will test on neurons 32, 64,96,128…,512.

Then we have added the output layer. While compiling the model Adam optimizer is used with different values of learning rate which is the next hyperparameter for tuning. hp.Choice( ) function is used which will test on any one of the three values provided for the learning rate.

#feeding the model and parameters to Random Search
tuner=RandomSearch(build_model,
    objective='val_accuracy',
    max_trials=5,
    executions_per_trial=3,
    directory='tuner1',
    project_name='Clothing')

The code above uses the Random Search Hyperparameter Optimizer. The following variables are provided to the Random Search. The first is model i.e build_model, next objective is val_accuracy that means the objective of the model is to get a good validation accuracy. Next, the value of trails and execution per trail provided which is 5 and 3 respectively in our case meaning 15 (5*3) iterations will be done by the model to find the best parameters. Directory and project name are provided to save the values of every trial.

#this tells us how many hyperparameter we are tuning
#in our case it's 2 = neurons,learning rate
tuner.search_space_summary()

#fitting the tuner on train dataset
tuner.search(X_train,y_train,epochs=10,validation_data=(X_test,y_test))

The above code will run 5 trails with 3 executions each and will print the trail details which provide the highest validation accuracy. In the below figure, we can see the best validation accuracy achieved by the model.

We can also check the summary of all the trails done and the hyperparameters chosen for the best accuracy using the below code. The best accuracy is achieved using 416 neurons in the hidden layer and 0.0001 as the learning rate.

tuner.results_summary()

That’s how we perform tuning for Neural Networks using Keras Tuner.

Let’s tune some more parameters in the next code. Here we are also providing the range of the number of layers to be used in the model which is between 2 to 20.

def build_model(hp):                 #hp means hyper parameters
    model=Sequential()
    model.add(Flatten(input_shape=(28,28)))
    #providing the range for hidden layers  
    for i in range(hp.Int('num_of_layers',2,20)):         
        #providing range for number of neurons in hidden layers
        model.add(Dense(units=hp.Int('num_of_neurons'+ str(i),min_value=32,max_value=512,step=32),
                                    activation='relu'))
    model.add(Dense(10,activation='softmax'))    #output layer
    #compiling the model
    model.compile(optimizer=keras.optimizers.Adam(hp.Choice('learning_rate',values=[1e-2, 1e-3, 1e-4])),   #tuning learning rate
                  loss='sparse_categorical_crossentropy',metrics=['accuracy'])
    return model

#feeding the model and parameters to Random Search
tuner=RandomSearch(build_model,
    objective='val_accuracy',
    max_trials=5,
    executions_per_trial=3,
    directory='project',
    project_name='Clothing')

#tells us how many hyperparameters we are tuning
#in our case it's 3 =layers,neurons,learning rate
tuner.search_space_summary()
#fitting the tuner
tuner.search(X_train,y_train,epochs=10,validation_data=(X_test,y_test))

Summary and the best accuracy of the model in the below code. This time we got 0.89 as the validation accuracy.

tuner.results_summary()

Endnotes:

This was the simplest possible way to tune the parameters in Neural Network. Please refer to the official documentation of Keras Tuner for more details: https://keras.io/keras_tuner/

About the Author:

I am Deepanshi Dhingra currently working as a Data Science Researcher, and possess knowledge of Analytics, Exploratory Data Analysis, Machine Learning, and Deep Learning. Feel free to content with me on LinkedIn for any feedback and suggestions.

Deepanshi