A Hands-On Discussion on Hyperparameter Optimization Techniques

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import pandas as pd
df=pd.read_csv('data.csv')
df.info()

df.isna().sum()

df=df.drop('Unnamed: 32',axis=1)

Next, let us proceed to understand the diagnosis column:

df['diagnosis'].value_counts()
#B-benign and M-malignant

df=df.drop('id',axis=1)

Observe that, in this example, we are planning to build an RFClassifier to predict if a given case comes under the category of B or M.Thus, the diagnosis column becomes our target variable(Y) and all the other columns are our inputs(features) denoted by the variable X.

X=df.drop('diagnosis',axis=1) 
y=df['diagnosis']

#### Train Test Split
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.20,random_state=0)

from sklearn.ensemble import RandomForestClassifier
rf_classifier=RandomForestClassifier().fit(X_train,y_train)
prediction=rf_classifier.predict(X_test)

from sklearn.metrics import confusion_matrix,classification_report,accuracy_score
print(confusion_matrix(y_test,prediction))
print(accuracy_score(y_test,prediction))
print(classification_report(y_test,prediction))

### Manual Hyperparameter Tuning
model=RandomForestClassifier(n_estimators=300,criterion='entropy',max_depth=100,min_samples_split=4,
                             max_features='sqrt',min_samples_leaf=10,random_state=100).fit(X_train,y_train)
predictions=model.predict(X_test)
print(confusion_matrix(y_test,predictions))
print(accuracy_score(y_test,predictions))
print(classification_report(y_test,predictions))

from sklearn.model_selection import RandomizedSearchCV
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt','log2']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 1000,10)]
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10,14]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4,6,8]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
              'criterion':['entropy','gini']}
print(random_grid)

rf=RandomForestClassifier()
rf_randomcv=RandomizedSearchCV(estimator=rf,param_distributions=random_grid,n_iter=100,cv=3,verbose=2,
                               random_state=100,n_jobs=-1)
### fit the randomized model
rf_randomcv.fit(X_train,y_train)

rf_randomcv.best_params_

best_random_grid=rf_randomcv.best_estimator_
from sklearn.metrics import accuracy_score
y_pred=best_random_grid.predict(X_test)
print(confusion_matrix(y_test,y_pred))
print("Accuracy Score {}".format(accuracy_score(y_test,y_pred)))
print(classification_report(y_test,y_pred))

from sklearn.model_selection import GridSearchCV

param_grid = {
    'criterion': [rf_randomcv.best_params_['criterion']],
    'max_depth': [rf_randomcv.best_params_['max_depth']],
    'max_features': [rf_randomcv.best_params_['max_features']],
    'min_samples_leaf': [rf_randomcv.best_params_['min_samples_leaf'], 
                         rf_randomcv.best_params_['min_samples_leaf']+2, 
                         rf_randomcv.best_params_['min_samples_leaf'] + 4],
    'min_samples_split': [rf_randomcv.best_params_['min_samples_split'] - 2,
                          rf_randomcv.best_params_['min_samples_split'] - 1,
                          rf_randomcv.best_params_['min_samples_split'], 
                          rf_randomcv.best_params_['min_samples_split'] +1,
                          rf_randomcv.best_params_['min_samples_split'] + 2],
    'n_estimators': [rf_randomcv.best_params_['n_estimators'] - 200, rf_randomcv.best_params_['n_estimators'] - 100, 
                     rf_randomcv.best_params_['n_estimators'], 
                     rf_randomcv.best_params_['n_estimators'] + 100, rf_randomcv.best_params_['n_estimators'] + 200]
}

#### Fit the grid_search to the data
rf=RandomForestClassifier()
grid_search=GridSearchCV(estimator=rf,param_grid=param_grid,cv=10,n_jobs=-1,verbose=2)
grid_search.fit(X_train,y_train)

Let’s now look at the optimal values of the parameters returned by this technique.

best_grid=grid_search.best_estimator_

y_pred=best_grid.predict(X_test)
print(confusion_matrix(y_test,y_pred))
print("Accuracy Score {}".format(accuracy_score(y_test,y_pred)))
print(classification_report(y_test,y_pred))

from hyperopt import hp,fmin,tpe,STATUS_OK,Trials

space = {'criterion': hp.choice('criterion', ['entropy', 'gini']),
        'max_depth': hp.quniform('max_depth', 10, 1200, 10),
        'max_features': hp.choice('max_features', ['auto', 'sqrt','log2', None]),
        'min_samples_leaf': hp.uniform('min_samples_leaf', 0, 0.5),
        'min_samples_split' : hp.uniform ('min_samples_split', 0, 1),
        'n_estimators' : hp.choice('n_estimators', [10, 50, 300, 750, 1200,1300,1500])
    }

def objective(space):
    model = RandomForestClassifier(criterion = space['criterion'], max_depth = space['max_depth'],
                                 max_features = space['max_features'],
                                 min_samples_leaf = space['min_samples_leaf'],
                                 min_samples_split = space['min_samples_split'],
                                 n_estimators = space['n_estimators'], 
                                 )
    #5 times cross validation fives 5 accuracies=>mean of these accuracies will be considered
    accuracy = cross_val_score(model, X_train, y_train, cv = 5).mean()

    # We aim to maximize accuracy, therefore we return it as a negative value
    return {'loss': -accuracy, 'status': STATUS_OK }

from sklearn.model_selection import cross_val_score
trials = Trials()
best = fmin(fn= objective,
            space= space,
            algo= tpe.suggest,
            max_evals = 80,
            trials= trials)

crit = {0: 'entropy', 1: 'gini'}
feat = {0: 'auto', 1: 'sqrt', 2: 'log2', 3: None}
est = {0: 10, 1: 50, 2: 300, 3: 750, 4: 1200,5:1300,6:1500}
print(crit[best['criterion']])
print(feat[best['max_features']])
print(est[best['n_estimators']])

trainedforest = RandomForestClassifier(criterion = crit[best['criterion']], max_depth = best['max_depth'], 
                                       max_features = feat[best['max_features']], 
                                       min_samples_leaf = best['min_samples_leaf'], 
                                       min_samples_split = best['min_samples_split'], 
                                       n_estimators = est[best['n_estimators']]).fit(X_train,y_train)
predictionforest = trainedforest.predict(X_test)
print(confusion_matrix(y_test,predictionforest))
print(accuracy_score(y_test,predictionforest))
print(classification_report(y_test,predictionforest))

What is TPOT Classifier?

TPOT stands for Tree-Based Pipeline Optimization Tool. It is used to automate Machine learning(AutoML). As the name suggests, it internally represents pipelines in the form of flexible ‘trees’.Its goal is to solve a given ML problem by automatically selecting one of the pipelines already available. Furthermore, it selects a pipeline among many based on its accuracy.

Genetic Programming(Genetic Algorithm)

It is based on the idea of ‘survival of the fittest. The popular theory proposed by Charles Darwin, states that only the best fit species will survive. Hence, the aim of our biological system is to pass on the best quality of parents to their children and thereby the best genes keep on getting transferred from a generation to its subsequent generations.

Similarly, Genetic programming is a hyperparameter optimization technique aiming to find the optimal solution from the given population. It is widely used to solve highly complex problems with wider search space and cannot be solved using the usual algorithms.

Phenotype refers to the raw and noisy inputs. Genotype, on the other hand, refers to fine and processed inputs. A phenotype has to thus be encoded to genotype and genotype can also be decoded to get phenotype back.

Initial Population:

The initial population must be as diverse as possible. More the diversity, the better will the performance of the algorithm be. For example, if you train your model to classify only black cats as cats, it will not work well on the images of golden cats because it has never seen golden cats before! That’s why diversity is important.

Fitness function:

The most stable output is given by the fitness function. Fitness is a measure of how relevant the prediction of the model is, to the true output. The fitness function is applied to all the values of the population.

Genetic operations:

1. Selection:

It is proportional to the fitness function. Here, the best parents having high values of fitness function will be selected. Generally, two parents are chosen to create a new population using cross-over and mutation.

2. Cross-over and Mutation:

The aim of cross-over and mutation is to generate a new population that takes us closer to the desired output for which their fitness values should be better than that of their previous generation.

There are several techniques of cross-over. For example, consider 2 words:horse and zebra.Say we must swap alphabet after 3rd position of the two words to get a new generation, then we will get the offsprings as horra and zebse respectively.

If our output is cake and a new generation results in the offsprings-bake, make and take, we can stop here as the fitness value of all the offsprings is the same and is equal to 4(as ‘ake’ is common in bake, make and take). Otherwise, if the fitness value decreases, we must backtrack and select parents with higher fitness values, as shown in the above flowchart. Cross-over should be repeated until an endpoint or, in other words, until convergence is reached.

Hyperparameter Optimization using TPOT

It is implemented using the tpot package. So, let’s go ahead and install it, in the first place:

!pip install tpot

Next, create an instance of the TPOTClassifier and obtain the optimal values for the parameters.

from tpot import TPOTClassifier
tpot_classifier = TPOTClassifier(generations= 5, population_size= 24, offspring_size= 12,
                                 verbosity= 2, early_stop= 12,
                                 config_dict={'sklearn.ensemble.RandomForestClassifier': param}, 
                                 cv = 4, scoring = 'accuracy')
tpot_classifier.fit(X_train,y_train)

As we have specified 5 generations, after running the 5th generation, the best pipeline’s values are displayed as shown in the above output. These are the required optimal values.

We then proceed to create a model with the optimal values as given by the TPOT technique.

rf_tpop=RandomForestClassifier(criterion='entropy', max_depth=120, max_features='auto', min_samples_leaf=2, min_samples_split=2, n_estimators=1400
                               )
rf_tpop.fit(X_train,y_train)

Now, it’s time to predict using the new model and thus check its performance as we did in the earlier techniques discussed so far.

predictionforest = rf_tpop.predict(X_test)
print(confusion_matrix(y_test,predictionforest))
print(accuracy_score(y_test,predictionforest))
print(classification_report(y_test,predictionforest))
acc = accuracy_score(y_test,predictionforest)

What is Optuna?

Optuna’s optimization algorithm ensures effective optimization using the following two components:

1. Sampling Strategy

2. Pruning Strategy

1. Sampling Strategy:

It tells us how to select the next hyperparameter and how to decide the next configuration using the previous history represented by Bayesian Optimization. A few examples include random search and CMA-ES which is a type of genetic algorithm.

We can also define our own sampling algorithm!!!

2. Pruning Strategy:

It detects the unpromising trials with the help of the intermediate learning curve. It performs early stopping of under-performing trials.

Hyperparameter Tuning using Optuna

It is implemented using the optuna package in Python.So, firstly, let us go ahead and install it:

!pip install optuna

Note that I have been using the exclamatory mark before the pip command as I have implemented all the code on Google Colab. If you are executing the above command in Google Colab or on Jupyter notebook, only then prefix the ‘!’ before pip in the above command.

Steps:

1. Define the objective function to minimize or maximize

2. Get hyperparameters with the suggest method

3. Run the search using study.optimize()

Steps 1 and 2:

import optuna
import sklearn.svm
def objective(trial):

    classifier = trial.suggest_categorical('classifier', ['RandomForest', 'SVC'])
    if classifier == 'RandomForest':
        n_estimators = trial.suggest_int('n_estimators', 200, 2000,10)
        max_depth = int(trial.suggest_float('max_depth', 10, 100, log=True))

        clf = sklearn.ensemble.RandomForestClassifier(
            n_estimators=n_estimators, max_depth=max_depth)
    else:
        c = trial.suggest_float('svc_c', 1e-10, 1e10, log=True)
        clf = sklearn.svm.SVC(C=c, gamma='auto')

    return sklearn.model_selection.cross_val_score(
        clf,X_train,y_train, n_jobs=-1, cv=3).mean()

Here, observe that we also want the Optuna technique to optimize between RandomForestClassifier and SVC. They suggest a method, as shown in the above code, has been used to achieve the same.

Step-3:

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

trial = study.best_trial

print('Accuracy: {}'.format(trial.value))

print("Best hyperparameters: {}".format(trial.params))

Now, we set off to get the best parameters given by the Optuna technique.

study.best_params

We should now re-train our RFClassifier with these optimal values and predict using it in order to check its performance.

rf=RandomForestClassifier(n_estimators=380,max_depth=15.4148845494433)
rf.fit(X_train,y_train)
y_pred=rf.predict(X_test)
print(confusion_matrix(y_test,y_pred))
print(accuracy_score(y_test,y_pred))
print(classification_report(y_test,y_pred))

Results

In this section, let’s finally compare the performances of all the models given by each of the hyperparameter optimization techniques discussed in this blog.

The above bar graph gives the test accuracy and the number of misclassifications, for both the categories, of each of the hyperparameter tuning techniques which we implemented. The test accuracy is nothing but the accuracy score of the model on the test dataset. It is denoted by the dark green bar. All such bars are tall which implies that all the models show good test accuracies!

‘B misclassified as M’ refers to the number of samples which are actually benign(B) but were misclassified by the model as malignant(M). It is denoted by the yellow bar.’M misclassified as B’ refers to the number of samples which are actually malignant(M) but were misclassified as benign(B). It is denoted by the red bar especially to indicate that this type of misclassification is dangerous. They can be noted in the confusion matrix as follows:

Thus, the best model is the one with the minimum number of dangerous misclassifications. Misclassifying M as B is dangerous because the patient is actually suffering from a harmful stage of breast cancer and will miss out on the important treatments required, as she was classified to be benign(B) which is a less harmful stage.

The models given by Bayesian and GridSearchCV optimization techniques are around 94.73% accurate but have 3 dangerous misclassifications. Coming to the model given by manual tuning, it shows an intermediate performance with around 95% accuracy. However, even in this case, the number of dangerous misclassifications is 2 which is also intermediate.

Finally, let’s talk about the models provided by the default model with default parameter values, RandomizedSearchCV, TPOT Classifier, and Optuna all of which perform the best, exhibiting around 96.49% accuracy. However, the model provided by RandomizedSearchCV’s optimal values outsmarts the rest with a comparatively lesser number of dangerous misclassifications. It has only 1 dangerous misclassification!

Therefore, the optimal values which are given by RandomizedSearchCV suit the best on our dataset, and thus RandomizedSearchCV is the best hyperparameter optimization technique for this problem. However, it will, of course, vary among datasets. So, every time, we must experiment with several hyperparameter tuning techniques before jumping to a conclusion.

You can get the entire code from here.

Hope you enjoyed reading my article.

Thank You.

References

1. All hyperparameter tuning techniques-implementation

2.GridSearchCV and RandomizedSearchCV-Theory

3. Bayesian optimization-Theory

4. Genetic Algorithm

5. Optuna-Theory