Beginner’s Guide to Machine Learning Explainability

Permutation Feature Importance

One simple method is Permutation Feature Importance, It is a model inspection technique that can be used for any fitted estimator when the data is tabular. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled. This procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature.

Algorithm

• Train a model on a dataset
• Calculate the error metrics or score(s) for the trained model for reference
• Shuffled the validation data columns values one column(features) at a time and can be repeated K times
• Using the shuffled data to evaluate the trained model using the same error or score metrics for each iteration
• The features that affect the error metrics most are the important ones as it indicates a model dependency on that features
• We can calculate the importance score for each feature as

We can use the python sklearn package for build-in permutation_importance function

from sklearn.inspection import permutation_importance

We will be using the Xgboost algorithm with default parameters on the Boston dataset

xg = xgboost.XGBRegressor()
xg.fit(Xtrain, ytrain)

Now we will use the permutation_importance function on the test set to calculate the feature importance, we need to provide a trained model and number of shuffling iteration to perform (n_repeats parameter)

r = permutation_importance(xg, Xtest, ytest,
                           n_repeats=30,
                           random_state=0)

Here, we have set n_repeats=30

The returned object, r will contain feature importance values for each feature which we can visualize using the matplotlib python library

plt.figure(figsize=(10,4))
plt.bar(boston.feature_names,r.importances_mean)
plt.xlabel('Features')
plt.ylabel('Mean Importance')
plt.title('Feature importance using Feature Permutation Importance');

Source: Github

Caveats

If two features are correlated and one of them is permuted then

• The model still has the other correlated feature and in that case, both features will have lower importance value even if those are actually important
• It can be biased by unrealistic data instances formed by permutation

Computationally expensive for a large number of features

A good practice is to drop one of the correlated features based on domain understanding and try to apply the Permutation Feature Importance algorithm which will provide better feature understanding.

Let’s discuss another method to interpret the black box models.

Global surrogate models

How does it work?

• Get the predictions(yhat) from the black-box model
• Select any simple and explainable model(linear reg., decision tree..) as per the use case
• Train the selected model on the same dataset used for training the black-box model, using predictions(yhat) as the target
• Measure the performance, as to how well the surrogate model approximates the behavior of the black-box model
• Finally, we can interpret the global surrogate model

Advantages:

Flexible, as a selection of surrogate model, does not depend on the black-box model. If at some time we have a better performing black-box model in place of an existing black box, we do not have to change the method of interpretation

Disadvantages 

▪️How much confidence is enough(% of variance explained) in deciding the surrogate model is close enough to the black-box model
▪️ Interpretation becomes irrelevant if the black-box model is not performing well

LIME (Local Interpretation Model agnostic Explanation)

import lime 
import lime.lime_tabular

xg = xgboost.XGBRegressor()
xg.fit(Xtrain, ytrain)

Now we will create a lime explainer object, for which we have to specify the target column, features names, categorical features, and mode of the algorithm(regression or classification)

explainer = lime.lime_tabular.LimeTabularExplainer(Xtrain, 
                                                  feature_names=boston.feature_names, 
                                                  class_names=['price'], 
                                                  categorical_features=categorical_features,
                                                  verbose=True, mode='regression')

We can use the explain_instance method of the explainer object to interpret a particular instance of data

exp = explainer.explain_instance(Xtest[i], xg.predict, num_features=5)

i is the index in test data that we need to interpret

we can visualize the interpretation output using the show_in_notebook method

exp.show_in_notebook(show_table=True)

We will get similar to below output

https://christophm.github.io/interpretable-ml-book/feature-importance.html

https://christophm.github.io/interpretable-ml-book/global.html

https://scikit-learn.org/stable/modules/permutation_importance.html

https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance_multicollinear.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-multicollinear-py

LIME Research Paper: https://arxiv.org/pdf/1602.04938.pdf

https://github.com/marcotcr/lime

About Me

I am a Machine Learning Engineer, Solving challenging business problems through data and machine learning

Linkedin

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.