ML and AI Model Explainability and Interpretability

In this article, we dive into the concepts of machine learning and artificial intelligence model explainability and interpretability. We explore why understanding how models make predictions is crucial, especially as these technologies are used in critical fields like healthcare, finance, and legal systems. Through tools like LIME and SHAP, we demonstrate how to gain insights into a model’s decision-making process, making complex models more transparent. The article highlights the differences between explainability and interpretability, and explains how these concepts contribute to building trust in AI systems, while also addressing their challenges and limitations.

Learning Objectives

• Understand the difference between model explainability and interpretability in machine learning and AI.
• Learn how LIME and SHAP tools enhance model transparency and decision-making insights.
• Explore the importance of explainability and interpretability in building trust in AI systems.
• Understand how complex models can be simplified for better understanding without compromising performance.
• Identify the challenges and limitations associated with AI model explainability and interpretability.

What Do Explainability and Interpretability Mean, and Why Are They Essential in ML and AI?

Explainability is a process of answering the why behind the model’s decision-making. For example, we can say an ML and AI model has explainability when it can provide an explanation and reasoning for the model’s decisions by explaining how the model split a particular node in the tree and explain the logic of how it was split.

On the other hand, Interpretability is a process that is involved with translating the model’s explanations and decisions to non-technical users. It helps Data Scientists understand things such as weights and coefficients contributing toward model predictions, and it helps non-technical users understand how the model made the decisions and to what factors the model gave importance in making those predictions.

As the AI and ML models are becoming more and more complex with hundreds of model layers and thousands to billions of parameters for example in LLM and deep learning models, it becomes extremely difficult for us to understand the model’s overall and local observation level decisions made by the model. Model explainability provides explanations with insights and reasoning for the model’s inner workings. Thus, it becomes imperative for Data Scientists and AI Experts to leverage explainability techniques into their model building process and this would also improve the model’s interpretability.

Benefits of Improving Model’s Explainability And Interpretability

Below we will look into the benefits of model’s explainability and interpretability:

Improved Trust

Trust is a word with broad meanings. It is the confidence in someone’s or something’s reliability, honesty, or integrity.

Trust is associated with people as well as non-living things. For example, relying on a friend’s decision-making or relying on a fully automated driving car to transport you from one place to another. Lack of transparency and communication can also lead to eroding of trust. Also, trust is built over time through small steps and repeated positive interactions. When we have consistent positive interactions with a person or thing, it strengthens our belief in their reliability, positive intentions, and harmlessness. Thus, trust is built over time through our experiences.

And, it plays an important role for us to rely on ML & AI models and their predictions.

Improved Transparency and Collaboration

When we can explain the inner workings of a machine or deep learning model, its decision-making process, and the intuition behind the rules and the choices made, we can establish trust and accountability. It also helps improve collaboration and engagement with the stakeholders and partners. 

Improved Troubleshooting

When something breaks or does not work as expected, we need to find the source of the problem. To do this, transparency into the inner workings of a system or model is crucial. It helps diagnose issues and take effective actions to resolve them. For example, consider a model predicting that person “B” should not be approved for a loan. To understand this, we must examine the model’s predictions and decisions. This includes identifying the factors the model prioritized for person “B’s” observations.

In such scenarios, model explainability would come very handy in looking deeper into the model’s predictions and decision-making related to person”B”. Also, while looking deeper into the model’s inner workings, we might quickly discover some biases that might be influencing and impacting model decisions.

Thus, having explainability with the ML and AI models and utilizing them would make the troubleshooting, monitoring, and continuous improvement efficient, and help identify and mitigate biases, and errors to improve model performance.

Popular Business Use Cases for ML and AI Explainability and Interpretability

We are always interested in the model’s overall prediction ability to influence and make data-driven informed decisions. There are numerous applications for the ML and AI models in various industries such as Banking and Finance, Retail, Healthcare, Internet. Commercial, Insurance, Automotive, Manufacturing, Education, Telecommunication, Travel, Space, and so on.

Following are some of the examples:

Banking and Finance

For the Banking and Finance industry, it is important to identify the right customer for giving loans or issuing credit cards. They are also interested in preventing fraudulent transactions. Also, this industry is highly regulated.

To make these internal processes such as application approvals and fraud monitoring efficient, the banking and finance leverage ML and AI modeling to assist with these important decisions. They utilize ML and AI models to predict outcomes based on certain given and known factors. 

Generally, most of these institutions continuously monitor transactions and data to detect patterns, trends, and anomalies. It becomes important for them to have the ability to understand the ML and AI model predictions for each application they process. They are interested in understanding the reasoning behind the model predictions and the factors that played an important role in making the predictions.

Now, let’s say an ML model predicted loan applications to be rejected for some of their customers with high credit scores, and this might not seem usual. In such scenarios, they can utilize model explanations for risk analysis and to gain deeper insights as to why the model decided to reject the customer application, and which of the customer factors played an important role in this decisionmaking. This discovery might help them detect, investigate, and mitigate issues, vulnerabilities, and new biases in their model decision-making and help improve model performance.

Healthcare

These days in the Healthcare industry, ML/AI models are leveraged to predict patient health outcomes based on various factors for example medical history, labs, lifestyle, genetics, etc.

Let’s say a Medical Institution uses ML/AI models to predict if the patient under their treatment has a high probability of cancer or not. Since these problems involve a person’s life, the AI/ML models are expected to predict outcomes with a very high level of accuracy.

In such scenarios, having the ability to look deeper into a model’s predictions, decision rules utilized, and understanding the factors influencing the predictions becomes important. The healthcare professional team would do their due diligence and would expect transparency from the ML/AI model to provide clear and detailed explanations related to the predicted patient outcomes and the contributing factors. This is where the ML/AI model explainability becomes essential.

This interrogation may sometimes help discover some hidden vulnerabilities and biases in the model decision-making and can be addressed to improve future model predictions.

Autonomous Vehicles

Autonomous vehicles are self-operating vehicles such as cars, freight trucks, trains, planes, ships, spaceships, etc. In such vehicles, AI and ML models play a crucial role in enabling these vehicles to operate independently, without human intervention. These models are built using machine learning and computer vision models. They enable autonomous cars/vehicles to perceive the information in their surroundings, make informed decisions, and safely navigate them.

In the case of autonomous vehicles designed to operate on roads, navigation means guiding the vehicle autonomously in real time i.e. without human intervention through crucial tasks such as detecting and identifying objects,  recognizing traffic signals and signs, predicting the object behaviors, maintaining lanes and planning paths, making informed decisions, and taking appropriate actions such as accelerating, braking, steering, stopping, etc.

Since autonomous road vehicles involve the safety of the driver, passengers, public, and public property, they are expected to work flawlessly and adhere to regulations and compliance, to gain public trust, acceptance, and adoption.

It is therefore very important to build trust in the AI and ML models on which these vehicles fully rely for making decisions. In autonomous vehicles, the AI and ML explainability is also known as Explainable AI(XAI). Explainable AI can used to improve user interaction by providing them feedback on AI actions and decisions in real-time, and these tools can also serve as tools to investigate AI decisions and issues, identify and eliminate hidden biases and vulnerabilities, and improve the autonomous vehicle models.

Retail

In the Retail industry, AI and ML models are used to guide various decisions such as product sales, inventory management, marketing, customer support and experience, etc. Having explainability with the ML and AI facilitates understanding of the model predictions, and a deeper look into issues related to predictions such as types of products not generating sales, or what will be the sales predictions for a particular store or outlet next month,  or which products would have high demand, and needs to be stocked, or what marketing campaigns have a positive impact on sales, etc.

From the above business use cases, we can see clearly that it is very important for the ML and AI models to have clear and usable explanations for the overall model as well as for individual prediction to guide business decisions and make business operations efficient.

Some of the complex models come with built-in explainability whereas some models rely on external tools for this. There are several model-agnostic tools available today that help us to add model explainability. We will look deeper into two of such tools available.

Tools for Improving ML and AI Model Explainability and Interpretability

Any tool that provides information related to the model decision-making process and the features contributions in model predictions is very helpful. Explanations can be made more intuitive through visualizations.

In this article, we will take a deeper look into two of the popularly used external tools to add ML and AI model explainability and interpretability:

• LIME (Local Interpretable Model-Agnostic Explanations)
• SHAP (SHapely Additive exPlanations)

LIME is model agnostic, meaning that it can be implemented with any machine learning and deep learning model. It can be used with machine learning models such as Linear and Logistic Regressions, Decision Trees, Random Forest, XGBoost, KNN, ElasticNet, etc. and with deep neural network models such as RNN, LSTM, CNN, pre-trained black box models, etc.

It works under the assumption that a simple interpretable model can be used to explain the inner workings of a complex model. A simple interpretable model can be a simple Linear Regression model or a Decision Tree Model. Here, we utilized a simple linear regression model as an interpretable model to generate explanations for the complex model using LIME/SHAP explanations.

LIME also called Local Interpretable Model-Agnostic Explanations works locally on a single observation at a time and helps us understand how the model predicted the score for this observation. It works by creating synthetic data using the perturbed values of features from the original observations.

What is Perturbed Data and How it is Created?

To create perturbed datasets for tabular data, LIME first takes all the features in the observation and then iteratively creates new values for the observation by slightly modifying the feature values using various transformations. The perturbed values are very close to the original observation value and from a neighborhood closer to the original value.

For text and image data types, LIME iteratively creates a dataset by randomly selecting features from the original dataset and creating new perturbed values from the features neighborhood for the features. The LIME kernel width controls the size of the data point neighborhood.

A smaller kernel size means the neighborhood is small and the points closest to the original value will significantly impact the explanations while for a large kernel size, the distant points could contribute to the LIME explanations.

Broader neighborhood sizes would lead to less precise explanations but could help uncover some broader trends in the data. For more precise local explanations, small neighborhood sizes should be preferred.

Understanding Figure

Through the figure (Fig-1) below we try to give some intuition into the perturbed values, kernel size, and the neighborhood.

For this discussion, we have used data examples from the Bigmart dataset and it is a regression problem. We utilized tabular data for the LIME.

Considering observation #0 from the Bigmart dataset. This observation has a feature ‘Item_Type’ with a value of 13. We calculated the mean and standard deviation for this feature and we got the mean value to be 7.234 and the standard deviation equal to 4.22. This is shown in the figure above. Using this information, we then calculated the Z-score equal to 1.366. 

The area to the left of the Z-score gives us the % of values for the feature that would fall below the x. For a Z-score of 1.366, we would have about 91.40% values for the feature that would fall below x=13. Thus, we get an intuition that the kernel-width would have to be below x=13 for this feature. And, the kernel width would help control the size of the neighborhood for perturbed data.

Below Fig-2 shows three original test data points from the Bigmart dataset and we have considered these for gaining intuition of the LIME process.  XGBoost is a complex model and it was used to generate predictions on the original observations instances.

For this article, we will be using the top 3 records from the Bigmart preprocessed and encoded dataset to provide examples and explanations to support the discussion.

LIME Distance Formula

LIME internally uses the distance between the original data point and the points in the neighborhood and calculates the distance using the Euclidean distance. Let’s say the point X = 13 has coordinates (x1,y1) and another point in the neighborhood has coordinates (x2, y2), the Euclidean distance between these two points is calculated using the below equation:

The figure (Fig-4) below shows the blue perturbed data points and the original value as the red data point. The perturbed data point at a shorter distance from the original data point will be more impactful for LIME explanations.

The above equation considers 2D. Similar equations can be derived for data points having N number of dimensions.

The kernel width helps LIME determine the size of the neighborhood for selecting the perturbed values for the feature. As the values or the data points move away from the original value, they would become less impactful in predicting the model outcomes.

The figure (Fig-6) below shows the perturbed feature values, along with their similarity score to the original value, and the perturbed instance predictions using the  XGBoost model, and figure (Fig-5) shows the information for a black box interpretable simple model (Linear Regression).

How In-Built Explainability and Interpretability Work in Complex Models

Complex models such as  XGBoost, Random Forest, etc. come with basic in-built model explainability features. The XGBoost model provides model explainability at a global level and is unable to explain the predictions at an observation local level.

Since for this discussion, we have utilized XGBoost as a complex model, we have discussed its in-built model explainability below. The XGBoost provides us with features to plot the decision tree for gaining intuition into the model’s global decision-making and its feature importance for predictions. Feature importance returns a list of features in order of their contribution significance towards the model’s outcomes.

First, we initiated an XGBoost model and then trained it using the independent and target features from the training set. The XGBoost model’s in-built explainability features were used to gain insights into the model.

To plot the XGBoost in-built explanations use the following source code:

# plot single tree
plot_tree(xgbr_model)
plt.figure(figsize=(10,5))
plt.show()

The figure (Fig-7) below shows the output decision tree of the above Bigmart complex XGBoost model.

From the above XGBoost model tree, we get some insights into the model’s decision-making and the conditional rules it utilized to split the data and make the final prediction. From the above, it seems for this XGboost model, the feature Item_MRP contributed the most towards the outcome, followed by the Outlet_Type in decision making. We can verify this by using XGBoost’s feature importance.

Source Code to Display the Feature Importance

To display the feature importance for the XGBoost model using the in-built explanation, use the following source code.

# feature importance of the model
feature_importance_xgb = pd.DataFrame()
feature_importance_xgb['variable'] = X_train.columns
feature_importance_xgb['importance'] = xgbr_model.feature_importances_

# feature_importance values in descending order
feature_importance_xgb.sort_values(by='importance', ascending=False).head()

The figure(Fig-9) below shows the feature importance generated using the above XGBoost model in-built explanations.

From the above XGBoost feature importances, interestingly we see that for the XGboost model, the Outlet_Type had a higher contributing magnitude than the Item_MRP. Also, the model provided information for the other contributing features and their impact on model predictions. 

As we notice, the XGBoost model explanations are at a global level and provide a good amount of information but some additional information such as the direction of feature contribution is missing and we do not have insights for local level observations. The direction would tell us if the feature is contributing towards increasing the predicted values or decreasing the predicted values. For classification problems, the direction of feature contributions would mean knowing whether the feature is contributing towards class “1” or class”0”.

This is where external explainability tools such as LIME and SHAP can be useful and complement the XGBoost model explainability with the information on the direction of feature contribution or feature impact. For models with no built-in functionalities for explaining the model decision-making process, LIME helps add this ability to explain its prediction decisions for local as well as global instances.

How does LIME Model Decision-Making Work and How to Interpret its Explanations?

LIME can be used with complex models, simple models, and also with black box models where we do not have any knowledge of the model working and have only the predictions.

Thus, we can fit the LIME model directly with a model needing explanations, and also we can use it to explain the black box models through a surrogate simple model.

Below we will use the XGBoost regression model as a complex as well as black box model and leverage a simple linear regression model to understand the LIME explanations for the black box model. This will also allow us to compare the explanations generated by LIME using both approaches for the same complex model.

To install LIME library, use the following code:

# install lime library
!pip install lime

# import Explainer function from lime_tabular module of lime library
from lime.lime_tabular import LimeTabularExplainer

Approach1: How to Implement and Interpret LIME Explanations using the Complex XGBR Model?

To implement the LIME explanation directly with the complex model such as XGBoost use the following code:

# Fit the explainer model  using the complex model and show the LIME explanation and score
explanation = explainer.explain_instance(X_unseen_test.values[0], xgbr_model.predict)
explanation.show_in_notebook(show_table=True, show_all=False)
print(explanation.score)

This would generate an output that looks like the figure shown below.

From above we see that the perturbed observation #0 has a similarity score of 71.85% and this indicates that the features in this observation were 71.85% similar to that of the original observation. The predicted value for observation #0 is 1670.82, with an overall range of predicted values between 21.74 and 5793.40. 

LIME identified the most contributing features for the observation #0 predictions and arranged them in descending order of the magnitude of the feature contributions.

The features marked in blue color indicate they contribute towards decreasing the model’s predicted values while the features marked in orange indicate they contribute towards increasing the predicted values for the observation i.e. local instance #0.

Also, LIME went further by providing the feature-level conditional rules utilized by the model for splitting the data for the observation.

Visualizing Feature Contributions and Model Predictions Using LIME

In the figure(Fig-13) above, the plot on the left indicates the overall range of predicted values (min to max) by all observations, and the value at the center is the predicted value for this specific instance i.e. observation.

The plot at the center displays the blue color represents the negatively contributing features towards model prediction and the positively contributing features towards model prediction for the local instance are represented by the color orange. The numerical values with the features indicate the feature perturbed values or we can say they indicate the magnitude of the feature contribution towards the model prediction, in this case, it is for the specific observation (#0) or local instance.

The plot on the very right indicates the order of feature importance given by the model in generating the prediction for the instance. 

Note: Every time we run this code, the LIME selects features and assigns slightly new weights to them, thus it may change the predicted values as well as the plots.

Approach 2: How to Implement and Interpret LIME Explanations for Black Box Model (XGBR) using Surrogate Simple LR Model?

To implement LIME with complex black box models such as XGBoost, we can use the surrogate model method.  For the surrogate model, we can use simple models such as Linear Regression or Decision Tree models. LIME works very well on these simple models. And, we can also use a complex model as a surrogate model with LIME.

To use LIME with the surrogate simple model first we will need predictions from the black box model.

# Black box model predictions
y_xgbr_model_test_pred

Second step

In the second step using the complex model, independent features from the train set, and the LIME, we generate a new data set of perturbed feature values, and then train the surrogate model (Linear Regression in this case) using the perturbed features and the complex model predicted values.

# Initiate Simple LR Model
lr_model = LinearRegression()

# Fit the simple model using the Train X 
# and the Complex Black Box Model Predicted Predicted values
lr_model.fit(X_train, y_xgbr_model_test_pred)

#predict over the unseen test data
y_lr_surr_model_test_pred = lr_model.predict(X_unseen_test)
y_lr_surr_model_test_pred.mean()

To generate the perturbed feature values using LIME, we can utilize the following source code shown below.

# Initialize the explainer function
explainer = LimeTabularExplainer(X_train.values, mode="regression", feature_names=X_train.columns)#i

# Copy the test data
X_observation = X_unseen_test

The above code works for regression. For the classification problems, the mode would have to be modified to “classification”.

Note

Finally, we fit the LIME for the local instance #0 using the surrogate LR model and view the explanations for it. This will also help to interpret the feature contributions for the black box model (XGBR). To do this, use the code shown below.

# Now we will use the mean of all observations to see the model explainability using LIME

#  fit the explainer model and show explanations and score
explanation = explainer.explain_instance(X_unseen_test.values[0], lr_model.predict)
explanation.show_in_notebook(show_table=True, show_all=False)
print(explanation.score)

On executing the above we got the following LIME explanations as shown in figure(Fig-13) below.

One thing that we immediately noticed was that when we used the LIME directly with the XGBoost model, the LIME explanations score was higher (71.85%) for observation #0 and when we treated it as a black box model and used a surrogate LR model to get the LIME explanations for the black box model(XGBoost), there is a significant drop in the explanation score (49.543%). This indicates with the surrogate model approach there would be less number of features in the observation that would be similar to the original features and therefore, there can be some difference in the predictions using the explainer as compared to the original model and LIME of original model.  

The predicted value for observation #0 is 2189.59, with an overall range of predicted values between 2053.46 and 2316.54. 

The predicted value for observation #0 using LIME XGBR was 1670.82.

How to Access LIME Perturbed Data?

To view the LIME perturbed values use the following code.

# Accessing perturbed data
perturbed_data = explanation.as_list()
perturbed_data

The output from above would look something like as shown in the figure below.

# Accessing Feature Weights
for feature, weight in perturbed_data:
    print(feature, weight)

LIME Feature Importance

Each instance in the model gives different feature importance in generating the prediction for the instance. These identified model features play a significant role in the model’s predictions. The feature importance values indicate the perturbed feature values or the new magnitude of the identified features for the model prediction.

What is the LIME Explanation Score and How to Interpret It?

The LIME explanation score indicates the accuracy of LIME explanations and the role of the identified features in predicting the model outcomes. The higher explainable score indicates that the identified features by the model for the observation played a significant role in the model prediction for this instance. From the above figure(Fig-13),  we see that the interpretable surrogate LR model gave a 0.4954 score to the identified features in the observation.

Now let’s look into another tool named SHAPely for adding explainability to the model.

Understanding SHAP (SHapley Additive Explanations)

Another popularly used tool for ML and AI model explanations is the SHAP (SHapely Additive exPlanations). This tool is also model agnostic. Its explanations are based on the cooperative game theory concept called “Shapley values”. In this game theory, the contributions of all players are considered and each player is given a value based on their contribution to the overall outcome. Thus, it provides a fair and interpretable insight into the model decisions.

According to Shapely, a coalition of players works together to achieve an outcome. All players are not identical and each player has distinct characteristics which help them contribute to the outcome differently. Most of the time, it is the multiple player’s contributions that help them win the game. Thus, cooperation between the players is beneficial and needs to be valued, and should not depend only on a single player’s contribution to the outcome. And, per Shapely, the payoff generated from the outcome should be distributed among the players based on their contributions. 

SHAP ML and AI model explanation tool is based on the above concept. It treats features in the dataset as individual players in the team(observation). The coalitions work together in an ML model to predict outcomes and the payoff is the model prediction. SHAP helps fairly and efficiently distribute the outcome gain among the individual features (players), thus recognizing their contribution towards model outcomes.

Fair Distribution of Contributions Using Shapley Values

In the figure (Fig-15) above, we have considered two players participating in a competition and the outcome is attained in the form of prize money earned.  The two players participate by forming different coalitions (c12, c10, c20, c0), and through each coalition they earn different prizes. Finally, we see how the Shapely average weights help us determine each player’s contribution toward the outcome, and fairly distribute the prize money among the participants.

In the case of “i”  players, the following equation shown in the figure(Fig-16) can be used to determine the SHAP value for each player or feature.

Let’s explore the SHAP library further.

How to Install SHAP Library Installation and Initialize it?

To install the SHAP library use the following source code as shown below.

# Install the Shap library
!pip install shap

# import Shap libraries
import shap

# Initialize the Shap js
shap.initjs()

# Import libraries
from shap import Explainer

How to Implement and Interpret Complex XGBR Model SHAP Explanations?

SHAP libraries can be used directly with the complex models to generate explanations. Below is the code to use SHAP directly with the complex XGBoost model (using same model instance as used for the LIME explanations).

# Shap explainer
explainer_shap_xgbr = shap.Explainer(xgbr_model)

How to Generate SHAP Values for Complex XGBR Model?

# Generate shap values
shap_values_xgbr = explainer_shap_xgbr.shap_values(X_unseen_test)

# Shap values generated using Complex XGBR model
shap_values_xgbr

The above will display the arrays of SHAP values for each of the feature players in the coalitions i.e. observations in the test dataset.

The SHAP values would look something like as shown in figure(Fig-19) below:

What are the SHAP Feature Importance for the Complex XGBR Model?

SHAP helps us identify which features contributed to the model’s outcome. It shows how each feature influenced the predictions and their impact. SHAP also compares the contribution of features to others in the model.

SHAP achieves this by considering all possible permutations of the features. It calculates and compares model outcomes with and without the features, thus calculating each feature contribution along with the whole team(all players a.k.a features considered).

How to Implement and Interpret SHAP Summary Plot for the Complex XGBR Model?

SHAP summary plot can be used to view the SHAP feature contributions, their importance, and impact on outcomes.

Following is the figure(Fig-20) shows the source code to generate the summary plot.

# Display the summary plot using Shap values
shap.summary_plot(shap_values_xgbr, X_unseen_test)

The figure(Fig-21) above shows a SHAP summary plot for the Bigmart data. From above we see that SHAP arranged the features from the Bigmart data set in the order of their importance. On the right-hand side, we see the features arranged from high-value features at the top and low value arranged at the bottom. 

Also, we can interpret the impact of model features on its outcome. The feature impact is plotted horizontally centered around the SHAP mean value. The SHAP values for the feature on the left of the SHAP mean value are indicated in pink color signifying its negative impact. The feature SHAP values on the right of the SHAP mean value signify the feature contribution towards positive impact. The SHAP values also indicate the magnitude or influence of the features on the outcome. 

Thus, SHAP presents an overall picture of the model indicating the magnitude and direction of the contribution of each feature towards the predicted outcome.

How to Implement and Interpret SHAP Dependence Plot for the Complex XGBR Model?

# Display SHAP dependence plot
shap.dependence_plot("Item_MRP", shap_values_xgbr, X_unseen_test, interaction_index="Outlet_Type")

The SHAP feature dependence plot helps us interpret the feature relationship with another feature. In the above plot, it seems the Item_MRP is dependent on the Outlet_Type. For Outlet_Types 1 to 3, the Item_MRP has an increasing trend, whereas as seen from the above for Outlet_Type  0 to Outlet_Type 1, Item_MRP has a decreasing trend.

How to Implement and Interpret SHAP Force Plot for the Complex XGBR Model?

So far we saw SHAP feature importance, impact, and decision-making at a global level. The SHAP force plot can be used to get an intuition into the model decision-making at a local observation level.

To utilize the SHAP force plot, we can use the code below. Remember to use your own dataset names. The following code looks into the first observation for the test dataset i.e. X_unseen_test.iloc[0]. This number can be changed to look into different observations.

#Shap force plots
shap.plots.force(explainer_shap_xgbr.expected_value, shap_values_xgbr[0,:], X_unseen_test.iloc[0, :], matplotlib = True)

We can interpret the above force plot as below. The base value indicates the predicted value for the local instance #0 using the SHAP surrogate LR model. The features marked in dark pink color are the ones that are pushing the prediction value higher whereas the features marked in blue color are pulling the prediction towards a lower value. The numbers with the features are the feature original values.

How to Implement and Interpret SHAP Decision Plot for the Complex XGBoost Model?

To display the SHAP dependence plot we can use the following code as shown in Fig-24 below.

# Shap dependence plot
shap.decision_plot(explainer_shap_xgbr.expected_value, shap_values_xgbr[0,:], X_unseen_test.columns)

The SHAP decision plot is another way of looking at the impact of different model features on the model prediction. From the decision plot below, we tried to visualize the impact of various model features on the predicted outcome i.e. Item Outlet Sales.

From the decision plot below, we observe that the feature Item_MRP positively impacts the predicted outcome. It increases the item outlet sales. Similarly, Outlet_Identifier_OUT018 also contributes positively by raising the sales. On the other hand, Item_Type negatively impacts the outcome. It decreases the item outlet sales. Likewise, Outlet_Identifier_27 also reduces the sales with its negative contribution.

The plot below shows the decision plot for the Big Mart Sales Data.

How to Implement and Interpret SHAP Force Plot for Complex XGBR Model using TreeExplainer?

# load the JS visualization code to notebook
shap.initjs()

# explain the model's predictions using SHAP values
explainer_shap_xgbr_2 = shap.TreeExplainer(xgbr_model)
shap_values_xgbr_2 = explainer_shap_xgbr_2.shap_values(X_unseen_test)

# visualize the first prediction's explainations
shap.force_plot(explainer_shap_xgbr_2.expected_value, shap_values_xgbr_2[0, :], X_unseen_test.iloc[0, :])

# visualize the training set predictions
shap.force_plot(explainer_shap_xgbr_2.expected_value, shap_values_xgbr_2, X_unseen_test)

How to Implement and Interpret Black Box Model SHAP Explanations using Surrogate Model?

To use the SHAP explanations with the surrogate model (Linear Regression Model used here) use the following code. The Linear Regression Model is trained using the predictions from the black box model and the training set independent features.

# Wrap the explainer in a function called Explainer and create a SHAP explainer object
explainer_shap = Explainer(lr_model.predict, X_train)
# Generate Shap values
shap_values = explainer_shap.shap_values(X_unseen_test)
shap_values[:3]

For the SHAP explainer surrogate model, the SHAP values would look something like below.

How to Implement and Interpret the SHAP Summary Plot for the Black Box Model using the Surrogate LR Model?

To display the SHAP summary plot for the Black Box Surrogate Model, the code would look like below.

# Display the summary plot using Shap values
shap.summary_plot(shap_values, X_unseen_test)

From the above SHAP summary plot for the black box surrogate LR model, the Item_Type and Item_MRP are among the highest contributing features with Item_Type having overall neutral impact while the Item_MRP seems to be pulling towards right hand side indicating it is contributing towards increasing the outcome (i.e. Item_Outlet_Sales).

How to Implement and Interpret the SHAP Dependence Plot for Black Box Surrogate Simple LR Model?

To Implement the SHAP Dependece Plot using the surrogate LR model, use the following code.

# Display SHAP dependence plot
shap.dependence_plot("Item_MRP", shap_values, X_unseen_test, interaction_index="Outlet_Type")

The output of this will look like below.

From the above plot we can say that for the Black Box Surrogate LR model, the MRP has an increasing trend for outlet types 0 and 1 while it has a decreasing trend for outlet types 3.

Comparison Table of Models

Below we will look into the table for comparing each model

Aspect	LIME	SHAP	Blackbox Surrogate LR Model	XGBR Model (Complex)
Explainability	Local-level explainability for individual predictions	Global-level and local-level explainability	Limited explainability, no local-level insights	Limited local-level interpretability
Model Interpretation	Uses synthetic dataset with perturbed values to analyze model’s decision rationale	Uses game theory to evaluate feature contributions	No local-level decision insights	Global-level interpretability only
Explanation Score	Average explanation score = 0.6451	Provides clear insights into feature importance	Lower explanation score compared to LIME XGBR	Higher prediction accuracy but lower explanation
Accuracy of Closeness to Predicted Value	Matches predicted values closely in some cases	Provides better accuracy with complex models	Low accuracy of closeness compared to LIME	Matches predicted values well but limited explanation
Usage	Helps diagnose and understand individual predictions	Offers fairness and transparency in feature importance	Not suitable for detailed insights	Better for high-level insights, not specific
Complexity and Explainability Tradeoff	Easier to interpret but less accurate for complex models	Higher accuracy with complex models, but harder to interpret	Less accurate, hard to interpret	Highly accurate but limited interpretability
Features	Explains local decisions and features with high relevance to original data	Offers various plots for deeper model insights	Basic model with limited interpretability	Provides global explanation of model decisions
Best Use Cases	Useful for understanding decision rationale for individual predictions	Best for global feature contribution and fairness	Used when interpretability is not a major concern	Best for higher accuracy at the cost of explainability
Performance Analysis	Provides a match with XGBR prediction but slightly lower accuracy	Performs well but has a complexity-accuracy tradeoff	Limited performance insights compared to LIME	High prediction accuracy but with limited interpretability

Insights from LIME’s Perturbed Features and Model Explainability

Also, on analyzing the LIME perturbed values, we get some intuition into how the LIME selected features and then assigned perturbed weights to them and try to bring predictions closer to the original.

Bringing all the LIME models and observations (for top 3 rows and selected features) we get following.

From the above, we see that for Observation #0, the original XGBR model prediction and the LIME XGBR model prediction are a match, while for the same original feature values, the Blackbox Surrogate Model predictions for Observation # 0 are way off. At the same time, the LIME XGBR model showcased a high Explanation Score( Similarity of features to original features).

The average of the explanation score for the complex LIME XGBR model is 0.6451 and the for the Black Box Surrogate LR LIME Model is 0.5701.  In this case, the average explanation score for LIME XGBR is higher than the black box model. 

Accuracy of Closeness of Predicted Value

 Below we analyzed the % accuracy of closeness of predicted values for the three models.

The % accuracy of the predicted values by the Simple LR model and the LIME complex XGBR model are the same, with both models achieving 100% accuracy for Observation #1. This indicates that the predicted values closely match the actual predictions made by the complex XGBR model. Generally, a higher % accuracy of closeness reflects a more accurate model.

When comparing predicted and actual values, a discrepancy is observed. For Observation #3, the predicted value (2174.69) is significantly higher than the actual value (803.33). Similarly, the % accuracy of closeness was calculated for the LIME Complex XGBR and Blackbox Surrogate LR models. The results highlight varying performance metrics, as detailed in the table.

From above we see that, for Observation # 1, the Blackbox Surrogate LR model performed best. At the same time for the other two observations (#2 and #3), both the model performance is equal. 

The average performance for the LIME Complex XGBR model is about 176 and the Blackbox Surrogate LR model is about 186.

Therefore, we can say that LIME Complex Model Accuracy < LIME Blackbox Surrogate LR Model Accuracy.

Conclusion

LIME and SHAP are powerful tools that improve the explainability of machine learning and AI models. They make complex or black-box models more transparent. LIME specializes in providing local-level insights into a model’s decision-making process. SHAP offers a broader view, explaining feature contributions at both global and local levels. While LIME’s accuracy may not always match complex models like XGBR, it is invaluable for understanding individual predictions.

On the other hand, SHAP’s game-theory-based approach fosters fairness and transparency but can sometimes be harder to interpret. Blackbox models and complex models like XGBR provide higher prediction accuracy but often at the cost of reduced explainability. Ultimately, the choice between these tools depends on the balance between prediction accuracy and model interpretability, which can vary based on the complexity of the model being used.

Key Takeaways

• LIME and SHAP improve the interpretability of complex AI models.
• LIME is ideal for gaining local-level insights into predictions.
• SHAP provides a more global understanding of feature importance and fairness.
• Higher model complexity often leads to better accuracy but reduced explainability.
• The choice between these tools depends on the need for accuracy versus interpretability.

References

For more details please use following

• Source Code Repository
• LIME Documentation
• SHAP Documentation

Frequently Asked Questions