Photo by Freepik

Concept of Shapley Value in Interpreting Machine Learning Models

The field of Artificial Intelligence powered by Machine Learning and Deep Learning has gone through some phenomenal changes over the last decade having widespread industry adoption across diverse domains including retail, technology, healthcare, science and many more. The key objective of data science and machine learning in the 21st century has changed to tackling and solving real-world problems, automating complex tasks, etc. Hence, in the industry, the main focus of data science or machine learning is more ‘applied’ rather than theoretical and for effective application of these models, understanding of feature contributions in model output, both from directional and magnitudinal aspects is of paramount importance. In this regard, Shapley Value plays a significant role in feature interpretability and complex model explainability.

There are two main parts in this discussion:

Section 1. Concept of Shapley Value along with an example.

Section 2. Illustration of the concept of Shapley applying on a model.

Section 1:

Concept of Shapley Value

In the literature of Game theory, “Game” is any situation where there are several decision- makers, and each of them wants to optimize their results. The optimizing decision will depend on the decisions of others. The game identifies the players’ identities, preferences, and available strategies and how these strategies affect the outcome. Game Theory attempts to define these situations in mathematical terms and determine what would happen if every player acts rationally.

· Perhaps an equilibrium can be reached (which is why all drive on the same side of the road within a country)

· Maybe this equilibrium will be worse for all players (which is why people litter or pollute common resources)

· Everyone will try to be as unpredictable as possible in their actions (as might happen with troop deployment in a war)

Shapley value is a concept in cooperative game theory. It was named in honor of Lloyd Shapley, who introduced it in 1951 and won the Nobel Prize in Economics for it in 2012. To each cooperative game it assigns a unique distribution (among the players) of a total surplus generated by the coalition of all players.

The setup is as follows: A coalition of players cooperates, and obtains a certain overall gain from that cooperation. Since some players may contribute more to the coalition than others or may possess different bargaining power (for example threatening to destroy the whole surplus), what final distribution of generated surplus among the players should arise in any particular game? Or phrased differently: how important is each player to the overall cooperation, and what payoff can he or she reasonably expect? The Shapley value provides one possible answer to this question. To summarize, the Shapley value for each variable (payout) is basically trying to find the correct weight such that the sum of all Shapley values is the difference between the predictions and average value of the model. In other words, Shapley values correspond to the contribution of each feature towards pushing the prediction away from the expected value.

The primary advantage in using Shapley values is the guarantee of a fair payout, that is, it holds the following properties:

· The difference between the average prediction and the sum of feature contributions are equal. (Efficiency)

· If two features contribute equally to all collations, their resultant contribution values will be equivalent. (Symmetry)

· If a feature has made no change to any predicted value in any collation, then the feature will have a Shapley value of 0. (Dummy)

· The average contribution of a feature across all iterations is equal to the sum of average contributes. (Additive). The Total contribution of each feature must sum up to the total contribution in the whole model.

Three benefits worth mentioning here are:

1. The first one is global interpretability — the collective SHAP values can show how much each predictor contributes, either positively or negatively, to the target variable. This is like the variable importance plot but it is able to show the positive or negative relationship for each variable with the target (see the SHAP value plot below).

2. The second benefit is local interpretability — each observation gets its own set of SHAP values (see the individual SHAP value plot below). This greatly increases its transparency. It can be explained using Shapley — why a case receives its prediction and the contributions of the predictors. Traditional variable importance algorithms only show the results across the entire population but not on each individual case. The local interpretability enables us to pinpoint and contrast the impacts of the factors.

3. Third, the SHAP values can be calculated for any tree-based model, while other methods use linear regression or logistic regression models as the surrogate models.

Photo by google

Example:

Calculating Cab fare from office to home for three persons using Shapley value

Let’s consider an example where Pratik, Rohan and Nupur share a cab. Normally it costs the following for various combinations of the 3 individuals:

· It costs Pratik INR 700 to reach home.

· It costs Rohan INR 800 to reach home.

· It costs Nupur INR 560 to reach home.

· It costs Pratik and Rohan INR 850 to reach home.

· It costs Rohan and Nupur INR 800 to reach home.

· It costs Nupur and Pratik INR 720 to reach home.

· It costs Nupur, Pratik and Rohan INR 900 to reach home.

Assuming they all live in the same direction, now if they share a cab, how much should each of them pay?

A visualization of the above example

Contributions made by each player or set of players over all permutations of player :

Set of fare combinations

Using this, different possible permutations of players were observed (different orderings imagining people could pay before they physically get off the cab). Looking at the 1st coalition, when Pratik gets off the cab he pays INR 700 to make up the coalition he formed, considering the coalition so far Rohan has to pay remaining INR 150 and finally Nupur pays INR 50. Now changing the order, imagining people could pay before they physically get off the cab. Under the second coalition Pratik pays INR 700, now coalition of Pratik and Nupur has value 720, so Nupur pays remaining INR 20 and Rohan pays 180. Every ordering like this is considered and the marginal contribution of each player is calculated and lastly an average of all these values is taken.

Different Combinations of paying cab fare.

The way to calculate the Shapley value: It is the average of the marginal contributions across all permutations.

Thus the Shapley value for:

1. Pratik =(700+700+50+100+160+100)/6 =301

2. Rohan = (150+180+800+800+180+240)/6 = 392

3. Nupur = (50+20+50+0+560+560)/6 = 207

Section 2:

Application of Shapley Value

· Shapley value is used for interpretation of complex machine learning model. It can also be used in various NLP related works like Sentiment analysis, Text generation and also in image data for image classification.

· It provides an unique and different perspective to interpret black-box machine learning models helping to describe the contribution of each feature in final prediction.

· Using this concept, CRM analytics can hyper-personalize their communications depending on the factors responsible for the final outcome.

Photo by google

Model Interpretation using SHAP in Python

The SHAP library in Python has inbuilt functions to use Shapley values for interpreting machine learning models. It has optimized functions for interpreting tree-based models and a model agnostic explainer function for interpreting any black-box model for which the predictions are known.

In the model agnostic explainer, SHAP leverages Shapley values in the below manner. To get the importance of feature X{i}:

1. Get all subsets of features S that do not contain X{i}
2. Compute effect on our predictions of adding X{i} to all those subsets
3. Aggregate all contributions to compute the marginal contribution of the feature

Now, for these subsets, SHAP does not go on and retrain the model for each subset. Instead, for the removed or left out feature, it just replaces it with the average value of the feature and generates the predictions.

Let’s look at the application of Shapley value in a model. You’ll find the codes here. Here the various types of plots are mentioned which helps in interpretation and explanation of the Model output.

We have used the famous German Credit data where the task is to classify people as Good or Bad Credit Risk. There are 20 attributes and 1000 instances. Full description of the data is available here.

1. Variable Importance Plot :

A variable importance plot lists the most significant variables in descending order. Features with large absolute Shapley values are important. The top variables contribute more to the model than the bottom ones and thus have high predictive power.

Here we see that account balance, duration of credit, value of savings/stocks, length of current employment, etc. are some of the important features in classifying people as good or bad credit risk customers.

2. Summary Plot :

Each point on the summary plot is a Shapley value for a feature and an instance. The position on the y-axis is determined by the feature and on the x-axis by the Shapley value. The color represents the value of the feature from low to high. Overlapping points are jittered in y-axis direction, so we get a sense of the distribution of the Shapley values per feature. The features are ordered according to their importance.
Here, all the values on the left represent the observations that shift the predicted value in the negative direction while the points on the right contribute to shifting the prediction in a positive direction.

From the above graph, we see that more the value of savings/stocks high, more is the chance of customers to payback the loan. Also if the customer is a tenured employee, that proves his financial stability which increases the chance of creditability of the customer.

3. Dependence Plot :

• Each dot is a single instance (row) from the dataset.
• The x-axis is the value of the feature.
• The y-axis is the SHAP value for that feature, which represents how the feature’s value changes the output of the model in presence of another feature for that sample’s prediction.

We see that although the length of current employment is same for two customers, higher account balance increases the chance of creditability. You can see this for each feature combination.

4. Force Plot :

Force plot explains the prediction of an instance.

The plot provides:

1. The model output value: Prediction for the particular instance.
2. The base value: this is the value would be predicted if we didn’t have any features for the current output.
3. In the x-axis, it shows the impact of each feature on the output.

Here we can see red and blue arrows associated with each feature.
Each of these arrows indicates:

• Feature’s impacts on the model: the bigger the arrow, the bigger the impact.
• How a feature impacts the model: a red arrow pushes the outcome to the right (increases the model output value) while a blue arrow pushes the model outcome to the left (decreases the model output value).

We see that high savings/stocks value, account balance etc. contributing to higher prediction value for this instance. You can see this both at instance level and also for a group of people(Cluster Plot).

This brings us to the end of our discussion. I hope you learnt something from this blog. Please share if you think it may help others.

Stay safe ! Take care ! Enjoy learning !!

References :

5.9 Shapley Values | Interpretable Machine Learning

Welcome to the SHAP documentation - SHAP latest documentation

A Unique Method for Machine Learning Interpretability: Game Theory & Shapley Values!

https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data)