Interpreting Classification Model with LIME


One of many things to consider when we want to choose a machine learning model is the interpretability: can we analyze what variables or certain values that contribute toward particular class or target? Some models can be easily interpreted, such as the linear or logistic regression model and decision trees, but interpreting more complex model such as random forest and neural network can be challenging. This sometimes drive the data scientist to choose more interpretable model since they need to communicate it to their manager or higher rank, who perhaps are not familiar with machine learning. The downside is, in general, interpretable model has lower performance in term of accuracy or precision, making them less useful and potentially dangerous for production. Therefore, there is a growing need on how to interpret a complex and black box model easily.

There exist a method called LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. Here we will see how LIME works on binary classification problem of employee attrition. By understanding on how our model works, we can have more advantage and could act wiser on what should we do.

Local Interpretable Model-Agnostic Explanations (LIME)

LIME characteristics

Let’s understand some of the LIME characteristic (Ribeiro et al., 2016):

  • Interpretable

Provide qualitative understanding between the input variables and the response. Interpretability must take into account the user’s limitations. Thus, a linear model, a gradient vector or an additive model may or may not be interpretable. For example, if hundreds or thousands of features significantly contribute to a prediction, it is not reasonable to expect any user to comprehend why the prediction was made, even if individual weights can be inspected. This requirement further implies that explanations should be easy to understand, which is not necessarily true of the features used by the model, and thus the “input variables” in the explanations may need to be different than the features. Finally, the notion of interpretability also depends on the target audience. Machine learning practitioners may be able to interpret small Bayesian networks, but laymen may be more comfortable with a small number of weighted features as an explanation.

  • Local Fidelity

Although it is often impossible for an explanation to be completely faithful unless it is the complete description of the model itself, for an explanation to be meaningful it must at least be locally faithful, i.e. it must correspond to how the model behaves in the vicinity of the instance being predicted. We note that local fidelity does not imply global fidelity: features that are globally important may not be important in the local context, and vice versa. While global fidelity would imply local fidelity, identifying globally faithful explanations that are interpretable remains a challenge for complex models.

  • Model-Agnostic

An explainer should be able to explain any model, and thus be model-agnostic (i.e. treat the original model as a black box). Apart from the fact that many state of the art classifiers are not currently interpretable, this also provides flexibility to explain future classifiers.

How LIME works

The generalized algorithm LIME applies is (Boehmke, 2018):

(1) Given an observation, permute it to create replicated feature data with slight value modifications.
(2) Compute similarity distance measure between original observation and permuted observations.
(3) Apply selected machine learning model to predict outcomes of permuted data.
(4) Select m number of features to best describe predicted outcomes.
(5) Fit a simple model to the permuted data, explaining the complex model outcome with m features from the permuted data weighted by its similarity to the original observation .
(6) Use the resulting feature weights to explain local behavior.

For more detailed description on how LIME work, you can check Ribeiro et al. paper works (

lime packages in R

You can implement LIME in R with lime package. See

Here is the list of packages you need to load before proceeding to the next section.


Example: Binary Classification

Let’s how LIME work on IBM HR attrition dataset from Kaggle ( We want to correctly target people who are likely to resign. We want to know what factors that drive people to resign/attrition and propose a plan to reduce the number of turnover next year. In order to effectively reduce turnover rate as many as possible, we want our model to have high Recall/Sensitivity.

Import Data

The data consists of information related to the employee who works from the company. Attrition refers to employees who quite the organization.

attrition <- read.csv("data_input/attrition.csv")

Data Preprocessing 1

Before do create our model, here is some of data wrangling that is done:

  • Sum all of the satisfaction score into total_satisfaction
  • Transform education into factor and rename each value (1 = Below College, 2 = College, 3 = Bachelor, 4 = Master, 5 = Doctor)
  • Transform job_level and stock_option_level into factor
  • Transform age into 3 level factors: Young (less than 25), Middle Age (25-54), and Senior (more than 54)
  • Transform monthly_income into 2 level factors: Below average and Above average)
  • Adjust the level of attrition, with the first level will be the positive class (attrition = yes)
  • Remove unnecessary variables
df <- attrition
df$total_satisfaction <- df %>% 
  group_by(employee_number) %>% 
  summarise(total_satisfaction = sum(environment_satisfaction, job_satisfaction, performance_rating,
                                  work_life_balance, job_involvement, relationship_satisfaction)) %>% 

df <- df %>% 
  mutate(education = as.factor(case_when(education == 1 ~ "Below College",
                                         education == 2 ~ "College",
                                         education == 3 ~ "Bachelor",
                                         education == 4 ~ "Master",
                                         TRUE ~ "Doctor")),
         age = as.factor(case_when(age <= 25 ~ "Young",
                                   age <= 54 ~ "Middle Aged",
                                   TRUE ~ "Senior")),
         monthly_income = if_else(monthly_income < median(monthly_income), "Below Average", "Above Average"),
         job_level = as.factor(job_level),
         stock_option_level = as.factor(stock_option_level),
         attrition = factor(attrition, levels = c("yes", "no"))) %>% 
  select(-c(environment_satisfaction, job_satisfaction, performance_rating, employee_number,
            work_life_balance, job_involvement, relationship_satisfaction))