Assessing the Local Interpretability of Machine Learning Models

Dylan Slack,1 Sorelle A. Friedler,1, Carlos Scheidegger2, Chitradeep Dutta Roy3

1Haverford College2University of Arizona

3University of Utah

Abstract

The increasing adoption of machine learning tools has led tocalls for accountability via model interpretability. But whatdoes it mean for a machine learning model to be interpretableby humans, and how can this be assessed? We focus on twodefinitions of interpretability that have been introduced in themachine learning literature: simulatability (a user’s ability torun a model on a given input) and “what if” local explain-ability (a user’s ability to correctly determine a model’s pre-diction under local changes to the input, given knowledge ofthe model’s original prediction). Through a user study with1000 participants, we test whether humans perform well ontasks that mimic the definitions of simulatability and “whatif” local explainability on models that are typically consid-ered locally interpretable. To track the relative interpretabil-ity of models, we employ a simple metric, the runtime opera-tion count on the simulatability task. We find evidence that asthe number of operations increases, participant accuracy onthe local interpretability tasks decreases. In addition, this ev-idence is consistent with the common intuition that decisiontrees and logistic regression models are interpretable and aremore interpretable than neural networks.

IntroductionRecently, there has been growing interest in interpreting ma-chine learning models. The goal of interpretable machinelearning is to allow oversight and understanding of machine-learned decisions. Much of the work in interpretable ma-chine learning has come in the form of devising methods tobetter explain the predictions of machine learning models.However, such work usually leaves a noticeable gap in un-derstanding interpretability (Lipton 2018; Doshi-Velez andKim 2017). The field currently stands on shaky foundations:papers mean different things when they use the word “in-terpretability”, and interpretability claims are typically notvalidated by measuring human performance on a controlledtask. However, there is growing recognition in the meritof such human validated assessments (Lage et al. 2018b;Lage et al. 2018a; Lakkaraju, Bach, and Leskovec 2016).In line with this goal, we seek concrete, falsifiable notionsof interpretability.

Copyright c© 2019, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

“Interpretability” can be broadly divided into global inter-pretability, meaning understanding the entirety of a trainedmodel including all decision paths, and local interpretabil-ity, the goal of understanding the results of a trained modelon a specific input and small deviations from that input. Wefocus on local interpretability, and on two specific defini-tions. We assess simulatability (Lipton 2018), the ability ofa person to—independently of a computer—run a modeland get the correct output for a given input, and “whatif” local explainability (Ribeiro, Singh, and Guestrin 2016;Lipton 2018): the ability of a person to correctly determinehow small changes to a given input affect the model out-put. We will refer to a model as locally interpretable if usersare able to correctly perform both of these tasks when givena model and input. The experiments we present here arenecessarily artificial and limited in scope. We see these aslower bounds on the local interpretability of a model; if peo-ple cannot perform these interpretability tasks, these modelsshould not be deemed locally interpretable.

In addition to considering the successful completion ofthese tasks a lower bound on the local interpretability ofa model, we might reasonably ask whether these are valu-able interpretability tasks at all. Though purposefully lim-ited in scope, we argue that these tasks are still valuable inreal-world settings. Consider a defense attorney faced with aclient’s resulting score generated by a machine learned riskassessment. In order to properly defend their client, the attor-ney may want to verify that the risk score was correctly cal-culated (simulatability) and argue about the extent to whichsmall changes in features about their client could change thecalculated score (local explainability). Despite being simpleinterpretability tasks, successfully completing them is im-portant to the attorney’s ability to defend their client frompotential errors or issues with the risk assessment.

We assessed the simulatability and “what if” local ex-plainability of decision trees, logistic regressions, and neu-ral networks through a crowdsourced user study using Pro-lific (Prolific 2014). We asked 1,000 participants to simulatethe model on a given input and anticipate the outcome ona slightly modified version of the input. We measured useraccuracy and completion time over varied datasets, inputs,and model types (described in detail in the User Study De-

arX

iv:1

902.

0350

1v2

[cs

.LG

] 2

Aug

201

9

sign section). The results are consistent with the folk hy-potheses (Lipton 2018) that decision trees and logistic re-gression models are locally interpretable and are more lo-cally interpretable than neural networks given the particularmodel representations, datasets, and user inputs used in thestudy.

As has been previously observed (Lipton 2018), it maybe the case that a small neural network is more interpretablethan a very large decision tree. To begin to answer questionssurrounding cross-model comparisons and generalizationsof these results to models not studied here, we investigateda measure for its suitability as a proxy for the users’ abilityto correctly perform both the simulation and “what if” lo-cal explainability tasks. We hypothesized that the number ofprogram operations performed by an execution trace of themodel on a given input would be a good proxy for the timeand accuracy of users’ attempts to locally interpret the modelunder both definitions; specifically, that as the total numberof operations increased, the time taken would increase andthe accuracy on the combined task would decrease.

Analyzing the results of this study, we find evidence thatas the number of total operations performed by the modelincreases, the time taken by the user increases and their ac-curacy on the combined local interpretability task decreases.We anticipated that as the number of operations increases,the model would become uninterpretable because all usersare eventually expected to make a mistake simulating a verylarge model. The operation count at which the users cannotlocally interpret a model can be considered an upper boundlimit to the interpretability of the model. Users reached thisupper bound when simulating the largest neural networksizes we considered. We see this work as a first step in amore nuanced understanding of the users’ experience of in-terpretable machine learning.

Related WorkWork on the human interpretability of machine learningmodels began as early as Breiman’s study of random forests(Breiman 2001). Since then, many approaches to the in-terpretability of machine learning models have been con-sidered, including the development of new globally inter-pretable models (Ustun and Rudin 2016), post-hoc local ex-planations (Ribeiro, Singh, and Guestrin 2016) and visual-izations (Olah et al. 2018), and post-hoc measurement of theglobal importance of different features (Henelius et al. 2014;Datta, Sen, and Zick 2016; Adler et al. 2018). We re-fer the interested read to Molnar and Guidotti et al. for amore detailed discussion of these methods (Molnar 2018;Guidotti et al. 2018).

Some of the recent activity on interpretability has beenprompted by Europe’s General Data Protection Regulation(GDPR). A legal discussion of the meaning of the regula-tion with respect to interpretability is ongoing. Initially, theGDPR regulations were described as providing a “right toan explanation” (Goodman and Flaxman 2016), althoughsubsequent work challenges that claim (Wachter, Mittel-stadt, and Floridi 2017), supporting a more nuanced rightto “meaningful information” about any automated decisionimpacting a user (Selbst and Powles 2017). Exactly what

is meant by interpretability to support the GDPR and in abroader legal context remains in active discussion (Selbstand Barocas 2018).

The uncertainty around the meaning of “interpretability”has prompted calls for more precise definitions and carefullydelineated goals (Lipton 2018). One thought-provoking pa-per makes the case for a research agenda in interpretabil-ity driven by user studies and formalized metrics that canserve as validated proxies for user understanding (Doshi-Velez and Kim 2017). Doshi-Velez and Kim argue that hu-man evaluation of the interpretability of a method in its spe-cific application context is the pinnacle of an interpretabilityresearch hierarchy followed by human evaluation of inter-pretability on a simplified or synthetic task and analysis ofproxy tasks without associated user studies. In order to per-form interpretability analysis without user studies, they ar-gue, it is necessary to first assess proxies for user behavior.Here, we propose one such metric and assess its suitabilityas a proxy for the local interpretability of a model.

Although we are unaware of existing metrics for the lo-cal interpretability of a general model, many measures de-veloped by the program analysis community aim at assess-ing the understandability of a general program, which couldbe seen as metrics for global interpretability. For example,the cyclomatic complexity counts the number of independentpaths through a program using its control flow graph (Mc-Cabe 1976). Metrics for specific model types have also beendeveloped. Lage et al. (Lage et al. 2018a) investigate howdifferent measures of complexity in decision sets affect ac-curacy and response time on tasks consisting of simulatabil-ity, verification, and counterfactual-reasoning. Via six differ-ent user studies of 150 people (for a total of 900 participants)they find that increased complexity in decision set logic re-sults in increased response time but do not find a signifi-cant connection with accuracy. They measure decision setcomplexity as a combination between the explanation size,clauses in the disjunctive normal form of the input (calledcognitive chunks), and number of repeated input conditionsto the decision set. Their work is specific to decision sets anddoes not generalize to other model types.

There have also been experimentally grounded assess-ments of model properties related to (but different from) in-terpretability. Poursabzi-Sangdeh et al. (Poursabzi-Sangdehet al. 2017) consider the impact of model attributes (e.g.black-box vs. clear) on user trust, simulatability, and mis-take detection using randomized user studies on a similarscale to what we will consider here. They find that clearmodels (models where the inner calculations are displayedto the user) are best simulated. Allahyari et al. (Allahyari andLavesson 2011) measure the perceived relative understand-ability of decision trees and rule-based models and find de-cision trees are seen as more understandable than rule-basedmodels.

Other methods are concerned with human in the loop opti-mization of the interpretability of machine learning models.Lage et. al. (Lage et al. 2018b) develop a method that op-timizes models for both interpretability and accuracy by in-cluding user studies in the optimization loop. Their methodminimizes the number of user studies needed to generate

models that are both interpretable and accurate. They per-form experiments on optimizing decision trees and find thatthe proxy interpretability metric optimized by the model(e.g. number of nodes, mean path length) varies based ondataset.

A Metric for Local Interpretability

Figure 1: A decision tree where the answer when run on theinput (a = −80, b = 200) is shown circled in blue and theresult of running the same model on the input (a = −64, b =115) is shown circled in red.

Motivated by the previous literature and its calls for user-validated metrics that capture aspects of interpretability, wewish to assess whether a candidate metric captures a user’sability to simulate and “what if” locally explain a model.The candidate metric we consider here is the total numberof runtime operation counts performed by the model whenrun on a given input. We consider two basic variants of op-erations, arithmetic and boolean, and track their totals sepa-rately. Effectively, we seek a proxy for the work that a usermust do (in their head or via a calculator) in order to simu-late a model on a given input, and will claim that the totalnumber of operations also impacts a user’s ability to performa “what if” local explanation of a model.

An ExampleAs an example of how this metric would work, consider thevisualization of a decision tree in Figure 1. The result of run-

ning the model on the input (a = −80, b = 200) is showncircled in blue and the result of running the same model onthe input (a = −64, b = 115) is shown circled in red. Thered answer is at a depth of 10 in the decision tree while theblue answer is at a depth of 5. Counting the operations thatthe model takes to run on the input (including each booleancomparison operation or memory access required, which wecount as an arithmetic operation) gives the total number ofruntime operations - our candidate metric. Using the be-low methodology to count these operations, to determine thenumber of runtime operations executed when evaluating thedecision tree model on the inputs from the example above,(blue: a = −80, b = 200 and red: a = −64, b = 115), theblue input is found to require 17 total operations (6 opera-tions are arithmetic and 11 are boolean) while the red inputrequires 32 total operations (11 arithmetic and 21 boolean).Essentially, at each branch point one arithmetic operation isperformed to do a memory access, one boolean operation isperformed to check if the node is a leaf node, and one moreboolean operation is performed for the branching operation.

Calculating Runtime Operation CountsIn order to calculate the number of runtime operations fora given input, we instrumented the prediction operationfor existing trained models in python’s scikit-learn pack-age (Buitinck et al. 2013). The source code for the tech-nique is available at URL removed for anonymization. Sincemost machine learning models in scikit-learn use (indirectly,via other dependencies) cython, Fortran, and C for speedand memory efficiency, we implemented a pure Python ver-sion of the predict method for the classifiers, and in-strumented the Python bytecode directly. We created pure-Python versions of the decision tree, logistic regression, andneural network classifiers in scikit-learn.1

Once working only with pure Python code, we used thetracing feature of python’s sys module and a custom tracerfunction to count the number of boolean and arithmetic op-erations. The default behavior of tracer in python is linebased, meaning the trace handler method is called for eachline of the source code. We used the dis module to mod-ify the compiled bytecode objects of useful modules storedin their respective .pyc files. In particular, we modified theline numbering metadata so that every bytecode is given anew line number, ensuring that our tracer function is calledfor every bytecode instruction (Ned 2008b; Ned 2008a;Ike-Nwosu 2018). Inside the tracer function we use the dismodule to determine when a byte corresponds to a valid op-eration and count them accordingly for our simplified pre-dict method implementations when run on a given input.

User Study DesignWe have two overall goals in this project: to assess thesimulatability and “what if” local explainability of machinelearning models, and to study the extent to which the pro-posed metric works as proxy for local interpretability. To

1Specifically, sklearn.tree.DecisionTreeClassifier,sklearn.linear model. LogisticRegression, andsklearn.neural network.MLPClassifier.

those ends, we designed a crowdsourced experiment thatwas given to 1000 participants. Participants were asked torun a model on a given input and then evaluate the samemodel on a locally changed version of the input. We startby describing the many potentially interacting factors thatrequired a careful experimental design.

Models and RepresentationsFor this study we consider the local interpretability ofthree models: decision trees, logistic regression, and neu-ral networks. We chose decision trees and logistic re-gression because they are commonly considered to be in-terpretable (Lipton 2018). In contrast, we picked neuralnetworks because they are commonly considered uninter-pretable. The models were trained using the standard pack-age scikit-learn.2

Our decision tree representation is a standard node-linkdiagram representation for a decision tree or flow chart. Inorder to allow users to simulate the logistic regression andneural network classifiers we needed a representation thatwould walk the users through the calculations without pre-vious training in using the model or any assumed mathemat-ical knowledge beyond arithmetic. The resulting representa-tion for logistic regression is shown in Figure 2. The neuralnetwork representation used the same representation as thelogistic regression for each node and one page per layer.

The representations described so far are for the first ques-tion a user will be asked about a model - the request to sim-ulate it on a given input. In order to allow users to assess the“what if” local explainability of the model, we also askedthem to determine the output of the model for a perturbedversion of the initial input they were shown. The represen-tations used here are the same as the ones described, but asnapshot of the participants’ previously filled in answers areshown for the logistic regression and neural network repre-sentations (see Figure 3) and users are not given blank en-tries to allow the re-simulation of the model.

Data and InputsIn order to avoid effects from study participants with do-main knowledge, we created synthetic datasets to train themodels. We created four synthetic datasets simple enoughso that each model could achieve 100% test accuracy. Thesedatasets consisted of a 2 dimensional dataset with rota-tion around an axis applied, 2 dimensional without rotationaround an axis, 3 dimensional with rotation around an axis,

2Decision trees were trained usingsklearn.tree.DecisionTreeClassifierwithout any depth restrictions and with default pa-rameters. Logistic regression was trained usingsklearn.linear model.LogisticRegressionwith the multi class argument set to ’multinomial’and ’sag’(Stochastic average gradient descent) as thesolver. The neural network was implemented usingsklearn.neural network.MLPClassifier. The neuralnetwork used is a fully connected network with 1 input layer, 1hidden layer with 3 nodes, and 1 output layer. The relu (rectifiedlinear unit) activation function was used for the hidden layer.

Figure 2: The logistic regression representation shown tousers.

and 5 dimensional with rotation around an axis. As the num-ber of dimensions increases, so does the operation count.These four datasets were used to train the three consideredmodels via an 80/20 train-test split. We generated user in-puts using the test data. For each test data point, we changedone dimension incrementally in order to create a perturbedinput.

From this set of input and perturbed input pairs, we thenchose a set of eight pairs for each trained model (i.e., foreach model type and dataset combination) to show to theparticipants. The set was chosen to fit the following condi-tions: 50% of the classifications of the original inputs areTrue, 50% of the classifications on the perturbed input areTrue, and 50% of the time the classification between inputand its perturbed input changes. We used this criteria in or-der to distribute classification patterns evenly across usersso that a distribution of random guesses by the participants

Figure 3: The “what if” local explainability question shownto users for the neural network model. Note that while thesimulatability question on the neural networks allowed usersto fill in the blanks, the shown blanks in the above image rep-resent where the variable values will be filled in, and usersare given no location to fill in partial simulations of the neu-ral network.

would lead to 50% correctness on each task, and guessingthat the perturbed input had the same outcome as the origi-nal input would also be correct 50% of the time.

Pilot StudiesIn order to assess the length of the study and work out anyproblems with instructions, we conducted three pilot studies.In the first informal study, one of us watched and took noteswhile a student attempted to simulate an input on each ofthe three types of models and determine the outcome for aperturbed input for each of those three models. In the secondtwo pilots we recruited about 40 participants through Prolificand gave the study for a few fixed models and inputs with thesame setup as we would be using for the full study. The maintakeaways from these pilot studies were that we estimated itwould take users 20-30 minutes to complete the survey, butthat some users would take much longer. We had originallyplanned to include a dataset with 10 dimensions, and basedon the time taken by users in the pilot survey decreased ourlargest dataset to 5 dimensions and added the 2-dimensionaldataset with no rotation.

Experimental SetupWe used Prolific to distribute the survey to 1000 users eachof whom was paid $3.50 for completing it. Participants wererestricted to those with at least a high school education (dueto the mathematical nature of the task) and a Prolific rat-ing greater than 75 out of 100. The full survey information(hosted through Qualtrics) and resulting data is available on-line.3

Each participant was asked to calculate the output of amachine learning model for a given input, and then to de-termine the output of a perturbed input applied to the samemodel. We showed each participant three trained models: alogistic regression, a decision tree, and a neural network in arandom order. Each participant was shown a model trainedon a specific dataset (chosen from the four described ear-lier) at most once to avoid memory effects across models.Each question began with the initial input and a brief de-scription of the task. As an attention check, we included aquestion in the survey that asked users to do some basic ad-dition. Lastly, we asked each user at the end of the study toindicate whether they fully attempted to determine correctanswers and that they would still be compensated in casethey selected no. We considered only the data of the 930users, who we will refer to as confident respondents, that se-lected they fully tried to determine correct answers and whocorrectly answered the basic addition problem.

Preregistered Hypotheses We preregistered two experi-mental hypotheses. Namely, that time to complete will bepositively related to operation count and that accuracy willbe negatively related to operation count. We also preregis-tered two exploratory hypotheses. These were that we wouldexplore the specific relationship between time and accuracyversus operation count and that we would explore how the

3URL removed for anonymization

“What If”Simulatability Local Explainability

DTCorrect 717 / 930 719 / 930p-Value 5.9× 10−63 5.16× 10−64

95% CI [0.73, 0.81] [0.73, 0.82]

LRCorrect 592 / 930 579 / 930p-Value 1.94× 10−15 2.07× 10−12

95% CI [0.59, 0.69] [0.57, 0.67]

NNCorrect 556 / 930 499 / 930p-Value 7.34× 5.5−8 0.7895% CI [0.55, 0.65] [0.49, 0.59]

Table 1: Per-model correct responses out of the total confi-dent respondents on the original input (simulatability task)and perturbed inputs (“what if” local explainability task) fordecision trees, logistic regression, and neural networks. p-values given are with respect to the null hypothesis that re-spondents are correct 50% of the time, using exact binomialtests.

perturbed input is related to time and operation count. Thesehypotheses can be found at the Open Science Framework at:url removed for anonymization

Study Setup Issues After running the user study, wefound that an error in the survey setup meant that the surveyexited prematurely for users given two of the eight inputs onthe decision tree models for one dataset. Since we did notreceive data from these participants, Prolific recruited otherparticipants who were allocated to other inputs and datasets,so the analyzed dataset does not include data for these twoinputs. Users who contacted us to let us know about theproblem were still paid.

Multiple Comparison Corrections In order to mitigatethe problem of multiple comparisons, all p-values and confi-dence intervals we report in the next section include a Bon-ferroni correction factor of 28. While we include 15 statisti-cal tests in this paper, we considered a total of 28. Reportedp-values greater than one arise from these corrections.

User Study ResultsBased on the results from the described user study, we nowexamine folk hypotheses regarding the local interpretabilityof different model types, consider the relative local inter-pretability of these models, and assess our proposed metric.

Assessing the Local Interpretability of ModelsIn order to assess the local interpretability of different modeltypes, we first separately consider the user success on thetask for simulatability (the original input) and the task for“what if” local explainability (the perturbed input). Sinceinputs were chosen so that 50% of the correct model out-puts were “yes” and 50% were “no”, we compare the result-ing participant correctness rates to the null hypothesis thatrespondents are correct 50% of the time. The resulting p-values and confidence intervals are shown in Table 1.

The results indicate strong support for the simulatabilityof decision trees, logistic regression, and neural networksbased on the representations the users were given. The re-sults also indicate strong support for the “what if” local ex-plainability of decision trees and logistic regression models,but neural networks were not found to be “what if” locallyexplainable.

Recall that we consider models to be locally interpretableif they are both simulatable and “what if” locally explain-able. Based on the results in Table 1, we thus have evidencethat decision trees and logistic regression models are locallyinterpretable and neural networks are not, partially validat-ing the folk hypotheses about the interpretability of thesemodels. Next, we’ll consider the relative local interpretabil-ity of these models.

Assessing Relative Local InterpretabilityIn order to assess the relative local interpretability of models— to evaluate the folk hypothesis that decision trees and lo-gistic regression models are more interpretable than neuralnetworks — we compared the distributions of correct andincorrect answers on both tasks across pairs of model types.We applied one-sided Fisher exact tests with the null hy-pothesis that the models were equally simulatable, “what if”locally explainable, or locally interpretable. The alternativehypotheses were that decision trees and logistic regressionmodels were more interpretable (had a greater number ofcorrect responses) than neural networks and that decisiontrees were more interpretable than logistic regression.

The results (see Table 2) give strong evidence that deci-sion trees are more locally interpretable than logistic regres-sion or neural network models on both the simulatability and“what if” local explainability tasks. While there was strongevidence that logistic regression is more “what if” locallyexplainable and more locally interpretable than neural net-works, there is not evidence that logistic regression is moresimulatable than neural networks using the given representa-tions. This may be because the logistic regression and neu-ral network representations were very similar. An analysisof the users who got both tasks right, i.e., were able to lo-cally interpret the model, shows that the alternative hypothe-sis was strongly supported in all three cases, thus supportingthe folk hypotheses that decision trees and logistic regres-sion models are more interpretable than neural networks.

Assessing Runtime Operations as a Metric forLocal InterpretabilityIn order to evaluate our preregistered hypotheses, we consid-ered the relationship between total operation counts, time,and accuracy on the simulatability, “what if” local explain-ability, and combined local interpretability tasks. The graphsshowing these relationships, including ellipses that depictthe degree to which the different measurements are linearlyrelated to each other, are shown in Figure 4. The time andaccuracy given for the simulatability and “what if” local ex-plainability tasks are separated individually for those tasksin the first two columns of the figure, while the final localinterpretability column includes the sum of the time taken

Table 2: Comparative correct / incorrect distributions and p-values between model types generated through Fisher Exact Testsfor confident responses. Relative correctness is shown for simulatability (correctness on the original input), “what if” localexplainability (correctness on the perturbed input), and local interpretability (correctness on both parts). DT stands for DecisionTree, LR stands for Logistic Regression, and NN stands for Neural Network.

Relative Simulatability:

Contingency Table DT > NN DT > LR LR > NNCorrect 717 556 717 592 592 556Incorrect 213 374 213 338 338 374p-value, 95% CI 1.5× 10−14 [1.69,∞] 3.7× 10−9 [1.43,∞] 1.3 [0.90,∞]

Relative “What If” Local Explainability:Contingency Table DT > NN DT > LR LR > NNCorrect 719 499 719 579 579 499Incorrect 211 431 211 351 351 431p-value, 95% CI 7.3× 10−26 [2.20,∞] 2.6× 10−11 [1.54,∞] 2.9× 10−3 [1.09,∞]

Relative Local Interpretability:Contingency Table DT > NN DT > LR LR > NNCorrect 594 337 594 425 425 337Incorrect 336 593 336 505 505 593p-value, 95% CI 9.3× 10−32 [2.36,∞] 5.9× 10−14 [1.60,∞] 5.7× 10−4 [1.13,∞]

by the user on both tasks and credits the user with an ac-curate answer only if both the simulatability and “what if”local explainability tasks were correctly answered. The ac-curacies as displayed in the figure are averaged over all usersgiven the same input into the trained model. All total oper-ation counts given are for the simulation task on the spe-cific input. In the case of the “what if” local explainabilitytask for decision trees, this operation count is for the simu-latability task on the perturbed input; the logistic regressionand neural network simulatability operation counts do notvary based on input. The local interpretability total opera-tion count is the sum of the counts on the simulatability and“what if” local explainability tasks. Additionally, we con-sidered the effect on time and accuracy of just the arithmeticoperation counts. The overall trends are discussed below.

Assessing the Relationship Between RuntimeOperations and TimeThe number of operations has a positive relationshipwith the time taken. Across all three interpretability tasksit appears clear that as the number of operations increases,the total time taken by the user also increases (see the firstrow of Figure 4). This trend is especially clear for the sim-ulatability task, validating Hypothesis 1. This effect is per-haps not surprising, since the operation count considered isfor the simulatability task and the representations given fo-cus on performing each operation.

Users were locally interpreting the “what if” local ex-plainability task. Users spent much less time on the lo-cal explainability task than the simulatability task across allmodels. The difference suggests that users were actually lo-cally interpreting the model on the “what if” local explain-ability task as opposed to re-simulating the whole model.

The time taken to simulate neural networks might not befeasible in practice. The neural network simulation timewas noticeably greater than that of the decision tree andlogistic regression. In some cases, the time expended wasgreater than 30 minutes. A user attempting the simulate theresults of a model might give up or be unable to dedicatethat much time to the task. The study takers likely fearedlack of compensation if they gave up. This result suggeststhat in time constrained situations, neural networks are notsimulatable.

Assessing the Relationship Between RuntimeOperations and AccuracyThe relationship between accuracy and operation countis clear for decision trees but not the other model types.As the total number of runtime operations increases, we hy-pothesized that the accuracy would decrease. In the secondrow of Figure 4 we can see that this trend appears to holdclearly for all three interpretability tasks for the decision treemodels, but there is no clear trend for the logistic regressionand neural network models. This lack of effect may be dueto the comparatively smaller range of operation counts ex-amined for these two model types, or it may be that the localinterpretability of these model types is not as related to oper-ation count as it is for decision trees. The lack of overlap inthe ranges for the operation counts of logistic regression andneural networks also makes it hard to separate the effects ofthe model type on the results.

Some users might not have understood the logistic re-gression and neural network tasks. Because the logis-tic regression and neural network tasks could be consideredmore challenging than the decision tree task, there may havebeen noise introduced by the variability in user ability to per-

Figure 4: Comparisons shown are between total operations for a particular trained model and input, the time taken by theuser to complete the task, and the accuracy of the users on that task for the simulatability (original input), “what if” localexplainability (perturbed input), and the combined local interpretability (getting both tasks correct) tasks. The total time shownis in seconds. The total operation count is for the simulatability task on the specific input; this is the same for both “what if”local explainability and simulatability except for in the case of the decision tree models, where operation counts differ based oninput. The local interpretability operation count is the sum for the simulatability and “what if” local explainability task operationcounts. Accuracy shown is averaged over all users who were given the same input for that task and trained model. The modelsconsidered are decision trees (DT), logistic regression models (LR), and neural networks (NN). The ellipses surrounding eachgroup depict the covariance between the two displayed variables, and capture 95% of the sample variance.

form the task. While operation counts might influence theaccuracy for users who are able to understand the base task,this trend may be hidden by the fact that some users whowere confident did not understand the task.

Discussion and ConclusionWe investigated the local interpretability of three commonmodel types: decision trees, logistic regression, and neuralnetworks, and our user study provides evidence for the folkhypotheses that decision trees and logistic regression modelsare locally interpretable, while neural networks are not. Wealso found that decision trees are more locally interpretablethan logistic regression or neural network models. We alsoshowed that as the number of runtime operations increase,participants take longer to locally interpret a model, and theybecome less accurate on local interpretation tasks. This run-time operations metric provides some insight into the localinterpretability of the discussed models and representations,and could indicate to practitioners the extent to which theirmodels fulfill a lower bound requirement of interpretabil-ity. Further work is needed to consider the extent to whichthe metric generalizes to other model types. In addition, wefound that users were consistently unable to locally inter-

pret the largest operation count neural networks shown tothem, and their inability to simulate such neural networkscould suggest that users struggle to locally interpret mod-els more than 100 operations. Because we did not give usersother models of similar operation count due to their poten-tial display size to the user, further work is needed to verifyif users inability to locally interpret large neural networkswas caused by the number of operation counts or neural net-works themselves.

Further, there are many caveats and limitations to thereach of this work. The domain-agnostic nature of our syn-thetic dataset has transferability advantages, but also has dis-advantages in that it does not study interpretability withinits target domain. The definitions of local interpretabilitythat we assess here — simulatability and “what if” localexplainability— are limited in their reach and the specificuser study setup that we introduce may be limited in captur-ing the nuance of these definitions. Still, this work providesa starting point for designing user studies to validate notionsof interpretability in machine learning. Such controlled stud-ies are delicate and time-consuming, but are ultimately nec-essary in order for the field to make progress.

References[Adler et al. 2018] Adler, P.; Falk, C.; Friedler, S. A.; Nix, T.;Rybeck, G.; Scheidegger, C.; Smith, B.; and Venkatasubra-manian, S. 2018. Auditing black-box models for indirectinfluence. Knowledge and Information Systems 54(1):95–122.

[Allahyari and Lavesson 2011] Allahyari, H., and Lavesson,N. 2011. User-oriented assessment of classification modelunderstandability. In 11th scandinavian conference on Arti-ficial intelligence. IOS Press.

[Breiman 2001] Breiman, L. 2001. Random forests. Ma-chine learning 45(1):5–32.

[Buitinck et al. 2013] Buitinck, L.; Louppe, G.; Blondel, M.;Pedregosa, F.; Mueller, A.; Grisel, O.; Niculae, V.; Pretten-hofer, P.; Gramfort, A.; Grobler, J.; Layton, R.; VanderPlas,J.; Joly, A.; Holt, B.; and Varoquaux, G. 2013. API designfor machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages forData Mining and Machine Learning, 108–122.

[Datta, Sen, and Zick 2016] Datta, A.; Sen, S.; and Zick, Y.2016. Algorithmic transparency via quantitative input in-fluence: Theory and experiments with learning systems. InSecurity and Privacy (SP), 2016 IEEE Symposium on, 598–617. IEEE.

[Doshi-Velez and Kim 2017] Doshi-Velez, F., and Kim, B.2017. Towards a rigorous science of interpretable machinelearning. arXiv preprint arXiv:1702.08608.

[Goodman and Flaxman 2016] Goodman, B., and Flaxman,S. 2016. European union regulations on algorithmicdecision-making and a” right to explanation”. presented atthe 2016 ICML Workshop on Human Interpretability in Ma-chine Learning (WHI 2016), New York, NY.

[Guidotti et al. 2018] Guidotti, R.; Monreale, A.; Ruggieri,S.; Turini, F.; Giannotti, F.; and Pedreschi, D. 2018. Asurvey of methods for explaining black box models. ACMComputing Surveys (CSUR) 51(5):93.

[Henelius et al. 2014] Henelius, A.; Puolamaki, K.;Bostrom, H.; Asker, L.; and Papapetrou, P. 2014. A peekinto the black box: exploring classifiers by randomization.Data mining and knowledge discovery 28(5-6):1503–1529.

[Ike-Nwosu 2018] Ike-Nwosu, O. 2018. Inside Python Vir-tual Machine. Lean publishing, 1st edition. chapter 5 CodeObjects, 68–78.

[Lage et al. 2018a] Lage, I.; Chen, E.; He, J.; Narayanan,M.; Gershman, S.; Kim, B.; and Doshi-Velez, F. 2018a.An evaluation of the human-interpretability of explanation.In Conference on Neural Information Processing Systems(NeurIPS) Workshop on Correcting and Critiquing Trendsin Machine Learning.

[Lage et al. 2018b] Lage, I.; Slavin Ross, A.; Kim, B.; J. Ger-shman, S.; and Doshi-Velez, F. 2018b. Human-in-the-loopinterpretability prior. In Conference on Neural InformationProcessing Systems (NeurIPS).

[Lakkaraju, Bach, and Leskovec 2016] Lakkaraju, H.; Bach,S. H.; and Leskovec, J. 2016. Interpretable decision sets:A joint framework for description and prediction. In ACM

SIGKDD Conference on Knowledge Discovery and DataMining (KDD).

[Lipton 2018] Lipton, Z. C. 2018. The mythos of modelinterpretability. Queue 16(3):30.

[McCabe 1976] McCabe, T. J. 1976. A complexity measure.IEEE Transactions on software Engineering (4):308–320.

[Molnar 2018] Molnar, C. 2018. Interpretable MachineLearning. https://christophm.github.io/interpretable-ml-book/. https://christophm.github.io/interpretable-ml-book/.

[Ned 2008a] Ned, B. 2008a. The structure of .pyc files.Blog. https://nedbatchelder.com/blog/200804/the structureof pyc files.html.

[Ned 2008b] Ned, B. 2008b. Wicked hack: Python byte-code tracing. Blog. https://nedbatchelder.com/blog/200804/wicked hack python bytecode tracing.html.

[Olah et al. 2018] Olah, C.; Satyanarayan, A.; Johnson, I.;Carter, S.; Schubert, L.; Ye, K.; and Mordvintsev, A.2018. The building blocks of interpretability. Distill.https://distill.pub/2018/building-blocks.

[Poursabzi-Sangdeh et al. 2017] Poursabzi-Sangdeh, F.;Goldstein, D. G.; Hofman, J. M.; Vaughan, J. W.; andWallach, H. 2017. Manipulating and measuring modelinterpretability. Transparent and Interpretable MachineLearning in Safety Critical Environments Workshop atNIPS.

[Prolific 2014] Prolific. 2014. https://prolific.ac/, last ac-cessed on June 5th, 2019.

[Ribeiro, Singh, and Guestrin 2016] Ribeiro, M. T.; Singh,S.; and Guestrin, C. 2016. Why should i trust you?: Ex-plaining the predictions of any classifier. In Proceedings ofthe 22nd ACM SIGKDD international conference on knowl-edge discovery and data mining, 1135–1144. ACM.

[Selbst and Barocas 2018] Selbst, A. D., and Barocas, S.2018. The intuitive appeal of explainable machines. Ford-ham Law Review. Forthcoming. Available at SSRN: https://ssrn.com/abstract=3126971.

[Selbst and Powles 2017] Selbst, A. D., and Powles, J. 2017.Meaningful information and the right to explanation. Inter-national Data Privacy Law 7(4):233–242.

[Ustun and Rudin 2016] Ustun, B., and Rudin, C. 2016. Su-persparse linear integer models for optimized medical scor-ing systems. Machine Learning 102(3):349–391.

[Wachter, Mittelstadt, and Floridi 2017] Wachter, S.; Mittel-stadt, B.; and Floridi, L. 2017. Why a right to explanationof automated decision-making does not exist in the generaldata protection regulation. International Data Privacy Law7(2):76–99.