Click here to load reader

Pathologies of Neural Models Make Interpretations Difficult · PDF file Pathologies of Neural Models Make Interpretations Difficult Shi Feng 1Eric Wallace Alvin Grissom II2 Mohit

  • View

  • Download

Embed Size (px)

Text of Pathologies of Neural Models Make Interpretations Difficult · PDF file Pathologies of Neural...

  • Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728 Brussels, Belgium, October 31 - November 4, 2018. c©2018 Association for Computational Linguistics


    Pathologies of Neural Models Make Interpretations Difficult

    Shi Feng1 Eric Wallace1 Alvin Grissom II2 Mohit Iyyer3,4 Pedro Rodriguez1 Jordan Boyd-Graber1 1University of Maryland 2Ursinus College

    3UMass Amherst 4Allen Institute for Artificial Intelligence {shifeng,ewallac2,entilzha,jbg},

    [email protected], [email protected]


    One way to interpret neural model predic- tions is to highlight the most important in- put features—for example, a heatmap visu- alization over the words in an input sen- tence. In existing interpretation methods for NLP, a word’s importance is determined by either input perturbation—measuring the de- crease in model confidence when that word is removed—or by the gradient with respect to that word. To understand the limitations of these methods, we use input reduction, which iteratively removes the least important word from the input. This exposes pathological be- haviors of neural models: the remaining words appear nonsensical to humans and are not the ones determined as important by interpreta- tion methods. As we confirm with human ex- periments, the reduced examples lack infor- mation to support the prediction of any la- bel, but models still make the same predic- tions with high confidence. To explain these counterintuitive results, we draw connections to adversarial examples and confidence cali- bration: pathological behaviors reveal difficul- ties in interpreting neural models trained with maximum likelihood. To mitigate their defi- ciencies, we fine-tune the models by encourag- ing high entropy outputs on reduced examples. Fine-tuned models become more interpretable under input reduction without accuracy loss on regular examples.

    1 Introduction

    Many interpretation methods for neural networks explain the model’s prediction as a counterfactual: how does the prediction change when the input is modified? Adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015) highlight the in- stability of neural network predictions by showing how small perturbations to the input dramatically change the output.

    SQUAD Context In 1899, John Jacob Astor IV invested

    $100,000 for Tesla to further develop and produce a new lighting system. In- stead, Tesla used the money to fund his Colorado Springs experiments.

    Original What did Tesla spend Astor’s money on ? Reduced did Confidence 0.78→ 0.91

    Figure 1: SQUAD example from the validation set. Given the original Context, the model makes the same correct prediction (“Colorado Springs experiments”) on the Reduced question as the Original, with even higher confidence. For humans, the reduced question, “did”, is nonsensical.

    A common, non-adversarial form of model in- terpretation is feature attribution: features that are crucial for predictions are highlighted in a heatmap. One can measure a feature’s importance by input perturbation. Given an input for text clas- sification, a word’s importance can be measured by the difference in model confidence before and after that word is removed from the input—the word is important if confidence decreases signifi- cantly. This is the leave-one-out method (Li et al., 2016b). Gradients can also measure feature im- portance; for example, a feature is influential to the prediction if its gradient is a large positive value. Both perturbation and gradient-based methods can generate heatmaps, implying that the model’s pre- diction is highly influenced by the highlighted, im- portant words.

    Instead, we study how the model’s prediction is influenced by the unimportant words. We use in- put reduction, a process that iteratively removes the unimportant words from the input while main- taining the model’s prediction. Intuitively, the words remaining after input reduction should be important for prediction. Moreover, the words

  • 3720

    should match the leave-one-out method’s selec- tions, which closely align with human percep- tion (Li et al., 2016b; Murdoch et al., 2018). How- ever, rather than providing explanations of the original prediction, our reduced examples more closely resemble adversarial examples. In Fig- ure 1, the reduced input is meaningless to a hu- man but retains the same model prediction with high confidence. Gradient-based input reduction exposes pathological model behaviors that contra- dict what one expects based on existing interpreta- tion methods.

    In Section 2, we construct more of these coun- terintuitive examples by augmenting input reduc- tion with beam search and experiment with three tasks: SQUAD (Rajpurkar et al., 2016) for read- ing comprehension, SNLI (Bowman et al., 2015) for textual entailment, and VQA (Antol et al., 2015) for visual question answering. Input re- duction with beam search consistently reduces the input sentence to very short lengths—often only one or two words—without lowering model confi- dence on its original prediction. The reduced ex- amples appear nonsensical to humans, which we verify with crowdsourced experiments. In Sec- tion 3, we draw connections to adversarial exam- ples and confidence calibration; we explain why the observed pathologies are a consequence of the overconfidence of neural models. This elucidates limitations of interpretation methods that rely on model confidence. In Section 4, we encourage high model uncertainty on reduced examples with entropy regularization. The pathological model behavior under input reduction is mitigated, lead- ing to more reasonable reduced examples.

    2 Input Reduction

    To explain model predictions using a set of impor- tant words, we must first define importance. Af- ter defining input perturbation and gradient-based approximation, we describe input reduction with these importance metrics. Input reduction dras- tically shortens inputs without causing the model to change its prediction or significantly decrease its confidence. Crowdsourced experiments con- firm that reduced examples appear nonsensical to humans: input reduction uncovers pathological model behaviors.

    2.1 Importance from Input Gradient Ribeiro et al. (2016) and Li et al. (2016b) de- fine importance by seeing how confidence changes when a feature is removed; a natural approxima- tion is to use the gradient (Baehrens et al., 2010; Simonyan et al., 2014). We formally define these importance metrics in natural language contexts and introduce the efficient gradient-based approx- imation. For each word in an input sentence, we measure its importance by the change in the con- fidence of the original prediction when we remove that word from the sentence. We switch the sign so that when the confidence decreases, the impor- tance value is positive.

    Formally, let x = 〈x1, x2, . . . xn〉 denote the in- put sentence, f(y |x) the predicted probability of label y, and y = argmaxy′ f(y

    ′ |x) the original predicted label. The importance is then

    g(xi | x) = f(y |x)− f(y |x−i). (1)

    To calculate the importance of each word in a sen- tence with n words, we need n forward passes of the model, each time with one of the words left out. This is highly inefficient, especially for longer sentences. Instead, we approximate the impor- tance value with the input gradient. For each word in the sentence, we calculate the dot product of its word embedding and the gradient of the output with respect to the embedding. The importance of n words can thus be computed with a single forward-backward pass. This gradient approxima- tion has been used for various interpretation meth- ods for natural language classification models (Li et al., 2016a; Arras et al., 2016); see Ebrahimi et al. (2017) for further details on the derivation. We use this approximation in all our experiments as it selects the same words for removal as an ex- haustive search (no approximation).

    2.2 Removing Unimportant Words Instead of looking at the words with high impor- tance values—what interpretation methods com- monly do—we take a complementary approach and study how the model behaves when the sup- posedly unimportant words are removed. Intu- itively, the important words should remain after the unimportant ones are removed.

    Our input reduction process iteratively removes the unimportant words. At each step, we remove the word with the lowest importance value un- til the model changes its prediction. We experi-

  • 3721

    ment with three popular datasets: SQUAD (Ra- jpurkar et al., 2016) for reading comprehension, SNLI (Bowman et al., 2015) for textual entail- ment, and VQA (Antol et al., 2015) for visual question answering. We describe each of these tasks and the model we use below, providing full details in the Supplement.

    In SQUAD, each example is a context para- graph and a question. The task is to predict a span in the paragraph as the answer. We reduce only the question while keeping the context paragraph unchanged. The model we use is the DRQA Doc- ument Reader (Chen et al., 2017).

    In SNLI, each example consists of two sen- tences: a premise and a hypothesis. The task is to predict one of three relationships: entailment, neutral, or contradiction. We reduce only the hy- pothesis while keeping the premise unchanged. The model we use is Bilateral Multi-Perspective Matching (BIMPM) (Wang et al., 2017).

    In VQA, each example consists of an image and a natural language question. We reduce only the question while keeping the image unchanged. The model we use is Show, Ask, Attend, and An- swer (Kazemi and Elqursh, 2017).

    During the iterative r

Search related