Explaining Models by Propagating Shapley Values

Hugh ChenPaul G. Allen School of CSE

University of WashingtonSeattle, WA, USA

Scott LundbergMicrosoft Research

Redmond, WA, USA

Su-In LeePaul G. Allen School of CSE

University of WashingtonSeattle, WA, USA

Abstract

In healthcare, making the best possible predictionswith complex models (e.g., neural networks, ensem-bles/stacks of different models) can impact patient wel-fare. In order to make these complex models explain-able, we present DeepSHAP for mixed model types, aframework for layer wise propagation of Shapley val-ues that builds upon DeepLIFT (an existing approachfor explaining neural networks). We show that in addi-tion to being able to explain neural networks, this newframework naturally enables attributions for stacks ofmixed models (e.g., neural network feature extractorinto a tree model) as well as attributions of the loss. Fi-nally, we theoretically justify a method for obtaining at-tributions with respect to a background distribution (un-der a Shapley value framework).

IntroductionNeural networks and ensembles of models are currently usedacross many domains. For these complex models, explana-tions accounting for how features relate to predictions is of-ten desirable and at times mandatory (Goodman and Flax-man 2017). In medicine, explainable AI (XAI) is impor-tant for scientific discovery, transparency, and much more(Holzinger et al. 2017). One popular class of XAI methodsis per-sample feature attributions (i.e., values for each fea-ture for a given prediction).

In this paper, we focus on SHAP values (Lundberg andLee 2017) – Shapley values (Shapley 1953) with a condi-tional expectation of the model prediction as the set func-tion. Shapley values are the only additive feature attribu-tion method that satisfies the desirable properties of localaccuracy, missingness, and consistency. In order to approxi-mate SHAP values for neural networks, we fix a problem inthe original formulation of DeepSHAP (Lundberg and Lee2017) where previously it used E[x] as the reference andtheoretically justify a new method to create explanations rel-ative to background distributions. Furthermore, we extend itto explain stacks of mixed model types as well as loss func-tions rather than margin outputs.

Popular model agnostic explanation methods that alsoaim to obtain SHAP values are KernelSHAP (Lundberg and

Copyright c© 2019, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Lee 2017) and IME (Strumbelj and Kononenko 2014). Thedownside of most model agnostic methods are that they aresampling based and consequently high variance or slow.

Alternatively, local feature attributions targeted to deepnetworks has been addressed in numerous works: Occlu-sion (Zeiler and Fergus 2014), Saliency Maps (Simonyan,Vedaldi, and Zisserman 2013), Layer-Wise Relevance Prop-agation (Bach et al. 2015), DeepLIFT, Integrated Gradients(IG) (Sundararajan, Taly, and Yan 2017), and GeneralizedIntegrated Gradients (GIG) (Merrill et al. 2019).

Of these methods, the ones that have connections to theShapley Values are IG and GIG. IG integrates gradientsalong a path between a baseline and the sample being ex-plained. This explanation approaches the Aumann-Shapleyvalue. GIG is a generalization of IG to explain losses andmixed model types – a feature DeepSHAP also aims to pro-vide. IG and GIG have two downsides: 1.) integrating alonga path can be expensive or imprecise and 2.) the Aumann-Shapley values fundamentally differ to the SHAP values weaim to approximate. Finally, DASP (Ancona, Oztireli, andGross 2019) is an approach that approximates SHAP val-ues for deep networks. This approach works by replacingpoint activations at all layers by probability distributionsand requires many more model evaluations than DeepSHAP.Because DASP aims to obtain the same SHAP values asin DeepSHAP it is possible to use DASP as a part of theDeepSHAP framework.

ApproachPropagating SHAP valuesDeepSHAP builds upon DeepLIFT; in this section we aim tobetter understand how DeepLIFT’s rules connect to SHAPvalues. This has been briefly touched upon in (Shriku-mar, Greenside, and Kundaje 2017) and (Lundberg and Lee2017), but here we explicitly define the relationship.

DeepSHAP is a method that explains a sample (fore-ground sample), by setting features to be “missing”. Missingfeatures are set to corresponding values in a baseline sam-ple (background sample). Note that DeepSHAP generallyuses a background distribution, however focusing on a sin-gle background sample is sufficient because we can rewritethe SHAP values as an average over attributions with respectto a single background sample at a time (see next section for

arX

iv:1

911.

1188

8v1

[cs

.LG

] 2

7 N

ov 2

019

a) Linear

b) Rescale

c) RevealCancel

d) Mixed Model Stack

x2

x1 h11

h12

+

+T y

g h21

h22g

xk

x2

x1

...+ h

+

+

h+

h−

xk

x2

x1

...

x2

x1

+ y+

+ h21

h11

g y

+ h g y

y = g(h)

h =k∑

i=1

wixi

y = w(2)1 h1

1 + w(2)2 h2

1

hj1 = w

(1)1,jx1 + w

(1)2,jx1

Figure 1: Visualization of models for understandingDeepLIFT’s connection to SHAP values. In the figure g is anon-linear function and T is a non-differentiable tree model.

more details). In this section, we define a foreground sampleto have features xfxi

and neuron values fh (obtained by aforward pass) and a background sample to have bxi

or bh.Finally we define φ(·) to be attribution values.

If our model is fully linear as in Figure 1a, we can getthe exact SHAP values for an input xi by summing the at-tributions along all possible paths between that input xi andthe model’s output y. Therefore, we can focus on a partic-ular path (in blue). Furthermore, the path’s contribution toφ(xi) is exactly the product of the weights along the pathand the difference in x1: w(2)

2 w(1)1,2(fx1 − bx1), because we

can rewrite the layers of linear equations in 1a as a singlelinear equation. Note that we can derive the attribution forx1 in terms of the attribution of intermediary nodes (as inthe chain rule):

φ(h21) = w(2)2 (fh2

1−bh2

1)

φ(x1) =φ(h21)

fh21−bh2

1

w(1)1,2(fx1−bx1) (1)

Next, we move on to reinterpreting the two variants ofDeepLIFT: the Rescale rule and the RevealCancel rule. First,a gradient based interpretation of the Rescale rule has beendiscussed in (Ancona et al. 2018). Here, we explicitly tie thisinterpretation to the SHAP values we hope to obtain.

For clarity, we consider the example in Figure 1b. First,the attribution value for φ(h) is g(fh)−g(bh) because SHAPvalues maintain local accuracy (sum of attributions equalsfy − by) and g is a function with a single input. Then, underthe Rescale rule, φ(xi)=

φ(h)fh−bhwi(fxi

− bxi) (note the re-

semblance to Equation (1)). Under this formulation it is easyto see that the Rescale rule first computes the exact SHAPvalue for h and then propagates it back linearly. In otherwords, the the non-linear and linear functions are treated asseparate functions. Passing back nonlinear attributions lin-early is clearly an approximation, but confers two benefits:1.) fast computation on order of a backward pass and 2.) aguarantee of local accuracy.

Next, we describe how the RevealCancel rule (originallyformulated to bring DeepLIFT closer to SHAP values) con-nects to SHAP values in the context of Figure 1c. Reveal-Cancel partitions xi into positive and negative componentsbased on if wi(fxi

− bxi) < t (where t=0), in essence form-

ing nodes h+ and h−. This rule computes the exact SHAPattributions for h+ and h− and then propagates the resultantSHAP values linearly. Specifically:

φ(g+) =1

2((g(fh++fh−)− g(bh++fh−)+

(g(fh++bh−)− g(bh+

+bh−))

φ(g+) =1

2((g(fh+

+fh−)− g(fh++bh−)+

(g(bh++fh−)− g(bh+

+bh−))

φ(xi) =

φh+

fh+−bh+

wi(fxi− bxi

), if wi(fxi− bxi

) > tφh−

fh−−bh−wi(fxi − bxi), otherwise

Under this formulation, we can see that in contrast to theRescale rule that explains a linearity and nonlinearity by ex-actly explaining the nonlinearity and backpropagating, theRevealCancel rule exactly explains the nonlinearity and apartition of the inputs to the linearity as a single functionprior to backpropagating. The RevealCancel rule incurs ahigher computational cost in order to get a an estimate ofφ(xi) that is ideally closer to the SHAP values.

This reframing naturally motivates explanations forstacks of mixed model types. In particular, for Figure 1d,we can take advantage of fast, exact methods for obtainingSHAP values for tree models to obtain φ(h2j ) using Indepen-dent Tree SHAP (Lundberg et al. 2018). Then, we can prop-agate these attributions to get φ(xi) using either the Rescaleor RevealCancel rule. This argument extends to explaininglosses rather than output margins as well.

Although we consider specific examples here, the linearpropagation described above will generalize to arbitrary net-works if SHAP values can be computed or approximated forindividual components.

SHAP values with a background distributionNote that many methods (Integrated Gradients, Occlusion)recommend the utilization of a single background/referencesample. In fact, DeepSHAP as previously described in(Lundberg and Lee 2017) created attributions with respectto a single reference equal to the expected value of the in-puts. However, in order to obtain SHAP values for a givenbackground distribution, we prove that the correct approachis as follows: obtain SHAP values for each baseline in yourbackground distribution and average over the resultant at-tributions. Although similar methodologies have been usedheuristically (Shrikumar, Greenside, and Kundaje 2017;Erion et al. 2019), we provide a theoretical justification inTheorem 1 in the context of SHAP values.Theorem 1. The average over single reference SHAP valuesapproaches the true SHAP values for a given distribution.

Proof. Define D to be the data distribution, N to be the setof all features, and f to be the model being explained. Ad-

ditionally, define X (x, x′, S) to return a sample where thefeatures in S are taken from x and the remaining featuresfrom x′. Define C to be all combinations of the set N \ {i}and P to be all permutations of N \ {i}. Starting with thedefinition of SHAP values for a single feature: φi(x)

=∑

S∈CW (|S|, |N |)(ED[f(X)|xS∪{i}]−ED[f(X)|xS ])

=1

|P |∑

S⊆PED[f(x)|do(xS∪{i})]−ED[do(f(x)|xS)]

=1

|P |∑

S⊆P

1

|D|∑

x′∈Df(X (x, x′, S ∪ {i}))−f(X (x, x′, S))

=1

|D|∑

x′∈D

1

|P |∑

S⊆Pf(X (x, x′, S ∪ {i}))−f(X (x, x′, S))

︸ ︷︷ ︸single reference SHAP value

where the second step depends on an interventional condi-tional expectation (Janzing, Minorics, and Blobaum 2019)which is very close to Random Baseline Shapley in (Sun-dararajan and Najmi 2019)).

ExperimentsBackground distributions avoid bias

DeepLIFT(rescale)

DeepSHAP(rescale)

CIFAR10Image

Figure 2: Using a single baseline leads to bias in explana-tions.

In this section, we utilize the popular CIFAR10 dataset(Krizhevsky, Hinton, and others 2009) to demonstrate thatsingle references lead to bias in explanations. We train aCNN that achieves 75.56% test accuracy and evaluate it us-ing either a zero baseline as in DeepLIFT or with a randomset of 1000 baselines as in DeepSHAP.

In Figure 2, we can see that for these images drawn fromthe CIFAR10 training set, DeepLIFT has a clear bias thatresults in low attributions for darker regions of the image.For DeepSHAP, having multiple references drawn from abackground distribution solves this problem and we see at-tributions in sensical dark regions in the image.

Explaining mortality predictionIn this section, we validate DeepSHAP’s explanations for anMLP with 82.56% test accuracy predicting 15 year mortal-

ity. The dataset has 79 features for 14,407 individuals re-leased by (Lundberg et al. 2018) based on NHANES I Epi-demiologic Followup Study (Cox et al. 1997).

Age

Sex (F/M)

Systolic BP

Serum Albumin

Sedimentation Rate

Hematocrit

SHAP Value-0.4 -0.2 0.0 0.2 0.4

Feat

ure

Valu

e

Low

High

-0.6 0.6

Figure 3: Summary plot of DeepSHAP attribution values.Each point is the local feature attribution value, colored byfeature value. For brevity, we only show the top 6 features.

In Figure 3, we plot a summary of DeepSHAP (with 1000random background samples) attributions for all NHANEStraining samples (n=8023) and notice a few trends. First,Age is predictably the most important and old age con-tributes to a positive mortality prediction (positive SHAPvalues). Second, the Sex feature validates a well-known dif-ference in mortality (Gjonca et al. 1999). Finally, the trendslinking high systolic BP, low serum albumin, high sedimen-tation rate, and high hematocrit to mortality have been in-dependently discovered (Port et al. 2000; Goldwasser andFeldman 1997; Paul et al. 2012; Go et al. 2016).

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9Output ValueExpected Value 0.740.34

Sex (Male)

Hema-tocrit

Age (67)Total Bilirubin

Pulse Pressure

Potass-ium

Red Blood Cells

a.) Background distribution: 1000 random samples (general population)

b.) Background distribution: 1000 random samples (Age > 60 & Male)0.74 0.80 Expected ValueOutput Value

0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

Hema-tocrit

BUN Potass-ium

Red Blood Cells

CalciumWhite Blood Cells

Figure 4: Explaining an individual’s mortality prediction fordifferent backgrounds distributions.

Next, we show the benefits of being able to specify a back-ground distribution. In Figure 4a, we see that explaining anindividual’s mortality prediction with respect to a generalpopulation emphasizes that the individual’s age and genderare driving a high mortality prediction. However, in practicedoctors are unlikely to compare a 67-year old male to a gen-eral population that includes much younger individuals. InFigure 4b, being able to specify a background distributionallows us to compare our individual against a more relevantdistribution of males over 60. In this case, gender and ageare naturally no longer important, and the individual actu-ally may not have cause for concern.

Interpreting a stack of mixed model typesStacks, and more generally ensembles, of models areincreasingly popular for performant predictions (Bao,

b) Keep Absolute (Mask)

Max fraction of features kept0.0 0.2 0.4 0.6 0.8 1.0

0.222 - DeepSHAP0.207 - IME Explainer0.134 - Kernel SHAP

-0.117 - Random

0.0

0.2

0.4

-0.2

-0.4

-0.6

Mod

el p

erfo

rman

ce (R

2 )

a) Runtime vs. Sampling Variability

Number of model evaluations0 1000 2000 3000 4000

10th largest |���

IME ExplainerKernel SHAP

DeepSHAP0.20

0.15

0.10

0.05

0.00(200,0.00)

Mea

n st

anda

rd d

evia

tion

of �

�

Figure 5: Ablation test for explaining an LSTM feature ex-tractor fed into an XGB model. All methods used back-ground of 20 samples obtained via kmeans. [a.] Conver-gence of methods for a single explanation. [b.] Model perfor-mance versus # features kept for DeepSHAP (rescale), IMEExplainer (4000 samples), KernelSHAP (2000 samples) anda baseline (Random) (AUC in the legend).

Bergman, and Thompson 2009; Gunes, Wolfinger, and Tan2017; Zhai and Chen 2018). In this section, our aim isto evaluate the efficacy of DeepSHAP for a neural net-work feature extractor fed into a tree model. For this ex-periment, we use the Rescale rule for simplicity and In-dependent TreeSHAP to explain the tree model (Lundberget al. 2018). The dataset is a simulated one called Cor-rgroups60. Features X ∈ R1000×60 have tight correlationbetween groups of features (xi is feature i), where ρxi,xi

=1,ρxi,xi+1

=ρxi,xi+2=ρxi+1,xi+2

=.99 if (i mod 3)=0, andρxi,xj

=0 otherwise. The label y ∈ Rn is generated lin-early as y=Xβ+ε where ε∼Nn(µ=0, σ2=10−4) and βi=1if (i mod 3)=0 and βi=0 otherwise.

We evaluate DeepSHAP with an ablation metric calledkeep absolute (mask) (Lundberg et al. 2018). The metricworks in the following manner: 1) Obtain the feature attribu-tions for all test samples 2) Mask all features (by mean im-putation) 3) Introduce one feature at a time (unmask) fromlargest absolute attribution value to smallest for each sampleand measure R2. The R2 should initially increase rapidly,because we introduce the “most important” features first.

We compare against two sampling-based methods (a nat-ural alternative for explaining mixed model stacks) that pro-vide SHAP values in expectation: KernelSHAP and IME ex-plainer. In Figure 5b, DeepSHAP (rescale) has no variabil-ity and requires a fixed number of model evaluations. IMEExplainer and KernelSHAP, benefit from having more sam-ples (and therefore more model evaluations). For the finalcomparison, we check the variability of the tenth largest at-tribution (absolute value) of the sampling based methods todetermine “convergence” across different numbers of sam-ples. Then, we use the number of samples at the point of“convergence” for the next figure.

In Figure 5c, we can see that DeepSHAP has a slightlyhigher performance than model agnostic methods. Promis-ingly, all methods demonstrate initial steepness in their per-formance; this indicates that the most important features hadhigher attribution values. We hypothesize that KernelSHAPand IME Explainer’s lower performance is due in part tonoise in their estimates. This highlights an important point:model agnostic methods often have sampling variability that

makes determining convergence difficult. For a fixed back-ground distribution, DeepSHAP does not suffer from thisvariability and generally requires fewer model evaluations.

Improving the RevealCancel rule

a.) RevealCancel vs. Rescale b.) Choice of RevealCancel threshold

Figure 6: Comparison of new RevealCancelMean rule for esti-mating SHAP values on a toy example. The axes correspondto mean absolute difference from the SHAP values (com-puted exactly). Green means RevealCancelMean wins and redmeans it loses.

DeepLIFT’s RevealCancel rule’s connection to the SHAPvalues is touched upon in (Shrikumar, Greenside, and Kun-daje 2017). Our SHAP value framework explicitly definesthis connection. In this section, we propose a simple im-provement to the RevealCancel rule. In DeepLIFT’s Reveal-Cancel rule the threshold t is set to 0 (for splitting h− andh+). Our proposed rule RevealCancelMean sets the thresholdto the mean value of wi(fxi

−bxi) across i. Intuitively, split-

ting by the mean better separates xi nodes, resulting in abetter approximation than splitting by zero.

We experimentally validate RevealCancelMean in Figure 6,explaining a simple function: ReLU(x1 + x2 + x3 + x4 +100). We fix the background to zero: bxi=0 and draw 100foreground samples from a discrete uniform distribution:fxi∼U{−1000, 1000}.

In Figure 6a, we show that RevealCancelMean offers alarge improvement for approximating SHAP values over theRescale rule and a modest one over the original RevealCan-cel rule (at no additional asymptotic computational cost).

ConclusionIn this paper, we improve the original DeepSHAP formula-tion (Lundberg and Lee 2017) in several ways: we 1.) pro-vide a new theoretically justified way to provide attributionswith a background distribution 2.) extend DeepSHAP to ex-plain stacks of mixed model types 3.) present improvementsof the RevealCancel rule.

Future work includes more quantitative validation on dif-ferent data sets and comparison to more interpretabilitymethods. In addition, we primarily used Rescale rule formany of these evaluations, but more empirical evaluationsof RevealCancel are also important.

References[Ancona et al. 2018] Ancona, M.; Ceolini, E.; Oztireli, C.;and Gross, M. 2018. Towards better understanding ofgradient-based attribution methods for deep neural net-works. In 6th International Conference on Learning Rep-resentations (ICLR 2018).

[Ancona, Oztireli, and Gross 2019] Ancona, M.; Oztireli,C.; and Gross, M. 2019. Explaining deep neural networkswith a polynomial time algorithm for shapley values approx-imation. arXiv preprint arXiv:1903.10992.

[Bach et al. 2015] Bach, S.; Binder, A.; Montavon, G.;Klauschen, F.; Muller, K.-R.; and Samek, W. 2015. Onpixel-wise explanations for non-linear classifier decisions bylayer-wise relevance propagation. PloS one 10(7):e0130140.

[Bao, Bergman, and Thompson 2009] Bao, X.; Bergman, L.;and Thompson, R. 2009. Stacking recommendation engineswith additional meta-features. In Proceedings of the thirdACM conference on Recommender systems, 109–116. ACM.

[Cox et al. 1997] Cox, C. S.; Feldman, J. J.; Golden, C. D.;Lane, M. A.; Madans, J. H.; Mussolino, M. E.; and Roth-well, S. T. 1997. Plan and operation of the nhanes i epidemi-ologic followup study, 1992. Vital and Health Statistics.

[Erion et al. 2019] Erion, G.; Janizek, J. D.; Sturmfels, P.;Lundberg, S.; and Lee, S.-I. 2019. Learning ex-plainable models using attribution priors. arXiv preprintarXiv:1906.10670.

[Gjonca et al. 1999] Gjonca, A.; Tomassini, C.; Vaupel,J. W.; et al. 1999. Male-female differences in mortality inthe developed world. Citeseer.

[Go et al. 2016] Go, D. J.; Lee, E. Y.; Lee, E. B.; Song, Y. W.;Konig, M. F.; and Park, J. K. 2016. Elevated erythrocyte sed-imentation rate is predictive of interstitial lung disease andmortality in dermatomyositis: a korean retrospective cohortstudy. Journal of Korean medical science 31(3):389–396.

[Goldwasser and Feldman 1997] Goldwasser, P., and Feld-man, J. 1997. Association of serum albumin and mortalityrisk. Journal of clinical epidemiology 50(6):693–703.

[Goodman and Flaxman 2017] Goodman, B., and Flaxman,S. 2017. European union regulations on algorithmicdecision-making and a “right to explanation”. AI Magazine38(3):50–57.

[Gunes, Wolfinger, and Tan 2017] Gunes, F.; Wolfinger, R.;and Tan, P.-Y. 2017. Stacked ensemble models for improvedprediction accuracy. In SAS Conference Proceedings.

[Holzinger et al. 2017] Holzinger, A.; Biemann, C.; Pat-tichis, C. S.; and Kell, D. B. 2017. What do we need tobuild explainable ai systems for the medical domain? arXivpreprint arXiv:1712.09923.

[Janzing, Minorics, and Blobaum 2019] Janzing, D.; Mi-norics, L.; and Blobaum, P. 2019. Feature relevancequantification in explainable ai: A causality problem. arXivpreprint arXiv:1910.13413.

[Krizhevsky, Hinton, and others 2009] Krizhevsky, A.; Hin-ton, G.; et al. 2009. Learning multiple layers of featuresfrom tiny images. Technical report, Citeseer.

[Lundberg and Lee 2017] Lundberg, S. M., and Lee, S.-I.2017. A unified approach to interpreting model predic-tions. In Advances in Neural Information Processing Sys-tems, 4765–4774.

[Lundberg et al. 2018] Lundberg, S. M.; Erion, G.; Chen, H.;DeGrave, A.; Prutkin, J. M.; Nair, B.; Katz, R.; Himmelfarb,J.; Bansal, N.; and Lee, S. 2018. Explainable ai for trees:From local explanations to global understanding. CoRRabs/1905.04610.

[Merrill et al. 2019] Merrill, J.; Ward, G.; Kamkar, S.;Budzik, J.; and Merrill, D. 2019. Generalized integratedgradients: A practical method for explaining diverse ensem-bles. CoRR abs/1909.01869.

[Paul et al. 2012] Paul, L.; Jeemon, P.; Hewitt, J.; McCal-lum, L.; Higgins, P.; Walters, M.; McClure, J.; Dawson, J.;Meredith, P.; Jones, G. C.; et al. 2012. Hematocrit predictslong-term mortality in a nonlinear and sex-specific mannerin hypertensive adults. Hypertension 60(3):631–638.

[Port et al. 2000] Port, S.; Demer, L.; Jennrich, R.; Walter,D.; and Garfinkel, A. 2000. Systolic blood pressure andmortality. The Lancet 355(9199):175–180.

[Shapley 1953] Shapley, L. S. 1953. A value for n-persongames. Contributions to the Theory of Games 2(28):307–317.

[Shrikumar, Greenside, and Kundaje 2017] Shrikumar, A.;Greenside, P.; and Kundaje, A. 2017. Learning importantfeatures through propagating activation differences. In Pro-ceedings of the 34th International Conference on MachineLearning-Volume 70, 3145–3153. JMLR. org.

[Simonyan, Vedaldi, and Zisserman 2013] Simonyan, K.;Vedaldi, A.; and Zisserman, A. 2013. Deep inside convo-lutional networks: Visualising image classification modelsand saliency maps. arXiv preprint arXiv:1312.6034.

[Strumbelj and Kononenko 2014] Strumbelj, E., andKononenko, I. 2014. Explaining prediction models and in-dividual predictions with feature contributions. Knowledgeand information systems 41(3):647–665.

[Sundararajan and Najmi 2019] Sundararajan, M., and Na-jmi, A. 2019. The many shapley values for model expla-nation. arXiv preprint arXiv:1908.08474.

[Sundararajan, Taly, and Yan 2017] Sundararajan, M.; Taly,A.; and Yan, Q. 2017. Axiomatic attribution for deep net-works. arXiv preprint arXiv:1703.01365.

[Zeiler and Fergus 2014] Zeiler, M. D., and Fergus, R.2014. Visualizing and understanding convolutional net-works. In European conference on computer vision, 818–833. Springer.

[Zhai and Chen 2018] Zhai, B., and Chen, J. 2018. Develop-ment of a stacked ensemble model for forecasting and ana-lyzing daily average pm 2.5 concentrations in beijing, china.Science of the Total Environment 635:644–658.