When does data augmentation help generalization in NLP?

Rohan Jha and Charles Lovering and Ellie PavlickBrown University

{first} {last}@brown.edu

Abstract

Neural models often exploit superficial(“weak”) features to achieve good perfor-mance, rather than deriving the more general(“strong”) features that we’d prefer a modelto use. Overcoming this tendency is a centralchallenge in areas such as representationlearning and ML fairness. Recent work hasproposed using data augmentation–that is,generating training examples on which theseweak features fail–as a means of encouragingmodels to prefer the stronger features. Wedesign a series of toy learning problems toinvestigate the conditions under which suchdata augmentation is helpful. We show thataugmenting with training examples on whichthe weak feature fails (“counterexamples”)does succeed in preventing the model fromrelying on the weak feature, but often doesnot succeed in encouraging the model touse the stronger feature in general. Wealso find in many cases that the number ofcounterexamples needed to reach a given errorrate is independent of the amount of trainingdata, and that this type of data augmentationbecomes less effective as the target strongfeature becomes harder to learn.

1 Introduction

Neural models often perform well on tasks by us-ing superficial (“weak”) features, rather than themore general (“strong”) features that we’d preferthem to use. Recent studies expose this tendencyby highlighting how models fail when evaluated ontargeted challenge sets consisting of “counterexam-ples” where the weak feature begets an incorrect re-sponse. For example, in visual question answering,models failed when tested on rare color descrip-tions (“green bananas”) (Agrawal et al., 2018); incoreference, models failed when tested on infre-quent profession-gender pairings (“the nurse caredfor his patients”) (Rudinger et al., 2018); in nat-

ural language inference (NLI), models failed onsentence pairs with high lexical overlap with differ-ent meanings (“man bites dog”/“dog bites man”)(McCoy et al., 2019).

One proposed solution has been to augmenttraining data to over-represent these tail events, of-ten with automatic or semi-automatic methods forgenerating such counterexamples at a large scale.This technique has been discussed for POS tagging(Elkahky et al., 2018), NLI (McCoy et al., 2019),and as a means of reducing gender bias (Zhao et al.,2018; Zmigrod et al., 2019), all with positive initialresults. However, it is difficult to know whetherthis strategy is a feasible way of improving systemsin general, beyond the specific phenomena targetedby the data augmentation. Understanding the con-ditions under which adding such training examplesleads a model to switch from using weaker featuresto stronger features is important for both practicaland theoretical work in NLP.

We design a set of toy learning problems to ex-plore when (or whether) the above-described typeof data augmentation helps models learn strongerfeatures. We consider a simple neural classifier ina typical NLP task setting where: 1) the labeled in-put data exhibits weak features that correlate withthe label and 2) the model is trained end-to-endand thus adopts whichever feature representationperforms best. Our research questions are:

• How many counterexamples must be seen intraining to prevent the model from adoptinga given weak feature? Do larger training setsrequire more counterexamples or fewer?

• Does the relative difficulty of representing afeature (strong or weak) impact when/whethera model adopts it?

• How does the effectiveness of data augmen-tation change in settings which contain many

1

arX

iv:2

004.

1501

2v1

[cs

.CL

] 3

0 A

pr 2

020

weak features but only a single strong feature?

Terminology: This work relates to a very largebody of work on learning, generalization, and ro-bustness in neural networks. Our goal with the cur-rent set of experiments is to establish a frameworkwithin which we can begin to isolate the phenom-ena of interest (data augmentation in NLP) and ob-serve patterns empirically, without the confoundsthat exist in the applied settings where the data aug-mentation techniques in which we are interestedhave been previously used. The results presentedare intended to probe the problem setting and helpfocus on interesting questions, prior to beginningto formalize the phenomena. As such, our choiceof terminology (“strong”, “weak”, “hard”, “coun-terexample”) is meant to be informal. Much of theexisting relevant vocabulary carries connotationswhich we do not specifically intend to invoke here,at least not yet. E.g. the work is related to adver-sarial examples and attacks, but we do not assumeany sort of iterative or model-aware component tothe way the counterexamples are generated. It isrelated to “spurious correlations”, “heurisitics”,and “counterfactual” examples, but we do not (yet)make formal assumptions about the nature of theunderlying causal graph. We also view the prob-lems discussed as related to work on perturbationsof the training distribution, to cross-domain gen-eralization, and to non-iid training data; there arelikely many options for casting our problem intothese terms, and we have not yet determined thebest way for doing so. Again, our present goal is tohome in on the phenomena of interest and share ourinitial informal findings, with the hope that doingso will enable more formal insights to follow.

2 Experimental Setup

2.1 Intuition

Our study is motivated by two empirical findingspresented in McCoy et al. (2019). Specifically, Mc-Coy et al. (2019) focused on models’ use of syntac-tic heuristics in the context of the natural languageinference (NLI) task: given a pair of sentences–thepremise p and the hypothesis h–predict whether ornot p entails h. They showed that when 1% of the300K sentence pairs seen in training exhibit lexicaloverlap (i.e. every word in h appears in p) and90% of lexical-overlap sentence pairs have the la-bel ENTAILMENT, the model adopts the (incorrect)heuristic that lexical overlap always corresponds

to ENTAILMENT. However, after augmenting thetraining data with automatically generated trainingexamples so that 10% of the 300K training pairsexhibit lexical overlap and 50% of lexical-overlapsentence pairs have the label ENTAILMENT, thesame model did not adopt the heuristic and ap-peared to learn features which generalized to anout-of-domain test distribution.

From these results, it is hard to say whichchanges to the training setup were most importantfor the model’s improved generalizability. Thenumber of lexical-overlap examples seen in train-ing? The probability of ENTAILMENT given that apair exhibits lexical overlap? Or some other pos-itive artifact of the additional training examples?Thus, we abstract away from the specifics of theNLI task in order to consider a simplified settingthat captures the same intuition but allows us toanswer such questions more precisely.

2.2 Assumptions and Terminology

We consider a binary sequence classification task.We assume there exists some feature which directlydetermines the correct label, but which is non-trivial to extract given the raw input. (In NLI, ide-ally, such a feature is whether the semantic meaningof h contains that of p). We refer to this feature asthe strong feature. Additionally, we assume theinput contains one or more weak features whichare easy for the model to extract from the input.(This is analogous to lexical overlap between p andh). In our set up, the correct label is 1 if and onlyif the strong feature holds.1 However, the strongand weak features frequently co-occur in trainingand so a model which only represents the weak fea-tures will be able to make correct predictions muchof the time. We can vary their co-occurence rateby adding counterexamples to the training data inwhich either the strong feature or the weak featureis present, but not both. This setup is shown inFigure 1.

2.3 Implementation

Task. We use a synthetic sentence classificationtask with sequences of numbers as input and bi-nary {0, 1} labels as output. We use a symbolicvocabulary V consisting of the integers 0 . . . |V |.In all experiments, we use sequences of length 5and set |V | to be 50K. We do see some effects asso-

1See Appendix B.1 for results which assume varying levelsof label noise.

2

- -

-

-

--

-

- --

-

--

-

-

++

+

+

++

+

+-

--

-

-

-

--

-

- --

-

--

-

-

++

+

+

++

+

+-

--

-

++ ++

++

--

---

Initial Training Set After Data Augmentation

strong feature

weak feature

strong-only counter examples

weak-only counter-

examples

Figure 1: Schematic of experimental setup.

ciated with vocabulary size, but none that affect ourprimary conclusions; see Appendix A.1 for details.

Model. We use a simple network comprising anembedding layer, a 1-layer LSTM, and a 1-layerMLP with a RELU activation. We found enoughinteresting trends to analyze in the behavior ofthis model, and thus leave experiments with morecomplex model architectures for future work. Ourcode, implemented in PyTorch, is released for re-producibility.2 Appendix A discusses how our ex-perimental results extend across changes in modeland task parameters. All models are trained untilconvergence, using early-stopping.3

Strong and Weak Features. In all experiments,we set the weak feature to be the presence of thesymbol 2 anywhere in the input. We considerseveral different strong features, listed in Table 1.These features are chosen with the intent of varyinghow difficult the strong feature is to detect given theraw sequential input. In all experiments, we designtrain and test splits such that the symbols which areused to instantiate the strong feature during train-ing are never used to instantiate the strong featureduring testing. For example, for experiments usingadjacent duplicate, if the model sees thestring 1 4 3 3 15 at test time, we enforce that itnever saw any string with the duplicate 3 3 duringtraining. This is to ensure that we are measuringwhether the model learned the desired pattern, anddid not simply memorize bigrams.

To quantify the difficulty of representing eachstrong feature, we train the model for the task ofpredicting directly whether or not the feature holdsfor each of our candidate feature, using a set of200K training examples evenly split between caseswhen the feature does and does not hold. For each

2http://bit.ly/counterexamples3The results shown here used the test error for early-

stopping; we have re-run some of our experiments with early-stopping on validation error, and it did not make a difference.

feature, Figure 2 shows the validation loss curve(averaged over three runs), flat-lined at its min-imum (i.e. its early stopping point). We see thedesired gradation in which some feature require sig-nificantly more training to learn than others. As aheuristic measure of “hardness”, we use the approx-imate area under this flat-lined loss curve (AUC),computed by taking the sum of the errors across allepochs. Table 1 contains the result for each feature.Note that the weak feature (whether the sequencecontains 2) is exactly as hard as contains 1.

0 5 10 15 20# epochs

103

102

101

erro

r

Strong Featureprefix duplicatefirst-last duplicate

adjacent duplicatecontains first

Figure 2: Learning curve for classifying whether thefeatures hold. Averaged over three re-runs and then flat-lined after the average reaches its minimum. Train sizeis 200K. The error for contains 1 and the weak fea-ture (not shown) reaches 0 after the first epoch.

2.4 Error Metrics

Definition. We partition test error into fourregions of interest, defined below. We use afiner-grained partition than standard true positiverate and false positive rate because we are particu-larly interested in measuring error rate in relationto the presence or absence of the weak feature.

weak-only: P (pred = 1 | weak,¬strong)strong-only: P (pred = 0 | ¬weak, strong)both: P (pred = 0 | weak, strong)neither: P (pred = 1 | ¬weak,¬strong)

For completeness, we define and compute botherror and neither error. However, in practice, sinceour toy setting contains no spurious correlationsother than those due to the weak feature, we findthat these two error metrics are at or near zero forall of our experiments (except for a small numberof edge cases discussed in §C). Thus, for all resultsin Section 3, we only show plots of strong-only andweak-only error, and leave plots of the others toAppendix C.

3

Feature Nickname Description Loss AUC Example

contains 1 1 occurs somewhere in the sequence 0.50 2 4 11 1 4prefix duplicate Sequence begins with a duplicate 0.52 2 2 11 12 4first-last duplicate First number equals last number 0.55 2 4 11 12 2adjacent duplicate Adjacent duplicate is somewhere in the sequence 1.43 11 12 2 2 4contains first First number is elsewhere in the sequence 1.54 2 11 2 12 4

Table 1: Features used to instantiate the strong feature in our experimental setup. Features are intended to differ inhow hard they are for an LSTM to detect given sequential input. We use the the AUC of the validation loss curveas a heuristic measure of “hardness”, as described in Section 2.3.

Interpretation. Intuitively, high weak-only errorindicates that the model is associating the weakfeature with the positive label, whereas high strong-only error indicates the model is either failing todetect the strong feature altogether, or is detectingit but failing to associate it with positive label. Inpractice, we might prioritize these error rates dif-ferently in different settings. For example, withinwork on bias and fairness (Hall Maudslay et al.,2019; Zmigrod et al., 2019), we are primarily tar-geting weak-only error. That is, the primary goalis to ensure that the model does not falsely asso-ciate protected attributes with specific labels oroutcomes. In contrast, in discussions about improv-ing the robustness of NLP more generally (Elkahkyet al., 2018; McCoy et al., 2019), we are presum-ably targeting strong-only error. That is, we arehoping that by lessening the effectiveness of shal-low heuristics, we will encourage models to learndeeper, more robust features in their place.

2.5 Data Augmentation

Definition. Adversarial data augmentation aimsto reduce the above-described errors by generatingnew training examples which decouple the strongand weak features. Parallel to the error categories,we can consider two types of counterexamples:weak-only counterexamples in which the weakfeature occurs without the strong feature and thelabel is 0, and strong-only counterexamples inwhich the strong feature occurs without the weakfeature and the label is 1.

Interpretation. Again, in terms of practical in-terpretation, these two types of counterexamplesare meaningfully different. In particular, it is likelyoften the case that weak-only counterexamples areeasy to obtain, whereas strong-only counterexam-ples are more cumbersome to construct. For ex-ample, considering again the case of NLI and thelexical overlap heuristic from McCoy et al. (2019),

it is easy to artificially generate weak-only coun-terexamples (p/h pairs with high lexical overlapbut which are not in an entailment relation) using aset of well-designed syntactic templates. However,generating good strong-only counterexamples (en-tailed p/h pairs without lexical overlap) is likely torequire larger scale human effort (Williams et al.,2018). This difference would likely be exacerbatedby realistic problems in which there are many weakfeatures which we may want to remove and/or it isimpossible to fully isolate the strong features fromall weak features. For example, it may not be pos-sible to decouple the “meaning” of a sentence fromall lexical priors. We explore the case of multipleweak features in Section 3.4.

2.6 Limitations

This study is intended to provide an initial frame-work and intuition for thinking about the relation-ships between data augmentation and model gen-eralization. We simplify the problem significantlyin order to enable controlled and interpretable ex-periments. As a result, the problems and modelsstudied are several steps removed from the whatNLP looks like in practice. In particular, we con-sider a very small problem (binary classification ofsequences of length 5 in which only two variableare not independent) and a very small model (asimple LSTM). Although we provide many addi-tional results in the Appendix to show consistencyacross different model and task settings, we donot claim that the presented results would holdfor more complex models and tasks–e.g. multi-layer transformers performing language modeling.Clearly many details of our setup–e.g. which fea-tures are easier or harder to detect–would changesignificantly if we were to change the model ar-chitecture and/or training objective to reflect morerealistic settings. Exploring whether the overarch-ing trends we observe still hold given such changeswould be exciting follow up work.

4

We also make the assumption that the input tothe model contains a single “true” feature which,once extracted, perfectly explains the output. Formany of the tasks currently studied in NLP, thisassumption arguably never holds, or rather, modelsonly ever get access to correlates of the strong fea-ture. That is, they see text descriptions but neverthe underlying referents. Thus, for a task like NLI,it might be that the best our models can do is findincreasingly strong correlates of “meaning”, butmay not ever extract “meaning” itself. This is amuch larger philosophical debate on which we donot take a stance here. We let it suffice that, inpractice, the presented results are applicable to anytask in which there exists some stronger (deeper)feature(s) that we prefer the model to use and someweaker (shallower) feature(s) which it may choseto use instead, whether or not those strong featuresare in fact a “true” feature that determines the label.

3 Results and Discussion

Our primary research questions, reframed in termsof the above terminology, are the following. First,how many counterexamples are needed in order toreduce the model’s prediction error? In particular,how is this number influenced by the hardness ofthe strong features (§3.1), the type of counterexam-ples added (i.e. strong-only vs. weak-only) (§3.2),and the size of the training set (§3.3)? Second,given a setting in which multiple weak features ex-ist, does the model prefer to make decisions basedon multiple weak features, or rather to extract asingle strong feature (§3.4)? We report all resultsin terms of strong-only error and weak-only error;results for other error categories are given in Ap-pendix C.

3.1 Effect of Strong Feature’s Hardness

We first consider prediction error as a function ofthe number of counterexamples added and the hard-ness of detecting the strong feature (with “hardness”defined as in Section 2.3). To do this, we constructan initial training set of 200K examples in whichthere is perfect co-occurrence between the strongand weak features, with the dataset split evenly be-tween positive examples (e.g. has both strong andweak features) and negative examples (with neitherfeature). We then vary the number of counterexam-ples added4 from 10 (≪ 0.1% of the training data)

4The results presented assume that the counterexamplesare added on top of the training data that is already there, and

to 100K (33% of the training data) and measurethe effect on strong-only and weak-only error. Fornow, we assume that the counterexamples addedare evenly split between strong-only and weak-onlytypes.

Figure 3 shows the results. We see that thenumber of counterexamples needed is substantiallyinfluenced by the hardness of the strong feature.For example, after only 10 counterexamples areadded to training, test error has dropped to nearzero when the strong feature is contains 1(which is trivial for the model to detect) but remainsvirtually unchanged for the contains firstand adjacent duplicate features. For theharder features, we don’t reach zero error until athird of the training data is composed of counterex-amples.

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0er

ror

weak-only error

5K 25K 50K 100K# examples

strong-only error

Strong Featurecontains 1prefix duplicatefirst-last duplicate

adjacent duplicatecontains first

Figure 3: The harder the strong feature, the more coun-terexamples the model requires to reduce a given errorrate. The legend shows models in order from hardest toleast hard, where “hardness” is determined by the AUCof the loss curve for a classifier trained to detect the fea-ture (§2.3). We run all experiments over five randomseeds. In all plots, the error band is the 95% confidenceinterval with 1,000 bootstrap iterations.

3.2 Effect of Counterexample TypeWe get a better understanding of the model’s be-havior when looking separately at strong-only andweak-only errors, considering each as a function ofthe number and the type (e.g. strong-only vs. weak-only) of counterexamples added (Figure 4). We see

thus models trained with larger numbers of counterexampleshave a slightly larger total training size. We also experimentedwith adding counterexamples in place of existing trainingexamples so that the total training size remains fixed acrossall runs. We do not see a meaningful difference in results.Results from the constant-training-size experiments are givenin Appendix D.2. We also perform a control experiment toverify that the additional number of training examples itself isnot impacting the results in Appendix D.1.

5

that adding weak-only counterexamples leads toimprovements in weak-only error, but has minimaleffect on strong-only error. The interesting excep-tion to this is when the strong feature is no harderto represent than the weak feature. In our setting,this happens when the strong feature is contains1, but it is unclear if this pattern would hold forother weak features. In this case, we see that bothstrong-only and weak-only error fall to zero afteradding only a small number of weak-only coun-terexamples.

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error

weak-only exam

ples

strong-only error

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

strong-only examples

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

both examples

Strong Featurecontains 1prefix duplicatefirst-last duplicate

adjacent duplicatecontains first

Figure 4: Adding weak-only counterexamples im-proves weak-only error but does not affect strong-onlyerror. In contrast, adding strong-only counterexamplesmay lead to improvements of both error types (see textfor discussion).

In contrast, we see some evidence that addingstrong-only counterexamples has impact on weak-only error as well as on strong-only error. The im-pact on weak-only error, however, is limited to thesettings in which the strong feature is sufficientlyeasy to detect. That is, when the strong feature isdifficult to detect, adding strong-only counterex-

amples does lead the model to detect the strongfeature and correctly associate it with the positivelabel, but does not necessarily lead the model toabandon the use of the weak feature in predictinga positive label. This behavior is interesting, sinceany correct predictions made using the weak fea-ture are by definition redundant with those madeusing the strong feature, and thus continuing tohold the weak feature can only hurt performance.

3.3 Effect of Training Data Size

In Section 3.1, we observed that, for most of ourfeatures, the model did not reach near zero error un-til 20% or more of its training data was composedof counterexamples. This raises the question: iserror rate better modeled in terms of the absolutenumber of counterexamples added, or rather thefraction of the training data that those counterex-amples make up? For example, does adding 5Kcounterexamples produce the same effect regard-less of whether those 5K make up 5% of a 100Ktraining set or 0.05% of a 10M training set? Ourintuition motivating this experiment is that largerinitial training sets might “dilute” the signal pro-vided by the comparably small set of counterex-amples, and thus larger training sets might requiresubstantially more counterexamples to achieve thesame level of error.

In Figure 5, we again show both weak-only andstrong-only error as a function of the number ofcounterexamples added to training, but this time fora range of models trained with different initial train-ing set sizes. Here, for simplicity, we again assumethat the counterexamples added are evenly splitbetween the strong-only and weak-only types. Gen-erally speaking, for most features, we see that ourintuition does not hold. That is, increasing the train-ing size while holding the number of counterexam-ples fixed does not substantially effect either errormetric, positively or negatively. That said, thereare several noteworthy exceptions to this trend. Inparticular, we see that when the training set is verylarge (10M) relative to the number of counterex-amples (< 50K), the efficacy of those counterex-amples does appear to decrease. We also againsee differences depending on the strong feature,with the harder features (contains first andadjacent duplicate) behaving more in linewith the “diluting” intuition described above thanthe easier features (first-last duplicateand prefix duplicate).

6

0.0

0.2

0.4

0.6

0.8

1.0er

ror

weak-only error

prefix duplicate

strong-only error

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

first-last duplicate

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

adjacent duplicate

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

contains first

Training Size (Millions)0.1 M 1 M 10 M

Figure 5: Error rate vs. number of counterexamplesadded for models trained with different initial trainingset sizes (100K to 10M). Feature contains 1 is notshown since it is always learned instantly. In general,increasing the training size while holding the numberof counterexamples fixed does not significantly affectthe efficacy of those counterexamples, although thereare exceptions (see text for discussion).

3.4 Multiple Weak Features

In our experiments so far, we have assumed thatthere is only one weak feature, which is an unreal-istic approximation of natural language data. Wetherefore relax this assumption and consider thesetting in which there are multiple weak featureswhich are correlated with the label. The question

is: does the option of minimizing loss by com-bining multiple weak features lessen the model’swillingness to extract strong features?

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error

prefix duplicate

strong-only error

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

first-last duplicate

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

adjacent duplicate

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

contains first

# distractorsone two three idealized

Figure 6: Error rate vs. number of counterexamplesadded for models trained in settings with different num-bers of weak features, assuming that strong-only coun-terexamples can guarantee the removal of at most oneweak feature at a time. Initial training size is 200K.Gold line shows performance in an idealized setting inwhich there are 3 weak feature but strong-only coun-terexamples are “pure”, i.e. free from all weak features.When more weak features are present, we see higherstrong-only error.

Our experimental setup is as follows. We as-sume k weak features d1, . . . dk, each of which

7

is correlated with the strong feature t, but whichare not correlated with each other beyond theircorrelation with t. As in previous experiments,P (di|t) and P (t|di) are determined by the numberof strong-only and weak-only counterexamples thathave been added, respectively. In the initial trainingset, before any counterexamples have been added,where P (t|di) is 1, and P (di|t) is close to 1/25.We assume that weak-only counterexamples can begenerated in the same way as before, but that goodstrong-only counterexamples cannot be generatedperfectly. For intuition, again consider the NLI set-ting. It is easy for a person to generate a p/h pairwith the label CONTRADICTION, but doing so whilemeeting the constraints that the sentences exhibitlexical overlap (McCoy et al., 2019) but do not con-tain any antonym pairs (Dasgupta et al., 2018) orexplicit negations (Gururangan et al., 2018) wouldlikely lead to unnatural and unhelpful training sen-tences. Thus, we assume that we can only reliablyremove at most one6 weak feature at a time. Specif-ically, we assume that, for a given weak featuredi, we can generate strong-onlyi counterexampleswhich 1) definitely exhibit the strong feature, 2)definitely do not exhibit di, and 3) exhibit everyother weak features dj with probability P (dj |t).

We again plot strong-only and weak-only erroras a function of the number of counterexamplesadded, assuming we add counterexamples whichare evenly split between weak-only and strong-onlytypes and evenly split among the k weak features.Note that, while our strong-only counterexamplesare not “pure” (i.e. they might contain weak fea-tures other than the one specifically targeted), ourstrong-only error is still computed on examplesthat are free of all weak features. Our weak-onlyerror is computed over examples that contain nostrong feature and at least one weak features. Theresults are shown in Figure 6 for k = {1, 2, 3}.7For comparison, we also plot the performance ofa model trained in an idealized setting in which“pure” strong-only counterexamples are added. Wesee that, holding the number of counterexamples

5We independently sample each feature with probability1/2 but then resample if the example does not contain anyweak features. This is done such that the reported number ofcounterexamples is correct.

6We ran experiments in which it was possible to removemore than one weak feature at a time (but still fewer than k)and did not see interestingly different trends from the at-most-one setting.

7Given our sequence length of 5, we were unable to gener-ate results for larger values of k without interfering with ourability to include the strong feature.

fixed, the more weak features there are, the higherthe strong-only error tends to be. We see also that,in an idealized setting in which it is possible togenerate “pure” strong-only counterexamples, thistrend does not hold, and there is no difference be-tween the single weak features and the multipleweak features settings. In terms of weak-only error,we see only small increases in error as the numberof weak features increases.

Taken together, our interpretation of these re-sults is that access to both weak-only counterexam-ples and to imperfect strong-only counterexamplesis sufficient for the model to unlearn undesirableweak features–that is, the model does learn thatnone of the di alone causes a positive label, andthus is able to achieve effectively zero weak-onlyerror. However, as evidenced by the substantiallyhigher strong-only error, the model does not ap-pear to learn to use the strong feature instead. Itsworth acknowledging that the model appears tolearn some information about the strong feature,as we do see that strong-only error falls as we addcounterexamples, just more slowly for larger val-ues of k. This trend might be taken as evidencethat the high strong-only error is not due to a failureto detect the feature (as was the case in Figure 4when only weak-only errors were added), but rathera failure to consistently associate the strong featurewith the positive label.

4 Related Work

4.1 Adversarial Data Augmentation

A wave of recent work has adopted a strategy ofconstructing evaluation sets composed of “adver-sarial examples” (a.k.a. “challenge examples” or“probing sets”) in order to analyze and expose weak-nesses in the decision procedures learned by neu-ral NLP models (Jia and Liang, 2017; Glockneret al., 2018; Dasgupta et al., 2018; Gururanganet al., 2018; Poliak et al., 2018b, and others). Ourwork is motivated by the subsequent research thathas begun to ask whether adding such challengeexamples to a model’s training data could help toimprove robustness. This work has been referredto by a number of names including “adversarial”,“counterfactual”, and “targeted” data augmentation.In particular, Liu et al. (2019) show that fine-tuningon small challenge sets can sometimes (thoughnot always) help models perform better. Similarapproaches have been explored for handling noun-verb ambiguity in syntactic parsing (Elkahky et al.,

8

2018), improving NLI models’ handling of syntac-tic (McCoy et al., 2019) and semantic (Poliak et al.,2018a) phenomena, and mitigating gender biasesin a range of applications (Zmigrod et al., 2019;Zhao et al., 2018, 2019; Hall Maudslay et al., 2019;Lu et al., 2018).

4.2 Adversarial RobustnessAdversarial robustness concerns whether a modelproduces the same output when the input has beenperturbed such that its underlying semantics are un-changed. In computer vision, these perturbationsmight be low-level noise added to the pixels. Butdefining the set of valid perturbations is open prob-lem in NLP, where small changes in surface-formcould dramatically change the underlying mean-ing of an utterance (removing ‘not’, for example).There has, however, been work on constructing per-turbations by replacing words in an utterance withtheir synonyms (Alzantot et al., 2018; Hsieh et al.,2019; Jia et al., 2019) and generating new sentencesvia paraphrases (Ribeiro et al., 2018; Iyyer et al.,2018). In particular, Jia et al. (2019) derive boundson error within this well-defined set of perturba-tions. In the context of evaluation of generatedperturbations, recent works have discussed the ex-tent to which they induce models to make a wrongprediction (Ribeiro et al., 2018; Iyyer et al., 2018;Hsieh et al., 2019; Jia et al., 2019) or change theiroutput (Alzantot et al., 2018). Hsieh et al. (2019)also analyze these perturbations’ effect on attentionweights.

Outside of NLP, Ilyas et al. (2019) make a dis-tinction between useful features (that generalizewell) and those that are robustly-useful (that gen-eralize well, even if an example is adversariallyperturbed). They are able to create a data set withonly robust features. And related to this work byIlyas et al. (2019), there’s been significant recentinterest in training models such that they’re robustto adversarial examples and in building adversarialdatasets that foil such defenses. Highlighting justtwo recent papers, Madry et al. (2017) describetraining that’s robust against adversaries with ac-cess to a model’s gradients, while Athalye et al.(2018) show that many defenses are “obfuscating”their gradients in a way that can be exploited.

4.3 Encoding Structure in NLP ModelsAnother related body of work focuses on under-standing what types of features are extracted byneural language models, in particular looking for

evidence that SOTA models go beyond bag-of-words representations and extract “deeper” featuresabout linguistic structure. Work in this vein has pro-duced evidence that pretrained language modelsencode knowledge of syntax, using a range of tech-niques including supervised “diagnostic classifiers”(Tenney et al., 2019; Conneau et al., 2018; Hewittand Manning, 2019), classification performanceon targeted stimuli (Linzen et al., 2016; Goldberg,2019), attention maps/visualizations (Voita et al.,2019; Serrano and Smith, 2019), and relational sim-ilarity analyses (Chrupała and Alishahi, 2019). Ourwork contributes to this literature by focusing ona toy problem and asking under what conditionswe might expect deeper features to be extracted,focusing in particular on the role that the trainingdistribution plays in encouraging models to learndeeper structure. Related in spirit to our toy dataapproach is recent work which attempts to quan-tify how much data a model should need to learna given deeper feature (Geiger et al., 2019). Stillother related work explores ways for encouragingmodels to learn structure which do not rely on dataaugmentation, e.g. by encoding inductive biasesinto model architectures (Bowman et al., 2015; An-dreas et al., 2016) in order to make “deep” featuresmore readily extractable, or by designing trainingobjectives that incentivize the extraction of specificfeatures (Swayamdipta et al., 2017; Niehues andCho, 2017). Exploring the effects these modelingchanges on the results presented in this paper is anexciting future direction.

4.4 Generalization of Neural Networks

Finally, this work relates to a still larger body ofwork in which aims to understand feature repre-sentation and generalization in neural networks ingeneral. Mangalam and Prabhu (2019) show thatneural networks learn “easy” examples (as definedby their learnability by shallow ML models) beforethey learn “hard” examples. Zhang et al. (2016)and Arpit et al. (2017) show that neural networkswith good generalization performance can nonethe-less easily memorize noise of the same size, sug-gesting that, when structure does exist in the data,models might have some inherent preference tolearn general features even though memorizationis an equally available option. Zhang et al. (2019)train a variety of over-parameterized models on theidentity mapping and show that some fail entirelywhile others learn a generalizable identify func-

9

tion, suggesting that different architectures havedifferent tendencies for learning structure vs. mem-orizing. Finally, there is ongoing theoretical workwhich attempts to characterize the ability of over-parameterized networks to generalize in terms ofcomplexity (Neyshabur et al., 2019) and implicitregularization (Blanc et al., 2019).

5 Conclusion

We propose a framework for simulating the effectsof data augmentation in NLP and use it to explorehow training on counterexamples impacts modelgeneralization. Our results suggest that addingcounterexamples in order to encourage a model to“unlearn” weak features is likely to have the im-mediately desired effect (the model will performbetter on examples that look similar to the gener-ated counterexamples), but the model is unlikely toshift toward relying on stronger features in general.Specifically, in our experiments, the models trainedon counterexamples still fail to correctly classifyexamples which contain only the strong feature.We see also that data augmentation may becomeless effective as the underlying strong features be-come more difficult to extract and as the number ofweak features in the data increases.

ReferencesAishwarya Agrawal, Dhruv Batra, Devi Parikh, and

Aniruddha Kembhavi. 2018. Don’t just assume;look and answer: Overcoming priors for visual ques-tion answering. In IEEE Conference on ComputerVision and Pattern Recognition (CVPR).

Moustafa Alzantot, Yash Sharma, Ahmed Elgohary,Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang.2018. Generating natural language adversarial ex-amples. arXiv preprint arXiv:1804.07998.

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, andDan Klein. 2016. Neural module networks. In Pro-ceedings of the IEEE Conference on Computer Vi-sion and Pattern Recognition, pages 39–48.

Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Bal-las, David Krueger, Emmanuel Bengio, Maxin-der S. Kanwal, Tegan Maharaj, Asja Fischer, AaronCourville, Yoshua Bengio, and Simon Lacoste-Julien. 2017. A closer look at memorization indeep networks. In Proceedings of the 34th Interna-tional Conference on Machine Learning - Volume 70,ICML’17, pages 233–242. JMLR.org.

Anish Athalye, Nicholas Carlini, and David Wagner.2018. Obfuscated gradients give a false sense of se-curity: Circumventing defenses to adversarial exam-ples. arXiv preprint arXiv:1802.00420.

Guy Blanc, Neha Gupta, Gregory Valiant, and PaulValiant. 2019. Implicit regularization for deep neu-ral networks driven by an ornstein-uhlenbeck likeprocess. arXiv preprint arXiv:1904.09080.

Samuel R. Bowman, Christopher Potts, and Christo-pher D. Manning. 2015. Recursive neural networkscan learn logical semantics. In Proceedings of the3rd Workshop on Continuous Vector Space Modelsand their Compositionality, pages 12–21, Beijing,China. Association for Computational Linguistics.

Grzegorz Chrupała and Afra Alishahi. 2019. Corre-lating neural and symbolic representations of lan-guage. In Proceedings of the 57th Annual Meet-ing of the Association for Computational Linguis-tics, pages 2952–2962, Florence, Italy. Associationfor Computational Linguistics.

Alexis Conneau, German Kruszewski, Guillaume Lam-ple, Loıc Barrault, and Marco Baroni. 2018. Whatyou can cram into a single $&!#* vector: Probingsentence embeddings for linguistic properties. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 2126–2136, Melbourne, Aus-tralia. Association for Computational Linguistics.

Ishita Dasgupta, Demi Guo, Andreas Stuhlmuller,Samuel J Gershman, and Noah D Goodman. 2018.Evaluating compositionality in sentence embed-dings. arXiv preprint arXiv:1802.04302.

Ali Elkahky, Kellie Webster, Daniel Andor, and EmilyPitler. 2018. A challenge set and methods for noun-verb ambiguity. In Proceedings of the 2018 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 2562–2572, Brussels, Belgium.Association for Computational Linguistics.

Atticus Geiger, Ignacio Cases, Lauri Karttunen, andChristopher Potts. 2019. Posing fair generalizationtasks for natural language inference. In Proceed-ings of the 2019 Conference on Empirical Methodsin Natural Language Processing and the 9th Inter-national Joint Conference on Natural Language Pro-cessing (EMNLP-IJCNLP), pages 4484–4494, HongKong, China. Association for Computational Lin-guistics.

Max Glockner, Vered Shwartz, and Yoav Goldberg.2018. Breaking NLI Systems with Sentences thatRequire Simple Lexical Inferences. In Proceed-ings of the 56th Annual Meeting of the Associa-tion for Computational Linguistics (Volume 2: ShortPapers), pages 650–655. Association for Computa-tional Linguistics.

Yoav Goldberg. 2019. Assessing bert’s syntactic abili-ties. arXiv preprint arXiv:1901.05287.

Suchin Gururangan, Swabha Swayamdipta, OmerLevy, Roy Schwartz, Samuel R Bowman, andNoah A Smith. 2018. Annotation artifacts innatural language inference data. arXiv preprintarXiv:1803.02324.

10

Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell,and Simone Teufel. 2019. It’s all in the name: Mit-igating gender bias with name-based counterfactualdata substitution. In Proceedings of the 2019 Con-ference on Empirical Methods in Natural LanguageProcessing and the 9th International Joint Confer-ence on Natural Language Processing (EMNLP-IJCNLP), pages 5266–5274, Hong Kong, China. As-sociation for Computational Linguistics.

John Hewitt and Christopher D. Manning. 2019. Astructural probe for finding syntax in word repre-sentations. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4129–4138, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, WeiWei, Wen-Lian Hsu, and Cho-Jui Hsieh. 2019. Onthe robustness of self-attentive models. In Proceed-ings of the 57th Annual Meeting of the Associationfor Computational Linguistics, pages 1520–1529.

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras,Logan Engstrom, Brandon Tran, and AleksanderMadry. 2019. Adversarial examples are not bugs,they are features. In Advances in Neural Informa-tion Processing Systems, pages 125–136.

Mohit Iyyer, John Wieting, Kevin Gimpel, and LukeZettlemoyer. 2018. Adversarial example generationwith syntactically controlled paraphrase networks.arXiv preprint arXiv:1804.06059.

Robin Jia and Percy Liang. 2017. Adversarial exam-ples for evaluating reading comprehension systems.In Proceedings of the 2017 Conference on Empiri-cal Methods in Natural Language Processing, pages2021–2031. Association for Computational Linguis-tics.

Robin Jia, Aditi Raghunathan, Kerem Goksel, andPercy Liang. 2019. Certified robustness toadversarial word substitutions. arXiv preprintarXiv:1909.00986.

Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.2016. Assessing the ability of LSTMs to learnsyntax-sensitive dependencies. Transactions of theAssociation for Computational Linguistics, 4:521–535.

Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019.Inoculation by fine-tuning: A method for analyz-ing challenge datasets. In Proceedings of the 2019Conference of the North American Chapter of theAssociation for Computational Linguistics: HumanLanguage Technologies, Volume 1 (Long and ShortPapers), pages 2171–2179, Minneapolis, Minnesota.Association for Computational Linguistics.

Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Aman-charla, and Anupam Datta. 2018. Gender bias in

neural natural language processing. arXiv preprintarXiv:1807.11714.

Aleksander Madry, Aleksandar Makelov, LudwigSchmidt, Dimitris Tsipras, and Adrian Vladu. 2017.Towards deep learning models resistant to adversar-ial attacks. arXiv preprint arXiv:1706.06083.

Karttikeya Mangalam and Vinay Uday Prabhu. 2019.Do deep neural networks learn shallow learnable ex-amples first?

R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019.Right for the wrong reasons: Diagnosing syntac-tic heuristics in natural language inference. arXivpreprint arXiv:1902.01007.

Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli,Yann LeCun, and Nathan Srebro. 2019. The roleof over-parametrization in generalization of neuralnetworks. In International Conference on LearningRepresentations.

Jan Niehues and Eunah Cho. 2017. Exploiting lin-guistic resources for neural machine translation us-ing multi-task learning. In Proceedings of the Sec-ond Conference on Machine Translation, pages 80–89, Copenhagen, Denmark. Association for Compu-tational Linguistics.

Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Ed-ward Hu, Ellie Pavlick, Aaron Steven White, andBenjamin Van Durme. 2018a. Collecting diversenatural language inference problems for sentencerepresentation evaluation. In Proceedings of the2018 Conference on Empirical Methods in NaturalLanguage Processing, pages 67–81, Brussels, Bel-gium. Association for Computational Linguistics.

Adam Poliak, Jason Naradowsky, Aparajita Haldar,Rachel Rudinger, and Benjamin Van Durme. 2018b.Hypothesis only baselines in natural language infer-ence. arXiv preprint arXiv:1805.01042.

Marco Tulio Ribeiro, Sameer Singh, and CarlosGuestrin. 2018. Semantically equivalent adversar-ial rules for debugging nlp models. In Proceedingsof the 56th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers),pages 856–865.

Rachel Rudinger, Jason Naradowsky, Brian Leonard,and Benjamin Van Durme. 2018. Gender bias incoreference resolution. In Proceedings of the 2018Conference of the North American Chapter of theAssociation for Computational Linguistics: HumanLanguage Technologies, Volume 2 (Short Papers),pages 8–14, New Orleans, Louisiana. Associationfor Computational Linguistics.

Sofia Serrano and Noah A. Smith. 2019. Is attentioninterpretable? In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics, pages 2931–2951, Florence, Italy. Associa-tion for Computational Linguistics.

11

Swabha Swayamdipta, Sam Thomson, Chris Dyer, andNoah A Smith. 2017. Frame-semantic parsing withsoftmax-margin segmental rnns and a syntactic scaf-fold. arXiv preprint arXiv:1706.09528.

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang,Adam Poliak, R Thomas McCoy, Najoung Kim,Benjamin Van Durme, Sam Bowman, Dipanjan Das,and Ellie Pavlick. 2019. What do you learn fromcontext? probing for sentence structure in contextu-alized word representations. In International Con-ference on Learning Representations.

Elena Voita, David Talbot, Fedor Moiseev, Rico Sen-nrich, and Ivan Titov. 2019. Analyzing multi-headself-attention: Specialized heads do the heavy lift-ing, the rest can be pruned. In Proceedings of the57th Annual Meeting of the Association for Com-putational Linguistics, pages 5797–5808, Florence,Italy. Association for Computational Linguistics.

Adina Williams, Nikita Nangia, and Samuel Bowman.2018. A broad-coverage challenge corpus for sen-tence understanding through inference. In Proceed-ings of the 2018 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume1 (Long Papers), pages 1112–1122. Association forComputational Linguistics.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Ben-jamin Recht, and Oriol Vinyals. 2016. Understand-ing deep learning requires rethinking generalization.CoRR, abs/1611.03530.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, andYoram Singer. 2019. Identity crisis: Memorizationand generalization under extreme overparameteriza-tion.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cot-terell, Vicente Ordonez, and Kai-Wei Chang. 2019.Gender bias in contextualized word embeddings.arXiv preprint arXiv:1904.03310.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or-donez, and Kai-Wei Chang. 2018. Gender bias incoreference resolution: Evaluation and debiasingmethods. In Proceedings of the 2018 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 2 (Short Papers), pages 15–20,New Orleans, Louisiana. Association for Computa-tional Linguistics.

Ran Zmigrod, Sebastian J. Mielke, Hanna Wallach,and Ryan Cotterell. 2019. Counterfactual data aug-mentation for mitigating gender stereotypes in lan-guages with rich morphology. In Proceedings of the57th Annual Meeting of the Association for Com-putational Linguistics, pages 1651–1661, Florence,Italy. Association for Computational Linguistics.

12

A Effect of Hyperparameter Settings

We vary several model and task settings, and wefind that the models exhibit similar behavior as inthe default case. The default settings are summa-rized in Table 2. We run all experiments over fiverandom seeds, arbitrarily set to 42, 43, 44, 45, 46.In all plots the error band is the 95% confidenceinterval with 1,000 bootstrap iterations. We do notaverage over all the random seeds for experimentswith dataset size because of the higher computa-tional cost.

We find that batch size (16, 32, 64) and dropout(0.0, 0.01, 0.1, 0.5) within the MLP do not affectmodel performance. We also find that embeddingand hidden sizes (50, 125, 250, 500) do not signif-icantly impact results, except that performance isworse for 500. This is likely because more trainingdata is needed for these higher capacity models.We don’t include figures for these experiments forthe sake of conciseness. We present results belowin Section A.1 for different vocabulary sizes, andwe find that the results don’t change significantly,though the model is unable to learn the strong fea-ture for the smallest vocabulary size.

Hyperparameter Value

Model Settings

optimizer Adamearly stopping patience 5batch size 64number of LSTM layers 1hidden dimensionality 250embedding dimensionality 250initial learning rate 0.001dropout 0

Task Settings

vocabulary size 50Ktraining size 10Ksequence length 5

Table 2: Hyperparameter settings. We set hyper-paramters as default values for our experiments above.Separately, we investigate various values for batch size,dropout, and embedding and vocab sizes and we findthat they do not significantly affect the results, with theexception of models with high embedding and hiddensizes. We find the same for vocabulary size (discussedbelow in A.1).

A.1 Vocabulary Size

We re-run the main experiment – in which we varythe number of adversarial counterexamples – atdifferent vocabulary sizes. The results are shownin Figure 7. We observe similar results when thevocabulary size is 5K as for the default of 50K.The models learn the strong features to some ex-tent as we increase the number of counterexam-ples, and more counterexamples are needed for themodel to generalize well when the strong featureis harder. However, we note that first-lastduplicate is harder than the other strong gea-tures at this vocabulary size. We aren’t sure whythis is the case because we see relative hardnesssimilar to the default case in classification exper-iments with vocabulary size of 5K (the model istrained on identifying whether the strong featureholds as in Figure 2). We also note that in thesame classification experiments with a vocabularysize of 0.5K, the model was not able to learn anyof these strong features, which explains why themodel didn’t learn the strong features at any num-ber of counterexamples.

B Complicating the Strong Feature

In a real-world setting, there often isn’t an under-lying strong feature that exactly predicts the label.We consider two variants of the experimental set-up that weaken this assumption. Specifically, weintroduce label noise, and the case when the strongfeature is a union of multiple other features.

B.1 Random Label Noise

For some noise level ε, we independently flip eachlabel (0 becomes 1, and 1 becomes 0) with prob-ability ε. The results are in Figure 8. The trendsare fairly resistant to noise. The model continuesto switch to learning the strong feature (though ittakes slightly more adversarial counterexampleswith more noise), and we continue to observe therelationship between the features’ hardness andprediction error as a function of the number of ad-versarial counterexamples.

B.2 Multiple Strong Features

We relax the assumption that there’s a single featurethat exactly predicts the label, and we instead con-sider k strong features t1, . . . tk that occur one-at-a-time with equal probability in examples for whichthe strong feature holds (i.e. examples for whichboth the strong and weak features hold and strong-

13

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error strong-only error neither error

prefix duplicate

both error

0.0

0.2

0.4

0.6

0.8

1.0

erro

rfirst-last duplicate

0.0

0.2

0.4

0.6

0.8

1.0

erro

radjacent duplicate

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

5K 25K 50K 100K# examples

5K 25K 50K 100K# examples

contains first

Vocabulary Size0.5 K 5 K 50 K

Figure 7: Vocabulary size. The model isn’t able to learn the strong features at a vocabulary size of 0.5K, but weobserve similar behavior at 5K and 50K.

only adversarial counterexamples). Intuitively, thestrong feature becomes t1 ∨ t2 ∨ . . . tn. Figure 9shows the results. We find that the model, in mostcases, switches from relying on the weak featureto learning to use the collection of strong features.We observe that collections of harder features takemore adversarial counterexamples to achieve lowerror than collections of easy features, which isconsistent with our previous results. Interestingly,in two of the three cases, we find that a collectionof features might take more adversarial counterex-amples to achieve low error than any one of thefeatures in the collection. However, in the thirdcase (the rightmost plot in Figure 9), the error ofcontains first exceeds that of the combinedstrong feature.

C Other Error Metrics

We include additional figures for our main results– hardness (Figure 10), strong-only and weak-onlycounterexamples (Figure 11), training size (Fig-ure 12), multiple weak feature (Figure 13) – thatpresent the error for the other two regions of the testdata (where both or neither of the features hold).With a single exception, we observe that addingcounterexamples doesn’t lead the model to gener-alize worse on data that isn’t adversarial, whichmeans that data augmentation helps with overalltest error, in addition to test error on adversarialexamples. This is expected; if the model switchesfrom using the weak feature to using the strongfeature, its performance shouldn’t change on exam-ples where both or neither feature holds. However,we note that error on these examples increases (andthen decreases) for first-last duplicate

14

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error strong-only error neither error

prefix duplicate

both error

0.0

0.2

0.4

0.6

0.8

1.0

erro

rfirst-last duplicate

0.0

0.2

0.4

0.6

0.8

1.0

erro

radjacent duplicate

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

5K 25K 50K 100K# examples

5K 25K 50K 100K# examples

contains first

Noise ( )0 0.001 0.01 0.05

Figure 8: Noise. We observe that adding noise does not affect the results, though more noise seems to make themodel less prone to learning the strong feature.

and contains first at a training size of 10M(Figure 12).

D Controls for Training Size

D.1 Adding Non-adversarial ExamplesWhen adding adversarial counterexamples, the to-tal number of training examples also increases. Wecontrol for this change by showing here that addi-tional “default” (or non-adversarial) training exam-ples do not help the model on either weak-only orstrong-only error. Figure 14 shows that the addi-tion of these examples does not impact the results.Therefore, it matters that added examples are coun-terexamples; the model doesn’t improve simplybecause there’s more data.

D.2 Fixed Training SizeWe’ve shown above that more training data with-out counterexamples is not sufficient to induce the

model to use the strong feature in classification;see Figure 15. However, one might argue that inthe presence of some counterexamples, more train-ing data is helpful, whether or not it’s adversarial.This would make it hard to disentangle the roleof more training data with that of an increasednumber of counterexamples. Here, we fix trainingsize as we add counterexamples (meaning thereare fewer non-adversarial examples) and we ob-serve similar results as in our main experimentsabove (Figure 3). Naturally, this is not the casefor extreme numbers of counterexamples: if weremove all non-adversarial examples, the model isnegatively impacted. But these results – taken to-gether with those above – indicate that the benefitsof adding counterexamples in general (§3.1) andincreasing the number of counterexamples (theseresults) should not be attributed to the larger train-ing size.

15

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error

5K 25K 50K 100K# examples

strong-only error

Strong Featureprefix duplicatefirst-last duplicateadjacent duplicate

contains first[all]

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error

5K 25K 50K 100K# examples

strong-only error

Strong Featureadjacent duplicatecontains first

[adjacent + contains first]

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error

5K 25K 50K 100K# examples

strong-only error

Strong Featureprefix duplicatecontains first

[prefix + contains first]

Figure 9: Multiple strong features. We find similar trends as for a single strong feature. Collections of harderfeatures require more counterexamples to achieve low generalization error than collections of easier features. Wealso find that collections of features might take more counterexamples to achieve low generalization than anyindividual feature in the collection.

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

neither error

5K 25K 50K 100K# examples

both error

Strong Featurecontains 1prefix duplicatefirst-last duplicate

adjacent duplicatecontains first

Figure 10: Hardness. Other error cases (neither leftand both right) for Figure 3.

0.0

0.2

0.4

0.6

0.8

1.0er

ror

neither error

weak-only exam

ples

both error

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

strong-only examples

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

both examples

Strong Featurecontains 1prefix duplicatefirst-last duplicate

adjacent duplicatecontains first

Figure 11: Counterexamples of varying types. Othererror cases (neither left and both right) for Figure 4.

16

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

neither error

prefix duplicate

both error

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

first-last duplicate

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

adjacent duplicate

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

contains first

Training Size (Millions)0.1 M 1 M 10 M

Figure 12: Changing training size. Other error cases(neither left and both right) for Figure 5.

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

neither error

prefix duplicate

both error

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

first-last duplicate

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

adjacent duplicate

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

5K 25K 50K 100K# examples

contains first

# distractorsone two three idealized

Figure 13: Multiple weak features. Other error cases(neither left and both right) for Figure 6.

17

5K 25K 50K 100KAdditional Examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error

5K 25K 50K 100KAdditional Examples

strong-only error

Strong Featureprefix duplicatefirst-last duplicate

adjacent duplicatecontains first

Figure 14: When adding adversarial counterexamples,the total number of training examples also increases.We control for this change by showing here that ad-ditional “default” training examples do not help themodel on either weak-only or strong-only error. Thisis unsurprising given our previously shown results.

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

weak-only error strong-only error

5K 25K 50K 100K# examples

0.0

0.2

0.4

0.6

0.8

1.0

erro

r

neither error

5K 25K 50K 100K# examples

both error

Strong Featurecontains 1prefix duplicatefirst-last duplicate

adjacent duplicatecontains first

Figure 15: Static training size; counterexamples re-place training examples.

18