ONLINE SUPPLEMENT

How do you deal with uncertainty?Cochlear Implant users differ in the dynamics of lexical processing of non-canonical inputs.

Bob McMurrayDept. of Psychological and Brain Sciences

Dept. of Communication Sciences and DisordersDept. of Otolaryngology

University of Iowa

Tyler P. EllisDept. of Communication Sciences and Disorders

University of Iowa

and

Keith S. ApfelbaumDept. of Psychological and Brain Sciences

University of Iowa, andFoundations in Learning, Inc.

Corresponding AuthorBob McMurrayW314 SSHDept. of Psychological and Brain SciencesUniversity of IowaIowa City, IA 52242319-335-2408 (voice)Bob-mcmurray@uiowa.edu

S1. Description of CI users in the experiment. For patients with no age of onset of deafness listed, onset was gradual starting in adulthood. PTA refers to the pure tone average without the CI (hearing aid only), either for low frequencies (≤1000) or all. For bimodal users, an audiogram was computed on the ear contralateral to the CI; for hybrid users, audiograms included whichever ear(s) used a hearing aid. Bilateral and unilateral users were all profoundly deaf as assessed in a pre-implantation audiogram.

Configuration ID Gender Age (yrs) Etiology Onset of

deafnessAge at

implant.Length of device use Ear Manufacturer Implant model PTA

(≤1K)PTA (all)

unilateral 3 F 55 hereditary 29 41 14.25 L Cochlear Nucleus CI 24M -- --unilateral 7 F 57 unknown 35 43 15.01 L Cochlear Nucleus CI 24M -- --unilateral 9 F 57 hereditary 34 44 14.10 L Cochlear Nucleus CI 24M -- --unilateral 11 F 52 unknown 46 51 1.25 L AB Clarion HiRes 90K -- --unilateral 12 M 49 unknown 36 38 11.09 R AB Clarion HiFocus II-CII -- --unilateral 15 F 44 unknown 29 33 12.11 L AB Clarion HiFocus II-CII -- --unilateral 19 F 63 unknown 17 43 20.09 R AB Clarion Radial -- --unilateral 22 F 59 hereditary/infection 17 32 27.36 L AB Clarion HiFocus II-CII -- --unilateral 56 F 58 unknown 40 45 13.05 R AB Clarion HiFocus 1.2/I-CII -- --unilateral 63 M 64 hereditary 33 38 26.07 L AB Clarion HiRes 90K -- --bilateral 2 M 58 unknown 42 42 16.36 B Cochlear R- Nucleus CI 24M;

L- Nucleus CI512(CA)-- --

bilateral 4 F 63 hereditary 55 57 6.12 B Cochlear Nucleus CI 24RE (CA) -- --bilateral 10 F 53 unknown 48 49 4.01 B Cochlear R- Nucleus CI 24 RE

L- Nucleus CI 512-- --

bilateral 13 F 54 unknown 25 50 4.15 B Cochlear R- Nucleus CI 24 RE;L- Nucleus CI 512

-- --

bilateral 16 F 60 Toxic Shock Syndrome 37 39 21.00 B AB R- Bipolar/Standard 1.0;

L- Clarion HiRes 90k-- --

bilateral 17 M 41 Meningitis 18 32 9.05 B AB Clarion HiRes 90k -- --bilateral 18 F 58 Otosclerosis 32 33 25.08 B AB R- Clarion HiRes 90k;

L- Ineraid-- --

bilateral 21 F 37 hereditary 26 26 12.10 B AB Clarion HiFocus II-CII -- --bilateral 27 F 53 unknown 40 43 10.22 B AB R- Clarion HiRes 90K;

L- Clarion Hi Focus II-CII-- --

bilateral 29 F 52 unknown 27 32 21.26 B AB R- Clarion 1.0;L- Clarion HiRes 90K

-- --

bilateral 34 M 64 unknown 44 60 4.02 B Cochlear Nucleus CI 512 (CA) -- --bilateral 53 F 52 unknown 31 40 11.96 B AB Clarion HiFocus II-CII -- --bilateral 61 F 54 hereditary 42 43 11.48 B AB Clarion HiFocus II-CII -- --bimodal 5 M 60 unknown 45 47 12.99 R Cochlear Nucleus CI 24M 35 44bimodal 28 M 62 Meniere's Disease 60 62 1.06 L Cochlear Nucleus CI 422 20 22.5bimodal 50 M 60 infection 32 48 12.27 L AB Clarion HiFocus II-CII 53.3 71.7bimodal 54 M 51 Type II

Neurofibromatosis 40 44 7.35 L Cochlear Nucleus CI 24RE (CA) 38.3 72.5

bimodal 118 M 21 unknown 8 18 2.98 R AB Clarion HiRes 90K 16.7 29.2bimodal 132 F 62 unknown 20 57 4.99 R Cochlear Nucleus CI 512 (CA) 23.3 57.5bimodal 148 M 56 unknown 47 51 5.05 R AB Clarion HiRes 90K 36.7 47.5hybrid 23 F 48 unknown 32 45 3.55 R Cochlear Nucleus Hybrid S12 28.3 56.7hybrid 39 F 47 unknown 19 46 1.09 L Cochlear Nucleus Hybrid L24 33.3 52

+ For this subject hearing aid only audiograms were not conducted. We report unaided PTA in the contra-lateral ear (in the ipsi-lateral ear, PTAs were 86.7 and 97.6, respectively).

Configuration ID Gender Age (yrs) Etiology Onset of

deafnessAge at

implant.Length of device use Ear Manufacturer Implant model PTA

<1KPTA all

freqhybrid 24 M 66 unknown 48 64 2.03 R Cochlear Nucleus Hybrid L24 36.7 60.8hybrid 25 M 27 unknown 17 23 4.16 R Cochlear Nucleus Hybrid L24 46.7 63.3hybrid 44 F 53 unknown 24 52 1.07 R Cochlear Nucleus Hybrid L24 58.3 71.7hybrid 52 M 59 unknown 48 54 5.00 R Cochlear Nucleus Hybrid S12 30 68.3hybrid 66 F 44 unknown 25 41 3.38 L Cochlear Nucleus Hybrid S12 40 73.3hybrid 128 F 37 unknown - 37 1.06 R Cochlear Nucleus Hybrid L24 43.3+ 54+

hybrid 135 M 50 unknown 42 50 0.92 L Cochlear Nucleus Hybrid L24 25 55hybrid 137 M 60 hereditary 15 48 11.99 R Cochlear Nucleus EAS2 36.7 60.8hybrid 138 F 62 Cogan’s Syndrome 45 59 3.30 R Cochlear Nucleus Hybrid L24 50 67.5hybrid 139 F 46 unknown 40 41 5.05 R Cochlear Nucleus Hybrid L24 33.3 59.2hybrid 141 F 64 unknown 55 56 9.04 R Cochlear Nucleus EAS3 - Hybrid S8 23.3 54.2hybrid 146 F 58 unknown 28 55 3.97 L Cochlear Nucleus Hybrid L24 21.7 53.3

S2. Words Used in the Experiment

Table S2A: Sets of words for visual world trials

beach foot shake gasboat check sash maidboot cup teeth vasecab bone face deckcake dome path knifedice keg mop shapedog fawn pipe jetfish book top kitefog nun peach tape

judge bike cat pig

Table S2B: Mispronounced forms of each word.

Correctly pronounced

Onset mispronunciation Offset mispronunciation

Single-feature Multi-feature Single-feature Multi-featurebeach deach heach beaj beagbike gike fike bipe bimeboat poat foat boak bosebone goen vone bome bopebook pook yook boog buhfboot poot joot buk boojcab pab rab kag cazcake gake nake cague cazecat gat wat cad cass

check jeck veck chep cheffcup gup nup cuk cuv

deck geck meck deg dechdice tice sice dishe ditedog gog shog dob dov

dome gome thome doene docheface sace tace fafe fagefawn vawn chawn fom fothfish sish nish fiss fidfog thog chog fod fossjet chet thet jek jech

judge chudge tudge jutch jufkeg geg Cheg kek kethkite pite thite kide kime

knife mife tife nive nipemaid naid daid mayb maychmop nop djop mot monnun mun vun nung nuthpath tath nath pathe pap

peach keach meach peadj peathgas kas zass gaz gajpig tig sig pid piz

pipe kipe chipe pibe picefoot voot noot fuhd foojsash fash tash saf sab

shake thake yake shague shageshape thape pape shabe shayztape dape mape tabe taveteeth deeth cheath teef teejtop dop zop tob todje

vase zase nase vafe vake

S3. Model Selection for Mixed Model AnalysesFor mixed effects models, random slope structure was selected by a process of model comparison (Matuschek, Kliegl, Vasishth, Baayen, & Bates, 2017). Here, the goal was to select the most complex set of random slopes (most conservative) that was supported by the data. We thus held fixed effects constant and compared nested models using the 2 test of model comparison, stopping when the comparison of adjacent models was no longer significant. Here, we report the statistics supporting this decision for each of the analyses reported in the main text. Model structures are reported in lmer notation.

S3A. Analysis of AccuracyThis used a binomial model with the following fixed effects

Correct ~ Location * (Mispron + Degree) * (ATHvCI + EvAE) Mispronunciation was coded as two contrast variables. Mispron: Mispron (.33) vs.

Correct (-.66); and Degree: Correct (0), Single Feature(-.5), Multi Feature (+.5) Location of mispronunciation (Location): onset (.5) or offset (-.5) Listener type: two contrast variables: ATH (.5) vs. CI (-5); CIE (-.5) vs. CIA+E (.5,

ATH=0To facilitate convergence, covariance terms between random slopes were set to 0. They are not notated as such in the following table for ease of explication.

Random Slopes df (model) AIC 2 df (2) p(1 | subject) 19 5117.7(1 | subject) + (1 | item) 20 4836.5 283.2 1 <.0001(Mispron + Degree + Location | subject) + (1 | item)

23 4838.0 4.5 3 0.21

(Mispron * Location + Degree * Location | subject) + (1 | item)

25 4839.2 2.8 2 0.25

(Mispron * Location + Degree * Location | subject) + (Mispron + Degree Location | item)

28 4650.5 194.7 3 <.0001

(Mispron * Location + Degree * Location | subject) + (Mispron * Location + Degree * Location | item)

30 4614.7 39.8 2 <.0001

While random slopes of the main effects on subject did not appear necessary (row 3, 4), follow-up analyses suggested that the effect of location was individually useful so both of these these were retained (as retaining slopes is more conservative; Barr, Levy, Scheepers, & Tily, 2013). This resulted in the maximal model.

Correct ~ (CIvNH + CIeVae) * (degCorr + degMPdiff) * OnsetN + (1 | subjectID) + (0 + degCorr | subjectID) + (0 + degMPdiff | subjectID) + (0 + OnsetN | subjectID) + (0 + degCorr:OnsetN | subjectID) + (0 + degMPdiff:OnsetN | subjectID) + (1 | baseword1) + (0 + degCorr | baseword1) + (0 + degMPdiff | baseword1) + (0 + OnsetN | baseword1) + (0 + degCorr:OnsetN | baseword1) + (0 + degMPdiff:OnsetN | baseword1) (1)

S3B. Documenting the mispronunciation effect in ATH listeners.Our analysis of the curvefit parameters began with two models examining the ATH listeners only (one for the Maximum, one for Timing). Each had fixed effects of location of mispronunciation (onset = .5, offset = .5), presence of a mispronunciation (-.66 for correct; +.33 for either incorrect form), and Degree (correct: 0, single feature: -.5, multi-feature: .5)

Property ~ Location * (MisPron + Degree) Both were within-subject, so both factors were potential random slopes. We could not include random slopes for item as it was necessary to average data across items to obtain the necessary curvefits. Separate model comparisons were run for models examining each property. Because the fully saturated model was rank deficient (there was only one data point per cell in the design), we dropped covariance terms.

Table S3B.1: Maximum (note random effects here are shown in R notation, assuming a full covariance matrix for brevity, rather than the expanded syntax for no covariance).Random Slopes df (model) AIC 2 df (2) p1 | subject 8 -399.5Mispron + Degree | subject 10 -396.4 0.8 2 .67Mispron + Degree + Location | Subject 11 -397.5 3.2 1 0.075(Mispron + Degree)*Location | Subject 13 -394.7 1.2 2 .56

Thus, the final model used intercepts only.

Max ~ (Mispron + Degree) * Location+ (1 | Subject) (2)

For Timing, a similar procedure was used with similar results

Table S3B.2: Timing (note random effects here are shown in R notation, assuming a full covariance matrix for brevity, rather than the expanded syntax for no covariance).Random Slopes df (model) AIC 2 df (2) p1 | subject 8 194.6Mispron + Degree | subject 10 198.6 0.0 2 1Mispron + Degree +Location | Subject 11 198.7 2.0 1 0.16(Mispron + Degree)*Location | Subject 13 197.9 4.8 2 0.09

Again, random intercepts only were used.

Timing ~ (Mispron + Degree) * Location+ (1 | Subject) (3)

S3C. Omnibus AnalysisThe primary, omnibus, analysis examined Max and Timing as a function of Mispronunciation, Degree (Single- vs. Multi-feature), and CI user (CI vs. ATH and CIE vs. CIAE). Thus, fixed effects were

Property ~ (Mispron + Degree) * Location * (ATHvCI + EvAE)

Potential random effects included Mispronunciation, Degree and Location (the other terms were between subjects, and the data had to be averaged across items to obtain the fits). As before model comparison procedures were followed to determine the optimal random effects structure, and covariance terms were assumed to be 0.

Table S3C.1: Max.Random Slopes df (model) AIC 2 df (2) p1 | subject 20 -825.9Mispron + Degree | subject 22 -828.5 6.6 2 0.037Mispron + Degree +Location | Subject 23 -834.1 7.6 1 0.0059(Mispron + Degree)*Location | Subject 25 -834.4 4.4 2 0.11

Thus, we used random slopes of both main effects, but no interactions.

Max ~ (Mispron + Degree) * Location * (ATHvCI + EvAE) + (1 | Subject)+ (0 + Mispron | Subject) + (0 + Degree | Subject) + (0 + Location | Subject) (4)

For slope, results were similar

Table S3C.2: Slope.Random Slopes df (model) AIC 2 df (2) P1 | subject 20 668.1Mispron + Degree | subject 22 671.7 0.4 2 0.83Mispron + Degree +Location | Subject 23 660.9 12.8 1 0.00034(Mispron + Degree)*Location | Subject 25 659.4 5.5 2 0.065

Here, a random slope of Location improved model fit over and above Mispronunciation and Degree, however, neither of these latter two terms improved fit over intercepts alone. Thus, we conducted additional model comparisons examining the effect of location alone

Table S3C.3: Slope. Models tested with an alternative order for the main effects.Random Slopes df (model) AIC 2 df (2) P1 | subject 20 668.1Mispron + Degree | subject 21 659.8 10.3 1 0.0013Mispron + Degree +Location | Subject 23 660.9 2.9 2 0.24(Mispron + Degree)*Location | Subject 25 659.4 5.5 2 0.065

This suggested that a random slope of location along was sufficient.

Timing ~ (Mispron + Degree) * Location * (ATHvCI + EvAE) + (1 | Subject) + (0 + Location | Subject) (5)

S4. Additional VisualizationsFor efficiency, in the main text, visualizations are only provided for correctly pronounced and single-feature mispronounced forms. This is done in part because multi-feature mispronunciations showed a similar (but more extreme) pattern of what was observed with single-feature mispronunciations. These are shown here to ease interpreting the interactions.

Effect of Location and Degree in ATH listeners (Complements Figure 3A).Figure S1 shows just the ATH listeners for both onset (A) and offset (B) mispronounced forms. This complements Figure 3A in the main text. The pattern in the multi-feature mispronunciations is not qualitatively different from single feature mispronunciations; rather multi-feature mispronunciations reveal a more extreme version of what was observed for single feature single-feature. In fact, the pattern appears roughly additive: the size of the difference between correctly pronounced and single feature mispronunciations is similar to the difference between single and multi-feature mispronunciations.

Effect of Location and Degree in CI users (Complements Figure 3B, C)A similar pattern is seen in Figure S2, which shows the same data for the CI users. Given the lack of statistical differences between CIE and CIAE users, we collapsed these two groups.

While overall, CI users’ target fixations rise more slowly and reach a lower amplitude, the magnitude of the difference between a single feature mismatch and the correctly pronounced form is roughly the same as between single and multi-feature mismatched stimuli.

Ultimately, the effect of mispronunciation appears a more or less linear function of the number of features that mismatch (at least within a location), and the multi-feature mismatching stimuli do not lead to a qualitatively different pattern of lexical activation dynamics.

0 500 1000 1500 2000 2500Time (msec)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Fixa

t ions

A.Onset MP

CorrectSingle Feature MPMulti Feature MP

0 25000

1B. Offset MP

Figure S1: Fixations to the target in ATH listeners as a function of location and degree of mispronunciation. A) For onset mispronounced (MP) forms, the primary effect was to delay and slow the rise of fixations. B) For offset mispronounced forms, the primary effect was to reduce the ultimate asymptote.

0 500 1000 1500 2000Time (msec)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Fixa

tions

A. Onset MP

CorrectSingle Feature MPMulti Feature MP

0 25000

1B. Offset MP

Figure S2: Fixations to the target in CI listeners as a function of location and degree of mispronunciation. A) For onset mispronounced (MP) forms, the primary effect was to delay and slow the rise of fixations. B) For offset mispronounced forms, the primary effect was to reduce the ultimate asymptote.

S5. Effect of Mispronunciation within CI users

In the main text, we presented an analysis of the effect of mispronunciation within just the ATH listeners. It showed a large effect on the Max fixations for offset mispronunciations, but no effect on the Timing; whereas onset Mispronunciations created a pronounced delay and smaller effects on Max. This is in accord with the timing of the mispronunciation in the stimuli – for offset mispronunciations, the beginning of a correctly and incorrectly pronounced word is identical so there was no effect on the Timing, even as the ultimate confidence (Max) is impaired by a mismatching phoneme. Conversely, for onset mispronunciations, the mismatching phoneme is at onset, slowing the timing.

We also conducted a parallel analysis with CI users to document that CI users showed similar patterns. As before, we examined the Max and Timing properties in separate models including mispronunciation location (Onset/Offset) and degree (Correct/Incorrect, and Single/Multi-feature) coded as in the main text. The model for Max included random slopes of Mispronunciation, Degree and Location (but no interactions). The model for Timing included a random slope of Location.

Results are presented in Table S5. The first column indicates the coefficient from the ATH listeners for comparisons (all effects were significant in the ATH listeners and can be found in Table 3 in the main text).

As in the main text, all effects and interactions were significant. Importantly, all of the coefficients were in a similar direction and magnitude in the CI users as in the ATH listeners. As in the main text, follow-up analyses split the data by location of mispronunciation. For Max, there were significant effects for both onset (Mispron: B=-.13, p<.0001; Degree: B=-.10, p<.0001) and offset (Mispron: B=-.24, p<.0001; Degree: B=-.25, p<.0001) mispronunciations, though as in the ATH analyses, effects were substantially larger for offset. For Timing, there

Table S5: Results of three mixed models examining properites of the target fixations as a function of mispronunciation location and degree for CI users (B(ATH) refers to the corresponding B from ATH analysis reported in Table 3 in the main text). * = p<.05.

DV Effect B (ATH) B SE df t p

Max

Location 0.07 0.06 0.011 43.9 5.6 <.0001 *Mispronunciation -0.14 -0.19 0.010 42.5 -17.8 <.0001 *Single vs. Multi -0.13 -0.18 0.017 41.4 -10.4 <.0001 *Location Mispron 0.12 0.10 0.018 78.8 5.8 <.0001 *Location Degree 0.16 0.15 0.021 81.8 6.9 <.0001 *

Tim

ing

Location -1.04 -0.79 0.079 43.9 -9.9 <.0001 *Mispronunciation -0.91 -0.81 0.062 167.7 -13.0 <.0001 *Single vs. Multi -0.35 -0.34 0.073 169.3 -4.7 <.0001 *Location Mispron -1.66 -1.12 0.125 167.7 -9.0 <.0001 *Location Degree -1.04 -0.66 0.146 169.3 -4.5 <.0001 *

was a large effect for onset mispronunciations (Mispron: B=-1.36, p<.0001; Degree: B=-.66, p<.0001), and a smaller effect for offset (Mispron: B=-.25, p=.0044; Degree: B-.016, p=.87).

The overall similarity in the coefficients suggests that, consistent with Figure 3B and 3C in the main text, at the coarsest level, CI users show similar sensitivity to mispronunciations as do ATH listeners. However, the fact that some properties differed in magnitude (in particular, the interactions in the Timing analysis) raises the possibility of interactions that were explored in the omnibus analysis in the main text.

S6. Comparison of CIE and CIAE users in MP Disruption

A BDOTs (Oleson, Cavanaugh, McMurray, & Brown, 2015) analysis of the MP Disruption measure was conducted to compare CIE and CIAE users on these measures. As before, we considered each of the four types of mispronounced forms in separate analyses. Table S2, and Figure S3 show the results of the analysis. For simplicity, we do not report separate analysis with liberal or conservative inclusion of curve fits. All analyses here use all of the fits (unless one could not be obtained). Results did not differ with a more liberal approach.

As Table S2 shows, none of these comparisons reached statistical significance. Moreover, Figure S3, which shows the raw data along with BDOTs computed confidence intervals suggest this is not due a numerical difference that simply did not reach significance due to the reduced sample size. Rather, there does not appear to be any difference between CIE and CIAE users in how they cope with non-canonical inputs.

Table S2: Results of BDOTs analysis comparing CIE and CIAE listeners within each mispronunciation condition.Panel Comparison Fit quality Subjects Statistics Significant Regions

7A

Onset / single feature

ATH vs. CI

AR1 R2>=0.95 72R2>=0.8 1R2<0.8 0

Non R2>=0.95 8R2>=0.8 1R2<0.8 0

Dropped 3

CIE: 23CIAE: 18

ɑ* = 0.0017 = 0.9984

7B

Onset / multi feature

ATH vs. CI

AR1 R2>=0.95 49R2>=0.8 6R2<0.8 0

Non R2>=0.95 9R2>=0.8 2R2<0.8 0

Dropped 12

CIE: 16CIAE: 16

ɑ* = 0.0018 = 0.9972

7C

Offset / single feature

ATH vs. CI

AR1 R2>=0.95 68R2>=0.8 4R2<0.8 0

Non R2>=0.95 13R2>=0.8 1R2<0.8 0

Dropped 1

CIE: 22CIAE: 21

ɑ* = 0.0011 = 0.9945

7D

Offset / Multi feature

ATH vs. CI

AR1 R2>=0.95 37R2>=0.8 18R2<0.8 4

Non R2>=0.95 4R2>=0.8 10R2<0.8 1

Dropped 8

CIE: 19CIAE: 17

ɑ* = 0.0016 = 0.9973

Figure S3: Comparison of MP disruption effect (difference in target fixations between correct and incorrect forms) between CIE and CIAE users. No regions were significant (see Table 6 for corresponding statistics). A high MP disruption indicates more difficulty created by non-canonical form. A) Single feature onset mispronunciations; B) Multi-feature onset mispronunciation; C) Single feature offset mispronunciation; D) Multi-feature offset mispronunciation.

References

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255-278.

Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94, 305–315.

Oleson, J. J., Cavanaugh, J. E., McMurray, B., & Brown, G. (2015). Detecting time-specific differences between temporal nonlinear curves: Analyzing data from the visual world paradigm. Statistical Methods in Medical Research. doi:10.1177/0962280215607411

0 500 1000 1500 2000 25000

0.1

0.2

0.3

0.4

Time (msec)

MP

Dis

rupt

ion

0 25000

0.1

0.2

0.3

0.4

0.5

0.6

0 25000

0.05

0.1

0.15

0.2

0 25000

0.1

0.2

0.3

0.4

0.5

A. Single Feature Onset B. Multi Feature Onset

C. Single Feature Offset D. Multi Feature Offset

CIAECIE