Intonationally marked contrast in instructed visual search: Intersective

color vs. subsective scalar adjectives

Shari R. Speer

Ohio State University

Acknowledgements

NSF BCS-018464

NSF BCS-0617609NIDCD R01-

DC007090

Ping Bai

Allison Blodgett

Laurie Maynell

Mary Beckman

Co-I, Kiwako Ito

Contrast Similarities and differences comparisons

Bolinger (1961) “… cases where one or more individual items are

singled out from a larger (but limited) set as being true as regards some relationship whereas others in the same set are untrue…”

Zeevat (2004)Contrastors (alternatives)… “must be obtainable from the actual utterance by substituting something else for the intonationally prominent constituent.”

Contrast

Discourse coherence

prerequisite for anaphora resolution Must be in focus domainOften accompanied by structural

parallelism

Matter of degree?

Accentuation for expressing contrast

B (vs. A) accent (Bolinger, 1961; Jackendoff, 1972)

L+H* (vs. H*) (Pierrehumbert, 1980; ToBI)

Pierrehumbert & Hirschberg (1990)

“the accented item - and not some alternative related item - should be mutually believed (p 296.)”

L+H*: +AGREED, theme accent (Steedman, 2003)

Intonationally marked contrast in sentence/discourse processing

Faster comprehension of short contrastive discourse when prominent accents were appropriately located (vs. inappropriate) (Bock & Mazella, 1983)

Faster comprehension and higher acceptance of appropriately accented Q-A pairs (English pitch accent: Birch & Clifton, 1995; Japanese, Basque pitch range expansion/peak alignment: Ito, 2002).

Faster phoneme monitoring when the contrastive entity was negated with prominent accent (vs. no accent). Intonationally marked narrow focus in negation generates contrast set (Davidson, 2001).

Accentuation shows immediate effects on instructed visual search(Dahan, Tanenhaus & Chambers, 2002)

Prominent accent interpreted non-anaphorically, lack of prominent accent interpreted anaphorically.

1. “Put the candle/candy below the triangle.”

2. “NOW, put the CANDLE above the square.” “NOW, put the candle ABOVE THE SQUARE.”

fixation to fixation to

candycandyfixation to fixation to candlecandle

candle candle “ “CANDLECANDLE” ” candle candle “ “candycandy” ”

L+H* on a modifier evokes a contrast set(Ito & Speer, 2008)

Manipulated pitch accent type on a modifier adjective or modified noun

L+H*: Evokes a contrast set that contains alternative related item(s) (Pierrehumbert & Hirschberg, 1990)

GREENGREEN ballballGREENGREEN ballballblue ballblue ball

Holiday Tree task

(Ito & Speer, 2008)

Eyemovements monitored as naïve participants followed recorded audio instruction to decorate Christmas trees.

(ASL e5000) at 60Hz.

Visual Stimuli (Ito & Speer, 2008)

44 ornaments displayed by type on a grid with 11 cells (8 target + 3 filler)

Sound Stimuli (Ito & Speer, 2008)

Accentual patterns of stimuli were selected based on the analysis of spontaneous speech in a similar task (Ito, Speer & Beckman, 2003).

A trained female phonetician produced stimuli with the intended pitch accent patterns. Adjective-noun pairs were presented in discourse context.

Recorded 44.1KHz, 16bits All stimuli were ToBI-annotated by two native speakers

of American English. Items on which they did not agree were re-recorded.

Experiment 1: Stimuli

Felicitous L+H* Patternsgreen onion ORANGE onion

L+H* no accent brown ball brown ANGEL

H* L+H* Infelicitous L+H* Patterns

gray stocking brown STOCKING H* L+H*

orange candy ORANGE onion L+H* no accent

Experiment 1 Results: Felicitous vs. Infelicitous L+H*

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

-600 -300 0 300 600 900 1200 1500

Time from noun onset (ms)

Fixation proportion to the target object cell

Felicitous Adj Contrast BLUE drum

Infelicitous Adj Contrast blue DRUM

Felicitous Noun Contrast blue DRUM

Infelicitous Noun Contrast BLUE drum

blue

BLUE

DRUM

drum

(Ito & Speer, 2008)(Ito & Speer, 2008)

Experiment 1: Summary

Felicitous L+H* on the color term in contrastive environments (green drum, BLUE drum) facilitated visual search compared to infelicitous L+H* on non-contrastive noun (green drum, blue DRUM).

Felicitous L+H* on the object noun in contrastive environments (blue egg, blue DRUM) showed a smaller advantage over infelicitous L+H* on the adjective (blue egg, BLUE drum).

Listeners ‘tune’ their sensitivity to contrastive accent on the basis of the visual task?

Experiment 2: stimuli

Does L+H* on the contrastive adjective lead to anticipatory eye-movement compared to H*? e.g. brown drum --> RED/red drum

Does infelicitous Does infelicitous L+H*L+H* on the color adjective lead on the color adjective lead to an expectation strong enough to lead a to an expectation strong enough to lead a “garden-path” in eye-movement?“garden-path” in eye-movement?

e.g. red onion --> e.g. red onion --> GREENGREEN drum drum

Experiment 2 Results: Felicitous/Infelicitous L+H* vs. H*

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

-600 -300 0 300 600 900 1200 1500

Time from noun onset (ms)

Fixation proportion to the target object cell

Felicitous L+H* Contrast brown drum...GREEN drum

H*!H* Adj Contrast brown drum...green drum

H*!H* Both Already red onion...green drum

Infelicitous L+H* red onion...GREEN drum

GREEN drum

green drum

Experiment 2 results: “Garden-path” due to L+H*

Experiment 2 results: Refixation of previous cell w/out L+H*?

The effect of infelicitous L+H*

QuickTime™ and aDV/DVCPRO - NTSC decompressor

are needed to see this picture.

Ito & Speer 2008: Summary

For cases with repetition of the immediately preceding noun, L+H* showed a processing advantage as compared to H* on the adjective (brown drum --> RED/red drum)

For cases without repetition of the noun, infelicitous L+H* led to incorrect anticipatory fixations to the most-recently mentioned ornament type.

significant delay in fixations to the real target significant delay in fixations to the real target

NO incorrect anticipatory fixationsNO incorrect anticipatory fixations were observed when were observed when non-repeated non-repeated targets were felicitously presented with [H* !H*] targets were felicitously presented with [H* !H*]

Discussion Dahan et al results showed that L+H* accent led to

the expectation of contrast. Contrastive accent on a (potentially) repeated word led listeners to fixate the object type that had *not* just been mentioned.

Results here show contrast effects from L+H* accent on the modifier that preceded the noun. L+H* led listeners to restrict the set of expected referents to those of the object type that *had* been most recently mentioned.

Discussion

Evidence suggests that L+H* evokes generation of a contrast set in the listener’s discourse representation. The set is based on the accented word interpreted within the structure of the utterance. Here, contrastive accent on the adjective restricted expectations about the coming head noun to the set of objects specified by the immediately preceding noun.

Facilitative anticipatory fixations and prosodic ‘garden-path’ effects due to prominence and have also been shown for German (Weber et al, 2006) and Japanese (Ito et al, under review).

Remaining questions

In Ito&Speer 2008, rapid integration of contrastive L+H* involved refixation of the immediately preceding ornament cell.

Were facilitation effects of L+H* dependent on the visual layout, where contrastive trials required re-fixation of a ‘just-visited’ ornament cell? Was predictive use of L+H* in the ‘garden-pathing’ trials dependent on the organization of the ornament array by objects (nouns)?

Are primary color adjectives, which describe a context- independent distinction between modified objects, more likely to require intonational contrast than other types of adjectives (e.g. scalars)?

Followup studies:

We used adjective-sorted arrays, where contrastive sequences (e.g., red ball green ball, small ball medium ball) require fixations to a new cell (instead of returning fixations).

We compared 2 adjective types: Color (context-independent, intersective) Size (scalar,subsective) Because relative size terms are inherently contrastive and context-dependent, prominent pitch accent may be redundant as a cue to contrast.

Exp3 Visual stimuli - Colors

Exp4 Visual stimuli - Sizes

Sound stimuli

Articles rather than determiners, e.g. ‘Hang a green ball,’ ‘Hang a large ball,’ as there were multiple like ornaments.

Same native English ToBI-trained speaker Same procedures for annotation, presentation

and eyemovement monitoring Four conditions:

Contrastive vs. non-contrastive

H* !H* vs. L+H* no accent

Anticipatory effect w/out ‘refixation’

Exp3, color adjectives:

“Hang a yellow/YELLOW star”

Exp4, scalar adjectives:

“Hang a large/LARGE tree”

Garden-path effects

Color:

fixations to green tree hearing ‘GREEN ball’

Scalar:

fixations to large tree hearing “LARGE ball”

QuickTime™ and aH.264 decompressor

are needed to see this picture.

Summary: Fixation proportions L+H* facilitates eyemovements to the targets for

both color and size adjectives in contrastive sequences.

Visually more complex size-sorted boards led to slower eye movements to the target than color-sorted boards.

L+H* led to more frequent fixations to the incorrect targets for both color and size adjectives in non-contrastive sequences.

Non-prominent size adjective (with H*) did not lead to looks to contrastive cells, i.e., size adjectives are not automatically interpreted contrastively.

Prominent pitch accent (L+H*) on a pre-nominal modifier evokes a contrast set and facilitates reference resolution with both intersective color adjectives and subsective scalar adjectives.

L+H* evokes the anticipation of contrast, and thus may lead to intonation-based “garden-pathing” regardless of the adjective type used.

The interpretation of pre-nominal adjectives depends on their efficacies within each referential environment rather than on their inherent semantics. Intonational prominence is used immediately and predictively for reference resolution regardless of the modifier type.

Summary

Current directions

L+H* vs. H* controversy: Phonetic analyses of stimuli from 4

comprehension experiments underway Correlation of first fixation latency and acoustic

measures Parallel analysis of spontaneous speech from

production version of holiday tree task

Issue on categorical distinction:Is L+H* a kind of H*?

Color adj with H* weaker interpretation of contrast? Categorical interpretation but non-categorical perception

(Ladd & Morton, 1997) Great overlap between H* and L+H* in pitch scale, shape &

alignment (Tayler, 2000) Frequent uncertainty between H* and L+H* in expert-ToBI

labeling (Brugos et al. 2008). More frequent use of H* than L+H* to mention contrastive

discourse entities in story continuation (Metusalem & Ito, 2008).

1. How can we define categories of prosodic prominence?2. What factors contribute to recognition & processing of

contrastiveness?

Step-wise Multiple Linear Regressions:Predicting First fixation latency: L+H*

Step Var t p

1 subject -2.91 <.01

2 F0 diff -2.76 <.01

Total R2: .051

F(2,256) = 7.96, p<.001

Step Var t p

1 F0 diff -5.66 <.001

2 F0 peak 4.44 <.001

3 color -2.57 <.05

4 size 2.53 <.05

Total R2: .245

F(2,296) = 25.38, p<.0001

EXP1: Color-sorted EXP2: Size-sorted

Step-wise Multiple Regressions:Predicting First fixation latency: H*

Step Var t p

1 size -4.78 <.0001

2 color 4.03 <.0001

3 subject -2.97 <.01

Total R2: .108

F(3,257) = 11.59, p<.0001

Step Var t p

1 size -2.44 <.05

2 F0 diff -3.21 <.01

3 color -2.64 <.01

Total R2: .088

F(3,293) = 10.57, p<.0001

EXP1: Color-sorted EXP2: Size-sorted

Further exploration needed:

Different dependent variable?Additional phonetic variables:

Normalized F0 scaling Peak, ‘low elbow’ alignment from vowel onset Word/syllable intensity Vowel quality (F1, F2, breathiness, etc.) Following noun’s phonetic status

F0 prominence intensity vowel quality, etc.