Upload
daisy-turner
View
226
Download
0
Tags:
Embed Size (px)
Citation preview
PSY 369: Psycholinguistics
Language ComprehensionSpeech recognition
Announcements Homeworks
Extended due date for Homework 5 to Thursday Posted Homework 6 (due Apr 3)
Briefly go over that at start of class Cut the number of homework from 11 down to 8
still will drop lowest grade, so top 7 count) So class “total” will be out of 925 points instead of 1000 Hope to have Hwks 7 & 8 in there soon (one speech
error collection, 1 journal summary), they’ll be for after exam 3
Different features than visualVisual word recognition Speech Perception
Some parallel input Orthography
Letters Clear delineation Difficult to learn
Serial input Phonetics/Phonology
Acoustic features Usually no delineation “Easy” to learn
Where are you going
Acoustic features
Spectrogram Time on the x-axis Amplitude is
represented by the darkness of the lines
Frequency (pressure under which the air is pushed) on the y-axis
Acoustic features
<-- Formant transitions ------->
<-- F1 -->
<-- F 2
<-- F 3Burst -->
Formants - bands of resonant frequencies Formant transitions - up or down movement of formants Steady states - flat formant patterns
Bursts - sudden release of air
40 ms5 msbit pit
Formants - bands of resonant frequencies Formant transitions - up or down movement of formants Steady states - flat formant patterns
Bursts - sudden release of air Voice onset time (VOT) - when the voicing begins
relative to the onset of the phoneme
Acoustic features
The confusion of palatalized labials > dentals & alveolars
[
What looks similar to the eye will probably seem similar to the ear!
Hard problems: Ambiguity in speech signal
Chest Jew Wade Aim In ItJust you wait a minute
Delights Haven DimeDaylight Savings Time
Canoes He Wad Ice HeCan You See What I See?
Free Quaintly As Quest Shuns Frequently Asked Questions
http://www.playmadgabonline.com/
TV ad
Hard Problems in Speech Perception
Segmentation problem Lack of Invariance Linearity (parallel transmission) Co-articulation Trading relations
Hard Problems in Speech Perception
Segmentation problem: Unlike visual input, the acoustic input is not physically segmented
Illusion of silence. There are no silent gaps in the wave form, even though we may “hear” some.
Hard Problems in Speech Perception
Segmentation problem: Unlike visual input, the acoustic input is not physically segmented
Here the silence that we see in the acoustics isn’t perceived as a gap in the word
Hard Problems in Speech Perception
Lack of Invariance: One phoneme should have a one waveform
This is not the case. The /i/ (‘ee’) in ‘money’ and ‘me’ are different
Show me the money
Hard Problems in Speech Perception
Lack of Invariance: One phoneme should have a one waveform
Another example: Here is the phoneme /d/ followed by different vowels
Hard Problems in Speech Perception
Lack of Invariance: One phoneme should have a one waveform
And another. The phrase has five /t/ phonemes, but there are not 5 identical sweeps in the spectrogram
There aren’t invariant cues for phonetic segments Although the search continues
Peter buttered the burnt toast
Hard Problems in Speech Perception
Linearity (parallel transmission): Acoustic features often spread themselves out over other sounds
Where does show start and money end?
Wave form
Show me the money
Hard Problems in Speech Perception Co-articulation: the influence of the articulation
(pronunciation) of one phoneme on that of another phoneme. Essentially, producing more than one speech sound at once May be helpful because it allows some parallel transmission of information
(possibly helping predict what’s coming next) Each sound partially shaped by sounds before & after it
keel vs kill vs cool / kil / vs / kIl / vs / kul / (IPA characters)
place of articulation and rounding on the k differ a lot different versions of “the same sound” in
different contexts from different speakers
This is what allows us to talk so fast May be helpful because it allows some parallel transmission of
information (possibly helping predict what’s coming next)
Hard Problems in Speech Perception
Trading relations Most phonetic distinctions have more than one acoustic cue
as a result of the particular articulatory gesture that gives the distinction.
Voice-onset-time (VOT) Energy in burst Onset frequency of the first formant Placement in syllable
e.g., slit–split – the /p/ relies on silence and rising formant, different mixtures of these can result in the same perception
Perception must establish some "trade-off" between the different cues.
Hard Problems in Speech Perception
Many factors that may be important Acoustic Information Visual information Prosodic information Semantic context Syntactic structure
Top-down
UNDERSTANDING
Bottom-up
Using Visual information
McGurk effect
The McGurk effect: McGurk and MacDonald (1976)• Showed people a video where the audio and the video don’t
match (Think “dubbed movie”)• Visual /ga/ with auditory /ba/ often hear /da/
Implications• Phoneme perception is an active process • Influenced by both audio and visual information
Beyond the segment Prosodic factors (supra segmentals)
English: Speech is divided into phrases. Every phrase has a focus. Word stress is meaningful in English. Stressed syllables are aligned in a fairly regular rhythm, while
unstressed syllables take very little time. An extended flat or low-rising intonation at the end of a phrase can
indicate that a speaker intends to continue to speak. A falling intonation sounds more final.
Beyond the segment Prosodic factors (supra segmentals)
Stress Emphasis on syllables in sentences
On meaning “black bird” versus “blackbird”
Top-down effects on perception Better anticipation of upcoming segments when syllable is stressed
Rate Speed of articulation: Faster talking - shorter vowels, shorter
VOT Normalization: taking the speaker’s rate into account
Intonation Use of pitch to signify different meanings across sentences
Top-down effects on Speech Perception
Sentence context effects Excised speech Sentence context effects Phoneme restoration effect
Top-down
UNDERSTANDING
Bottom-up
Excised Speech
Syntactic and semantic cues can help
Pollack & Pickett (1964)
Task: Recorded conversations and excised individual words. Presented the words to listeners for identification
Within context Out of context
Results:Words out of context were only recognized 47% of time, identification was greatly improved with contextSuggests that clarity in speech reflects processing (top-down as well as bottom-up)
Semantic Influences
Garnes & Bond (1976):
16 tokens, spanning the spectrum of bait-date-gate (/b/ /d/ /g/) So some were clear examples (unambiguous), others in between (ambiguous)
3 carrier sentences (context): Here’s the fishing gear and the ______. Check the time and the _______. Paint the fence and the _______.
Results If unambiguous, get semantically implausible sentences
(Paint the fence and the bait.) If ambiguous (near a phoneme boundary), semantic context effects
– interpreted the word as contextually appropriate
Phoneme restoration effect
Task: Listen to a sentence which contained a word from which a phoneme was deleted and replaced with another noise (e.g., a cough)
The state governors met with their respective legi*latures convening in the capital city.
* /s/ deleted and replaced with a cough
Click here for a demo and additional informationWarren (1970)
Results:
• Participants heard the word normally, despite the missing phoneme
• Usually failed to identify which phoneme was missing
Interpretation:
We can use top-down knowledge to “fill in” the missing information
Phoneme restoration effect
Warren and Warren (1970)
What if the missing phoneme was ambiguous?
The *eel was on the axle.
Results:
Participants heard the contextually appropriate word normally, despite the missing phoneme
The *eel was on the shoe. The *eel was on the orange. The *eel was on the table.
Phoneme restoration effect
Possible loci of phoneme restoration effects Perceptual loci of effect:
Lexical or sentential context influences the way in which the word is initially perceived.
Post-perceptual loci of effect: Lexical or sentential context influences decisions
about the nature of the missing phoneme information.
Samuel (2001) attempts to look at this issue
Cross-modal priming
Shillcock (1990)
hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming)
The scientist made a new discovery last year.
Hear:
NUDIST
Cross-modal priming
The scientist made a novel discovery last year.
Hear:
Shillcock (1990)
hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming)
NUDIST
Cross-modal priming
The scientist made a novel discovery last year.
Hear:
The scientist made a new discovery last year. faster
Shillcock (1990)
hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming)
NUDIST
Cross-modal priming
The scientist made a novel discovery last year.
Hear:
NUDIST gets primed by segmentation error
faster
Although no conscious report of hearing “nudist”
The scientist made a new discovery last year.
Shillcock (1990)
hear a sentence, make a lexical decision to a word that pops up on computer screen (cross-modal priming)
Theories of speech perception Motor Theory Direct Realist Theory General Auditory Approach Cohort TRACE Model
Motor theory of speech perception
A. Liberman (initially proposed in late 50s, recent Liberman & Mattingly, 1985)
Direct translation of acoustic speech into articulatory categories Holds that speech perception and motor control involved linked
(or the same) neural processes Theory held that categorical perception was a direct reflection of articulatory
organization Categories with discrete gestures (e.g., consonants) will be perceived
categorically Categories with continuous gestures (e.g., vowels) will be perceived continuously
There is a speech perception module that operates independently of general auditory perception
Frontal slices showing differential activation elicited during lip and tongue movements (Left), syllable articulation including [p] and [t] (Center), and listening to syllables including [p] and
[t] (Right)
Pulvermüller F et al. PNAS 2006;103:7865-7870
©2006 by National Academy of Sciences
Speech Perception & the brain
Motor theory of speech perception Some problems for MT
Categorical perception found in non-speech sounds (e.g., music)
Categorical perception for speech sounds in non-humans Chinchillas can be trained to show categorical perception of /t/ and /d/
consonant-vowel syllables (Kuhl & Miller, 1975)
Other theories of speech perception Direct Realist Theory (C. Fowler and others)
Similar to Motor theory, articulation representations are key, but here they are directly perceived (related to Gibson’s perceptual theory)
Perceiving speech is part of a more general perception of gestures that involves the motor system
General Auditory Approach (e.g., Diehl, Massaro) Do not invoke special mechanisms for speech
perception, instead rely on more general mechanisms of audition and perception
For nice reviews see: Diehl, Lotto, & Holt (2003) Galantucci, Fowler, Turvey (2006)
Other theories of spoken word rec. Cohort Model (Marslen-Wilson & Welsh, 1978; Discussed last time)
1) The acoustic information at the beginning of a word activates a “cohort” of possible words
2) Syntax and semantics influence the selection of the target word from the cohort
TRACE Model (Elman and McClelland 1984, 1986) Connectionist, parallel distributed model Processing occurs through excitatory and inhibitory
connections – in processing units called nodes 3 levels of nodes: features, phonemes, and words all highly
interconnected