CS 4705

Automatic Speech Recognition

• Opportunity to participate in a new user study for Newsblaster and get $25-$30 for 2.5-3 hours of time respectively.

• http://www1.cs.columbia.edu/~delson/study.html• More opportunities will be coming….

What is speech recognition?

• Transcribing words?• Understanding meaning?• Today:

– Overview ASR issues

– Building an ASR system

– Using an ASR system

– Future research

“It’s hard to ... recognize speech/wreck a nice beach”

• Speaker variability: within and across• Recording environment varies wrt noise• Transcription task must handle all of this and

produce a transcript of what was said, from limited, noisy information in the speech signal– Success: low word error rate (WER)

• WER = (S+I+D)/N * 100– Thesis test vs. This is a test. 75% WER

• Understanding task must do more: from words to meaning

– Measure concept accuracy (CA) of string in terms of accuracy of recognition of domain concepts mentioned in string and their values

I want to go from Boston to Baltimore on September 29

– Domain concepts Values

– source city Boston

– target city Baltimore

– travel date September 29

– Score recognized string “Go from Boston to Washington on December 29” (1/3 = 33% CA)

– “Go to Boston from Baltimore on September 29”

Again, the Noisy Channel Model

Input to channel: spoken sentence s

– Output from channel: an observation O

– Decoding task: find s’ = P(s|O)

– Using Bayes Rule

– And since P(O) doesn’t change for any hypothetical s’

– s’ = P(O|s) P(s)

– P(O|s) is the observation likelihood, or Acoustic Model, and P(s) is the prior, or Language Model

Source Noisy Channel Decoder

maxargVs

)()()|(maxarg

OPsPsOP

Vs

maxargVs

What do we need to build use an ASR system?

• Corpora for training and testing of components• Feature extraction component• Pronunciation Model• Acoustic Model• Language Model• Algorithms to search hypothesis space efficiently

Training and Test Corpora

• Collect corpora appropriate for recognition task at hand– Small speech + phonetic transcription to associate

sounds with symbols (Acoustic Model)

– Large (>= 60 hrs) speech + orthographic transcription to associate words with sounds (Acoustic Model)

– Very large text corpus to identify unigram and bigram probabilities (Language Model)

Representing the Signal

• What parameters (features) of the speech input– Can be extracted automatically– Will preserve phonetic identity and distinguish it from

other phones– Will be independent of speaker variability and channel

conditions– Will not take up too much space

• Speech representations (for [ae] in had): – Waveform: change in sound pressure over time– LPC Spectrum: component frequencies of a waveform– Spectrogram: overall view of how frequencies change

from phone to phone

• Speech captured by microphone and sampled (digitized) -- may not capture all vital information

• Signal divided into frames• Power spectrum computed to represent energy in

different bands of the signal– LPC spectrum, Cepstra, PLP – Each frame’s spectral features represented by small set

of numbers

• Frames clustered into ‘phone-like’ groups (phones in context) -- Gaussian or other models

• Why this works?– Different phonemes have different spectral

characteristics

• Why it doesn’t work?

– Phonemes can have different properties in different acoustic contexts, spoken by different people …

– Nice white rice

Pronunciation Model

• Models likelihood of word given network of candidate phone hypotheses (weighted phone lattice)

• Allophones: butter vs. but• Multiple pronunciations for each word• Lexicon may be weighted automaton or simple

dictionary• Words come from all corpora; pronunciations

from pronouncing dictionary or TTS system

Acoustic Models

• Model likelihood of phones or subphones given spectral features and prior context

• Use pronunciation models• Usually represented as HMM

– Set of states representing phones or other subword units

– Transition probabilities on states: how likely is it to see one phone after seeing another?

– Observation/output likelihoods: how likely is spectral feature vector to be observed from phone state i, given phone state i-1?

• Initial estimates for• Transition probabilities between phone states

• Observation probabilities associating phone states with acoustic examples

• Re-estimate both probabilities by feeding the HMM the transcribed speech training corpus (forced alignment)

• I.e., we tell the HMM the ‘right’ answers -- which words to associate with which sequences of sounds

• Iteratively retrain the transition and observation probabilities by running the training data through the model and scoring output until no improvement

Language Model

• Models likelihood of word given prior word and of entire sentence

• Ngram models:– Build the LM by calculating bigram or trigram

probabilities from text training corpus

– Smoothing issues very important for real systems

• Grammars– Finite state grammar or Context Free Grammar (CFG)

or semantic grammar

• Out of Vocabulary (OOV) problem

• Entropy H(X): the amount of information in a LM, grammar– How many bits will it take on average to encode a

choice or a piece of information?

– More likely things will take fewer bits to encode

• Perplexity 2H: a measure of the weighted mean number of choice points in e.g. a language model

Search/Decoding

• Find the best hypothesis P(O|s) P(s) given– Lattice of subword units (Acoustic Model)

– Segmentation of all paths into possible words (Pronunciation Model)

– Probabilities of word sequences (Language Model)

• Produces a huge search space: How to reduce?– Lattice minimization and determinization

– Forward algorithm: sum of all paths leading to a state

– Viterbi algorithm: max of all paths leading to a state

– Forward-backward (Baum-Welch, Expectation-Maximization) algorithm: computes probability of sequence at any state in search space

– Beam search: prune the lattice

Varieties of Speech Recognition

Telephone, microphonesInput device

> 30dB (high) < 10dB (low)SNR

<10 > 100Perplexity

Finite state, ngrams, CFGs, CSGsLanguage Model

<20 5K --> 60K -->~1MVocabulary size

Speaker-dependent or independentEnrollment

Read, prepared, spontaneousStyle

Isolated words continuousMode

Challenges for Transcription

• Robustness to channel characteristics and noise• Portability to new applications• Adapatation: to speakers, to environments• LMs: simple ngrams need help• Confidence measures• OOV words• New speaking styles/genres• New applications

Challenges for Understanding

• Recognizing communicative ‘problems’– ASR errors

– User corrections

– Disfluencies and self-repairs

• Possibilities:– Recognizing speaker emotion

– Identifying speech acts: okay

– Locating topic boundaries for topic tracking, audio browsing, speech data mining

An Unsuccessful Dialogue

S: Hi, this is AT&T Amtrak schedule system. This is TOOT. How may I help you?

U: I'd like to find a train from Ch- to Chicago from Baltimore.

S: Sorry, I can't understand you. Please repeat your utterance.

U: I'd like to find a train from Baltimore to Chicago. S: I heard you say “leave today around 5 thirty in the

morning .“ Which city do you want to go to? U: I would like to leave Saturday at eight in the

morning.

S: You can say the arrival city name, such as “New York City."U: Baltimore to Chicago. S: You can say the arrival city name, such as “New York City.”U: Baltimore to Chicago. (I hate it.)U: Train from Baltimore ...S: ...I heard you s...U: to Chicago.... Hello?S: You can say the name of your departure city, such as "New York City."U: Baltimore.

U: Bal-ti-more....S: You can say...U: Baltimore.S: Do you want to continue and find a new set of trains?U: Yes.S: You can say "yes" or "I am done here" to find a new set of trains.U: Yes.S: Thank you for using AT&T Amtrak train time table system. See you next time.U: I nev-

Summary

• ASR technology relies upon a large number of phenomena and techniques we’ve already seen to convert sound into words– Phonetic/phonological, morphological, and lexical events

– FSA’s, Ngrams, Dynamic programming algorithms

• Better modeling of linguistic phenomena will be needed to improve performance on transcription and especially on understanding

• For next class: we’ll start talking about larger structures in language above the word (Ch 8)

Disfluencies and Self-Repairs

• Disfluencies abound in spontaneous speech – every 4.6s in radio call-in (Blackmer & Mitton ‘91)hesitation: Ch- change strategy.

filled pause: Um Baltimore.

self-repair: Ba- uh Chicago.

• Hard to recognizeCh- change strategy. --> to D C D C today ten fifteen.

Um Baltimore. --> From Baltimore ten.

Ba- uh Chicago. --> For Boston Chicago.