Applying Speech and Language Technology to Foreing Language Education

Grażyna Demenko, Natalia Cylwik, Agnieszka Wagner

Adam Mickiewicz University, Institute of Linguistics, Department of Phonetics

Applying Speech and Language Technology to

Foreing Language Education

2nd International Symposium on Multimedia – Applications and Processing (MMAP'09)


Introduction

reading - vocabulary writing - grammarlistening - perceptionspeaking – pronunciation &

prosody

L2 learning – to acquire and develop different skills

Computer-assisted language learning(CALL)

Computer-assisted pronunciation training

(CAPT)


OutlineRequirements on an intelligent tutoring

systemSoftware – tutoring system AzARPronunciation training

curriculum feedback

Prosody training curriculum feedback

allow for training of both pronunciation and prosody: weak pronunciation can sometimes preclude full intelligibility of speech, but prosody is important too, because it helps listeners to process the segmental content

identify precisely the location and type of the errorprovide scoring of learner’s utterance that gives

immediate information on the overall output qualityprovide effective feedback via different channels

(visual, aural, also descriptive, contrastive feedback) – the feedback should be relevant to the type of error made by the learner, easy to interpret and constructive, so that the learner understands how to self-correct and get improvement

keep track of the learner’s performance, so that identification of features that should be practiced is possible and the learner’s progress can be monitored

user-friendliness - it should be clear how to interpret displays and evaluate results


Requirements on an intelligent CAPT system


AzAR(Automat for Accent Reduction)focuses on specific language pairs: Slavonic

(Polish, Slovak, Russian, Czech) and Germanis a knowledge-based system: it uses expert’s

knowledge on typical errors made by L2 learners caused by interference with their native language (L1) phonology and phonetics (analyses of large non-native speech corpora)

includes an extensive curriculum for production and perception training of difficult segmental and prosodic contrasts

learner’s task is to listen to the recording of the utterance produced by the reference voice and to repeat it (in the production scenario) or to discriminate between utterances realized by the reference voice (in the perception scenario)

Pronunciation training: curriculum (1)•Production exercises created by expert phoneticians on the basis of analyses of non-native speech corpora•Goal (1): elicit specific pronunciation errors•Goal (2): concentrate on the most common problems


• pronunciation of sounds which do not exist in one’s mother tongue (L1) or are too difficult to pronounce (e.g. production of Polish vowels instead of the German ones by speakers with L1 Polish)• carry-over of pronunciation regularities from L1 (e.g. assimilation rules with respect to devoicing)• overgeneralizations of target (L2) language regularities (e.g. mapping of polish graphemes [ą] and [ę] to /ow~/ and /ew~/ irrespective of the following consonantal context)


L1 PL – L2 DE L1DE– L2 PL

Pronunciation training: curriculum (2)


•multimodality: visual and audio feedback•the software uses HMM-based speech recognition and speech signal analysis on the learner’s input which makes a visual and aural comparison of the user’s own performance with that of the reference voice possible•in the detection and assessment of pronunciation errors the system relies on “mispronunciation hypotheses” defined by experts

Features of the feedback system (1)mięsień (a muscle)

canonical pronunciation: /m j e j~ s' e n'/

mispronunciation hypotheses: 1) m j e-E~ j~- s' e n'2) m j e-E~ j~- s' e n'-n 3) m j e-E~ j~- s'-S' e n'-n4) m j e-E~ j~- s'-S e n'-n


Features of the feedback system (2)

User interface in AzAR, template for training vowel contrasts in German (here: long tense /i:/ versus short lax /I/)

learner’s utterance is displayed and scored

all uttered phones are marked using a color scaleAn oscillogram of the model utterance is presented simultaneously to allow for comparisonThe learner can listen to his/her own realization of the utterance and to that

produced by the reference voice.

animated visualization of the vocal tract (lips area and articulators movements) and a formants graph for particular phones


Features of the feedback system (3)• Tutorial: gives introduction to acoustic

and articulatory phonetics and explains how to interpret the acoustic displays

Features of the feedback system (4)


For each exercise in the curriculum a passage containing information on the classification, features and articulation of the phone is providedas well as a sagittal slice of the vocal tract during the phone production and pictures of the lip area


Prosody training: curriculum (1)

• Production and perception exercises created by expert phoneticians on the basis of analyses of non-native speech corpora

• Goal (1): elicit specific prosodic errors• Goal (2): enable an efficient training of

the realization of prosodic features in smaller and larger syntactic units

Prosody training: curriculum (2)Word-level prosody


regular word stress:łata - sałata stress shift: chodził - chodziłem

similar word form in Polish and German, but different accentuation: student (vs. Student), Irak (vs. Irak)ABW /a.be.vu/menueksmążAcha!matematykabyliśmydał, dałby, dałby mi, dałby mi godo domunie wiem

Prosody training: curriculum (3)Sentence-level prosody

Intonation patterns (accentuation and phrasing) in simple and complex neutral statements

Intonation in questions:Are you going home? (rising)What are you doing? (falling)Are you going by car or by plane? (falling)-Do you have it? - What? – A dress. (falling)-Do you have it? - What? – Do you have it? (rising)

Intonation patterns in commands, requests, warnings, contrastive and emphatic sentences



Prosody training: curriculum (4)Perception exercises

Asking for additional information/repetition: assigning answer to the question, e.g.– I ‘ve eaten it. – What? a) A banana. b) I’ve eaten it.Completing sentence with focus, e.g.– It’s a big car… a) not a small one. b) not a bus.Assigning question to a sentence with focus, e.g.– Martha is going to the seaside.a) Who is going to the seaside? b) Where is Martha going?


Prosody training: feedback• an accurate visual representation of student’s and native speaker’s pitch contour in real time paired with auditory feedback

• pitch contours are stylized (Pitch Line software) to provide a nearly continuous representation and to ensure that only perceptually relevant pitch variations are displayed

Polishnative

German learner

• relevant portions of the intonation contour (accented and phrase-boundary words) are described parametrically

• a higher-level categorical representation of utterance's intonation is derived from the parametric description; intonation contours which have different categorical representations convey different meanings

• automatic assessment: qualitative (categorical) and quantitative (parametric)

• results are displayed on a color scale (red - green)

• the learner is instructed to compare his/her realization to that of the native speaker – quantitative measurements of both realizations and explanations are provided as a support

#mam #ją #co #sukienkę

Training session in AzAR

Thank you for your attention!

Documents

Applying Speech and Language Technology to Foreing Language Education