Upload
others
View
14
Download
0
Embed Size (px)
Citation preview
1
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Sp-ToBI (Tones and break indices) DIME-Sp ToBI2006
� Spanish-ToBI� Five levels
� DIME Labeling
� Labeling experiment
� DAMSL and Intonation
� Conclusions
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Sp-ToBI
� Spanish Tones and break indices (Sp-ToBI)� Mary Beckman (1999)
� Referential system to transcribe intonational and prosodic information of any Spanish dialect
� Researchers may use it for :
� Unify linguistic vocabulary
� Make the analysis easier with a phonological transcription
� Label different corpus and develop databases
� Based upon ToBi which is based on the Metric and Auto
Segmental Model
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Metric and AutosegmentalModel (AM)
� Phonological analysis intended to identify contrastable elements from an intonational system whose combination produces melodic contours founded on the possible elocutions of language.
� Autosegmental phonology theory assumes that the prosodic level of representation (i.e. melody or tunes) is independent of the segmental and
El Proyecto DIME, DCC-IIMAS, UNAM 2006
AM Model
� It is characterized by tones anchored to
the tonic syllables.
� Tones could be :
�Phrase (H high, L Low, HL, LH) (Pitch)
�Boundary (H%, L%)
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Sp- ToBI Five levels
Syllables
Words
BI
Tones
Code
Miscelaneous
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DIME- ToBI labeling
Analyze if the DIME corpus can be labeled
with Sp-ToBI
�Pick up as much information as possible
�Analyze speech phenomena (natural speech)
�Align the labels and the audio files (time)
�Find the best computational tool that gives the needed information
2
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Sosa’s example
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DIME’s exampleBefore, Before that, could you move the air extractor
above the, the stove
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Problems with Sp-ToBI
� DIME Corpus needs to describe real
language
� Laboratory and controlled examples in
other studies provide ideal intonation paterns
� DIME corpus presents several
spontaneous speech phenomena that need to be solved for tone labeling.
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tools
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tools
� MES (Motif editor for speech-signals) )
� FO without interpretation problems (stops, unvoiced, pauses)
� Good visual alignment
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DIME Corpus characteristics:
�Spontaneous speech phenomena:
� pauses
� speech repairs
� repetitions
� interjections
�Non “ideal” recording quality
�Different utts length (1 word to 20 words)
3
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Labeling experiment
� Two phases: 1. Tone recognition:
� Preliminary study
� New tones proposal
� DIME conventions
2. Tagging validation: � Manual writing
� Tone labeling by three different taggers
� Analysis of results
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Phase I
� Descriptive study of 117 utterance
dialogue (d12) using 5 level (Allophones, allophonic syllables, words (orthographic),
break Indices and tones) in order to describe the DIME corpus and find out
possible solutions.
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Allophones, syllables and words
Alophones
Phonetic Sylables
Words
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Break Indices
� Break indices
�Original
� “0” vowel coalescence (synalefa)
� “1” ordinary interword juncture
� “4” Intonational phrase, different tonal phrase
�DIME Proposal
� “2” Interruption point of speech repairs
� “3” Mid tones (eg.enumerations)
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Break indices (2)
Could you move the, the sink a little toward, toward to the minibar?
2
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Break Indices (3)Cuboards, shelves, stoves and air extractors,sinks and diswashers, cabinets, tables, refrigerators,
chairs,
3 3 3 3 33 3
4
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tones1. Phrase
� L+H* � L*+H
� H+L*
� H*+L� H*
� L*
2. Boundary� H%� L%
3. Middle� H-
� L-
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DIME Tones
Could you show me the sink and dishwasher catalogue?
Middle
L-
Boundary
H%
Phrase
L+H*
4
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tones1. Phrase
� L+H*
� L*+H� H+L*
� H*+L
� H*� L*
2. Boundary� H%
� L%
3. Middle� H-� L-
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tone asignment
L*+H
Could you show me the shelves ?
El Proyecto DIME, DCC-IIMAS, UNAM 2006
TonesPhrase tones
� Original:� L+H* : Accented peak after a valley
� L*+H: Stress followed by a peak
� H+L*: Accented valley after a peak
� H*: high-tone stress
� DIME Proposal:� H*+L: Accented peak followed by a valley
� L*: low-tone stress
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Original tones
Accented peak after a valley Stress followed by a peak
Accented valley after a peak High-tone stress
5
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DIME proposal
Accented peak followed by a valley Low-tone stress
El Proyecto DIME, DCC-IIMAS, UNAM 2006
TonesPhrase tones
� Original:
�L*+H: Stress followed by a peak
� L+H* : Accented peak after a valley
� H+L*: Accented valley after a peak
� H*: high-tone stress
� DIME Proposal:
� H*+L: Accented peak followed by a valley
� L*: low-tone stress
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Example (L*+H)
L*+H L*+H
¿Is that all right?
El Proyecto DIME, DCC-IIMAS, UNAM 2006
TonesPhrase tones
� Original:� L*+H: Stress followed by a peak
�L+H* : Accented peak after a valley
� H+L*: Accented valley after a peak
� H*: high-tone stress
� DIME Proposal:� H*+L: Accented peak followed by a valley
� L*: low-tone stress
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Example (L+H*)
L+H*
To the other side of the air extractor
El Proyecto DIME, DCC-IIMAS, UNAM 2006
TonesPhrase tones
� Original:
� L+H* : Accented peak after a valley
� L*+H: Stress followed by a peak
�H+L*: Accented valley after a peak
� H*: high-tone stress
� DIME Proposal:
� H*+L: Accented peak followed by a valley
� L*: low-tone stress
6
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Example (H+L*)
H+L*
This, right to the stove
El Proyecto DIME, DCC-IIMAS, UNAM 2006
TonesPhrase tones
� Original:
� L+H* : Accented peak after a valley
� L*+H: Stress followed by a peak
� H+L*: Accented valley after a peak
�H*: high-tone stress
� DIME Proposal:
� H*+L: Accented peak followed by a valley
� L*: low-tone stress
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Example (H*)
H*H*
El Proyecto DIME, DCC-IIMAS, UNAM 2006
TonesPhrase tones
� Original:
� L+H* : Accented peak after a valley
� L*+H: Stress followed by a peak
� H+L*: Accented valley after a peak
� H*: high-tone stress
� DIME Proposal:
�H*+L: Accented peak followed by a valley
� L*: low-tone stress
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Examples (H*+L)
H*+L
This?
El Proyecto DIME, DCC-IIMAS, UNAM 2006
TonesPhrase tones
� Original:
� L+H* : Accented peak after a valley
� L*+H: Stress followed by a peak
� H+L*: Accented valley after a peak
� H*: high-tone stress
� DIME Proposal:
� H*+L: Accented peak followed by a valley
�L*: low-tone stress
7
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Example (L*)
L*
Yes
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tones1. Phrase
� L+H* � L*+H
� H+L*
� H*+L� H*
� L*
2. Boundary� H%� L%
3. Middle� H-
� L-
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Boundary tones
H%
� F0 rise, after any pitch accent.
L%
� F0 fall, with regard to the last stress.
H%
L%
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Boundary H% tone
H%
¿Where do you want me to put it?
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Boundary L% tone
L%
To the other side of the air extractor
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tones1. Phrase
� L+H* � L*+H
� H+L*
� H*+L� H*
� L*
2. Boundary� H%� L%
3. Middle� H-
� L-
8
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Mid tones
� H or L tone preceded with a “-”
� Shows a break in the tonal phrase, that
changes the normal intonation.
� L- o H- :
�Enlargements
�Audible changes
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Mid tone (enlargment)Could you show me the sink and dishwasher catalogue?
H-
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Mid tone (Change)
Would you like me to move or bring any object tothe kitchen?
L-
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DIME conventions
Break Indices
0
1
4
Break Indices:
0
1
4
2
3
Tagging layers:
syllables
words
BI
tones
miscelaneous
Tagging layers:
alophones
syllables
words
BI
tones.
BeckmanDIME
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DIME conventions
Tones:
Phrase:
L*+H
L+H*
H+L*
L*
Boundary:
L%
H%
Tones:
Phrase:
L+H*
L*+H
H+L*
H*+L
L*
H*
Boundary:
L% H%
Mid:
L- H-
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Phase II
1. Write a Manual of DIME-ToBI principles,
based on the found results of the first phase of the experiment.
2. Train three different labelers the DIME-ToBIconventions.
3. Tone labeling of a dialogue (4 layers were
done before)
4. Measure agreement between the three
taggers.
9
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Agreement between taggers.
� Parameters considered:
�Speech phenomena
�Total agreement percentajes between taggers
�Toneme match (last tone ands boundary
tone) according to the tonal intention.
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Monosyllable
L%L%L%
L*
L*
L*
Yes
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Bisyllable
Here
H+L*L+H*
L*+H
10
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech repair (Repetition)
L+H*
L+H*
L+H*
L%
L%
L%
By, by the stove
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Interjection
Eh, mh, below the window
H-
H-
NO
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Enlargments
What i want now is a dishwasher machine
L-L-
L-
11
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
One intonational phrase
H+L*
H+L*
H+L*
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Intonational phrasesArriba, arriba de este mueble…
L+H*
L+H*
L+H*
L*+H
L*+H
L*+H
L%
L%
L%
va_7 ri_7
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Intonational phrases…del mueble que acabamos de rotar noventa grados
L%
L%
L%
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Speech phenomena:
Monosyllables
Bisylables
Speech repair
Interjection
Enlargment
One intonational phrase
Intonational phrases
Speech phenomena
12
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Miscelaneous
L*+H L- L*+H
L*+H H- L+H*
L*+H L*+¡H
Puedo, puedo mover, eh, el es que quiero…
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Agreement between taggers.
� Parameters considered:
�Total agreement percentajes between taggers
�Toneme match (last tone ands boundary tone) according to the tonal intention.
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Total agreement percentajesbetween taggers
0
20
40
60
80
100
Labeling agreement percentaje
Between 100% and80%
Less than 80%
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Total tone agreement
0
10
20
30
40
50
60
Agreement %
Less tan 60%
From 70% to80%
From 80% to90%
From 90% to100%
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Agreement between taggers.
� Parameters considered:
�Total agreement percentajes between taggers
�Toneme match (last tone ands boundary tone) according to the tonal intention.
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tonemes. Single word
MatchesAgreementCasesUtt
L+H* H- H%60%5Interrogative
H+L* L- L%
H+L* L- L%
60%21“Okey“
L* L- L%85%28Declarative
13
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Tonemes. DeclarativesMatchesAgreementCasesUtt
H+L* L- L%
L* L- L%
L+H* L- L%
85%20With pauses
( + 12 tones)
H+L* L- L%L+H* L- L%
H+L* L- L%
48%29Withoutpauses
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Toneme. Interrogatives
L+H* H- H%
L*+H H- H%
91%12Interrogative
(Check)
MatchesAgreementCasesUtt
L+H* H-H%54%11Is that allright?
L+H* H- H%80%10Interrogative
(Genuine)
El Proyecto DIME, DCC-IIMAS, UNAM 2006
E.g. “Where do you want me to
place it?”
utt75 : dónde quieres que la
ponga ? L*+H H+L* L+H* H- H%
L+H* H+L* L+H* H- H%
L*+H H+L* L+H* H- H%
utt90 : s: dónde quieres que lo
ponga ? L*+H H+L* L- L+H* H- H%
L*+H H+L* L- L+H* H- H%
L*+H H+L* L- L+H* H- H%
utt129: s: dónde quieres que lo
ponga ? L*+H H+L* L- L+H* H- H%
L*+H H+L* L+H* H- H%
L*+H H+L* L- L+H* H- H%
El Proyecto DIME, DCC-IIMAS, UNAM 2006
E.g “Is that all right?”
utt44 : <ruido> <sil> <ruido> <ruido>
<sil> <ruido> <sil> ahí está bien ? + + L* !L* L*+H H- H%
L* !L* L+H* H- H%
L* !L* L*+H L- H%
utt59 : ahí está bien ? L*+H L+H* H- H%
L*+H L+H* H- H%
L*+H L+H* H- H%
utt69 : s: ahí está bien ? + + L* !L* L*+H H- H%
L* !L* L+H* H- H%
L* !L* L*+H H- H%
El Proyecto DIME, DCC-IIMAS, UNAM 2006
E.g. “Okey”
utt58 : s: okey H*+L L- L%
H*+L L- L%
H*+L L- L%
utt62 : s: okey H+L* L- L%
H+L* L- L%
H+L* L- L%
utt67 : s: okey H+L* L- L%
H+L* L- L%
H+L* L- L%
El Proyecto DIME, DCC-IIMAS, UNAM 2006
DAMSL and intonation
� Original hypothesis
�There is a relation between intonation andDAMSL tagging* and it will be possible to find
behavior paterns between labels.
�The same toneme matches with the same
dialogue act.
14
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Action directives
L%L-H+!L*H+L*
L%L-H+!L*H+L*
L%L-H+!L*H+L*utt81 : ahora quiero un estante
L%L-H+L*H-
L%L-L*H-
L%L-L*+!HH-utt120: y <sil> ahora quiero otro gabinete
H%H-L*+HL*
H%H-L+H*!L*
H%H-L*+HH+L*utt25 : ahora quiero un refrigerador
L%L-H*+LL*
L%L-!L*L*
L%L-!H*+LH*+Lutt14 : <ruido> mm mh <no-vocal> <sil> ahora
quiero una campana
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Action directives
L%L-H+!L*H+L***** utt102: u: dame la blanca superior doble
L%L-H+!L*H+L*utt81 : ahora quiero un estante
L%L-H+L*H-
L%L-L*H-
L%L-L*+!HH-utt120: y <sil> ahora quiero otro gabinete
H%H-L*+HL*
H%H-L+H*!L*
H%H-L*+HH+L*utt25 : ahora quiero un refrigerador
L%L-H*+LL*
L%L-!L*L*
L%L-!H*+LH*+Lutt14 : <ruido> mm mh <no-vocal> <sil> ahora quiero una campana
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Information request
H%H-L*+H!L*
H%H-L+H*!L*
H%H-L*+H!L*utt94 : s: ahí está bien ? + +
L%L-H+L*L*+H
L%L-H+L*L*+H
L%L-H+L*L*+Hutt102: quieres que <sil> sustituya <sil> este mueble por éste ?
H%H-L+H*
H%H-L+H*
H%H-L+H*utt28 : s: éste ?
H%H-L+H*H+L*
H%H-L+H*H+L*
H%H-L+H*H+L*utt49 : s: dónde quieres que la ponga ?
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Information request
H%H-L*+HH+!L****utt101: s: quieres alguno en particular?
H%H-L+H*!L*
H%H-L*+H!L*utt94 : s: ahí está bien ? + +
L%L-H+L*L*+Hutt102: quieres que <sil> sustituya <sil> este
mueble por éste ?
H%H-L+H*utt28 : s: éste ?
H%H-L+H*H+L*utt49 : s: dónde quieres que la ponga ?
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Commit vs. Acknowledge
L%L-H+L*utt22 : s: okey
L%L-H+L*
L%L-H+L*
L%L-H+L*
L%L-H*+L
L%L-L+H*utt134: s: okey
L%L-H*+L
L%L-H*+L
L%L-H*+Lutt32 : s: okey
L%L-H*+L
L%L-H*+L
L%L-H*+Lutt11 : s: okey
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Commit vs. Acknowledge
Acknowledge
L%L-H+L***utt135: s: okey
utt22 : s: okey
L%L-H+L*Commit
L%L-H+L*
L%L-H*+LAcknowkedge
L%L-L+H*utt134: s: okey
L%L-H*+LCommit
utt32 : s: okey
L%L-H*+LCommit
utt11 : s: okey
15
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Conclusions
� Most popular tones are L*+H and H+L*
� Mostly declarative intonation is related
to L- L%, and interrogative to H- H%
� Before performing deep research
involving more than two dialogues thetone set proposal could be validated.
El Proyecto DIME, DCC-IIMAS, UNAM 2006
Conclusions
� DIME labeling has a good agreement levelbetween taggers, which means thatconventions are understandable.
� The relation between agreement betweentaggers and dialogue acts is notrepresentative.
� No pattern between DAMSL anotation andToBI labeling has been found so far.
� The relation between tonal intention andtones has not been found so far.