Upload
elaina
View
64
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Каналы мультимодальной коммуникации: относительный вклад в понимание дискурса. «Мультимодальная коммуникация» 15 ноября 201 3. А.А. Кибрик (ИЯз РАН и МГУ) Н.Б. Молчанова (BearingPoint) [email protected]. What is the contribution of different communication channels?. - PowerPoint PPT Presentation
Citation preview
Каналы мультимодальной коммуникации:
относительный вклад в понимание дискурса
А.А. Кибрик (ИЯз РАН и МГУ)Н.Б. Молчанова (BearingPoint)
«Мультимодальная коммуникация»15 ноября 2013
2
What is the contribution of different communication channels?
Traditional approach of mainstream linguistics:
the verbal channel is so central that prosody and the visual channel are at best downgraded as “paralinguistics”
Applied psychology It is often stated that (figures go back to Mehrabian
1971):• body language conveys 55% of information• prosody conveys 38% of information• the verbal component conveys 7% of information
Who is right?
3
Relative contribution of three communication channels?
DISCOURSE
Vocal channels Visual channel
Verbal channel Prosodic channel
4
Experimental design
Isolate the three communication channels Present a sample discourse in all possible
variants (23=8) Present each of the eight variants to a
group of subjects Assess the degree of understanding in
each case Such assessment may lead to estimates of
the contributions of communication channels
5
Studies in this line of research
Èl’bert 2006, year paper Èl’bert 2007, diploma thesis
Reinterpreted and refined in Kibrik and Èl’bert 2008
Molchanova 2008, year paper Molchanova 2009, year paper Molchanova 2010, diploma thesis
Reinterpreted and refined in Kibrik and Molchanova 2013
6
Èl’bert 2007, Kibrik and Èl’bert 2008
Russian TV serial “Tajny sledstvija” – “Mysteries of the investigation”
Context excerpt: 8 minutes Experimental excerpt: 3 min. 20 sec.
consisting of conversation alone, to ensure that we are testing the understanding of discourse rather than of the film in general
Two vocal channels have been separated: Verbal: running subtitles Prosodic: superimposed filter creating the “behind a
wall” effect
Participants: Native speakers of Russian Eight groups of 10 to 17 participants
7
Eight experimental groups
Group 0: only the context excerpt Groups 1 (one communication channel)
Verbal: subtitles, temporally aligned Prosodic: filtered sound Visual: video
Groups 2 (two communication channels): Verbal + prosodic = original sound Verbal + visual: subtitles and video Prosodic + visual: filtered sound and video
Group 3: original material
8
Verbal + visual
9
Procedure
The context and the experimental excerpts were shown to a group of subjects on a large screen
Each subject answered 23 multiple-choice questions concerned with the experimental excerpt alone
What Tamara Stepanovna offers Masha before the beginning of the conversation: a. to take off her coat b. to have a cup of tea c. to have a seat d. to have a drink
Percentage of correct answers is used as an assessment of a subject’s degree of understanding
10
Results
All three channels are substantially informative
Verbal > visual > prosodic
Integration of visual and prosodic channels is difficult
Group 0
Verbal
Prosodic
Visual
Ver+Pro
Ver+vis
Vis+pro
Group 3
0,00%
20,00%
40,00%
60,00%
80,00%
100,00%
11
Molchanova 2010Kibrik and Molchanova 2013
Methodological issues The following aspects of the prior
study have been changed (improved) Stimulus material Methods of isolating the channels Questionnaire Participants and interviewing procedure
12
Stimulus material: discourse type
Shortcomings of movies Plot facilitates guessing Possible familiarity with the movie Quasi-natural behavior of actors
Solution: natural dialogue Guessing game
original.avi, 0:19 – 0:57
13
Stimulus material: speakers
Shortcomings of the prior studies Same-sex speakers indistinguishable
in the prosody-only version
Solution: Different sexes: F0 range is different
14
Methods of isolating the channels: Verbal channel
Shortcomings of subtitles Subtitles belong to the visual mode Hard to read without punctuation
• Especially at the rate of speech• And especially in the “verbal + visual”
condition
Solution: spoken prosody-free signal Each word in transcript is recorded
individually from the corresponding person All thus elicited words are glued together
in the right order
15
Visual + verbal (the robot condition)
16
Verbal channel
Remaining problem Unnatural input
• No reduction• No intonation• etc.
17
Methods of isolating the channels: Prosodic channel
Shortcomings of the prosodic material as used in previous studies Excessive noise
Solution: Loudness is decreased radically at all
frequencies except for the speaker’s average F0 frequency
This has led to a more satisfactory “behind the wall” (or “behind the glass”) effect
18
Visual + prosodic (the mermaid condition)
19
Questionnaire
Shortcomings of prior studies Èl’bert 2007: gap between Group 0 (38.3%)
and Group 3 (87.4%) is insufficient
Solution Testing stage
• Identify trivial questions (high Group 0) –5 • Identify unfortunate questions (low Group 3) –2 • 30 23
Group 0: 34.5% correct answers Group 3: 88.0% correct answers
20
Participants and interviewing procedure
Shortcomings of prior studies Uncontrolled social status and geographical origin of
participants Multiple participants in one room may affect each
other’s performance Need for a big screen
Solutions Control for social status and geographical origin;
homogeneous group Comparable, independent, and comfortable conditions
• Detailed guidelines Remote implementation
• Stimulus materials at Youtube.com• Questionnaire at Googledocs
21
Kibrik and Molchanova 2013: Results
Each individual channel is substantially informative and prevails over the null condition (34.5%)
F-test: verbal and visual: p<0.05, prosodic: p=0.127
Verbal (58.8%) > visual (52.2%) > prosodic (40.2%)F-test: verbal > prosodic,
visual > prosodic: p<0.05, verbal > visual: p=0.071
22
Kibrik and Molchanova 2013: Results
Two-channel conditions prevail over the one-channel conditions much more clearly than in the previous experiment (verbal+prosodic – 73.5%, verbal+visual – 88.2%)
F-test: all pairwise comparisons but “visual+prosodic > visual”:
p<0.05; all two-channel conditions > all
one- channel conditions: p<0.0001
A dramatic dip in the visual+prosodic condition is even clearerF-test: significant difference from the two other two-channel conditions, p<0.0001
23
Kibrik and Èl’bert 2008 vs. Kibrik and Molchanova 2013
0,00%
20,00%
40,00%
60,00%
80,00%
100,00%
General picture is remarkably similarIn the new study all effects are clearer
24
Normalized contribution of three channels
Suppose the three channels are independent
Sum up all percentages of individual channel contributions and normalize to 100%
Identify normalized contribution
25
Normalized contribution of three channels
Kibrik and Èl’bert 2008
Kibrik and Molchanova 2013
Summed percentages
72+51+62=185 59+52+40=151
Normalized contributions
Verbal 72%:1.85≈39%
59%:1.51≈39%
Prosodic
51%:1.85≈28%
46%:1.51≈30%
Visual 62%:1.85≈33%
49%:1.51≈32%
26
Gender differences
Molchanova 2010: gender advantages Percentages of correct answers
Condition Men Women Advantage
Verbal only 59.1 69.9 Women: +10.7
Visual + prosodic
66.1 51.6 Men: +14.5
27
Conclusions
All communicatioin channels are highly significant the traditional linguistic viewpoint is incorrect
The verbal channel is the leading one the viewpoint popular in applied psychology is incorrect
Information from the prosodic and the visual channels is primarily used through integration with the verbal channel
Very similar results have been attained in different studies, in spite of very different methodological details
28
Further questions
Auditory or graphic presentation of the “verbal alone” channel?
Explore different discourse types, such as monologic discourse
…and: Other suggestions on this approach?
29
Acknowledgements
Olga FedorovaAnna LaurinavičiuteAndriy Myachykov
RGNF #11-04-00153
30
Thanks for your attention
verbal channel
visual channel
prosodic channel
language