Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Quality in the Eye of the Beholder?Quality measures and user satisfaction
in live subtitling.
Monika SZCZYGIELSKA, Łukasz DUTKA
Imagine subtitles with disappearing letters
2
3
Nawiedź dusz wiernych Tobie krąg, Niebieską łaskę zesłać racz, Sercom co dziełem są Twych rąk.
Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
4
Nawiedź dusz wiernych Tobie krąg, Niebieską łaskę zesłać racz, Sercom co dziełem są Twych rąk.
Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
5
Subtitles with no Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
Nawiedć dusz wiernych Tobie krąg, Niebieskąśaskę zesśał racz, Sercom co dzieśem są Twych rąk.
ź -> ć ł -> ś ć -> ł
S
Someone messed up with diacritics
6
Subtitles with Polish diacritics. Dancing with the Stars, Polsat, Poland 2017.
On powiedział że opanowałeśpodstawy. Jakie podstawy?
7
Subtitles with no Polish diacritics. Dancing with the Stars, Polsat, Poland 2017.
On powiedzial, ze opanowalespodstawy. Jakie podstawy? S
Diacriticsdisappearedaltogether
8
Disaster!What happened?
POSZEDŁ DO SĄDU / SADUHe went to .... the court / the orchard
9
ZROBIĆ ŁASKĘ / LASKESomeone turned diacritics
Make .... grace / a stick
10
11
Niebieską łaskę zesłać racz.(“Give us your heavenly grace”)
Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
12
Disaster, really ?What do users think about it?
The errors went unnoticed
13
How is it possible?
14
Research---
objective & subjective measures
● NER value● USERS'
OPINIONS
15
SPEECHES FROM THE POLISH PARLIAMENT [2016]
16
USERS' OPINION
N = 55 [Hearing 24, HoH 15, Deaf 16]
online questionnairerecorded speeches with live transcriptioninstructions in sign language or written instructions
Live subtitling priorities according to users [Szczygielska & Dutka 2016] 17
1 LEGIBILITY
3 LATENCY2VERBATIM
4DISPLAY MODE
5SEGMEN- TATION
4
6GRAMMAR
& SPELLING
5
3
evaluated the subtitles with the highest NER value as the worst
12%
MYTHNER value >99%
NER value & users’ opinions [Szczygielska, Dutka 2016] 18
said that subtitles were highly edited and simplified
15%
MYTHreduction=0.5%
NER value is in line with users' opinions
overall
NER > 99%users rating: 3.6 (1-5)
Stereotypes among users?
● too much delay● too much text reduction● too slow / too fast
19
DANCING WITH THE STAIRS TV POLSAT [2017]
20
USERS' OPINION
N = 203 [Hearing 8, HoH 68, Deaf 127]
Questionnaire & instruction in sign language
NER value Dancing with the Stars [Szczygielska & Dutka 2017] 21
NER value in final episode
99.5%
> 98 OFCOM recommendation
average delay in final episode
<5-6 sec.
real average delay in the UK after 10 years
lowest average delay in an episode(episode 3)
3.4 sec.
3 sec. OFCOM recommendation
of the users were satisfied or very satisfied with the quality of live subtitles for “Dancing...”
22
67.5%
BUT….Users’ opinions. Dancing with the Stars [Szczygielska & Dutka 2017]
23
Users’ opinions. Dancing with the Stars [Szczygielska & Dutka 2017] 24
the text reduction is very high with many utterances missing
22 %
MYTHreduction > 10%
the delay was high or very high
18.70 %
MYTHdelay > 6 sec.
subtitles are too fast, I can’t keep up with the pace
19.20 %
18,72%subtitles are too slow
Conclusions 1/2
25
Users might not be rational in their opinions.
Users can be very critical.
Users’ needs can be so diverse as to be mutually exclusive.
Our approach is user-focused and user-informed but not user-lead.
We should listen to users but at some points we have to make an informed decision.
26
Quality in the Eye of the Beholder. Quality measures and user satisfaction
in live subtitling.
Monika SZCZYGIELSKA, Łukasz DUTKA