Embed Size (px)
Citation preview
Joke DaemsPhD student
Lieve Macken, Sonia Vandepitte, Robert Hartsuiker
Comparing HT and PE using advanced research tools
Overview
• Experimental setup• Process
– Speed– Translation units– Cognitive load– Gaze behavior– External resources
• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources
Overview
• Experimental setup• Process
– Speed– Translation units– Cognitive load– Gaze behavior– External resources
• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources
Experimental setup
• Text selection• Participants• Session• Tools• Data collected
Text selection
• 15 newspaper articles (+/- 150 words) from Newsela.com
• Readability metrics• Translation problems• MT quality 8 texts selected
Participants
• 10 master’s students of translation (pass general translation exam)
• No previous experience post-editing• Reward: 2 coupons of 50 euros each
Session overview• Session 1:– Survey – LexTALE test (proficiency)– Warmup task– 2 texts HT, 2 texts PE
• Session 2:– Warmup task– 2 texts PE, 2 texts HT– Retrospection– Survey
Experimental setup: DesignParticipant P1 P3 P5 P7 P9 P2 P4 P6 P8 P10
Session1 task1 PE_1 PE_8 PE_7 PE_6 PE_5 HT_1 HT_8 HT_7 HT_6 HT_5
task2 PE_2 PE_1 PE_8 PE_7 PE_6 HT_2 HT_1 HT_8 HT_7 HT_6
task3 HT_3 HT_2 HT_1 HT_8 HT_7 PE_3 PE_2 PE_1 PE_8 PE_7
task4 HT_4 HT_3 HT_2 HT_1 HT_8 PE_4 PE_3 PE_2 PE_1 PE_8
Session2 task5 HT_5 HT_4 HT_3 HT_2 HT_1 PE_5 PE_4 PE_3 PE_2 PE_1
task6 HT_6 HT_5 HT_4 HT_3 HT_2 PE_6 PE_5 PE_4 PE_3 PE_2
task7 PE_7 PE_6 PE_5 PE_4 PE_3 HT_7 HT_6 HT_5 HT_4 HT_3
task8 PE_8 PE_7 PE_6 PE_5 PE_4 HT_8 HT_7 HT_6 HT_5 HT_4
Tools
• EyeLink (eye-tracking)• Casmacat (keystrokes + compatibility EyeLink)• Inputlog (keystrokes + logging of external
resources)
Data
• 80 sessions– 40 HT, 40 PE– 10 for each text– 8 for each translator
• Keystroke logging (Casmacat + Inputlog) & eye-tracking
• Pre- & post surveys• Annotation of problems
Overview
• Experimental setup• Process
– Speed– Translation units– Cognitive load– Gaze behavior– External resources
• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources
Translation speed
523.43 ms (± 202.14 standard error) more in HT
Production units
0,1 PU (± 0,019 se) more per ST token in HT 1295ms (± 227 se) more per PU in HT
Cognitive load
• Post-task survey: Which translation method was most tiring?–5 respondents: HT and PE equally tiring–1 respondent: PE most tiring–4 respondents: HT most tiring
Cognitive load
• What makes HT so tiring?– “Having to start from nothing”– “Insecurity, I need to double-check everything because
there’s no basic structure as with PE”– “Looking up synonyms, finding the correct words”
• What makes PE so tiring?– “Letting go of the MT output”– “Noticing errors in the MT output”– “Making sure the structure is still okay after fixing one
part of the sentence”
Gaze behavior: duration
Longer fixations on TT (cognitive load?)Difference greatest for HT (harder to understand MT
without ST? No effect found for MT quality)
Gaze behaviour: # fixations
More fixations on TT Difference greatest for PE (MT output? No effect found for MT
quality)
Gaze behaviour: # fixations ST and TT
ST: 2,3 (± 0,4 se) more fixations per ST token in HTTT: 1,3 (± 0,5 se) fewer fixations per ST token in HT
External resources
External resources
297.46 ms (± 105.36 standard error) more time in HTNo significant effect of MT quality
Overview
• Experimental setup• Process
– Speed– Translation units– Cognitive load– Gaze behavior– External resources
• Product– Acceptability– Adequacy – Influence of MT on PE quality– Overall quality: Influence of external resources
Quality: acceptability• Task in itself no significant predictor• Participant * Task: R²=0,44, p<0,001
Quality: adequacy
• Task in itself no significant predictor• Participant * Task no significant predictor• Post-editing adequacy error score: negatively
influenced by MT acceptability error score
PE adequacy ~ MT acceptability
Having to solve many acceptability issues might make students more aware of adequacy issues as well.
Overall quality: influence of external resources
Strategies used when translating not always successful when post-editing
Conclusions• PE faster than HT• More & longer production units in HT• HT perceived as more tiring• Longer fixations on TT (more so in HT)• More fixations on TT (more so in PE)• HT: more fixations ST, PE: more fixations TT• More time in external resources in HT• Acceptability more dependent on participant than task• PE adequacy influenced by MT acceptability• Consulting external resources more effective in HT
In sum
PE is faster than HT, cognitively less demanding, and requires less usage of external resources to obtain a product of comparable quality.
But, students need to be trained to better use external resources when post-editing.
Future research
• More fine-grained analysis:– Segment level– Comparison per participant– Comparison per text– Problem-solving strategies– Difficult passages
• Repeat with professional translators
