1. Kotaro Hara and Shamsi Iqbal makeability lab Effect of Machine Translation in Interlingual Conversation 2. 3. 4. Hello! 5. Successful intercultural communication is important in: Education Business Health-care Yet, a language barrier can be a challenge 6. 133/2011 euroscola by rosipaw 7. Simultaneous interpretation is only for privileged few 8. The infrequent use of interpreters* in the delivery ward was among the most important reasons for the reduced quality of [health] care. Kale and Syed (2010) * Interpreter: a person who translates the words that someone is speaking into a different language (Merriam-Webster) 9. Automatic Spoken Language Translation 10. Speech Recognition & Machine Translation Performance Fgen et al. (2007) noted that automatic systems can already provide usable information for people. 2015 11. 2015 Little research has paid attention to how people interact with spoken language translation technology 12. Very little knowledge about Spoken Language Translation in HCI Jigsaw puzzle by James Petts 13. The void that we are trying to fill Very little knowledge about Spoken Language Translation in HCI 14. Background 15. Hamon O., et al., End-to-End Evaluation in Simultaneous Translation, ECACL2009 Hello! Speech Text Spoken Language Translation System Spoken Language Translation Automatic speech recognition Machine translation Speech synthesis a 16. Hamon O., et al., End-to-End Evaluation in Simultaneous Translation, ECACL2009 Hello! Speech Text Spoken Language Translation System Spoken Language Translation Automatic speech recognition Machine translation Speech synthesis a 17. Danger. Do not enter Effect of machine translation on text-based communication Gao et al. 2014; Yamashita et al. 2009; Yamashita and Ishida 2006 18. Hello! Native English SpeakerNon-native English Speaker Hello! Effect of speech recognition on interlingual conversation Gao et al. CHI2014 19. Key Point Speech Recognition Text Output Machine Translation 20. Effect of spoken language translation system to interlingual communication is under-explored. Key Point 21. Evaluation of NESPOLE! Constantini et al. 2002 No detailed discussion on how people adapted in using the system Push-to-talk interface made it hard to analyze turn-taking behavior Examined usability of spoken language translation system 22. Explore how spoken language translation affects a natural conversation between people speaking different languages Goal 23. Translator Tool Study Method Quantitative Analysis Content Analysis 24. Translator Tool 25. Video to show how the system work: One side speaks in English, another side in German Translator Tool | Demo Skype Translator Demo, English-German Conversation 26. Video to show how the system work: One side speaks in English, another side in German Translator Tool | Demo 27. Video to show how the system work: One side speaks in English, another side in German Video chat interface Translator Tool | Interface Components 28. Video to show how the system work: One side speaks in English, another side in German Closed Caption (CC) Translator Tool | Interface Components 29. Video to show how the system work: One side speaks in English, another side in German Translated Text-to-Speech (TTS) Audio Translator Tool | Interface Components 30. Video to show how the system work: One side speaks in English, another side in German Headset Microphone Translator Tool | Interface Components 31. Translator Tool Study Method Quantitative Analysis Content Analysis 32. Translator Tool Study Method Quantitative Analysis Content Analysis 33. Study Method | Participants 8 French-German pairs (N=16 participants) 15 English-German pairs (N=30 participants) 34. Study Method | Participants 8 French-German pairs (N=16 participants) 15 English-German pairs (N=30 participants) No common language With common language (English) All participants could speak English reasonably well. 35. Study Method | Participants 8 French-German pairs (N=16 participants) 15 English-German pairs (N=30 participants) With common language (English) I will mainly talk about this study No common language 36. Study Method | Conversation Task Olivia trainierte fur den Tanzwettbewerb. (Olivia was practicing for the dance-off) Conversation Task German Speaker French Speaker 37. A starting sentence was provided to the participant Study Method | Conversation Task Olivia trainierte fur den Tanzwettbewerb. (Olivia was practicing for the dance-off) Conversation Task German Speaker French Speaker Translate 38. Study Method | Conversation Task Olivia trainierte fur den Tanzwettbewerb. (Olivia was practicing for the dance-off) Je ne savais pas Olivia dans. Combien de temps elle s'y adonne ? (I didnt know Olivia danced. How long has she been practicing?) Conversation Task German Speaker Translate French Speaker 39. Study Method | Conversation Task Olivia trainierte fur den Tanzwettbewerb. (Olivia was practicing for the dance-off) Seit sechs Jahren (For six years) C'est une longue priode. (Thats a long time.) Conversation Task German Speaker French Speaker Je ne savais pas Olivia dans. Combien de temps elle s'y adonne ? (I didnt know Olivia danced. How long has she been practicing?) 40. We asked participants to perform 9 conversation tasks (~3 min ea.) in their respective languages and an additional conversation task in English The goal of each task was to collaboratively construct a coherent story Study Method | Conversation Task Tasks were conducted using three different settings of the translator tool Conversation Task 41. Study Method | Interface Settings Closed Caption & Text-to-Speech (CC & TTS) CC Closed Caption only (CC) CC Text-to-Speech only (TTS) Interface Settings 42. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech With Closed Caption and Text-to-Speech CC CC 43. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC 44. Closed Caption only Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC 45. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC 46. Text-to-Speech only Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC 47. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC 48. Closed Caption and Text-to-Speech CC Closed Caption CC Text-to-Speech Round 1 Study Method | Interface Settings Interface Settings Order was permuted for each pair for counter balancing 49. CC CCRound 1 Study Method | Interface Settings Interface Settings CC CC CC CC Round 2 Round 3 9 tasks with different interface settings and different starting sentences 50. CC CCRound 1 Study Method | Interface Settings Interface Settings CC CC CC CC Round 2 Round 3 Eng A baseline English task to compare with/without translation settings 51. CC CCRound 1 Study Method | Data Data CC CC CC CC Round 2 Round 3 Post-task survey (after every task) E.g., We had a successful conversation. Strongly Agree Strongly Disagree Eng 52. Eng CC CCRound 1 Study Method | Data Data CC CC CC CC Round 2 Round 3 Post-round survey (after every 3 tasks) Interface setting preference ranking CC & TTS: __________, CC: __________, TTS: __________Best Neutral Worst 53. CC CCRound 1 Study Method | Data Data CC CC CC CC Round 2 Round 3 Eng Post-session survey and Interview 54. Translator Tool Study Method Quantitative Analysis Content Analysis 55. Translator Tool Study Method Quantitative Analysis Content Analysis Overall interface setting preferences Whether people became used to using the translator tool 56. Three interface settings Three rounds (nine tasks) Two languages 3 x 3 x 2 mixed design study With-in subject With-in subject Between subject Quantitative Analysis | Analysis Method Analysis Method 57. Transformed ordinal data in survey responses with aligned rank transformation We analyzed the data with restricted maximum likelihood model Quantitative Analysis | Analysis Method Analysis Method 58. Interface Preference Results Quantitative Analysis | Interface Preference Results Which interface setting did you favor the most and least? Closed Caption & Text-to-SpeechClosed Caption only Text-to-Speech only CC CC 59. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group CC CC Interface Settings Percentage of people who favored the interface setting 60. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group Interface settings with closed caption were preferred CC CC 61. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group CC CC 24% more people preferred to have text-to-speech 62. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group CC CC Set the Closed Caption & Text-to-Speech to the default, but allow users to turn on/off setting 63. 2.2 2.2 2.7 4.8 1 2 3 4 5 Round 1 Round 2 Round 3 English Overall Conversation Quailty Perceived Conversation Quality over Rounds Round Average 5-point Likert scale rating Quantitative Analysis | Perceived Conversation Quality 5-point Likert scale rating (higher is better) Rating 64. 2.2 2.2 2.7 4.8 1 2 3 4 5 Round 1 Round 2 Round 3 English Overall Conversation Quailty p < 0.01 p < 0.01 Participants felt that their conversation quality improved Perceived Conversation Quality over Rounds Quantitative Analysis | Perceived Conversation Quality F2,112=6.275, p