Embed Size (px)
Citation preview
Lessons Learned Using Latent Semantic Analysis to Reduce Researcher Bias in Back Translation of
Instruments in American Sign Language
Jared A. Embree Kyle Grove
Kathy Taylor Josephine Wilson
Wright State University
The Substance Abuse Resources and Disability Issues(SARDI) Program, Boonshoft School of Medicine
Instruments being translated and validated
1. Alcohol Use Disorders Identification Test (AUDIT) 2. Beck Depression Inventory(BDI) 3. Drug Abuse Screening Test (DAST) 4. Patient Health Questionnaire (PHQ-(9) 5. Generalized Anxiety Disorder Assessment (GAD-7) 6. Rosenberg Self-Esteem Scale (RSES) 7. The Satisfaction with Life Scale (SWLS) 8. The O*NET Short Form Interest Profiler, Work Importance
Profiler, and Ability Profiler, 9. Global Appraisal of Individual Needs Short Screener
(GAIN-SS)
CHALLENGESFORASLINSTRUMENTATION
1. SpecializedinstrumentsinAmericanSignLanguage(ASL)arescarce.
2. ValidatedversionsofASLinstrumentsaremostrare.3. Thiscontributestoanincreasedlikelihoodoferrorsinscreeningandcoststimeandmoneyforproviders.
4. Despiteeffortsataddressingthisdeficiency,theprocessislaboriousandsometimeshasinconsistentresults.
5. Eventhebesteffortshasbeentoolaborintensivetoscaleupinwaysthatcanaddresstheissue.
Funding for this program was made possible by a grant (90DP0067) from the National Institute on Disability, Independent Living, and
Rehabilitation Research (NIDILRR).
StandardProcess:ForwardTranslationEnglish versions of instruments were translated, in a process called forward translation, into ASL gloss by a team of Deaf experts and interpreters. Upon completion of the forward translation, another team of ASL experts, both Deaf and Hearing, recorded a video of each question in ASL. During this part of the process, ASL grammar and structure are fine-tuned.
StandardProcess:BackTranslation
Individuals (n = 30), both Deaf and Children of Deaf Adults (CODA), participated in back translation. Bi-lingual Deaf and CODAs were chosen because their native language is ASL in addition to having English fluency.
ASL Translation Process: Cognitive Interviewing and Field Testing
• Cognitive interviews were conducted in individual meetings with 20 deaf adults to get their feedback on the signed items of each instrument being validated. Individuals watched the video clip of each item and then provided feedback on the clarity of the signing and their understanding of the meaning of each item.
▪ Field testing for each instrument completed online by 300 VR and 150 non VR
Deaf individuals (each compensated for their time).
▪ Field testing doesn’t occur until we satisfactorily complete: ▪ Forward Translation, ▪ Back Translation, ▪ Back Translation Reconciliation, ▪ Cognitive Interviewing.
StandardProcess:SME-GuidedReconciliation
English-ASL bilingual subject matter experts directly assess similarity of meaning between original and back-translated text. Process is subject to low inter-rater reliability.
ComparisonofProcesses
Englishbacktranslation
Englishsource
ASLtranslation
manualtranslation manualtranslation
STSscoring
Englishbacktranslation
Englishsource
ASLtranslation
manualtranslation manualtranslation
SMEsmanualreconciliation
SMEsmanualreconciliation
StandardProcess ProposedProcess
ProposedProcess:AutomaticReconciliation
We utilize Semantic Textual Similarity (STS) scoring (https://www-nlp.stanford.edu/wiki/STS) to identify degree of similarity of semantic content between original and back-translated text.
ProposedProcess:AutomaticReconciliation
STS scoring utilizes a method called Latent Semantic Analysis to represent the semantics of a text as a mathematical matrix. The distance between two matrices is a measure of dissimilarity between the respective semantic content of the two texts.
Englishbacktranslation
Englishsource
cosinesimilarity
[0.2,0.9,-2.3][0.1,0.3,-1.8]
[0.1,1.9,-1.3][0.1,1.6,-1.8]
LSA LSA
ProposedProcess:AutomaticReconciliation
We use the STS score between source and back-translated text, as provided by the online SimService portal to measure the quality of the translation process.
Englishbacktranslation
Englishsource
ASLtranslation
manualtranslation manualtranslation
STSscoring
SMEsreconciliation
Mean:0.6854Range:0.12010–0.9137
Results:example:“IfeelIdonothavemuchtobeproudof.”
FutureSteps
• Conduct follow-on analysis of inter-rater reliability and method reliability.
• Develop intuitive analytics for SMEs to use when evaluating the results from back-translation efforts.
• Develop online tool for users to upload their own content to be back-translated by a community of bilingual raters.
References
SimService (http://swoogle.umbc.edu/SimService/) Kashyap et al. "Robust semantic text similarity using LSA, machine learning, and linguistic resources." Language Resources and Evaluation (2015): 1-37.
