Upload
paul-doyle
View
220
Download
2
Tags:
Embed Size (px)
Citation preview
1
Cross-cultural surveys and translation
ESRC Question Bank ConferenceSURVEY MEASUREMENT: ASSESSING THE
RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS
April 10Royal Statistical Society
Janet A Harkness
gesis-ZUMA
gesis-ZUMA 2
Outline
I. FrameworkII. Progress in survey translation
production and assessmentIII. Persistent problems & consequencesIV. Towards solutions
gesis-ZUMA 3
I. Framework
1. Survey translation uses2. Expectations3. Current practice
gesis-ZUMA 4
1. Survey translations uses
Within country research Groups may partially share larger
context Shared contexts, degree of
interaction, immigrant acculturation affect translations required
Across country research Different contexts and languages
gesis-ZUMA 5
2. Expectations for survey translations
Assumption is a good translation asks the same question
maintains semantic & pragmatic meaning maintains measurement properties
retains source design features satisfies multiple other requirements
(askable and answerable, burden, saliency, etc)
gesis-ZUMA 6
3. Common current practice
Depending on discipline Do not assess translation process quality Assess translation quality through back
translation Pretest translated questions Assess translated instrument quality on
basis of statistical analysis (dif, IRT)
gesis-ZUMA 7
II. Progress in survey translation production and assessment
gesis-ZUMA 8
Progress in survey translation production and assessment
1. Guidelines and know-how2. Tools3. Basic research
Procedures, strategies, outcomes
gesis-ZUMA 9
Progress in survey translation production and assessment
1. Guidelines and know-how2. Tools3. Basic research4. Procedures and strategies
gesis-ZUMA 10
1. Guidelines and Know-How
gesis-ZUMA 11
1. Guidelines and Know-How
ESS, USA Census Bureau, International Test Commission, QoL research and publications
Comprehensive sets of guidelines CSDI Workshop Guidelines Initiative (lead groups at ISR, Michigan, UNL and gesis-ZUMA)
gesis-ZUMA 12
2. Tools
gesis-ZUMA 13
2. Tools
Process documents Note-taking templates
Queries, decisions, rationales Harmonization templates
Decisions, rationales Version records
Decisions, rationales
gesis-ZUMA 14
2. Tools Technological options
Memory files -- repeated questions, instructions, answer scales …
Translation software support Parallel presentation of source and target
versions Importing modifications to existing text
Potential do-it-all tools Combine questionnaire production-documentation
with translation version production-documentation
gesis-ZUMA 15
3. Basic research
gesis-ZUMA 16
3. Basic research
Survey translation research on Assessment procedures & outcomes Translation procedures & outcomes Answer scale translation Oral translation and interpreting Tool options Impact of source questions
gesis-ZUMA 17
3. Basic research
Survey translation research on Assessment procedures & outcomes Translation procedures, strategies
& outcomes Answer scale translation Oral translation and interpreting Tool options Impact of source questions
gesis-ZUMA 18
Procedures, strategies, outcomes
Team translation efforts Interdisciplinary expertise Translators, reviewers, adjudicators,
consultants Iterative process
gesis-ZUMA 19
Team Translation
TRAPD model --an iterative cycleTranslationReviewAdjudicationPretesting and refinementDocumentation underpins all stages(cf. frameworks in ESS, SHARE, and US Bureau of Census, Westat, WMHI)
gesis-ZUMA 20
Why iterative TRAPD procedures may need to be repeated at different stages. For example, pre-testing and debriefing sessions with fielding staff and respondents will lead to revisions; these call for further testing of revised translations.
gesis-ZUMA 21
Translate and document
Review, adjudicateand document
SOURCEPretest and document,reiterate if necessary
gesis-ZUMA 22
Basic Team Players
Translators: selected, competent, briefed
Reviewers: selected, competent, briefed
Adjudicator: (takes final decision) selected as possible on basis of skills as well as seniority.
May need to work with Consultant.
gesis-ZUMA 23
Other Team Players Translators: Reviewers: Adjudicator-------------- (Co-ordinator) (Substantive experts) (External assessors)Copy-editors (Programmers)Back-ups (illness, vacation, leave) (Oral translation and interpreting extra)
gesis-ZUMA 24
Basic Procedures
Translators translate Review session: reviewers discuss and review each question Adjudicator decides/signs off
consults with senior reviewer; if sensible/possible also joins in review session(s).
gesis-ZUMA 25
Scenes from a review session …
gesis-ZUMA 26
The team
Senior reviewer, co-adjudicator Translator 1 Translator 2 Project coordinator, survey researcher Pre-tester, survey researcher Survey researcher, lay translation
talent
gesis-ZUMA 27
Clip 1: The meeting begins Four in room discussing the weather Paul and Margrit enter Greetings Seating Framing the session
Paul: I’ve got a parking space till 5.06 pm. Janet: …Yes, we need to finish by five.
gesis-ZUMA 28
gesis-ZUMA 29
Clip 2: reaching decisions
Team has been trying at length to find a phrase that includes atheist views on religion.Translations proposed so far imply people do have religious views, rather than just a view about religion (so the clip begins)
gesis-ZUMA 30
Clip 2 events
Discussion ongoing Janet halts discussion (hand motion) and
identifies the continuing problem Peter makes a new proposal Team consider it and accept Adjustments to other text discussed,
approval re-confirmed The core group takes notes
gesis-ZUMA 31
JH Let’s recapitulate. There’s the problem that some have no faith and no religion … our discussion … solves perhaps what we could do for Muslims and Protestants or among Protestant groups, but does not really solve the other problem.
PM Well, there’s an odd word in German “glaubensvorstellungen” that means other views from those you /they have
JH Ah.. And would that cover atheists too?
.
gesis-ZUMA 32
PM Yes, of course, they have a view about faith, namely none (= without faith)
MR And the atheists are covered
JH Good. Then we’ll take that. Then we have everything
MR And Muslims are also covered
JH Good
(More discussion of how good the solution is andthat earlier text should be modified)
.
gesis-ZUMA 33
gesis-ZUMA 34
What we learn from teams
Language challenges Source question issues Strategies Task knowledge available and needed
gesis-ZUMA 35
Quality improvements considerable, but…
gesis-ZUMA 36
III. Persistent problems
gesis-ZUMA 37
III. Persistent problems
1. Views on translation and what it can do
2. Established common practices3. Nature of questionnaires4. Researchers and good questions
gesis-ZUMA 38
1. Views on translation & what it can do
i. Anyone can translateii. Focus on wordsiii. Neglect/avoidance of adaptation
gesis-ZUMA 39
i) Anyone can translate
My secretary speaks Czech My son studies French Harry spent a year in Turkey
Language ability is not a guarantee for translation ability
gesis-ZUMA 40
ii) Focus on words
Have you felt blue or down recently?
gesis-ZUMA 41
ii) Focus on words
Have you felt blue or down recently? NOT a matter of colour or idiom Temporary state Degree of depressed state [Disclosure] [Symptom relevance]
gesis-ZUMA 42
iii) Neglect/avoidance of Adaptation
gesis-ZUMA 43
Adaptation
Deliberate modification of a question or questionnaire to meet new requirements
Frequently but not necessarily associated with translation
Undertaken in source questions and/or translated questions
Various forms (cf. Harkness 2008)
gesis-ZUMA 44
Adaptation
Do you have difficulty walking several blocks?
Do you have difficulty walking 100 yards?
Do you have difficulty walking 100 metres?
Do you have difficulty walking 200 metres? (Sweden)
gesis-ZUMA 45
Adaptation and Design are related
Measurement properties should remain Intended latent construct should remain Burden or difficulty should not change Relationship to other questionnaire
elements should not change Adaptation can be anticipated in source
instruments
gesis-ZUMA 46
Adaptation and Translation are related
Translation involves adaptation (Adaptation need not involve translation) In the context of translation, some general
types of adaptation can be identified Blends/entanglement of different
types not unusual
gesis-ZUMA 47
2. Established practices as persistent problems
gesis-ZUMA 48
2. Established practices as persistent problems
Back translation False economy Horse-and-cart structures in survey
implementation Horse and Cart are essentially
different and distinct Perceived interdependence minimal
gesis-ZUMA 49
3. Nature of questionnaires as persistent problem
gesis-ZUMA 50
3. Nature of questionnaires
Complicated text type Leads a double life
Covert measurement properties
Surveyspeak and scalespeak
gesis-ZUMA 51
3. Nature of questionnaires
Complicated text type Leads a double life
Covert measurement properties
Surveyspeak and scalespeak
gesis-ZUMA 52
Surveyspeak
Dentist When did you last visit a dentist?
Patient Dentist follow-up Are you sure?
Uh-huh.
I see.
gesis-ZUMA 53
Surveyspeak (2)Interviewer: When did you last visit a dentist?
Respondent: About two months ago.
Survey follow-upWould you say that you are very certain / somewhat certain / neither certain nor uncertain / somewhat uncertain or very uncertain about the date you just gave?
(or attempt to get date)
53
gesis-ZUMA 54
Scalespeak in an “importance” scale
very importantsomewhat importantneither important nor unimportantsomewhat unimportantvery unimportant
This is somewhat unimportant
This is very unimportant
This is somewhat important
gesis-ZUMA 55
4. Good questions (and researchers) as persistent problems
gesis-ZUMA 56
4. Good questions (and researchers) as persistent problems
What makes a question good?
Q measures what it should
Q understood as intended
Q salient for respondents
Q answerable and answered
Demonstrated quality through testing and use
gesis-ZUMA 57
Good questions "fit" their context
Validity and reliability are facilitated by common ground, shared speech community usage and social norms
Intended meaning of question and answer options = perceived meaning
gesis-ZUMA 58
A visual example of a good instrument
gesis-ZUMA 59
Chinese diagnostic doll
Cultural norms on disclosure observed to enable response
Patient remains clothed, doll is naked
Cultural norms on sensitive topics observed
Note: feet are covered
gesis-ZUMA 60
The diagnostic doll reflects and
accommodates the cultural embedding of the
instrument (doll), the researcher (doctor) and
the respondent (patient)
gesis-ZUMA 61
When "good" questions go travelling...
In different contexts, good questions may be poor cultural fits change in "meaning"
different conceptual coverage
socially difficult to ask or to answer
lose or gain saliency
gesis-ZUMA 62
When "good" questions go travelling...
In different contexts, good questions may be poor cultural fits
Translation may then be an inappropriate means to "ask the same question"
gesis-ZUMA 63
Consequences: what can go wrong ….and why
gesis-ZUMA 64
TRANSLATION
Translation (Spanish)We do not talk much about my illness athome
SourceCommunication about my illness at home is poor
Why? Remedy?
gesis-ZUMA 65
PERCEPTION
Translation (French)
Do you have difficulty standing for 2 hours?
SourceDo you have difficulty sitting for 2 hours?
Why? Remedy?
gesis-ZUMA 66
CULTURE or INTENDED MEANING
Translation (German)
Would you take part in a demonstration thatblocks the traffic?
SourceWould you take part in a demonstration?
Why? Remedy?
gesis-ZUMA 67
GLITCH
Translation (Turkish)
Have you ever felt like shooting someone
SourceHave you ever felt like hitting someone
Why? Remedy?
gesis-ZUMA 68
Translation process highlights design issues
Do you provide financial support for grown-up children or grandchildren?
How many hours TV do you watch on an average weekday?
Please give me the initials of your first and last name
Do you prefer OTC or prescription medicines?
gesis-ZUMA 69
Unavoidable design changesEnglish
neither satisfied nor dissatisfied
isiZulu and Hebrew
neither nor not ... not
dissatisfied not satisfied
gesis-ZUMA 70
Unavoidable design changesEnglish
neither satisfied nor dissatisfied
isiZulu and Hebrew "not satisfied and not not satisfied"
cf. Henningsson et al, 1998, Harkness et 2005cf. Henningsson et al, 1998, Harkness et 2005
gesis-ZUMA 71
IV Towards solutions:concluding remarks
gesis-ZUMA 72
IV Towards solutions
See the source as the source Source of information Source of challenges
Source needs to be appraised and tested for suitability for new contexts and languages
gesis-ZUMA 73
IV Towards solutions
See the target as the target Determine aims for target language
questions Appraise source question goals and
means for target realization Engage in deep processing
Target may need degrees of freedom
gesis-ZUMA 74
IV Towards solutions: more progress
Progress will be a process Base any change on evidence
Research and documentation Create critical mass
Collaboration in initiatives Record and share lessons learned
75
Thank you
gesis-ZUMA 76
Harkness, J. (2008) “Comparative Survey Research: Goals and Challenges.” Foundation chapter in: Dillman, D., Hox., J. and de Leeuw, E. (eds.) International Handbook of Survey Methodology, Hyattsville, VA: Erlbaum. Harkness, J., Schoebi, N., Joye, D., Mohler, P., Faass, T. and Behr, D. (2007) “Oral Translation in Telephone Surveys”. In: J.M. Lepkowski, C. Tucker, J.M. Brick, E. de Leeuw, L. Japec, P.J. Lavrakas, M.W. Link and R.L. Sangster. Advances in Telephone Survey Methodology, John Wiley and Sons, Hoboken, New Jersey.Harkness, J.; Pennell, B.-E., Schoua-Glusberg, A. (2004): Survey Questionnaire Translation and Assessment. In: Presser, Stanley, Rothgeb, Jennifer, Couper, Michael, Lessler, Judith, Martin, Elizabeth, and Singer, Eleanor (Eds.): Questionnaire Development Evaluation and Testing Methods, Wiley Series in Survey Methodology. New Jersey: John Wiley & Sons Inc.
Related Literature
gesis-ZUMA 77
Harkness, J. (2003): Questionnaire Translation. In: Harkness, Janet A., Van de Vijver, Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey Methods. Wiley Series in Survey Methodology. New Jersey: John Wiley and Sons Inc
Harkness, J.; Van de Vijver, F. J. R.; Johnson, T. P. (2003): Questionnaire Design in Comparative Research. In: Harkness, Janet A., Van de Vijver, Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey Methods. Wiley Series in Survey Methodology. New Jersey: John Wiley and Sons Inc
Harkness, J.; Schoua-Glusberg, A. (1998): Questionnaires in Translation. In: Harkness, J. (Hrsg.): Cross-Cultural Survey Equivalence. ZUMA-Nachrichten Spezial Band 3. Mannheim: Zentrum für Umfragen, Methoden und Analysen 1998, S. 87-128