1 Cross-cultural surveys and translation ESRC Question Bank Conference SURVEY MEASUREMENT: ASSESSING...

1

Cross-cultural surveys and translation

ESRC Question Bank ConferenceSURVEY MEASUREMENT: ASSESSING THE

RELIABILITY AND VALIDITY OF QUESTIONNAIRE ITEMS

April 10Royal Statistical Society

Janet A Harkness

gesis-ZUMA

gesis-ZUMA 2

Outline

I. FrameworkII. Progress in survey translation

production and assessmentIII. Persistent problems & consequencesIV. Towards solutions

gesis-ZUMA 3

I. Framework

1. Survey translation uses2. Expectations3. Current practice

gesis-ZUMA 4

1. Survey translations uses

Within country research Groups may partially share larger

context Shared contexts, degree of

interaction, immigrant acculturation affect translations required

Across country research Different contexts and languages

gesis-ZUMA 5

2. Expectations for survey translations

Assumption is a good translation asks the same question

maintains semantic & pragmatic meaning maintains measurement properties

retains source design features satisfies multiple other requirements

(askable and answerable, burden, saliency, etc)

gesis-ZUMA 6

3. Common current practice

Depending on discipline Do not assess translation process quality Assess translation quality through back

translation Pretest translated questions Assess translated instrument quality on

basis of statistical analysis (dif, IRT)

gesis-ZUMA 7

II. Progress in survey translation production and assessment

gesis-ZUMA 8

Progress in survey translation production and assessment

1. Guidelines and know-how2. Tools3. Basic research

Procedures, strategies, outcomes

gesis-ZUMA 9

Progress in survey translation production and assessment

1. Guidelines and know-how2. Tools3. Basic research4. Procedures and strategies

gesis-ZUMA 10

1. Guidelines and Know-How

gesis-ZUMA 11

1. Guidelines and Know-How

ESS, USA Census Bureau, International Test Commission, QoL research and publications

Comprehensive sets of guidelines CSDI Workshop Guidelines Initiative (lead groups at ISR, Michigan, UNL and gesis-ZUMA)

gesis-ZUMA 12

2. Tools

gesis-ZUMA 13

2. Tools

Process documents Note-taking templates

Queries, decisions, rationales Harmonization templates

Decisions, rationales Version records

Decisions, rationales

gesis-ZUMA 14

2. Tools Technological options

Memory files -- repeated questions, instructions, answer scales …

Translation software support Parallel presentation of source and target

versions Importing modifications to existing text

Potential do-it-all tools Combine questionnaire production-documentation

with translation version production-documentation

gesis-ZUMA 15

3. Basic research

gesis-ZUMA 16

3. Basic research

Survey translation research on Assessment procedures & outcomes Translation procedures & outcomes Answer scale translation Oral translation and interpreting Tool options Impact of source questions

gesis-ZUMA 17

3. Basic research

Survey translation research on Assessment procedures & outcomes Translation procedures, strategies

& outcomes Answer scale translation Oral translation and interpreting Tool options Impact of source questions

gesis-ZUMA 18

Procedures, strategies, outcomes

Team translation efforts Interdisciplinary expertise Translators, reviewers, adjudicators,

consultants Iterative process

gesis-ZUMA 19

Team Translation

TRAPD model --an iterative cycleTranslationReviewAdjudicationPretesting and refinementDocumentation underpins all stages(cf. frameworks in ESS, SHARE, and US Bureau of Census, Westat, WMHI)

gesis-ZUMA 20

Why iterative TRAPD procedures may need to be repeated at different stages. For example, pre-testing and debriefing sessions with fielding staff and respondents will lead to revisions; these call for further testing of revised translations.

gesis-ZUMA 21

Translate and document

Review, adjudicateand document

SOURCEPretest and document,reiterate if necessary

gesis-ZUMA 22

Basic Team Players

Translators: selected, competent, briefed

Reviewers: selected, competent, briefed

Adjudicator: (takes final decision) selected as possible on basis of skills as well as seniority.

May need to work with Consultant.

gesis-ZUMA 23

Other Team Players Translators: Reviewers: Adjudicator-------------- (Co-ordinator) (Substantive experts) (External assessors)Copy-editors (Programmers)Back-ups (illness, vacation, leave) (Oral translation and interpreting extra)

gesis-ZUMA 24

Basic Procedures

Translators translate Review session: reviewers discuss and review each question Adjudicator decides/signs off

consults with senior reviewer; if sensible/possible also joins in review session(s).

gesis-ZUMA 25

Scenes from a review session …

gesis-ZUMA 26

The team

Senior reviewer, co-adjudicator Translator 1 Translator 2 Project coordinator, survey researcher Pre-tester, survey researcher Survey researcher, lay translation

talent

gesis-ZUMA 27

Clip 1: The meeting begins Four in room discussing the weather Paul and Margrit enter Greetings Seating Framing the session

Paul: I’ve got a parking space till 5.06 pm. Janet: …Yes, we need to finish by five.

gesis-ZUMA 28

gesis-ZUMA 29

Clip 2: reaching decisions

Team has been trying at length to find a phrase that includes atheist views on religion.Translations proposed so far imply people do have religious views, rather than just a view about religion (so the clip begins)

gesis-ZUMA 30

Clip 2 events

Discussion ongoing Janet halts discussion (hand motion) and

identifies the continuing problem Peter makes a new proposal Team consider it and accept Adjustments to other text discussed,

approval re-confirmed The core group takes notes

gesis-ZUMA 31

JH Let’s recapitulate. There’s the problem that some have no faith and no religion … our discussion … solves perhaps what we could do for Muslims and Protestants or among Protestant groups, but does not really solve the other problem.

PM Well, there’s an odd word in German “glaubensvorstellungen” that means other views from those you /they have

JH Ah.. And would that cover atheists too?

.

gesis-ZUMA 32

PM Yes, of course, they have a view about faith, namely none (= without faith)

MR And the atheists are covered

JH Good. Then we’ll take that. Then we have everything

MR And Muslims are also covered

JH Good

(More discussion of how good the solution is andthat earlier text should be modified)

.

gesis-ZUMA 33

gesis-ZUMA 34

What we learn from teams

Language challenges Source question issues Strategies Task knowledge available and needed

gesis-ZUMA 35

Quality improvements considerable, but…

gesis-ZUMA 36

III. Persistent problems

gesis-ZUMA 37

III. Persistent problems

1. Views on translation and what it can do

2. Established common practices3. Nature of questionnaires4. Researchers and good questions

gesis-ZUMA 38

1. Views on translation & what it can do

i. Anyone can translateii. Focus on wordsiii. Neglect/avoidance of adaptation

gesis-ZUMA 39

i) Anyone can translate

My secretary speaks Czech My son studies French Harry spent a year in Turkey

Language ability is not a guarantee for translation ability

gesis-ZUMA 40

ii) Focus on words

Have you felt blue or down recently?

gesis-ZUMA 41

ii) Focus on words

Have you felt blue or down recently? NOT a matter of colour or idiom Temporary state Degree of depressed state [Disclosure] [Symptom relevance]

gesis-ZUMA 42

iii) Neglect/avoidance of Adaptation

gesis-ZUMA 43

Adaptation

Deliberate modification of a question or questionnaire to meet new requirements

Frequently but not necessarily associated with translation

Undertaken in source questions and/or translated questions

Various forms (cf. Harkness 2008)

gesis-ZUMA 44

Adaptation

Do you have difficulty walking several blocks?

Do you have difficulty walking 100 yards?

Do you have difficulty walking 100 metres?

Do you have difficulty walking 200 metres? (Sweden)

gesis-ZUMA 45

Adaptation and Design are related

Measurement properties should remain Intended latent construct should remain Burden or difficulty should not change Relationship to other questionnaire

elements should not change Adaptation can be anticipated in source

instruments

gesis-ZUMA 46

Adaptation and Translation are related

Translation involves adaptation (Adaptation need not involve translation) In the context of translation, some general

types of adaptation can be identified Blends/entanglement of different

types not unusual

gesis-ZUMA 47

2. Established practices as persistent problems

gesis-ZUMA 48

2. Established practices as persistent problems

Back translation False economy Horse-and-cart structures in survey

implementation Horse and Cart are essentially

different and distinct Perceived interdependence minimal

gesis-ZUMA 49

3. Nature of questionnaires as persistent problem

gesis-ZUMA 50

3. Nature of questionnaires

Complicated text type Leads a double life

Covert measurement properties

Surveyspeak and scalespeak

gesis-ZUMA 51

3. Nature of questionnaires

Complicated text type Leads a double life

Covert measurement properties

Surveyspeak and scalespeak

gesis-ZUMA 52

Surveyspeak

Dentist When did you last visit a dentist?

Patient Dentist follow-up Are you sure?

Uh-huh.

I see.

gesis-ZUMA 53

Surveyspeak (2)Interviewer: When did you last visit a dentist?

Respondent: About two months ago.

Survey follow-upWould you say that you are very certain / somewhat certain / neither certain nor uncertain / somewhat uncertain or very uncertain about the date you just gave?

(or attempt to get date)

53

gesis-ZUMA 54

Scalespeak in an “importance” scale

very importantsomewhat importantneither important nor unimportantsomewhat unimportantvery unimportant

This is somewhat unimportant

This is very unimportant

This is somewhat important

gesis-ZUMA 55

4. Good questions (and researchers) as persistent problems

gesis-ZUMA 56

4. Good questions (and researchers) as persistent problems

What makes a question good?

Q measures what it should

Q understood as intended

Q salient for respondents

Q answerable and answered

Demonstrated quality through testing and use

gesis-ZUMA 57

Good questions "fit" their context

Validity and reliability are facilitated by common ground, shared speech community usage and social norms

Intended meaning of question and answer options = perceived meaning

gesis-ZUMA 58

A visual example of a good instrument

gesis-ZUMA 59

Chinese diagnostic doll

Cultural norms on disclosure observed to enable response

Patient remains clothed, doll is naked

Cultural norms on sensitive topics observed

Note: feet are covered

gesis-ZUMA 60

The diagnostic doll reflects and

accommodates the cultural embedding of the

instrument (doll), the researcher (doctor) and

the respondent (patient)

gesis-ZUMA 61

When "good" questions go travelling...

In different contexts, good questions may be poor cultural fits change in "meaning"

different conceptual coverage

socially difficult to ask or to answer

lose or gain saliency

gesis-ZUMA 62

When "good" questions go travelling...

In different contexts, good questions may be poor cultural fits

Translation may then be an inappropriate means to "ask the same question"

gesis-ZUMA 63

Consequences: what can go wrong ….and why

gesis-ZUMA 64

TRANSLATION

Translation (Spanish)We do not talk much about my illness athome

SourceCommunication about my illness at home is poor

Why? Remedy?

gesis-ZUMA 65

PERCEPTION

Translation (French)

Do you have difficulty standing for 2 hours?

SourceDo you have difficulty sitting for 2 hours?

Why? Remedy?

gesis-ZUMA 66

CULTURE or INTENDED MEANING

Translation (German)

Would you take part in a demonstration thatblocks the traffic?

SourceWould you take part in a demonstration?

Why? Remedy?

gesis-ZUMA 67

GLITCH

Translation (Turkish)

Have you ever felt like shooting someone

SourceHave you ever felt like hitting someone

Why? Remedy?

gesis-ZUMA 68

Translation process highlights design issues

Do you provide financial support for grown-up children or grandchildren?

How many hours TV do you watch on an average weekday?

Please give me the initials of your first and last name

Do you prefer OTC or prescription medicines?

gesis-ZUMA 69

Unavoidable design changesEnglish

neither satisfied nor dissatisfied

isiZulu and Hebrew

neither nor not ... not

dissatisfied not satisfied

gesis-ZUMA 70

Unavoidable design changesEnglish

neither satisfied nor dissatisfied

isiZulu and Hebrew "not satisfied and not not satisfied"

cf. Henningsson et al, 1998, Harkness et 2005cf. Henningsson et al, 1998, Harkness et 2005

gesis-ZUMA 71

IV Towards solutions:concluding remarks

gesis-ZUMA 72

IV Towards solutions

See the source as the source Source of information Source of challenges

Source needs to be appraised and tested for suitability for new contexts and languages

gesis-ZUMA 73

IV Towards solutions

See the target as the target Determine aims for target language

questions Appraise source question goals and

means for target realization Engage in deep processing

Target may need degrees of freedom

gesis-ZUMA 74

IV Towards solutions: more progress

Progress will be a process Base any change on evidence

Research and documentation Create critical mass

Collaboration in initiatives Record and share lessons learned

75

Thank you

gesis-ZUMA 76

Harkness, J. (2008) “Comparative Survey Research: Goals and Challenges.” Foundation chapter in: Dillman, D., Hox., J. and de Leeuw, E. (eds.) International Handbook of Survey Methodology, Hyattsville, VA: Erlbaum. Harkness, J., Schoebi, N., Joye, D., Mohler, P., Faass, T. and Behr, D. (2007) “Oral Translation in Telephone Surveys”. In: J.M. Lepkowski, C. Tucker, J.M. Brick, E. de Leeuw, L. Japec, P.J. Lavrakas, M.W. Link and R.L. Sangster. Advances in Telephone Survey Methodology, John Wiley and Sons, Hoboken, New Jersey.Harkness, J.; Pennell, B.-E., Schoua-Glusberg, A. (2004): Survey Questionnaire Translation and Assessment. In: Presser, Stanley, Rothgeb, Jennifer, Couper, Michael, Lessler, Judith, Martin, Elizabeth, and Singer, Eleanor (Eds.): Questionnaire Development Evaluation and Testing Methods, Wiley Series in Survey Methodology. New Jersey: John Wiley & Sons Inc.

Related Literature

gesis-ZUMA 77

Harkness, J. (2003): Questionnaire Translation. In: Harkness, Janet A., Van de Vijver, Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey Methods. Wiley Series in Survey Methodology. New Jersey: John Wiley and Sons Inc

Harkness, J.; Van de Vijver, F. J. R.; Johnson, T. P. (2003): Questionnaire Design in Comparative Research. In: Harkness, Janet A., Van de Vijver, Fons J.R., Mohler, Peter Ph.(Hrsg.): Cross-Cultural Survey Methods. Wiley Series in Survey Methodology. New Jersey: John Wiley and Sons Inc

Harkness, J.; Schoua-Glusberg, A. (1998): Questionnaires in Translation. In: Harkness, J. (Hrsg.): Cross-Cultural Survey Equivalence. ZUMA-Nachrichten Spezial Band 3. Mannheim: Zentrum für Umfragen, Methoden und Analysen 1998, S. 87-128