Gile SURVEYS 1

SURVEYS IN TSSOME PRINCIPLES AND A CASE STUDY

daniel.gile@yahoo.com

www.cirinandgile.com

Gile SURVEYS 2

WHY SURVEYS?

IN EMPIRICAL RESEARCH“OBJECTIVE” DESCRIPTION AND MEASUREMENT

CANNOT COVER EFFICIENTLY ALL RELEVANT PHENOMENA

IN PARTICULAR:THOUGHTS, MOTIVATIONS, OPINIONS, INTENTIONSCANNOT BE OBSERVED OR MEASURED DIRECTLY

INDIRECT EXPERIMENTAL PROCEDURES ON INDICATORS ARE COMPLEX AND SOMETIMES

TRICKY

ASKING PEOPLE IS ANOTHER OPTION

Gile SURVEYS 3

CAUTION

PEOPLE’S UTTERANCES ARE NOT ALWAYS

TRUTHFUL OR ACCURATE

THEY ARE NOT NECESSARILY EASY TO INTERPRET

BUT THEY ARE OFTEN A USEFUL COMPONENT OF RESEARCH FOR:

- PRELIMINARY EXPLORATION

- COMPLEMENTARY COLLECTION OF INFORMATION

- VALIDATION OF INFORMATION OBTAINED THROUGH OTHER METHODS

RELEVANCE TO RESEARCH INTO TRANSLATION IS OBVIOUS BECAUSE OF:

COMPLEX PROCESSING IS PART OF TRANSLATION, ONLY PRODUCT IS VISIBLE, CORPORA ARE DIFFICULT TO OBTAIN, INTERPRETATION OF INDICATORS IS DIFFICULT

Gile SURVEYS 4

THERE ARE MANY TYPES OF SURVEYS

LARGE SCALE/SMALL SCALE

SPOKEN (INTERVIEWS) VS. QUESTIONNAIRES

COLLECT OBJECTIVE INFO VS. SUBJECTIVE INFO

FOR QUALITATIVE PURPOSES / SIMPLE QUANTITATIVE PURPOSES / ADVANCED QUANTITATIVE

PROCESSING

INDEPENDENT / PART OF A LARGE PROJECT WITH OTHER COMPONENTS

Gile SURVEYS 5

(SOME) OBSTACLES TO COLLECTION OF TRUTHFUL, ACCURATE INFO IN SURVEYS

LACK OF WILLINGNESS OF PEOPLE TO INVEST EFFORTS IN SURVEYS → LOW RESPONSE RATES, INCOMPLETE ANSWERS

- RESPONSES VARY AS A FUNCTION OF SELF-IMAGE AND IMAGE SUPPOSEDLY GIVEN TO INTERVIEWER

(SO LESS THAN TRUTHFUL RESPONSE POSSIBLE)

- RESPONSES VARY AS A FUNCTION OF ULTERIOR INTERESTS

- QUESTIONS (AND ANSWERS) MAY BE INTERPRETED DIFFERENTLY BY AUTHORS AND READERS (RESPONDENTS

AND ANALYSTS)

Gile SURVEYS 6

IMPORTANT STRATEGIES

- MINIMIZE EFFORTS FOR RESPONDENTS(CLEAR QUESTIONS, EASY TO ANSWER, LENGTH OF QUESTIONNAIRES)

- PILOT THE QUESTIONNAIRES TO MAKE SURE PEOPLE UNDERSTAND

PILOTING IS ALSO IMPORTANT TO DETECT MISSING ITEMS, AMBIGUITY AND OTHER WEAKNESSES

- TRY TO ELIMINATE OR REDUCE EFFECT OF SELF-IMAGE(QUESTIONING STRATEGIES, TRAINING OF INTERVIEWERS)

IF NOT POSSIBLE, USE RESULTS CONSERVATIVELY

Gile SURVEYS 7

SAMPLING AND REPRESENTATIVITY

SIZE IS NOT THE ONLY CONCERN

REPRESENTATIVITY IS PARAMOUNT

“POPULATION”“SUB-POPULATION”

“BIAS”(SYSTEMATIC ERROR – versus SAMPLING ERROR)

AVOIDING BIAS THROUGH RANDOM SAMPLING

MAY WANT STRATIFIED RANDOM SAMPLING(RANDOM SAMPLING WITHIN SUB-POPULATIONS THAT YOU

WANT TO INCLUDE)

Gile SURVEYS 8

SAMPLING ISSUES

SMALL SAMPLE SIZE DOES NOT IMPLY BIASTHROUGH IT MAY NOT HAVE FEATURES CLOSE

ENOUGH TO THOSE OF THE POPULATION DUETO SAMPLING ERROR

SAMPLING ERROR DECREASES WITH SAMPLE SIZE(BUT PROPORTIONALLY TO SQUARE ROOT OF SAMPLE

SIZE – MUTLIPLY SAMPLE SIZE BY 4 NAND YOU WILL ONLY REDUCE SAMPLING ERROR BY 2)

SO INCREASING SAMPLE SIZE IS ONLY EFFICIENT TO A MODERATE EXTENT

ESPECIALLY AS PATTERNS STABILIZE

Gile SURVEYS 9

CASE STUDY: PUBLISHING IN TS

EST SEMINAR IN GHENT IN 2005

EDITORS ASKED TO TALK ABOUT: - CRITERIA FOR PUBLICATION

- DIFFICULTIES IN PUBLISHING ACTIVITY- REFEREEING

DURING SEMINAR:JOURNAL EDITORS DID NOT SEEM AWARE OF THE

FACT THAT THEY ARE IN COMPETITION

DID NOT MENTION PROBLEMS WITH REFEREEING

SOSURVEY TO CHECK BOTH

Gile SURVEYS 10

PURPOSE OF THE SURVEY

• IDENTIFY FACTORS WHICH MAY BE RELEVANT/IMPORTANT(NOT QUANTIFY THEM)

• DO SOME EXPLORATIONIN PREPARATION FOR FURTHER STUDIES

IF INTERESTING ISSUES COME UP

SURVEY DESIGN TAKES THESE OBJECTIVES ON BOARD(LESS STRINGENT REQUIREMENTS THAN IF HYPOTHESES

WERE TO BE TESTED OR IF INFLUENCES WERE TO BE QUANTIFIED)

Gile SURVEYS 11

QUESTIONNAIRE OR INTERVIEWS?QUESTIONNAIRES - INTERNET

WHAT QUESTIONS?

ONE SET ON CRITERIA FOR SELECTIONONE SET ON REFEREEING PROBLEMS

(Not standard procedure, but for efficiency and higher response rate)

- THINK OF CERTAIN CRITERIA FOR SELECTION- BUT LET RESPONDENTS ADD

- ON REFEREEING, ASK ABOUT POSITIVE EXPERIENCE AND NEGATIVE EXPERIENCE

- ASK ABOUT REFEREE’S EXPERIENCE

- PLUS QUESTIONS ON RESPONDENTS TO ANALYZE BY EXPERIENCE, COUNTRY, LANGUAGES IF POSSIBLE

(MINIMAX)

Gile SURVEYS 12

CLASSIFY RESPONDENTS AS BEGINNERS / EXPERIENCED / IN THE MIDDLE

BECAUSE THIS MAY BE RELEVANT

TENTATIVE DUAL CRITERION WITH ARBITRARY THRESHOLDS (TURNED OUT TO BE ALL RIGHT – PERHAPS BECAUSE DESIGNER OF

SURVEY “KNEW” THE FIELD WELL ENOUGH)

- How long have you been in the field of TS?- Less than 5 years

- 5 to 10 years- More than 10 years

- How many papers have you published so far in the field, either in journals or in collective volumes?

- Less than 5- 5 to 10

- More than 10

Gile SURVEYS 13

SELECTION CRITERIA WHEN CHOOSING JOURNALS

• Reputation/Rating - ATTENTION this is subjective• Speed of publication

• Choosing the language of publication

Why these criteria?

What scale?- Not at all important- Not very important

- Quite important- I don’t know

Why not numerical? Why not more?

Gile SURVEYS 14

REFEREEING

- How often have you felt that the referee’s comments were useful to you? - ATT

- How often have you felt that referees misunderstood your statements? - ATT- If you have acted as a referee yourself, how often have you felt that you had

to perform this function because nobody else with the right qualification was available or willing to do the job, but were not fully qualified because of language problems or because you were not familiar enough with the

field, topic or research methods concerned? – ATT

• Often/Sometimes / Rarely / I don’t know

NOTE:• BOTH POSITIVE AND NEGATIVE

Why?

• AUTHOR’S AND REFEREE’S VIEWPOINTWhy?

Gile SURVEYS 15

PILOTING

DISTRIBUTION MODE : INTERNETADVANTAGES AND DISADVANTAGES?

SENT TO COLLEAGUES EXPERIENCED IN SURVEYS FOR INPUT (CHIARO AND NOCELLA)

Why?

SOME CHANGES ON BASIS OF THEIR COMMENTS AND SUGGESTIONS

- PILOTING 1: SENT TO ABOUT 15 SCHOLARS(AFTERWARDS, THEY ARE NO LONGER AVAILABLE AS RESPONDENTS)

- PILOTING 2: INITIAL PRODUCTION: FIRST BATCH – POSSIBILITY OF FINDING FURTHER PROBLEMS AND CORRECTING

- THEN FINAL “PRODUCTION”

Gile SURVEYS 16

PREPARATION OF DATA FOR ANALYSIS

• EXCEL FOR SIMPLICITY, PORTABILITY(DATA BASE MORE POWERFUL, BUT MORE DIFFICULT TO EXPORT TO

OTHER COLLEAGUES/SYSTEMS)

• “NAMES”, MEANINGFUL(FOR EASIER READING AND INTERPRETATION THAN CODES)

• W/ DIFFERENT INITIALS(FOR AUTOMATIC COPYING BY EXCEL – SAVES TIME)

• COLUMN FOR FURTHER COMMENTS

• IDENTIFIERS (NOT NAMES OF RESPONDENTS, UNLESS EXPLICIT PERMISSION)

• INITIAL INFO ON RESPONDENTS TO HELP FIND OUT ABOUT INFLUENCE OF POTENTIALLY RELEVANT FACTORS (COUNTRY,

EXPERIENCE)

Gile SURVEYS 17

Ident Country Experience

Publics Reputation of journal

Speed of publication

Language

Useful Misunderstings

Not qualified

Other criteria

1 AUS B 5-10 C 10+ QUITE 3 NOT VY ZERO OFTEN SMTMS NEVER INTERACTN W/ REFEREES

2 BELG B C Q N Z O S RARE POLITICS

3 CAN N/A

4 CHINA

5 CZ

6 DK

7 FINL

Gile SURVEYS 18

COUNTRIESn = 85

AUSTR 2 OEST 3

BELG 4

GERM 4

GR 1

IRL 1

ITAL 4

NL 1

PORT 1

SPAIN 14

UK 4

FRANCE 1

CAN 11

US 1

CHINA 2

IL 3

JAP 1

KOR 6

MALAY 1

CZ 2

POL 3

SLOV 1

DK 5

FINL 3

SWED 1

SOUTHAF 1

N/A 1

2 38 12 13 6 9 1

Gile SURVEYS 19

WHY RELEVANT AND WHAT CAN WE SAY ABOUT THIS SAMPLE?

• FROM ALL OVER THE WORLD

• BUT STRONG WEST EUROPEAN BIAS

• POSSIBILITY OF CHECKING BY REGIONS

• POSSIBILITY OF CHECKING BY COUNTRIES FOR SOME FACTORS

(IF HYPOTHESES ON LANGUAGE, REFEREEING W/O FULL QUALIFICATIONS)

Gile SURVEYS 20

Exper Publs Reput Speed Lang Useful Misunderst

Not qual Other crit

A:15/18%

B:16/18%

C:54/64%

A:14/17%

B:13/15%

C:58/68%

Q:75/88%

NV:10/12%

Q:50/60%

NV:33/37%

Z:3/4%

Q:39/45%

NV:32/38%

Z:13/15%

NA:1

O:42/50%

S:34/40%

R:5/5%

NA:4

O:3/4%

S:35/42%

R:42/49%

NA:5

O:2/3%

S:25/37%

R:24/37%

Z:15/23%

NA:19

INTL 8

SPCLZD 4

READERSHIP 4

POLITICS, OPEN MIND, INTEGRITY OF PAPER/EDITORS

N = 85

A 0-5 yrs

B 5-9 yrs

C > 10 yrs

A > 5

B 5-9

C > 10

Q quite impt

NV not very impt

NA: not available

O often

S stimes

R rarely

Z never

Z: zero imptance

Gile SURVEYS 21

INTERPRETING THE RESULTS

CRITERIA FOR CHOOSING JOURNAL

• REPUTATION: Q 88% CLEARLY IMPORTANT• SPEED: Q 60% - NV 37% OFTEN IMPORTANT – When? • LANGUAGE: Q 45% - NV 38% SOMETIMES NOT CLEAR BY COUNTRY IN SPITE OF HYPOTHESIS – FURTHER

EXPLORATION POSSIBLE

REFEREEING• USEFUL: O 50% - S 40% - R 5% POSITIVE ON THE WHOLE – OVERASSESSED BECAUSE

POLITICALLY CORRECT?• MISUNDERSTANDINGS: O: 4% - S: 42% - R: 49% OK ON THE WHOLE• I NOT COMPETENT: O: 3% - S: 37% • NOT INSIGNIFICANT – UNDERASSESSED? – NOT CLEAR BY

COUNTRY DESPITE SPECIFIC SITUATIONS – FURTHER EXPLORATION

Gile SURVEYS 22

SOME LIMITATIONS AND SHORTCOMINGS OF THE STUDY

- NOT RANDOM SAMPLE, DIFFICULT TO ASSESS BIAS

- QUESTIONNAIRE SENT IN ENGLISH ONLYPOSSIBLE LANGUAGE BIAS,

PARTICULARLY IMPORTANT FOR QUESTION ON RELEVANCE ON LANGUAGE

- IN LAST QUESTION, POSSIBLE AMBIGUITY BETWEEN NON APPLICABLE AND “NEVER”. SHOULD HAVE

ADDED “NON APPLICABLE” CATEGORY

Gile SURVEYS 23

WAS THE EXERCISE USEFUL?

• QUANTITATIVE RESULTS CONFIRM INTUITIVE KNOWLEDGE,

• GIVE GENERAL ORDER OF MAGNITUDE• NO BIG SURPRISES

• PERHAPS USEFUL TO KNOW THAT MORE THAN A THIRD OF THE RESPONDENTS SAY THAT

SOMETIMES THEY ARE NOT FULLY QUALIFIED

• “QUALITATIVE” RESULT: MANY RESPONDENTS WANT TO INTERACT WITH REFEREES – PERHAPS

USEFUL FOR EDITORS’ POLICY DECISIONS