D.Gile citations as indicators1 CITATIONS AS INDICATORS IN TS Daniel Gile [email protected]

D.Gile citations as indicators 1

CITATIONS AS INDICATORS IN TS

Daniel Gile

[email protected]

www.cirinandgile.com

mailto:[email protected]

http://www.cirinandgile.com/


AIMS OF THIS PRESENTATION

THIS PRESENTATION IS OFFERED IN THE FRAMEWORK OF RESEARCHER TRAINING

RESEARCH CONTEXT IS: RESEARCH ON TS AS AN OBJECT OF STUDY

TS IS A COMPLEX ENTITY, HENCE THE USE OF INDICATORS IN ITS STUDY

THE CONCEPT OF INDICATORS IS EXPLAINED

USES OF CITATIONS AS AN INDICATOR IN THE STUDY OF TS ARE EXPLAINED

METHODOLOGICAL ISSUES WHICH ARE RELEVANT TO OTHER EMPIRICAL STUDIES ARE DISCUSSED


INDICATORS

INDICATORS ARE

MEASURABLE UNITS

(or at least countable units)

WHICH ARE USED TO MEASURE

(or at least detect/identify)

ASPECTS OF ENTITIES WHICH ARE DIFFICULT TO MEASURE DIRECTLY

BECAUSE THEY ARE COMPLEX AND/OR DIFFICULT TO DEFINE OPERATIONALLY


EXAMPLES OF INDICATORS

- FOR HEALTH: BODY TEMPERATURE, BLOOD PRESSURE…

- FOR NATIONAL WEALTH: GNP

- FOR LEVEL OF CRIMINAL ACTIVITY: NUMBER OF CRIMINAL TRIALS, NUMBER OF COMPLAINTS TO THE POLICE…

- FOR TRANSLATION QUALITY: NUMBER OF ERRORS AND OMISSIONS, NUMBER OF COMPLAINTS FROM USERS

- FOR QUALITY OF TRANSLATOR TRAINING PROGRAM: PROPORTION OF GRADUATES WHO FIND WORK AS

TRANSLATORS


IMPORTANT FEATURES OF INDICATORS (1)

- VALIDITY(DO THEY MEASURE WHAT THEY ARE SUPPOSED TO MEASURE?)

- CONVENIENCE AND COST OF ACCESS(HOW LONG DOES IT TAKE TO MEASURE THEM? HOW MUCH

MONEY/MANPOWER etc. DOES IT COST?)

- EASE/COST OF PROCESSING(ONCE YOU HAVE THEM, IN ORDER TO DO SOMETHING USEFUL

WITH THEM, HOW MUCH EFFORT AND HOW MANY RESOURCES ARE REQUIRED?)


IMPORTANT FEATURES OF INDICATORS (2)

- COVERAGE(WHAT PROPORTION OF THE RELEVANT PHENOMENON IN THE

ENTITY TO BE STUDIED DO THEY COVER?)

- SENSITIVITY(HOW MUCH OF THE “SIGNAL” OR PHENOMENON THEY ARE

SUPPOSED TO PICK UP THEY ACTUALLY DO PICK UP?)

- SELECTIVITY(DO THEY PICK UP THE TARGET SIGNAL OR PHENOMENON ONLY, OR

DO THEY ALSO PICK UP OTHER SIGNALS WHICH INTERFERE WITH THE EXPLORATION PROCESS?)

COMPLAINTS FOR CRIMINAL ACTIVITY? VAL/COV/SENS/SELCOMPLAINTS OF TRANSLATION USERS? VAL/COV/SENS/SELPROPORTION OF GRADUATES WHO FIND EMPLOYMENT?

VAL/COV/SENS/SEL


CITATIONS

WHEN INVESTIGATING TS AS AN OBJECT OF STUDY:

WRITING NORMS IN SCIENCE IMPOSE REFERENCES TO AUTHORS AND WORKS

“USED”

THESE TAKE THE FORM OF CITATIONS

Operational definition:EXPLICIT REFERENCE IN A RESEARCH TEXT TO

ANOTHER AUTHOR’S PUBLISHED TEXT

i.e. NAME OF AUTHOR AND PARTICULAR TEXT WITH REFERENCES


CITATIONS AS AN INDICATOR OF INFLUENCE

VALID?

EASE AND COST OF ACCESS?

EASE AND COST OF PROCESSING?

COVERAGE?

SENSITIVITY?

SELECTIVITY?


IN THE FIELD

CONSIDERED VALID

WIDELY USED

By

GOVERNMENTS, SOCIOLOGISTS OF SCIENCE…

IN TS:

COVERAGE INSUFFICIENT

SENSITIVITY PERHAPS INSUFFICIENT

CORRELATION BETWEEN NUMBER OF CITATIONS AND ACTUAL INFLUENCE PERHAPS NOT STRONG ENOUGH


TENTATIVE METHODOLOGICAL IMPROVEMENT - 3 LEVELS OF INFLUENCE OF A CITED TEXT?

L1: In passing, as illustration, historical reference,NO VISIBLE IMPACT on citing author’s thought or work,

judging by the citing text

L2: SOME IMPACT: Clear signs in citing text that citing author has thought about the cited text significantly or is building on

it

L3: CENTRAL in citing text

EVALUATE INFLUENCE OF CITED TEXTS IN CITING TEXT


3 LEVELS OF INFLUENCE: IS IT A PROGRESS?

PERHAPS TOO MUCH VARIABILITY?

PERHAPS DIFFICULT TO CATEGORIZE?

HAVE TO CHECK

IF NOT GOOD ENOUGH, IMPROVE DEFINITION

IF NOT GOOD ENOUGH, GIVE IT UP

CHECK IN CLASS WITH PAPERS

SEE % OR AGREEMENT, DISAGREEMENT

IF OK, COMPARE RESULTS WITH 3 LEVELS AND WITHOUT


3 LEVELS OF INFLUENCE: TESTS?

A NUMBER OF TESTS WITH COLLEAGUES

TAKE PUBLISHED TEXT

ASK PANEL OF RESEARCHERS TO ASSESS LEVEL

CHECK CONVERGENCE

IF CONVERGENCE, OK, IMPLEMENT

(IF COLLECTION COST NOT TOO HIGH)

IF NOT ENOUGH CONVERGENCE

- TRY TO IMPROVE OPERATIONAL DEFINITION

- OR INFER THAT THIS IMPROVEMENT, HOWEVER ATTRACTIVE CONCEPTUALLY, DOES NOT REALLY WORK


OPERATIONALIZING CITATION ANALYSIS TO MEASURE AUTHOR INFLUENCE

- DETERMINE SAMPLING METHOD AND SAMPLE SIZE

(TYPE OF CITING TEXTS, PERIOD, LANGUAGES, GEOGRAPHIC AND SOCIOLOGICAL IDENTITY…)

IF THESE CONSIDERATIONS ARE NOT TAKEN UP, PROBLEM OF REPRESENTATIVITY

- PROCESSING

(SIMPLE COUNTS? PROPORTIONS? NUMBER OF TIMES CITED OR NUMBER OF PAPERS IN WHICH CITED?...)


WHAT QUESTIONS CAN CITANAL HELP WITH?INITIAL CONTRIBUTIONS

- WHO ARE THE MOST INFLUENTIAL AUTHORS AT TIME t?

IN TS? IN A PARTICULAR BRANCH OF TS? ON A PARTICULAR TOPIC? WITH A PARTICULAR METHODOLOGY?

- SAME QUESTION ABOUT GROUPS, “SCHOOLS”

- EVOLUTION OF INFLUENCE OVER TIME

- INFLUENCE OF PUBLICATION MEDIUM

(PAPERS IN JOURNALS, PAPERS IN COLLECTIVE VOLUMES, MONOGRAPHS, THESES AND DISSERTATIONS)

- INFLUENCE OF LANGUAGE OF PUBLICATION

…


WHAT QUESTIONS CAN CITANAL HELP WITH?ASSESSING QUALITY?

ARE CITATIONS A VALID INDICATOR FOR RESEARCH QUALITY?

QUESTION:

ARE THE MOST OFTEN CITED AUTHORS IN TS/IN OTHER DISCIPLINES

THE ONES WHO DO THE BEST RESEARCH?


WHAT QUESTIONS CAN CITANAL HELP WITH?DEVELOPING THEIR USE

- INTERACTIONS BETWEEN GROUPS

(INCLUDING INTERDISCIPLINARITY)COLLECT INFORMATION ON CITED TEXT - FROM TS OR OTHER

DISCIPLINES? Gile 2002/2006

TYPES OF CITATIONS- THEORIES

- FINDINGS

- RESEARCH METHODS

- OPINIONS

- TERMS…

INDICATIONS ON QUALITATIVE ASPECTS OF TS RESEARCHEMPIRICAL/NON EMPIRICAL


GENERAL METHODOLOGICAL NOTE:THRESHOLDS

ASSUME “EMPIRICAL”: FINDINGS, METHODOLOGY, REFERENCES TO EMPIRICAL STUDIES

NON EMPIRICAL: TERMS, OPINIONS,

IF 100% “EMPIRICAL” NO PROBLEM

IF 1% “EMPIRICAL” NO PROBLEM

IF 50% “EMPIRICAL”?

IF 30% “EMPIRICAL”?

NECESSARILY SUBJECTIVE/CONVENTIONAL THRESHOLDS

THIS IS COMMON TO MANY EMPIRICAL PROJECTS WITH METRICS SUCH AS PAUSE LENGTHS, NUMBER OF E/O’S, etc.

AT SOME TIME, SUBJECTIVE DECISIONS COME IN


GENERAL METHODOLOGICAL NOTE: CLASSIFICATIONS

FOR MANY USES OF INDICATORS IN EMPIRICAL RESEARCH, CLASSIFICATION IS CENTRAL

PRACTICAL ISSUE:

NOT EVERY PHENOMENON ENOUNTERED IS CLASSIFIABLE:

- MISSING INFORMATION

- BORDERLINE CASES

WHAT DOES ONE DO?

CONVENTIONS, PERSONAL DECISIONS, INFORM READER

PRACTICAL ASSIGNMENT TO GAIN FAMILIARITY WITH PROBLEMS ?


ASSIGNMENT (1)

CHOOSE AT LEAST 3 SPECIFIC RESEARCH QUESTIONS FOR WHICH YOU BELIEVE CITATIONS ARE A GOOD

INDICATOROr

THE EXPLORATORY QUESTION: WHAT CAN I FIND OUT ABOUT MY SAMPLE FROM CITATIONS?

CHOOSE APPROPRIATE SAMPLE OF TEXTS TO TRY TO ANSWER THE QUESTIONS

- At least 20 papers with a total of at least 200 citations if specific research questions

- At least 5 papers with a total of at least 50 citations if your question is exploratory


ASSIGNMENT (2)

- STUDY THE SAMPLEREPORT ALL THE CITATION DATA ON EXCEL FILE

(SEE SUGGESTIONS LATER IN THIS PRESENTATION)

- REPORT RESULTS AND ANSWER TO QUESTION(S) ON SEPARATE WORD FILE

- REPORT CAREFULLY ALL METHODOLOGICAL PROBLEMS YOU ENCOUNTERED AND ALL

METHODOLOGICAL LESSONS YOU LEARNED FROM THE EXERCISE


SUGGESTIONS FOR COLUMNS OF INFO ON DATA EXCEL FILE (1)

1. Citing author

2. (Affiliation)

3. Year of citing paper

4. References of citing paper

5. Page in which citation found

6. Cited author

7. (Affiliation of cited author)

8. (Discipline of cited author)

9. Year of cited reference

10.Cited reference


SUGGESTIONS FOR COLUMNS OF INFO ON DATA EXCEL FILE (2)

11. Type of medium (thesis, dissertation, article, article in collective volume, monograph, internet document…)

12. Language

13. Year of cited reference

14. Cited reference

15. Type of reference (opinion, definition of concept, term, finding, methodology, reference to empirical study, plain reference to publication, theory,…)

16. Other comments or key words

Documents

D.Gile citations as indicators1 CITATIONS AS INDICATORS IN TS Daniel Gile [email protected]