Holistic vs Analytical Assessment in Legal Translation

Carmen Valero-Garcés & Francisco Vigier – University of AlcaláMary Phelan – Dublin City University

• Assessment in Translation Studies and Professional Practice

• Introduction to HA

– Research Study on HA in Legal Translation

• Introduction to AA

– Research Study on AA in Legal Translation

• Conclusions2

Assessment in Translation Studies and Professional Practice

• Underresearched area

• Common problems in TQA (Williams 2009)

– The evaluator

– Level of target language rigour

– Seriousness of errors

– Sampling vs full-text assessment

– Quantification of quality

– TQA purpose 3

4

What is Holistic Assessment?

• The evaluator gives a TT a rating (0-10) or evaluative letter (e.g. A = excellent, B = very good) based on an overall impression

• Frequently used in both academia and industry

• Advantages less time-consuming and assessment of translations at the discourse/text level not at the sentence/word level (Garant 2009)

• Some attempts of systematization (Waddington, 2001)

• Disadvantages subjective, hence arbitrary, intuitive, unscientific, unsystematic and unreliable; does not provide a clear justification of the result (Waddington 2001)

5

Research Study on HA in Legal Translation

• Analyse strengths and weaknesses of holistic methods for the assessment of legal translation ( interrater reliability)

• One of the WS1 essential documents translated into SP by a student on MA in Translation

• That translation assessed numerically (0-10) by ten evaluators

• Evaluators surveyed on their assessment method

6

7

Results• Numeric assessment

8

Results (2)• Survey

– Most evaluators ranked pragmatic errors as those with highest relevance and linguistic errors as those with lowest relevance

– Very different opinions expressed by respondents as to the translation’s strengths and weaknesses (i.e. “The message is appropriately conveyed. It fulfills its communicative function” vs. “Errors regarding sense, coherence, punctuation... A poor quality translation” assessment is based on personal criteria, thus subjective and variable

9

ANALYTICAL ASSESSMENTAnalytical Assessment

10

ATA

• ATA system – (a) grid, (b) flowchart and (c) Explanation of Error Categories

11

ATA Grid

ATA CERTIFICATION PROGRAM Exam Number:

FRAMEWORK FOR STANDARDIZED ERROR MARKING Exam Passage:

Version 2009 Check here if for Review

1 2 4 8 16 Code Reason

Errors that concern the form of the exam

Treat missing material within the passage as an omission UNF Unfinished (if a passage is substantially unfinished, do not grade the

exam)

ILL Illegibility

IND Indecision, gave more than one option

Translation/strategic/transfer errors: Negative impact on understanding/use of target text

MT Mistranslation (use a subcategory if possible)

MU - Misunderstanding of source text (if identifiable)

A - Addition

O - Omission

T - Terminology, word choice

R - Register

F - Faithfulness

L - Literalness

FA - Faux ami (false friend)

COH - Cohesion

AMB - Ambiguity

ST - Style (inappropriate for specified type of text)

OTH - Other (describe)

Mechanical errors: Negative impact on overall quality of target text. Points may vary by language. Maximum 4 points

G Grammar

SYN - Syntax (phrase/clause/sentence structure)

P Punctuation

SP/CH Spelling/Character (usually 1 point, maximum 2, if more than 2 points,

another category must apply)

D Diacritical marks/ Accents

C Capitalization

WF/PS Word form/ Part of speech

U Usage

OTH Other (describe)

0 0 x 2

=0

0 x 4 = 0 0 x 8 = 0 0 Column totals

A grader may stop marking errors when the

score reaches 46 error points

A grader may award a quality point for each of

up to three specific instances of exceptional

translation

Quality points are subtracted from the error point total to yield a final

score. A passage with a score of 18 or more points receives a grade of

Fail.

Total error points (add column totals): 0 Quality points (maximum 3) 0 Final passage score (subtract quality points from error points) 0

12

ATA flowchart

13

• UAH text – holistic - 532 words in ST• DCU text – analytical – 256 words in ST

• 5 assessors – three in Europe plus two ATA assessors

14

Evaluators’ VerdictsAssessor Score Verdict

Spanish evaluator 2 9 Pass

Spanish evaluator 3 16 Pass

Spanish evaluator 1 23 Would accept it with reservations

ATA evaluator 1 45+ Fail

ATA evaluator 2 43 Fail

15

Conclusions

• HA: subjective method with a low degree of inter-rater reliability

• Cost and time efficiency HA as supplementary method for LT assessment?

• AA: even though the system appears self-explanatory, there is a lot of variation in the overall result.

• AA: The ATA evaluators have years of experience of using this method.

16

References

Garant, M. (2009). A case for holistic assessment. AFinLA-e Soveltavan kielitieteen tutkimuksia 2009, 1, 5-17.

Waddington, C. (2001b). Should translations be assessed holistically or through error analysis? Hermes, Journal of Linguistics, 26, 15-38.

Williams, M. (2009). Translation Quality Assurance. Mutatis Mutandis, Vol 2, No 1., 3-23

Thank you!

Carmen Valero-Garcés carmen.valero@uah.es Francisco Vigier francisco.vigier@uah.esMary Phelan mary.phelan@dcu.ie