Measuring the Effectiveness of Language Training

Hugh MorganLanguage Adviser

Defence School of Languages

DefinitionsDefinitions• Evaluation: Assessment of a total training

system, training course or programme, attempting to measure the overall cost-benefit of the course or programme and not just the achievement of laid down objectives.

• Validation of Training: A system of Quality Assurance that is implemented to ensure that the processes and products of training meet the operational training requirement.

Source: Defence Glossary - OC Trg Sp ASTS

Moderation at Moderation at MODLEBMODLEB

(University of Westminster and Ministry of (University of Westminster and Ministry of Defence Language Examination Board)Defence Language Examination Board)

…… the “practical “ application

The MODLEB ExamsThe MODLEB Exams

• Advanced Certificate (SLP 3333)

• Diploma (SLP 4444)

Advanced Certificate Advanced Certificate - exam content - exam content

• Listening – Task 1: Listening for specific

information– Task 2: Gist

• Speaking– Transacting Business– Conversation– Situation Interpreting

Advanced Certificate Advanced Certificate - exam content (cont) - exam content (cont)

• Reading– Task 1: Reading for specific

information (defence-related)– Task 2: Reading for specific

information (eg newspaper article)

• Writing– Task 1: Business / Social letter– Task 2: Draft Translation (military)

Diploma Diploma - exam content- exam content

• Listening– Task 1: Radio-magazine style

discussion– Task 2: Gist - from news item– Task 3: Military Briefing (specific info)

• Speaking– Task 1: Transacting Business– Task 2: Conversation– Task 3: Consecutive Interpreting– Task 4: Situational Interpreting

DiplomaDiploma

-- examexam content (cont)content (cont)• Reading

– Task 1: Translation of professional military article

– Task 2: Translation of a newspaper article

• Writing– Task 1: Social Letter– Task 2: Translation of Report (military)

The Moderation The Moderation ProcessProcess

• Three parts:

– Setting Board

– Moderation Board

– Confirmation

Setting BoardSetting Board• To identify suitable tasks for each of the

4 skills.• To ensure that the parameters of the

materials meet the stated exam criteria.• To ensure that the language knowledge

required by candidates meets the appropriate SLP level descriptors.

• To remove any items that may be culturally inappropriate in certain languages.

The Moderation BoardThe Moderation Board

• Before

– All tasks marked twice.– ‘Blind marking’ occurs - no reference

between markers.– Provisional grades placed on marking

matrix

The Moderation BoardThe Moderation Board

• During– 10% of all scripts, for each language,

checked for appropriateness of marking.– Tasks are checked for consistent poor

performance ie was the task reasonable?– If OK, existing gradings (where there is no

dispute) stand.– Where a dispute exists, or a ‘Referred to

Board’ grading has been recorded, individual task markings are checked by whole board.

– Gradings are checked for inconsistencies across the skills areas.

ConfirmationConfirmation

• Effectively the ‘Conclusion’ of the Moderation Board.

• Confirmation is the consolidation of results, including ‘brought forward’ results from previous exams or previous attempts, checked by all Board members.

• It fulfils the University of Westminster’s own academic validation requirements for granting their accredited qualification.

Follow-up to the MODLEB Follow-up to the MODLEB BoardBoard

• Consolidated results go back to the Directorate of Educational & Training Services (Army) - the Tri-Service language sponsor - to match candidate numbers to names.

• Candidates (and Schools) advised of results

Motivating and Monitoring Language Learning

Mike ZolloHead of Modern Languages

Dept,Britannia Royal Naval College

DefinitionsDefinitions

• Motivation: The stimulus, incentive, motives for action towards a goal, especially as resulting from psychological or social factors; the factors giving purpose or direction to behaviour; enthusiasm, drive.

• Monitoring: A method used in a production process to check, regulate observe and supervise; to keep under review, measure or test at intervals, especially for purpose of regulation and control.

Source: Oxford English Dictionary

Priorities

• Do we really want to risk killing off motivation by over-testing?

• If good motivation leads to effective learning, then ongoing monitoring can provide enough evaluation

Course Design > < Motivation Monitoring > < Evaluation

Course design for Course design for MotivationMotivation

(from “Effective Language Training Strategies”, (from “Effective Language Training Strategies”, Stephen Hagen, in “Languages in International Stephen Hagen, in “Languages in International

Business”)Business”)• Course structure• Learning environment• Methodology• Learning technology• Learner type• Tutor type

Course structure

• Mode: regular / crash course • Duration: total number of hours • Programme: Long + thin / short + fat• Open learning: student autonomy• Balance of skills: L/Sp/R/Wr• Evaluation: by tests or ongoing?• Language > < Culture

Learning environment

Location– ‘at home’ or in country– classroom or open/distance

learning– residential or ‘day visits’

Atmosphere– motivation or excessive pressure– learning or testing– progress or frustration

–confidence or terror ! ! !

Methodology

• traditional: grammar-based• audio-lingual: audio-visual• direct method: immersion• communicative: function / task• balanced approach: pick and mix

Learning technology• Hardware: learning/teaching equipment

• Software: CALL, distance e-learning

• Printed material: books, press,

• Audio-visual:TV, video, sound recording

Learner type

• language aptitude we all have it!• previous experience variable• motivation obligation/ need?• attitude positive/ interest• personality open-minded• educational background variable• cultural baggage inevitable!• trust essential

Tutor type• personality essential• teaching technique versatile• commitment total• attitude flexible• knowledge and experience essential• qualifications essential• background variable• quality of preparation essential

Effective learning • knowledge• understanding• ability to manipulate• self-monitoring• willingness to take risks• strategies for retention• self-reliance and resourcefulness• effectiveness of recall

• >>>>>>>>>>>>>>>>> CONFIDENCE

Monitoring Monitoring forfor EvaluationEvaluation

(after “ESP Today” Pauline Robinson and “The (after “ESP Today” Pauline Robinson and “The Communicative Approach to Language Teaching” Communicative Approach to Language Teaching”

Brumfitt and Johnson”)Brumfitt and Johnson”)• Formative value of evaluation• Evaluation to enhance motivation• Avoidance of negative evaluation• Ongoing assessment• Observation• Recording progress• Effectiveness of evaluation

Formative value of evaluationStudents learn by mistakesSuccess gives positive reinforcementFeeling of progress enhances motivation

Evaluation to enhance motivationSet realistic objectivesSet valuable targetsRemember importance of confidence

Avoidance of negative evaluationRisks demotivating studentsCreates frustration, resentmentRuins trust and student-tutor relationship

Ongoing assessmentquestionnaires based on texts/recordingschecklists of vocabulary/phrases in textsregular achievable ‘homework’‘self-monitoring’ checklists of tasks achieved

Observation … of pairwork/groupwork in oral lessons… by marking regular achievable ‘homework’… by discussion in groups… by interview of individual

Recording progressKeep regular records using suitable codesBuild up student language-learning profileInform students of their progress

Evaluation: effectiveness Evaluation: effectiveness

criteriacriteria (“Effective Language Training Strategies”, Stephen (“Effective Language Training Strategies”, Stephen

Hagen)Hagen) Effectiveness: Skills used by whom, and for what

purpose?• end user (language task done satisfactorily)

• individual (job satisfaction, confidence)

Other factors:• professionalism (impresses customers)

• cost effectiveness (time, efficiency etc)

• accountability (confidence / reliability)

• desired results (satisfaction on all sides)