Evaluation at the Defense Language InstituteDr. Thomas S. Parry

Directorate of Evaluation and StandardizationDefense Language Institute

Bureau for International Language CoordinationBudapest, Hungary

May, 2006

Evaluation can make you rethink what you have been doing!

Why Evaluation?

What is Evaluation?

The process of finding the value or amount of something—to determine it’s

significance or quality.

-- Webster’s New World Dictionary

DLI Facts

• Average daily resident attendance: 3,079• Languages taught: 22• Number of students taught annually

through all Continuing Education programs: 5,000

• Highest enrollment languages:

ArabicChineseKorean

Persian-FarsiRussianSpanish Serbian-Croatian

• 0515: Physical Training

• 0630: Breakfast

• 0730: Formation with Service Unit

• 0755: Class start

• 1530: Class end

• 1600: Military Training• Cumulative:

– 6 hours of class instruction

– 45 minute special assistance or enhancement training

– 3 hours of homework plus weekend assignments

– 2-3 hours of study hall (M-Th)

– Developing & sustaining basic warrior-linguist skills, common task training, and physical readiness

Typical DLI Student Day

Basic Course

Language FY06 Student Load Faculty

Category IV Languages (63 weeks)Category IV Languages (63 weeks) ArabicArabic 1165 1165 272 272 KoreanKorean 833 833 216 216 ChineseChinese 558 558 126 126 JapaneseJapanese 3535 10 10

2591 (71%) 6242591 (71%) 624 Category III Languages (47 weeks)Category III Languages (47 weeks) DariDari 87 87 24 24 PashtoPashto 134 134 34 34 Persian FarsiPersian Farsi 248 248 68 68 RussianRussian 144 144 58 58 Uzbek, Hindi, Uzbek, Hindi, Urdu, & 6 other langUrdu, & 6 other lang 211211 9494

824 (23%) 278824 (23%) 278Category II Languages (34 weeks)Category II Languages (34 weeks) GermanGerman 2727 88 27 (1%)27 (1%) 88Category I Languages 26 weeks)Category I Languages 26 weeks) Spanish, French,Spanish, French, Italian, PortugueseItalian, Portuguese 196196 5656

196 (5%) 196 (5%) 56 56

Totals Totals 3638 966 3638 966

DLIFLC

Student Load by DifficultyResident Courses at Presidio of Monterey

94% of students taking CAT III & IV Languages

DLI Evaluation Goals

• Ensure student and teacher accountability• Provide for continuous academic program

improvement– Faculty and staff development– Curriculum development and implementation

• Gauge our return on investment• Ensure quality of life and support services

are conducive to learning

Teachers Evaluate Students

Students are held to high levels of accountability for their learning

Teachers EvaluateStudents

• Teachers evaluate student performance in the classroom through:– Participation in classroom activities– Teacher-developed chapter/unit tests– Periodic counseling– Diagnostic assessments– Learning style assessment– Performance tests– Oral Proficiency Interviews (OPI)– Defense Language Proficiency Tests (DLPT)

• Listening• Reading

Students Evaluate

Teachers Program of Study Quality of Life

Evaluation at DLI is Bi-directional

Teachers Evaluate Students

Students Evaluate Teachers

Teacher Evaluation

• Teacher evaluation at the DLI includes:– Complete student evaluation of the teacher– Scheduled and unscheduled classroom observations by

the supervisor– Periodic reviews of performance objectives with the

supervisor

• The combination of these three teacher evaluation elements provides an accurate assessment of the quality and effectiveness of foreign language instruction

Teacher Evaluation

• Each teacher is evaluated on classroom performance by all the students he/she teaches

• A personal performance profile is created for each teacher by tracking his/her performance throughout each of their courses

Teacher evaluations by students have implications for merit pay and rank advancement

Interim Student Questionnaire

• Administered via computer to all students enrolled in 47- and 63-week language courses 60% of the way through the program

• Takes roughly 60-90 minutes to complete• Survey sections include:

– Program Evaluation (PE)– Teacher Evaluation (TE)– Quality of Life (QL)

ISQ

End-of Course Student Questionnaire

• Administered via computer at the end of language courses to all graduating students

• Takes roughly 60-90 minutes to complete

• Survey sections include:– Program Evaluation (PE)

– Teacher Evaluation (TE)

– Quality of Life (QL)

ESQ

Analysis and Reporting

• Evaluation feedback from ISQ/ESQ delivered within two working days from survey administration– Deans

– Chairs

– Team leaders

– Individual teachers

• Special monthly reports provided to service commanders and individual units via separate questionnaires

• Annual summaries for each instructor sent to Provost, Vice Chancellors and school Deans

ISQ/ESQ Response Scale

• Items are rated from 1-4 with an option for no opinion:

0 No opinion

1 Strongly disagree

2 Disagree

3 Agree

4 Strongly agree

Teacher Evaluation (TE)

• Class preparation and quality of instruction

• Ability to teach language and related concepts effectively

• Adjustment of teaching strategies

• Ability to provide a variety of instructional activities

Sample Criteria

Teacher Evaluation (TE)

• Active use of foreign language in classroom

• Ability to provide challenging instruction

• Counseling and assistance offered

• Feedback given to students

Sample Criteria

Team and Individual Instructor TE Analysis

Teacher Improvement

Program Evaluation (PE)

• Testing and grading

• Team teaching

• Homework

• Curriculum and supporting materials

• Overall evaluations

Sample Criteria

Quality of Life (QL)

• Barracks, family housing and child care

• Quality of food, mess hall and related services

• Medical and dental care

• Chaplain and community services

• Military support

• Sport and recreation opportunities

• Transportation and travel office services

Sample Criteria

Quality of Life (QL)

• Equal Opportunity– Prejudice– Discrimination– Sexual harassment

• Military– Training– Duties– General quality of life

Other Areas of Concern

PE and QL Analysis

Serious Issues

• Depression

• Suicide threats

• Sexual assaults and harassment

• Fraternization and other misconduct

Immediate turnaround; sent directly to Command Group for action

Evaluation of Continuing Education Programs

• ESQ questionnaires distributed world-wide to training sites and include:– Video tele-training sites– Mobile training team deployments– Language Teaching Detachments– Command Language Programs– Special, customized programs

• Questionnaires completed and returned to DLI via on-site program managers

• Analysis and feedback provided to instructor’s school and training site within two weeks

Conclusion

• DLI evaluation practices: – Are well established—honed through years of

practice– Are rigorous and thorough– Bring about faculty and program improvement– Establish baselines for student/teacher

accountability– Give us cause to rethink what we are doing and

consider how we might do things better

Evaluation at the Defense Language Institute

Questions?

Discussion?