Access to the World and Its Languages Evaluation Access to the World and Its Languages I N T E R N A T I O N

Access to the World and Its Languages

www.ed.gov/ope/iegps

Evaluation

Access to the World and Its Languages www.ed.gov/ope/iegps

I N T E R N A T I O N A L E D U C A T I O N

P R O G R A M S S E R V I C E

CIBE, FLAS, IIPP, LRC, NRC Technical Assistance Workshop

February 2-3, 2009



Title VI 50th Anniversary Update• Madeline Albright is the Keynote

speaker• Please make separate reservations

– Conference registration– Hotel registration– Rooms are going quickly

• Contacts Rob Glew and Tomas Hult, Michigan State University

• http://titlevi50th.msu.edu



Evaluation

Love



Panelists

• Eleanor Briscoe, Senior Budget Analyst• John Clement, Director, Strategic

Planning Services• Susan Lehmann, Evaluation Analyst,

FIPSE• Karla Ver Bryck Block, Senior Advisor,

IEPS



Why???

• ED believes in and supports high quality program evaluations to help inform budgetary and program decisions

• ED Strategic Plan– Goal 3, Objective 1: Increase success in and

completion of quality postsecondary education.• Strategy 3. Prepare more graduates for

employment in areas of vital interest to the United States, especially in critical-need languages, mathematics, and the sciences.



Administration Update

• “Accountability and Transparency”• GPRA continues• PART may or may not continue but

the need to measure program effectiveness remains



Congress

• Clear interest in program evaluations

• Directed a $1.5 million review of Title VI and Fulbright-Hays programs by the National Research Council of the National Academies– International Education and Foreign

Language – Keys to Securing America’s Future (March 2007)



Keys to Securing America’s Future

• National Academies recommendation– The Department of Education should

commission independent outcome and impact evaluations of all programs every 4 to 5 years



IEPS Program Level Initiatives• Evaluations are underway with AIR

– Language Resource Centers– Fulbright-Hays Group Projects Abroad –

Overseas Intensive Language Projects

• Analysis of the International Research and Studies Program is concluding

• National Resource Centers are next• Future plans include Centers for

International Business Education



IRIS

• IEPS gathers information via IRIS that is used for program evaluations

• Breakouts for 4 programs today (CIBE, LRC) and tomorrow (NRC, FLAS) to discuss:– Current and future program performance

and efficiency measures– Changes to IRIS for the next clearance– Evaluation efforts– Trivia, prizes, chocolate!!!



Keys to Securing America’s Future

• National Academies Recommendation– The Department of Education should work

with universities to create a system of continuous improvement for Title VI and Fulbright-Hays programs. The system would help develop performance indicators and other improvement tools and should include networks of similar centers and university officials with overall responsibilities in language, area, and international studies.



Your Project

• Think…– Goal-oriented– Data driven

ED does not have specific models, timelines, budgets, etc for your evaluation plans. Develop a plan that serves your needs.



Performance Objectives & Measures

• A measurable indication used to determine how well goals are being met– How will progress be measured?– How much progress will constitute success?– How will it be known if an objective or part

of an objective has been achieved?

• Are we measuring process or outcome?



Objective & Measure Types

• A specific achievement/accomplishment

• Change in a level of achievement/accomplishment

• Achievement/accomplishment (or change) compared with other students or institutions



Components of Objectives & Measures

• What will change• How much change is expected• Who will achieve the change• When the change will take place



Performance Objective Example

“20% (how much) of local businesses with more than 200 employees (who) will open an office in a foreign country or enter into a joint venture with a foreign company (what) within two years (when).”

• Help local businesses to internationalize



Performance Objective Example

• Teacher workshop – Developing Materials for LCTLs

“50 (how much) LCTL teaching professionals (who) will complete LCTL workshop (what) annually (when).”

“90% (how much) of LCTL workshop participants (who) respond with 4 or higher on survey (what) one year after completing program (when).”



Performance Measure Example

• Revise public health curriculum to enhance international knowledge“The required MPH courses (who) will be revised (what) to cover the 10 important international health areas (how much) by next fall (when).”



Performance Measure Example

•Improve enrollment in summer language institute“Undergraduate student enrollment in the summer language institute (who) will increase (what) by five percent (how much) next summer (when).”



Improving Performance Measures

• Original Performance Measure: Provide overseas internships.

• What/Who:– Provide dynamic student internships with partners in a

foreign country.

• When:– Provide dynamic student internships with partners in a

foreign country each semester.

• How Much:– 20 students will engage in dynamic student internships

with partners in a foreign country each semester.



Improving Performance Measures• Original Performance Measure: Enhance

Business Resource Center.

• What/Who:– Enhance Business Resource Center to advise

businesses on international trade regulations.

• When/How Much:– Business Resource Center documents and staff will

be updated on new international trade regulations within one week of passage.



Remember!!!

• Activities are not performance measures!



Resolving the “Activity-as-Measures” Dilemma

• “Create opportunities for local faculty to strengthen area skills.” (activity)– Create “visiting” faculty training programs

and train 2 per year. (outcome)

• “Enhance study abroad.” (activity)– Increase number of students studying

abroad by X% each year. (outcome)



Summary: Good Objectives & Measures

• Each objective should have one or a few specific performance measures to demonstrate how progress will be measured toward meeting the objective

• More objectives and measures ≠ better objectives and measures!

• Ensure data is collectible

• Quantify and define success



Summary: Strong Evaluation• Partner with a competent evaluator• Develop quantifiable objectives and

measures• Measure what is important• Measure what can be changed• Get frequent progress reports• Keep key stakeholders informed of

results• Use results to adjust project regularly• Keep analyzing…

Documents

Access to the World and Its Languages Evaluation Access to the World and Its Languages I N T E R N A T I O N