Evaluating Your IEPS Grant

Dr. Susan Lehmann Education Research Analyst, FIPSE Presented at the 2009 International Education Programs Service Technical Assistance Workshop held on 2/2/2009 in Washington, D.C.

Your job, and ours, is to turn vague, general program goals into specific benchmarks

that you hope to achieve.

For each award you should ask:

What knowledge do you expect fellows to acquire?

What gains in skills do you expect fellows to make?

What counts as a gain in professional skills and how will you measure gain?

Things to Evaluate in Title VI Programs

1.Are you advancing the internationalization of academic disciplines?

You are probably keeping track of course materials that are being produced as a result of the grant.

Can you tell us the number of:

new interdisciplinary courses that have been adopted,

field-specific courses that have added new units on internationalization using your materials,

collaborative faculty projects that have resulted from your award?

Before your study abroad trip, had you ever taken a course about your host country in any of the following subject areas?

American Councils' FIPSE Survey of Short-term Curriculum-based Study AbroadChart Date: February 22, 2007 (N=326)

67

27

23

19

18

15

11

0 10 20 30 40 50 60 70 80 90 100

Foreign languagecourse

History course

Literature course

Contemporary societycourse

International affairscourse

Political science course

Sociology course

Percent Yes

58% of students studying abroad on FIPSE Programs learned about challenges and problems that face Americans in their field who choose to work abroad.

The American Councils study found that:

56% of students studying abroad on FIPSE programs learned about international laws and regulations governing work in their field.

The American Councils study found that:

By having an alumni newsletter you could collect information that would allow you compile a list like this:

Recent Selected Alumni Publications

Building Democracy in Contemporary Russia: Western Support for Grassroots Organizations. 2003.

The North Korean Nuclear Program: Security, Strategy, and New Perspectives from Russia. 2000.

America’s New Allies: Poland, Hungary, and the Czech Republic in NATO. 1999.

Enlarging Europe: The Industrial Foundations of a New Political Reality. 1998.

Guide to Scholars of the History and Culture of Central Asia. 1995.

2.Can you

document gains in foreign language

skills?

Ways to Track and Discuss Language GainPre and post fellowship language testing

using standardized tests.

Pre and post language testing using a specialized list of terms or concepts that you have designed.

Self-reported ratings of language speaking, reading, writing, and listening ability.

Self-reported ability to use foreign language primary source materials in various contexts.

Self-reported ability to create original materials in the foreign language – surveys, interviews, research presentations.

If a new and interesting project were proposed to you at work, and it required you to read non-technical material in a foreign language,

would you...

Accept the project with NO hesitation.

42%

Accept the project with some hesitation.

43%

Decline to participate due to inadequate foreign language

ability.15%

If a new and interesting project were proposed to you at work, and it required you to conduct written correspondence in a foreign language,

would you...

Accept the project with NO hesitation.

16%

Accept the project with some hesitation.

56%

Decline to participate due to inadequate foreign language

ability.28%

3.Are you tracking

postgraduate employment,

education, and training?

Kathryn Hendley

1995 Social Science Research Council Post-Doctoral Fellow 1995 Publishes Developing Commercial Law in Transition Economies: Examples from Hungary and Russia. Washington, D.C.: The World Bank. 1996 Kennan Institute Research Scholar “Coping with Chaos: Enterprise Adjustment in Russia” 1996 Publishes Trying to Make Law Matter: Legal Reform and Labor Law in the Soviet Union. Ann Arbor: University of Michigan Press. 2000 National Council for Eurasian and East European Research National Research Competition “Courts and Markets in Russia: An Analysis of the Implementation of Arbitrage Court Decisions” 2001-2002 Kennan Institute Fellow “Revitalizing Law: An Analysis of the Role of Law in Russian Enterprises”

What term best describes your current employment sector? American Councils' Outbound Alumni Survey

(N=674)

79

14

52 1

33

11

4

14

0

5

10

15

20

25

30

35

40

45

50

Pe

rce

nt

Has knowledge of Russian ever been a factor in receiving a new job or promotion?

American Councils' Outbound Alumni Survey (N=674)

Yes, crucial43%

Yes, it helped19%

No38%

What % of ALL Jobs Every Held by the Respondent Required Knowledge of Russian or Russian Society?

American Councils' Outbound Alumni Survey (N=623 people who have ever held jobs)

35

8

20

13

24

37

7

22

11

23

0

5

10

15

20

25

30

35

40

45

50

None < 25% 26-50% 51-75% 76-100%

Pe

rcen

t o

f R

es

po

nd

en

ts

Required Russian

Required Knowledge ofRussian Society

For which of the following do you use Russian on a regular basis? American Councils' Outbound Alumni Survey

(N=309, only those who use Russian regularly)

4543 42

39

2925

23

1714 13

0

5

10

15

20

25

30

35

40

45

50

Pe

rce

nt

4.Can you document

that you are producing experts in areas of vital interest to the

U.S.?

By looking up alumni names in Lexus/Nexus or Thomas at www.loc.gov/thomas, you could make up a chart like this:

House Committees that Title VIII Alumni Have Testified Before (Data based on Lexis/Nexus searches on 2,000 alumni.)

1

2

2

1

1

3

1

7

2

1

1

1

0 1 2 3 4 5 6 7 8

Agriculture

Appropriations

Armed Services

Banking and Financial Services

Education and the Workforce

Foreign Affairs

Government Operations

International Relations

Judiciary

National Security

Science

Small Business

Number of Individual Instances

2002 US-Russian Relations: An Assessment.2001 Silencing Central Asia: The Voice of Dissidents.2000 Academic Achievement for All.2000 Hearing on the Balkans: What are U.S. Interests and the Goals of U.S. Engagement?2000 State of Democratization and Human Rights in Turkmenistan.1999 Assisting Russia: What Have We Achieved After Seven Years?1999 Russia's Foreign Policy Objectives: What are they?

1998Examination of the Russian Economic Crisis and the International Monetary Fund Aid Package.

1998 U.S. and Russia, Part II: Russia in Crisis.1997 U.S.-Russian Cooperation in Human Spaceflight.1996 Effects of Bank Consolidation on Small Business Lending.1996 Proliferation Threats and Missile Defense Responses.1994 Foreign Operations, Export Financing, and Related Programs Appropriations.1994 U.S. Policy Toward Bosnia and the Balkans.1993 Trade and Investment Between the U.S. and Russia/ CIS.1993 Current Agricultural Situation in Russia.1992 Democracy Building in the Former Soviet Union.1992 U.S. Assistance to Central and Eastern Europe.

1991Impact of the Persian Gulf War & the Decline of the U.S.S.R. on how the U.S. does its Defense Business.

1991 Romanian Adoptions.1988 Defense Burdensharing: The Costs, Benefits, and Future of U.S. Alliances.

Title VIII Alumni TestimonyBefore House Committees

By collecting information on the affiliation of people who use your centers, you could re-package the information in a chart like this:

Title VIII Award Recipients with U.S. Print Media Affiliations

(Data based on a review of 3,174 awards made between 1985-2003)

Forbes MagazineThe Washington PostPrinceton Research ForumJournal of International Wildlife Law & Policy

East European Politics & SocietyPost-Communist StudiesThe Ukrainian Quarterly

By asking fellows whether they subsequently shared information that they learned as a result of your award, you might find:

Diffusion Effect: 1 36

We found that every person who went to the U.S. on exchange, subsequently trained more than 36 people (superiors, co-workers, professional colleagues, government officials, and young people) when they returned home.

2003 Community Connections Alumni Survey, (N=5,429).

Assessment requirements did not start under the last administration and they are not going to go away.

As taxpayers you should demand value for your money. Please assist us in making the case for the value of your project and your programs in general.

Measuring language gain efficiently and inexpensively.

Measuring language gain in a manner that allows for comparisons across languages.

Tracking students once they graduate.

Documenting the long-term career impact of your program on fellows.

Documenting the long-term policy impact of your programs.

Title VI Evaluation Challenges

Don’t leave us in the dark! Give us some suggestions about what you could reasonably assess.

We could come up with an endless list of data that you need to submit. But we respect your judgment as educators. We would rather make assessment a collaborative process.

Set out to obtain data that move knowledge forward by documenting what works, what does not work, and why.

At the end of the project, you may conclude that your original model didn’t get you the results you had hoped for. If so, tell us what you would change next time – the schedule, the activities, the partners, the type of students that your program targets, etc.

YOU WILL NOT BE PUNISHED FOR PROVIDING AN HONEST ASSESSMENT. YOU WILL EDUCATE US ABOUT WHAT WORKS.

1: Refining the Project

2: Identifying the Data to Collect

3: Setting the Timeline

4: Identifying Interested Parties to Share Results With

A Good Evaluation Plan Has 4 Parts

Part 1: Refining the Project:Things to Consider

1-3 main themes, a.k.a “project goals”.

1-2 key questions per theme. These are your “project objectives”.

What will be documented or measured so that you will be able to determine whether you met your objectives?

Speed and extent of change.

Part 2: Data Collection

Baseline measures.

Data collection instruments.

Who is the respondent, interview subject, etc. Who is changing?

Comparison or control group, if possible.

Part 3: Timeline

When will your evaluation instruments be drafted?

When will you collect your data?

When will you analyze your data?

Will your evaluation results provide feedback during the project that will enable you to modify project activities? Which activities might you modify?

When will your written evaluation findings be ready for an outside audience?

Part 4: Dissemination:Who should hear about your

findings?

On campus?In the local community?At similar institutions?Professional groups & associations?Professional colleagues?Local, state, and federal agencies & officials?Interested others?

Now that you have identified who should know about your project, review the project

objectives and data collection plan one more time.

Did you address issues of high interest to the people you intend to share results with?

Are you collecting data that will be convincing to a skeptic?

Will the data you collect help you rule out alternative explanations for any changes you notice?

Role of the Outside EvaluatorAssists the Project Director in completing the initial evaluation plan.

Assists the Project Director in designing the evaluation instruments.

This includes making the PD aware of existing evaluation instruments and comparison data available in the public domain.

Together the Project Director and Outside Evaluator

Work together on the analysis of assessment data that are collected.

Decide how you divide up data entry and analysis tasks.

The Project Director’s Evaluation Responsibilities Include:

Collecting baseline data.

Collecting project evaluation data.

Implementing any changes to the project as a result of the preliminary evaluation.

Disseminating evaluation results.