Assessing Impact: Study Assessing Impact: Study Abroad at Two InstitutionsAbroad at Two Institutions

Elaine Meyer-LeeElaine Meyer-Lee Saint Mary’s CollegeSaint Mary’s College

Carol WarfieldCarol WarfieldAuburn University Auburn University

2006 AIEA Annual Conference2006 AIEA Annual Conference February, 2006February, 2006

Why Assess Outcomes?Why Assess Outcomes?

1,022 US institutions reported information 1,022 US institutions reported information on their study abroad programson their study abroad programs

174,629 U.S. students studied abroad174,629 U.S. students studied abroad

Data from Open Doors Report on International Educational Exchange 2003-2004Data from Open Doors Report on International Educational Exchange 2003-2004

Inform continuous quality Inform continuous quality improvementimprovement

Advocate for international educationAdvocate for international education

Satisfy regional accrediting bodiesSatisfy regional accrediting bodies

Add to basic understanding of Add to basic understanding of student growth in this areastudent growth in this area

Young, wide-open fieldYoung, wide-open field

Much need for better methods and more Much need for better methods and more data on many different dimensionsdata on many different dimensions

This presentation is not a step-by-step This presentation is not a step-by-step overview, but two examples overview, but two examples

(See Forum’s upcoming Guide to (See Forum’s upcoming Guide to Outcomes Assessment in Education Outcomes Assessment in Education Abroad)Abroad)

Multi-method Longitudinal Assessment of Multi-method Longitudinal Assessment of Intercultural Engagement, IC Sensitivity, Intercultural Engagement, IC Sensitivity, Identity, and Goals: Preliminary ResultsIdentity, and Goals: Preliminary Results

Elaine Meyer-LeeElaine Meyer-Lee Director of International & Intercultural LearningDirector of International & Intercultural Learning

Joy EvansJoy EvansResearch Analyst-Center for Women’s InterCultural LeadershipResearch Analyst-Center for Women’s InterCultural Leadership

Saint Mary’s CollegeSaint Mary’s CollegeNotre Dame, Indiana Notre Dame, Indiana

206 AIEA Annual Conference206 AIEA Annual Conference February 25, 2006February 25, 2006

Presentation OverviewPresentation Overview

ContextContext

Assessment Methods Assessment Methods

Review of Instruments Review of Instruments

Preliminary ResultsPreliminary Results

Contributions to Study Abroad AssessmentContributions to Study Abroad Assessment

Saint Mary’s CollegeSaint Mary’s College

Catholic Women’s Liberal ArtCatholic Women’s Liberal Art Established 1844Established 1844

About 1600 enrollmentAbout 1600 enrollment• 98% full-time98% full-time• 82% on-campus82% on-campus• 85% Catholic85% Catholic• 9% students of color9% students of color

20% of students study abroad20% of students study abroad

Mission of the Mission of the Center for Women’s InterCultural LeadershipCenter for Women’s InterCultural Leadership

Saint Mary's College's mission includesSaint Mary's College's mission includespreparing women to make a differencepreparing women to make a difference

in an increasingly complex andin an increasingly complex andinterdependent world. The role of theinterdependent world. The role of the

Center for Women's InterCulturalCenter for Women's InterCulturalLeadership (CWIL) within this missionLeadership (CWIL) within this mission

is to foster the intercultural knowledgeis to foster the intercultural knowledgeand competence critical to educatingand competence critical to educating

the next generation of women leaders.the next generation of women leaders.

CWIL at Saint Mary’sCWIL at Saint Mary’s

Encompasses much more than study Encompasses much more than study abroad (see brochures)abroad (see brochures)

Is adopting Byram’s definition of Is adopting Byram’s definition of intercultural competenceintercultural competence

Is launching a portfolio-based Is launching a portfolio-based certificate program in Women’s certificate program in Women’s Intercultural LeadershipIntercultural Leadership

Study Abroad ProgramsStudy Abroad Programs

Year/ Semester• Ireland (Year only)• India (Semester only)• Rome• Spain• France• Australia• Austria• South Africa

4-2 Weeks• European Studies (4)• Internships in Europe (4+)• Honduras (3)• Ecuador (3-2) • Greece (2 ½) • S. Korea (2)

1 Week• Prague/Poland• Jamaica• Mexico

Study Abroad ProgramsStudy Abroad Programs

Assessing Study Abroad… Assessing Study Abroad… Going Beyond Going Beyond

Student SatisfactionStudent Satisfaction

Evaluation & Assessment GoalsEvaluation & Assessment Goals

Evaluate programs’ effectivenessEvaluate programs’ effectiveness Measure Students’:Measure Students’:

• Changes in sensitivity to cultural differences Changes in sensitivity to cultural differences • American identity developmentAmerican identity development• Own sense of growth toward goalsOwn sense of growth toward goals

Explore correlations of change with:Explore correlations of change with:• Intercultural engagement while abroad Intercultural engagement while abroad

(interaction and reflection)(interaction and reflection)• Program or demographic characteristicsProgram or demographic characteristics

Assessment Methods/DesignAssessment Methods/Design

Longitudinal Research DesignLongitudinal Research Design• Pre-test/Post-test (some grad follow-up)Pre-test/Post-test (some grad follow-up)• Procedures – 70% response rate so farProcedures – 70% response rate so far

Control Group Control Group • 30 First-year students30 First-year students

Focus GroupFocus Group• Study Abroad Re-entry CourseStudy Abroad Re-entry Course• Written reflection and Kolb’s LSI Written reflection and Kolb’s LSI

Assessment Instruments: Pre-TestAssessment Instruments: Pre-Test

Study Abroad ParticipantsStudy Abroad Participants

Intercultural Development Intercultural Development Inventory (IDI)Inventory (IDI)

American Identity American Identity QuestionnaireQuestionnaire

Goals/motivations from Goals/motivations from Saint Mary’s College Study Saint Mary’s College Study Abroad SurveyAbroad Survey

Control GroupControl Group

Intercultural Intercultural Development Inventory Development Inventory (IDI)(IDI)

American Identity American Identity QuestionnaireQuestionnaire

Assessment Instruments: Post-TestAssessment Instruments: Post-Test

Study Abroad ParticipantsStudy Abroad Participants

Intercultural Development Intercultural Development Inventory (IDI)Inventory (IDI)

American Identity American Identity Questionnaire (AIQ)Questionnaire (AIQ)

SMC Study Abroad Survey SMC Study Abroad Survey Program Impact & Program Impact & Measure of Intercultural Measure of Intercultural Engagement (MIE)Engagement (MIE)

Control GroupControl Group

Intercultural Intercultural Development Development Inventory (IDI)Inventory (IDI)

American Identity American Identity QuestionnaireQuestionnaire

Data Collection TimelineData Collection Timeline2004-20052004-2005 2005-20062005-2006 2006-20072006-2007

Fall Fall SpringSpring FallFall SpringSpring FallFall SpringSpring

Pre-testPre-test

N=16----N=16----

Pre-testPre-test

N=44----N=44----

--------------------

--------------------

Pre-testPre-test

N=83----N=83----

Control Control

N=32----N=32----

Post-testPost-test

N=(16)N=(16)

Post-testPost-test

N=(44)N=(44)

Post-testPost-test

N=(6)---------N=(6)---------

--------------------------------

Pre-testPre-test

N=(?)---------N=(?)---------Spring SemesterSpring Semester

Post-testPost-test

N=(48?)--N=(48?)--Fall Fall SemesterSemester

----------------------

----------------------

Pre-test?Pre-test?

N=(?)----N=(?)----SummerSummer

Post-testPost-test

N=(29?)N=(29?)

YearYear

ControlControl

N=(32)N=(32)

Post-testPost-test

N=(?)N=(?)

Post-test?Post-test?

N=(?)N=(?)

Review of InstrumentsReview of Instruments

Intercultural Development Inventory Intercultural Development Inventory (IDI)(IDI)

50-item Self-Assessment50-item Self-Assessment

Valid & Reliable Psychometric Measure Valid & Reliable Psychometric Measure

Culture general in focusCulture general in focus

Quantifies the subjective experience of cultural differenceQuantifies the subjective experience of cultural difference

Most widely used currentlyMost widely used currently

Based on Milton Bennett’s Developmental Model of Based on Milton Bennett’s Developmental Model of Intercultural Sensitivity (constructivist stage model)Intercultural Sensitivity (constructivist stage model)

Slide Available Online:http://www.coe.int/T/E/North-South_Centre/Programmes/3_Global_Education/c_GENE/GENE%20Web-pageSlide Available Online:http://www.coe.int/T/E/North-South_Centre/Programmes/3_Global_Education/c_GENE/GENE%20Web-page%20-%20(3)%20Hesse%20article.asp%20-%20(3)%20Hesse%20article.asp

IDI Measurement ScalesIDI Measurement Scales

Denial/DefenseDenial/Defense

ReversalReversal

MinimizationMinimization

Acceptance/AdaptationAcceptance/Adaptation

Encapsulated MarginalityEncapsulated Marginality

Overall ProfileOverall Profile

American Identity QuestionnaireAmerican Identity Questionnaire 10-item Likert-style Self Assessment10-item Likert-style Self Assessment

Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record)track record)

Based on Eriksonian identity development theory. Measures 2 Factors: Based on Eriksonian identity development theory. Measures 2 Factors:

• identity search (e.g. I have often talked to other people about what it means to identity search (e.g. I have often talked to other people about what it means to be an American.)be an American.)

• affirmation, belonging, and commitment (e.g. affirmation, belonging, and commitment (e.g. Being American plays an Being American plays an important part in my life.)important part in my life.)

One goal of study abroad is more complexity in understanding own culture, One goal of study abroad is more complexity in understanding own culture, and students in pilot focus group felt it captured an important dimensionand students in pilot focus group felt it captured an important dimension

Currently refining adaptation’s psychometric propertiesCurrently refining adaptation’s psychometric properties

Saint Mary’s College Saint Mary’s College Study Abroad SurveyStudy Abroad Survey

Semester/Year-Long Semester/Year-Long ProgramsPrograms

Core SurveyCore Survey• 111 Items111 Items• 4 open-ended 4 open-ended

questions questions

Customized AddendumCustomized Addendum

Summer/Short-term Summer/Short-term ProgramsPrograms

Core SurveyCore Survey• 58 Items58 Items• 1 open-ended 1 open-ended

questions questions

Customized AddendumCustomized Addendum

Saint Mary’s College Saint Mary’s College Study Abroad SurveyStudy Abroad Survey

Semester/Year-Long ProgramsSemester/Year-Long Programs

Question Sections Include:Question Sections Include:• Demographics/BackgroundDemographics/Background• GoalsGoals• Pre-DeparturePre-Departure• Academic ProgramAcademic Program• Cultural Tours & ExcursionsCultural Tours & Excursions• On-Site Student ServicesOn-Site Student Services• Intercultural EngagementIntercultural Engagement• Overview of ProgramOverview of Program• Program Impact Program Impact (Goals)(Goals)• Volunteer InterestsVolunteer Interests

Summer/Short-term Summer/Short-term ProgramsPrograms

Question Sections Question Sections Include:Include:• DemographicsDemographics• Pre-DeparturePre-Departure• Living ArrangementsLiving Arrangements• Intercultural Intercultural

EngagementEngagement• Overview of ProgramOverview of Program• Volunteer InterestsVolunteer Interests

Self-reported Growth in:Self-reported Growth in: Understanding of different culturesUnderstanding of different cultures

Different perspective on American cultureDifferent perspective on American culture

Greater understanding of the US in world affairs and historyGreater understanding of the US in world affairs and history

Different perspective on your ethnic heritageDifferent perspective on your ethnic heritage

Understanding of my faithUnderstanding of my faith

Self-awarenessSelf-awareness

Maturity and self-confidenceMaturity and self-confidence

Interpersonal skills and the ability to adaptInterpersonal skills and the ability to adapt

Intellectual development in the liberal artsIntellectual development in the liberal arts

Ability to communicate in a foreign language Ability to communicate in a foreign language

Measure of Intercultural Measure of Intercultural EngagementEngagement

Asks students to quantify the Asks students to quantify the frequency of their engagement frequency of their engagement through:through:

Interaction with cultural resourcesInteraction with cultural resources Interaction with local peopleInteraction with local people Explicit reflection on these Explicit reflection on these

interactionsinteractions

Interaction with cultural resourcesInteraction with cultural resources

Theater/opera/symphony/concerts Theater/opera/symphony/concerts

Museums/ historical sitesMuseums/ historical sites

Forests/parks/beaches Forests/parks/beaches

Traveling outside of the host country Traveling outside of the host country

Traveling within the host country Traveling within the host country

Interaction with local peopleInteraction with local people Shopping/visiting local marketsShopping/visiting local markets Movies/entertainment catering to localsMovies/entertainment catering to locals Restaurants/pubs/cafes catering to localsRestaurants/pubs/cafes catering to locals Visiting local families/friends in their homesVisiting local families/friends in their homes Participating in worship servicesParticipating in worship services On-campus informal conversationsOn-campus informal conversations Events sponsored by the study abroad programEvents sponsored by the study abroad program Extra-curricular activities (student clubs, Extra-curricular activities (student clubs,

intramurals, etc.) intramurals, etc.) Volunteering, service-learning, internships, Volunteering, service-learning, internships,

shadowingshadowing Reading or listening to local news Reading or listening to local news

Explicit reflection on these Explicit reflection on these interactionsinteractions

Journaling/BloggingJournaling/Blogging

Photography Photography

Other artistic expression (painting, drawing, collages, etc.) Other artistic expression (painting, drawing, collages, etc.)

Class discussionsClass discussions

Informal conversations with localsInformal conversations with locals

Informal conversations with students studying abroadInformal conversations with students studying abroad

Calling/e-mailing family/friends/faculty in America about Calling/e-mailing family/friends/faculty in America about experience experience

Pre-test ResultsPre-test Results IDI Worldview Profile: Pre-AbroadIDI Worldview Profile: Pre-Abroad

IDI Worldview Profile “Pre”: IDI Worldview Profile “Pre”: ControlsControls

Pre-test Comparison to Control: Pre-test Comparison to Control: American Identity Questionnaire American Identity Questionnaire

Affirmation, belonging and Affirmation, belonging and commitment commitment

no significant differenceno significant difference

Identity searchIdentity search

statistical significance statistical significance (at the .05 level) (at the .05 level)

Summary of Pre-test FindingsSummary of Pre-test Findings

IDI: Students’ scores do not support IDI: Students’ scores do not support developmental modeldevelopmental model

IDI: Low Minimization Score IDI: Low Minimization Score • Catholic beliefs ….We’re all God’s childrenCatholic beliefs ….We’re all God’s children

Control on both: Heritage Seeker FactorControl on both: Heritage Seeker Factor• Are study abroad participants a different Are study abroad participants a different

population than other SMC students?population than other SMC students?

Preliminary Analysis of Change Preliminary Analysis of Change Data for Matched SetData for Matched Set

No significant change in means for No significant change in means for IDI or AIQ yet (on paired T-test on IDI or AIQ yet (on paired T-test on pre-post mean scores, n=42 and 27)pre-post mean scores, n=42 and 27)

BUT, high self-reported growth (1.7 BUT, high self-reported growth (1.7 when 1 is high) and wide range in when 1 is high) and wide range in change scores and program meanschange scores and program means

What about those who do change?What about those who do change?

Correlation of AIQ with IDI scales (bivariate Correlation of AIQ with IDI scales (bivariate non-parametric correlations)non-parametric correlations)

Correlations with Intercultural Correlations with Intercultural Engagement:Engagement:• Cultural resource scale with AIQ changeCultural resource scale with AIQ change• Overall and reflection with program impact Overall and reflection with program impact

(especially interpersonal items)(especially interpersonal items) 50% did grow on at least one scale 50% did grow on at least one scale

(defined as 2 standard deviations of (defined as 2 standard deviations of positive change, n=21)positive change, n=21)

How do the “growers” compare?How do the “growers” compare?

CharacteristicCharacteristic GrowersGrowers Non-GrowersNon-GrowersFrom N.America From N.America 91%91% 9595%%WhiteWhite 67%67% 8686%%LatinaLatina 2424%% 4.5%4.5%Never been abroadNever been abroad 53%53% 63%63%Abroad < 3 monthsAbroad < 3 months 24%24% 26%26%Whole year programWhole year program 10%10% 46%46%Semester programSemester program 76%76% 46%46%Summer programSummer program 14%14% 8%8%Rome programRome program 24%24% 3232%%Seville programSeville program 2424%% 5%5%MexicoMexico 1414%% 9%9%France France 1414%% 9%9%IrelandIreland 10%10% 4646%%

Methodological FindingsMethodological Findings

Importance and Challenges of Importance and Challenges of Control GroupControl Group

Procedural effects (e.g. on-line vs. Procedural effects (e.g. on-line vs. paper)paper)

Signs of internal consistency in Signs of internal consistency in measures (inter-item correlations)measures (inter-item correlations)

Intercultural Intercultural Development Development Inventory:Inventory: Standardized Standardized

instrumentinstrument

Ability to compare Ability to compare study abroad program study abroad program outcomes with outcomes with domestic intercultural domestic intercultural programsprograms

? Intercultural Intercultural Development Development Inventory:Inventory:? Requires qualifying Requires qualifying

training to administertraining to administer

? Highly rigorous and Highly rigorous and proprietaryproprietary

? Labor- and cost-Labor- and cost-intensive for large intensive for large scale usescale use

Benefits to Benefits to

Study Abroad AssessmentStudy Abroad Assessment & Challenges to

& Challenges to

Institutional Research Institutional Research

Program Impact & Program Impact & MEIMMEIM Comparable & program Comparable & program

specific data on students’ specific data on students’ intercultural interactionsintercultural interactions

AIQAIQ Examine identity Examine identity

dimension, connect to dimension, connect to literature on thisliterature on this

? Program Impact & Program Impact & MEIMMEIM? Labor-intensive Labor-intensive

development and development and administrationadministration

? AIQAIQ ? Reliability and Validity of Reliability and Validity of

adaptationadaptation? Needs refiningNeeds refining

Benefits to Benefits to

Study Abroad AssessmentStudy Abroad Assessment & Challenges to

& Challenges to

Institutional Research Institutional Research

Next StepsNext Steps AnalysisAnalysis

• More on change and changers with full datasetMore on change and changers with full dataset• More on correlations with intercultural engagement, More on correlations with intercultural engagement,

program variables (including Engle & Engle’s category), program variables (including Engle & Engle’s category), and student variables (including original motivation)and student variables (including original motivation)

• QualitativeQualitative• Psychometrics on AIQ, Program Impact, MIEPsychometrics on AIQ, Program Impact, MIE

Data collectionData collection• Add 1 year follow-up Add 1 year follow-up • Draw on Women’s Intercultural Leadership Portfolios?Draw on Women’s Intercultural Leadership Portfolios?

ApplicationsApplications• Many improvements to (and rationales for) programs Many improvements to (and rationales for) programs

already. Overhauling re-entry course. already. Overhauling re-entry course.

For more handouts or bibliography on For more handouts or bibliography on assessing intercultural outcomes of assessing intercultural outcomes of study abroad programs, go tostudy abroad programs, go to http://www.saintmarys.edu/~cwil/php/intercultural.learning/IIhttp://www.saintmarys.edu/~cwil/php/intercultural.learning/IILOutcomes.phpLOutcomes.php

or email me:or email me: meyerlee@saintmarys.edu meyerlee@saintmarys.edu