Upload
donald-ray
View
217
Download
1
Embed Size (px)
Citation preview
Lessons Learned from Industry: Achieving Diversity & Efficacy in
College Success
ETS - College Board Invitational Conference
Washington, DC
Wayne Camara & Krista Mattern
September 8, 2008
2
Job Analysis• Organizations use job analysis to determine what
work outcomes are desired.
• Sample individual outcomes (productivity, job performance, retention) and organizational outcomes (efficiency, quality, innovation, team work)
• Identify performance components (pc)pc = {Declarative knowledge x Procedural knowledge x
Motivation}
Knowledge x Cognitive skills x Level of effort Goals Interpersonal “ Persistence “
Ability, Interests, Education, Experience Importance & Prob. Of Outcomes
Predictors
3
Job Analysis – College Success1. Identify desired performance outcomes
for individuals and organizations (college success) (GPA, return, graduate, life after college – grad school, certification)
Each outcome likely has different predictors
2. Identify performance tasks associated with outcomes (persistence, academic ability, health, engagement)
3. Identify or develop performance measures (GPA, advisory ratings, self report data, dB of student engagement)
4
Predicting Performance
Goldstein, Zedeck, & Schneider (1993)
5
Group differences are not unique to tests: They are present across most educational measures
6
Large Mean Differences Persist on Cognitive Ability Tests by
Race/Ethnicity Remain
SA
T
CR
+M
7 Source: U.S. Census Bureau
College-Going Rates of High School Graduates Aged 18 to 24 by Ethnic Group,
1999-2006
20%
25%
30%
35%
40%
45%
50%
55%
60%
65%
70%
05-0604-0503-0402-0301-0200-0199-0098-99
Black
White
Hispanic
Asian
8
9
Disparities Exist in HS Graduation, HS Drop Out and College Ready
Source: Manhattan Institute, Public HS Graduation and College-Readiness Rates: 1991-2002, http://Manhattan-institute.org/html/ewp_08.htm;* Condition of Education, 2007 Table 23-2
10
Graduation Rates in 2004 by ethnicity
Published 3/7/2007 Title Awards conferred by Title IV institutions, by race/ethnicity, level of award, and gender: United States, academic year 2004–05 (recalculated to eliminate students who with other or no ethnicity reported). http://nces.ed.gov/ipeds/factsheets/pdf/fct_awards_conferred_03072007_5.pdf; Public HS graduation rates: WICHE 3/2008, http://www.wiche.edu/policy/knocking/1992-2022/index.asp
11
Rationale for looking beyond Grades and Tests
What is college success? Is it more than grades and GPA? (Camara & Kimmel)
Develop measures that predict your goal or desired outcome.
Employers test multiple measures:
openness, conscientiousness, extraversion, agreeableness, neuroticism
Military use today (GED).
Can do does not equal will do.
12
Predictors of College Success
Tests MeasureColleges Collect in some form (applications, transcripts)
Not Collected in Standardized form
College Skills Content Knowledge
Achievement Non-Cognitive
Personal Qualities/ Experiences/
Characteristics
School Performance/
Context
Interests - Vocations
Verbal Reasoning Math Motivation Letters Grades Career Interests
Math Reasoning Language Arts Follow-through Essay GPA Learning -Study
Skills
Writing Science Communication Community Service Weighted GPA Interest in Major
Metacognition Social Studies/
Humanities Conscientiousnes
s Extra-curricular Rank Self Efficacy
Creativity Foreign Language Leadership Work Experience Courses Completed Aspirations/
Practical Knowledge Language Proficiency Other Personality
Literacy in Second Lang Academic Rigor
Realistic Self-concept
Spatial Relations Health/Lifestyle Teacher Ratings AP/Honors Courses
Intellectual Curiosity Adaptability Gender School Size
Technology – Research Skills Ethnicity School Quality
Ability to Benefit
Family Education/
Income
Ability to Pay
Residence
13
• Identify a broader domain of college student performance:• Review university mission statements and
department objectives
• Interview with university staff responsible for student life
• Review of the education literature on student outcomes
• Our systematic search (A JOB ANALYSIS OF UNDERGRADUATE STUDENTS) resulted in 12 dimensions of student performance…
• Validate items with successful juniors – they are the experts.
Research collaboration with Michigan State University
14
12 Dimensions of Student PerformanceBroadening the Performance Domain in the Prediction of
Academic Success (Schmitt, Oswald, & Gillespie, 2004)
1. Knowledge, learning, mastery of general principles
2. Continuous learning, intellectual interest and curiosity
3. Artistic and cultural appreciation
4. Multicultural appreciation
5. Leadership
6. Interpersonal skills
7. Social responsibility, citizenship and involvement
8. Physical and psychological health
9. Career orientation
10.Adaptability and life skills
11.Perseverance
12.Ethics and integrity
15
Two “Noncognitive” Measures
• Situational judgment inventory• A situation is presented along with several
alternative courses of action.
• The respondent is asked to indicate what she/he would be most likely and least likely to do.
• Biodata• Short, multiple choice reports of past
experience/background and interests/preferences.
16
Sample SJI Item for LeadershipYou are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything.
a)-1 Look at them until someone eventually says something
b)Start the conversation yourself by introducing yourself
c)+1 Get to know everyone first and see what they are thinking about the project to make sure the project’s goals are clear to everyone
d)Try to start working on the project by asking everyone’s opinion about the nature of the project
e)You would take the leadership role by assigning people to do things or ask questions to get things rolling
17
Sample Biodata Items for Leadership
1. The number of high school clubs and organized activities (such as band, sports, newspapers, etc.) in which I took a leadership role was:a) 4 or more
b) 3
c) 2
d) 1
e) I did not take a leadership role
2. How often do you talk your friends into doing what you want to do during the evening?a) most of the time
b) sometimes (about half the time)
c) occasionally (about as often as others in my group
d) seldom or infrequently
e) never
18
Study 1: Develop and refine the measures
• 644 MSU freshmen completed one of the two parallel forms of the biodata and SJI instruments at the beginning of the academic year.
• Results indicated significant incremental validity for some of the scales above and beyond the validity of SAT/ACT scores and existing measures of personality in predicting college GPA.
• The biodata and SJI demonstrated the greatest incremental validity when absenteeism, students’ self ratings, and peer-ratings of performance were examined ( .19, .22, and .14, respectively).
19
Study 2 Examine Validity & Subgroup Differences:
10 Participating Institutions & 2,700 Freshmen HBCU N
Winston-Salem (public) 229
Spelman College (private) 254
Big Ten (public) N
University of Iowa 335
Michigan State University 546
Ohio State University 304
University of Michigan 297
Indiana University 170
Other Institutions N
University of Chicago (private) 168
Cal State – Fullerton (public) 223
Virginia Tech (public) 237
20
Predicting FYGPA: Total Sample across 10 Institutions (N = 2443)
Non cognitive measures contribute little beyond tests and grades in predicting academic outcomes
21
Predicting Class Absenteeism: Total Sample across 10 Institutions (N = 899)
However, non cognitive measures will predict non cognitive outcomes – better than tests or grades (graduation, attendance, leadership, engagement)
22
Percent of Students Selected:Two Composites and Three Selection Strategies
Top 85% Top 50% Top 15%
Group AB AB+ AB AB+ AB AB+
Hispanic 4.4 4.6 4.1 4.9 3.9 5.5
(+.2) (+.8) (+1.6)
Asian 7.6 7.7 9.9 9.5 17.5 12.9
(+.1) (-.4) (-4.6)
African-American 17.9 19.8 9.6 13.6 1.3 7.2 (+1.9) (+4.0) (+5.9)
White 70.2 67.9 76.4 71.9 77.2 74.4
(-2.3) (-4.5) (-2.8)
AB = equally weighted composite of HSGPA and SAT/ACT.
AB+ = equally weighted composite of HSGPA, SAT/ACT, Biodata, and SJI.
Less selective Moderately selective Very selective
23
Correlations of Non-cognitive Measures with Cumulative GPA
and GraduationVariable Cumulative GPA Graduation
SAT/ACT scores 0.59 0.24
HSGPA 0.56 0.28
Knowledge 0.24 0.12
Continuous Learning 0.11 0.03
Artistic Appreciation 0.22 0.13
Multicultural Appreciation 0.12 0.11
Leadership 0.10 0.14
Responsibility 0.12 0.15
Health 0.13 0.08
Career Orientation -0.18 0.02
Adaptability 0.01 0.07
Perseverance 0.02 0.10
Ethics 0.15 0.11
SJI 0.20 0.14
Note. Bold values are significant at p< .01. N ranges from 1560 to 1798 across variables. Graduation is dichotomously scored (1, 0).
24
Study 3: Purpose & Research Questions• 15 institutions (n = 4,164 for SJI and 7,645 for biodata)
• Purpose: evaluating the utility of the biodata and situational judgment measures in as close to a real admissions situation as is possible
• Administer new measures to college applicants rather than college freshmen.
• On an annual basis, collect class absenteeism, self rated performance of the noncognitve dimensions, and commitment to the university from enrolled students; institutions will provide course grades and retention.• University of Washington • Meredith College
• Michigan State University • University of Southern California
• Lafayette College • Furman University
• Earlham College • University of North Carolina, Chapel Hill
• Ohio State University • Kenyon College
• Purdue University • Gonzaga University
• Spelman College • University of Puget Sound
• Johnson and Wales University
25
Incremental Validity of Biodata Measures
Outcomes N R2 (HSGPA,SAT) Overall R2 R2
BARS 57 0.023 0.443* 0.420*
OCB 57 0.017 0.392 0.374*
Deviance 57 0.025 0.373 0.348
Turnover Intent 58 0.077 0.248 0.172
Academic Satisfaction 58 0.008 0.353 0.345
Social Satisfaction 58 0.077 0.294 0.218
FYGPA 84 0.201* 0.335* 0.134
Absenteeism 58 0.061 0.234 0.173
• To preserve N in these regressions, the SJI was not included because of a relatively low response rate to this measure.
• It is worth noting that small sample sizes, such as those observed in these analyses, can seriously limit the ability to detect significant relationships due to decreased statistical power.
26
Next Steps• In need of a demonstration project – Implement with Research
across a few colleges!
• Encourage applicants to complete on-line as part of admissions and only use data as a “plus factor.”
• Provide incentives for applicants to complete the new measures and institutions to track student success over time.
• Likely outcomes will be more diversity, broader talent, greater retention, and standardized – defensible measures to evaluate applicants fairly and objectively.
• Increased efficiency and judgmental decisions based on data and comparability.
• For more information, go to http://www/iopsych.msu/cbstudy