SURVEY OF RECENT GRADUATES AS AN INSTRUMENT FOR ASSISTING IN THE ASSESSMENT OF

INSTITUTIONAL EFFECTIVENESS

Dr. Teresa WardMs. Beth Katz

Office of Institutional ResearchGeorgia State University

Presentation Outline

• Conceptual Model• Technical Logistics• Dissemination of Survey Data• Utilization of Data

Objectives Content Questions

General educationlearning outcomes

critical thinking skills, writing skills, oral presentation skills, research and analytical skills,cooperative learning skills, independent learning skills, technicalskills

1, 2, 3, 4,5, 6, 7, 8,9, 10, 11,13

Program of study learning outcomes

instruction in major program, academicsupport, scholarship, preparation forAdvanced study/career, faculty support

23, 24, 25

StudentEngagement

interaction with faculty, interactionwith other students, social events,campus meetings, athletic events, useOf recreation facilities

6, 17, 27,28, 29, 30,31

Student Satisfaction

courses, instruction, advisement,Recommendation of program

14, 15, 16,18, 19, 20,21, 22, 26

Survey Matrix

• General Academic Skills – Questions 1-13 • Major Area of Study and University Support – Questions 14-26• Engagement – Questions 27-31• Comments

SurveyAdministration

Current Process Online Process

Time/Expense Level Time/Expense Level

Survey framework (matrix construction)

High High

Material costs High None

Data integrity Medium to High High

Timely dissemination of survey information

Low High

Response Rate Low to Medium Medium to High

Survey of Recent GraduatesResponse Rates

0

10

20

30

40

50

60

70

%

Recipients of GraduateDegrees

Recipients ofUndergraduate Degrees

Please rate the degree of competency you had in the following areas when you entered/graduated from Georgia State University

Graph 1 (Entering) Graph 2 (Exiting)

*Mean range: 1=very weak competency to 5=very strong competency**p<.001 all pairs

Learning Outcomes Communication Technology

Collaboration Quantitative Skills

Critical Thinking Contemporary Issues

Mean

5

4.5

4

3.5

3

English Majors

All Others

Ability to write clearly and effectively

Entering

Ability to write clearly and effectively

Exiting

3.7

3.9

4.7

4.5

English Majors

(Seniors)

All Others

(Seniors)

Prepared two or more drafts of a paper or assignment before turning it in

2.63 2.47

Number of written papers or reports of 20 PAGES OR MORE

1.50 1.57

Number of written papers or reports BETWEEN 5 AND 19 PAGES

2.93 2.57

Number of written papers or reports of FEWER THAN 5 PAGES

2.96 2.87

Institutional contribution: Writing clearly and effectively

3.56 3.11

2007 NSSE Writing Items

Ability to analyze quantitative problems Entering

Mean

5

4.5

4

3.5

3

4.3

Ability to analyze quantitative problems Exiting

Math Majors

All Others

3.9

3.7

4.7

4.3

Mathematics Majors

(Seniors)

All Others

(Seniors)

Number of problem sets (problem-based homework assignments) that take you MORE than an hour to complete

2.80 2.43

Number of problem sets (problem-based homework assignments) that take you LESS than an hour to complete

3.00 2.22

Institutional contribution: Analyzing quantitative problems

3.60 2.99

Coursework emphasized: MEMORIZING facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form

2.83 2.76

2007 NSSE Quantitative Items

Apply scientific reasoning in problem-solving

Entering

Mean

5

4.5

4

3.5

3

Physics Majors

All Others

Apply scientific reasoning in problem-solving Exiting

3.73.6

4.9

4.3

GPA

3.7-4.03.0-3.692.0-2.99

5.00

4.80

4.60

4.40

4.20

4.00

3.80

Mean of communication (exiting)

Mean of communication (entering)

GPA3.7-4.03.0-3.692.0-2.99

Mean of quantitative skills (entering)

5.00

4.80

4.60

4.40

4.20

4.00

3.80

3.60

Mean of quantitative skills (exiting)

GPA3.7-4.03.0-3.692.0-2.99

Mean of critical thinking (entering)

5.00

4.80

4.60

4.40

4.20

4.00

3.80

3.833.86

3.92

4.47*

4.564.63*

*p<.01

Mean of critical thinking (exiting)

GPA3.7-4.03.0-3.692.0-2.99

Mean of technology (entering)

5.00

4.80

4.60

4.40

4.20

4.00

3.80

Mean of technology (exiting)

*Mean range: 1=poor to 4=excellent

Spring 2005

Summer 2005

Fall 2005

Spring 2006

Summer 2006

Fall 2006

Spring 2007

Summer 2007

Fall 2007

Mean Mean Mean Mean Mean Mean Mean Mean Mean

Quality of instruction In your major program

3.44 3.52 3.43 3.42 3.45 3.46 3.43 3.51 3.45

Scholarly competency Of faculty in your major program 3.62 3.68 3.58 3.59 3.57 3.61 3.56 3.65 3.61

4 point scale: Poor to Excellent

Qualitative Variable Information

Variable Position Label date 1 Date level 2 Level K1 3 academic advisement K2 4 academic challenge K3 5 activities K4 6 administrative services K5 7 alumni K6 8 class size K7 9 career advisement K8 10 communication K9 11 continuing education K10 12 course content K11 13 course offerings K12 14 course schedule K13 15 diversity K14 16 environment K15 17 facilities K16 18 fees K17 19 general comment (negative) K18 20 general comment (neutral) K19 21 general comment (positive) K20 22 grading and assessment K21 23 Negative K22 24 Neutral K23 25 Positive K24 26 professors/instructors K25 27 program K26 28 program requirements K27 29 quality of students K28 30 safety K29 31 specific professors/instructors K30 32 staff K31 33 support Variables in the working file

0

20

40

60

80

100

Positive

Negative

Neutral

Utilization of Survey Data

• Department Level– Identify strengths and weaknesses using

comparative data– Identify trends and implications of program

changes using longitudinal data– Ability to triangulate survey data with other

assessment measures– Starting point for conversations

• University Level– Identify trends and implications of University

policy changes using longitudinal data– Ability to triangulate survey data with other

assessment measures– Contribute to strategic planning and policy

discussions

Conclusion

• High reliability index• Focused on indirect measures of learning outcomes

• Saved OIR a considerable amount of money and time via online administration

• Increased response rates• Facilitated access to the data• Contributed to department and University assessment activities

• Teresa Ward – tward@gsu.edu• Bethann Katz – bnorthup@gsu.edu• Office of Institutional Research, Georgia State University –

http://www2.gsu.edu/~wwwire/