Volkwein- Penn State Assessing Student Attainment in the Undergrad. Major: Examples of Discipline-Specific Strategies Senior Thesis or Writing Project

Volkwein- Penn State

Assessing Student Attainment in the Undergrad. Major:

Examples of Discipline-Specific Strategies•Senior Thesis or Writing Project•Senior Research Project•Capstone Course with Assessment Embedded in the

Course•Comprehensive Examination or Proficiency Test•Student Portfolio, Performance, or Exhibit

-- with expert or peer assessment•Senior Essay followed by Focus Group Discussions

with Faculty•Student/Faculty Retreat for Collective Assessment•Internship or Field Work•Analysis of Historical, Archival, or Transcript Data•Alumni and Employer Surveys and Interviews


Should I purchase an instrument or create my own??

If you develop your own instrument:

• can be less expensive (at least in dollars)• can address institutionally-specific issues• can be very time consuming• must attend to formatting, ease of completion

for respondents, printing, on-site scoring

• may require pilot testing• can require extensive commitments for

reliability, validity checks or can jeopardize credibility

• Has less external credibility, can increase chances for criticism


Literature Review

and Collection of Survey

Items

Item Reduction & Revisions by

Research Team

Pilot Test to Determine

Psychometric Properties &

Revise

Final Survey Items

Vet with Faculty, Employers, Alumni, and Administrators

AlumniFaculty

Development of Local Instruments

Seniors

Program Chairs

Employers


Tips for Improving Response Rates

Explain the Importance of the Survey

Explain How the Results will be Used

Assure Confidentiality

Thank Participants

Make the Instrument Interesting

Offer Incentives, if possible

Offer to Share the Results

Be sure the Survey is easy to understand and complete

Offer multiple modes of response (web, paper, telephone)


Should I purchase an instrument or create my own??

If you purchase an instrument:

Requires less time for design, printing, etc. Likely has established reliability and validity Often includes norms groups for inter-

institutional comparisons May include scoring and reporting services Likely to have external credibility, Increases

face validity May not fit your curriculum & goals Can be expensive in dollars


Advantages of Standardized Tests

Validity Established through content specification expert panel reviews for fairness pilot testing

Minimal Staff Time Involved Comparative Data and User Norms Available Short Turn-around Time for Results Access to Professional Expertise

Why not use the GRE for assessment? GRE Exams have the following deficiencies. 1. GRE scores are relational and only answer the comparison question.

You don’t know what a 600 means in terms of student achievement because you don’t know how many questions were answered correctly and incorrectly.

2. The GRE uses the wrong comparison group. Instead of comparing each GRE score against all college graduates, student GRE scores are in relation to a graduate school-bound population.

3. There are no GRE subfield scores within each major field, so you cannot tell if student performance is congruent with the curriculum.

The ETS Major Field Exam was constructed in response to these weaknesses in the GRE. The Major Field Exam scores not only are normed on populations of graduating seniors, and not only indicate the number answered correct (non relational), but also report scores by subfield, thus providing useful information for analyzing the curriculum. I recommend that the faculty send off for a specimen set, take the exam, discuss and decide.


Validity of Self-Reported Data

Self-reports vs. actual scores/tests in various studies shown to correlate (r = .50 to .90) with:

SAT & GRE Verbal and Quantitative scores National Teacher Examination scores Criterion-referenced achievement tests in

English, math, science, and social studies College and High School GPA Course material tests Behaviors/activities consistent with reported

gains


When Self-Reports are Reasonable Proxies for Objective Measures

The information requested is known to the respondent Questions are phrased clearly and unambiguously A moderate-to-high degree of overlap in content exists

between the measures The instruments measure the same constructs, and

questions refer to recent activities Respondent believes questions merit a serious, thoughtful

response Answering won’t threaten, embarrass, or violate the

respondent’s privacy, nor encourage socially desirable responses

Self-reported gains/proficiencies are for groups (averaged), rather than for individuals

(Based on Kuh, 2005 and Pike, 1995)


Strengths of Surveys

Great Flexibility and Universal Applicability

Ease of Construction Application to all Students Results do not Require Expert

Interpretation Relatively Inexpensive


Weaknesses of Surveys

Better for Measuring Group Responses than for Individual Responses

Best if Supplemented by other information from Interviews & Focus Groups.

To Construct Good Surveys avoid vague items Construct multi-item scales Link to constructs in your model

or to educational goals Effective visual layout


Value of Alumni Studies(Volkwein, 1990)

Alumni provide a ‘Janusian’ perspective - Internal and external experiences.

Alumni and employers have legitimacy with both internal and external stakeholders.

Internally, Alumni Studies can assess important outcomes and provide info for enhancing

curriculum, programs, and policy. Externally, Alumni studies can support accreditation,

accountability, recruitment, and fund raising. Such studies provide an opportunity for faculty and

administrative collaboration.

Centralized collection, decentralized uses


Outcomes Approach to Alumni Assessment

Assessment of alumni rests on the premise that institutional quality and effectiveness can be measured by examining:

Educational & Job satisfaction Income & Socio-economic Status Occupational Attainment Engagement in civic and political activities Values like Tolerance for diversity Collegiate experiences General Satisfaction with Institution Quality of Instruction received Preparedness for employment & Grad School Willingness to enroll again & Donate $

(Volkwein, 1990, 1998; Dellow & Romano, 2002)


Selected Alumni Surveys

ACT Alumni Survey (2-year & 4-year) SUNY Alumni Outcomes Survey Appalachian Region Alumni Outcomes SUNY-Albany Alumni Survey HEDS Alumni Survey HERI Alumni Survey NCHEMS Alumni Assessment Survey Clemson University Alumni Survey Georgia Tech Alumni Survey Penn State Engineering Alumni Survey


ACT Alumni Outcomes Survey Demographics/Background/Career (15) Employment History and Experiences (18) Educational Outcomes (22)

Critical Thinking & Problem Solving (4) Life-long Learning (2) Social/Moral Values/Ethics (3) Multi-Cultural/Diversity (4) Citizenship/Global Issues (2) Team Work (2) Communication (2) Overall (3)

Educational Experiences/Satisfaction (31) Activities and Organizations (11) Additional Questions (30 spaces) Space for Comments/Suggestions


Penn State - CSHE Engineering Alumni Survey

Demographics/Background/Career (13) Undergraduate Engineering Outcomes/Experiences (72)

Technical Skills and Abilities (15) Professional Skills (23) Analytical/Thinking Skills (10) Nature of Engineering Courses (15) Diversity & Tolerance (9)

Additional Information (9) Satisfaction (1) Test performance (1) Degrees, field, and minor (4) Planned and Actual employment (3)

Surveys available at Surveys available at http://www.ed.psu.edu/cshe/abet/instruments.htmlhttp://www.ed.psu.edu/cshe/abet/instruments.html

Percent of Alumni That Would "Attend All Over Again" and Select Same Major and Same Career (N=1743)

Attend All Over Again

Select Same Major

Select Same Career

020406080100

12

11

14

26

20

34

19

16

21

Probably & Definitely No

All Alumni Career Related to Major Career Not Related to Major

0 20 40 60 80 100

66

67

63

54

61

41

58

62

52

Probably & Definitely Yes

Mean Response on a 5 point Scale

3.9

3.9

3.8

3.5

3.7*

3.1

3.6

3.7*3.5

* = These mean responses are significantly higher than those in careers not related to the major.(p=<.01)

Perceived contribution of the College Experience

To Alumni Development as Undergraduates

Intellectual Growth

Personal Growth

Social Growth

Effective Preparation

Effective Preparation

020406080

44

912

1110

913

12

19

Percent responding 1 or 2 on a 5 point scale

Pre-2000

Post-2000

0 20 40 60 80100

68

70

63

71

58

70

6251

5642

Percent responding 4 or 5 on a 5 point scale

* Means are significantly different

from pre-2000 (p=<.01).

for Graduate School

for Employment

Mean Response(Scale of 1 to 5)

3.84.2*

3.73.9*

3.63.9*

3.73.5*

3.63.3*


Abilities and Skills Rated by Engineering

Employers as Highly Important for Success

99%99%97%97%95%94%

92%90%85%73%70%

Communicate effectively*Problem solving*Apply math, science, and engineeringUse modern engineering toolsTeamwork*Understand professional and ethical

responsibilitiesDesign a system to meet needsLife-long learning*Design and conduct experimentsKnowledge of contemporary issuesEngineering In global and social

contexts

Abilities and Skills Rated by Non-Engineering Alumni as Highly Important for Success

Function independently, without supervisionExercise personal responsibilityListen effectively*Exercise self-disciplineExercise problem-solving skills*Maintain openness to new ideas*Speak effectively*Evaluate and choose between alternative actions*Think analytically and logicallyAcquire new skill and knowledge on your own*Possess clear goalsCope with conflictUnderstand myselfWrite effectively*Lead and supervise tasks and groups of peopleFunction effectively as a member of a team*Learn how to learn*

96%96%93%91%91%90%90%89%89%87%84%82%80%80%78%75%75%

Consensus Liberal Education Outcomes

Over the past ten years, the AAC&U appears to have forged a national consensus among business, government, and accreditation leaders about the liberal education outcomes that all undergraduates should possess, summarized in Our Students’ Best Work: A Framework for Accountability Worthy of Our Mission (2004):

•Strong analytical, communication, quantitative, and information skills.•Deep understanding or hands-on experience with the inquiry practices of disciplines that explore the natural, social, and cultural realms.•intercultural knowledge and collaborative problem-solving

skills.•A proactive sense of responsibility for individual, civic, and

social choices.•Habits of mind that foster integrative thinking and the ability to transfer skills and knowledge from one setting to another.


The Value of a Model

•Encourages Clarity of Purpose

•Stands as a Road Map or Guide for Developing•Research Questions•Assessment Design•Data Collection•Hypothesis Testing•Statistical Analysis

•Serves as a Logic Chart (Otherwise Causality is not Clear)

•Assists Workload Conservation and Fog Dissipation (concentrates energy and attention, streamlines research design, reduces the amount of data collection, data storage, analysis, and reporting)

2. Five Questions:

1. Are you meeting your goals?2. Are you improving?3. Do you meet the standard?4. How do you compare?5. Are your efforts cost-effective?

4. Collect & Analyze Evaluation Evidence

3. Select Model, Methods, Measures

For Academic Effectiveness: Student Learning, Research & Scholarship

For Administrative Effectiveness:Planning & Resource Management

3. Select Methods & MeasuresFor Academic Effectiveness: Student Learning, Research & Scholarship

For Administrative Effectiveness:Strategic Planning & Resource Mgmt

1. Janusian Duality

Inspirational Purposes-for Internal Improvement

Pragmatic Purposes- for External Accountability

Volkwein Effectiveness Model

5. Communicate Results

Take Academic & Administrative Action

Improve & StrengthenPrograms

For Each Level --Institution

--Program --Individual


Today’s Effectiveness Approach

Undergraduate Focus, Goal driven, Evidence based, Improvement oriented


Assessment Resources

•Patrick Terenzini’s article, “Assessment with Open Eyes” (JHE, 1989).

•Middle States 2003 Handbook, Student learning assessment: Options and resources.

•Linda Suskie’s 2004 book, Assessing Student Learning: A Common Sense Guide

•Several books by Trudy Banta and her associates constitute helpful resources for campus assessment efforts, Assessment in practice: putting principles to work on college campuses (Banta, et al. 1996), Assessment essentials: planning, implementing, and improving assessment in higher education (Palomba & Banta 1999), and Building a scholarship of assessment (Banta et al., 2002).

•Additionally, there have been several constructive national attempts to develop guidelines and standards for good assessment practices. The most significant of these are the AAHE Nine Principles of Good Practice for Assessing Student Learning, and the recommendations of the AAC&U (from Our Students’ Best Work, 2004).

Documents

Volkwein- Penn State Assessing Student Attainment in the Undergrad. Major: Examples of Discipline-Specific Strategies Senior Thesis or Writing Project