Outcomes Assessment A Sustained Effort to Improve the Quality of Student Learning

Outcomes Assessment

A Sustained Effort to Improve the Quality of Student Learning

Peter Hernon (October 2004) 2

Four Perspectives

1. The institution and the communities served in the life of the library

2. The library and academic departments and programs in the life of the institution

3. The library, department/program, and institution in the life of the user/ customer

4. The library, program/ department, and institution in the life of stakeholders


Measurement Questions

1. How much?

2. How many?

3. How economical?

4. How prompt?

5. How accurate?

6. How responsive?

7. How well?

8. How valuable?

9. How reliable?

10.How courteous?

11.How satisfied?


Quality(examples)

Collections

Staff

Management

LeadershipServices

Facilities, equipment

Programs

Instruction, teaching,

Fostering learning

Examples of Quality Areas for Libraries


Measurement Indicators of Quality

Input

Output

Customer-focused Outcomes, including impacts

Choices


• “Assessment and evaluation are intended as means to document educational quality and institutional effectiveness, foster institutional improvement, and demonstrate accountability Unfortunately, without a clear sense of the purpose for assessment, knowledge of what is to be assessed, and understanding how the results will be used, assessment efforts, all too often, become an end in themselves. If the results of assessment are not used to information planning and decisions, college and universities often find themselves in positions of being data rich and information poor.” Ronald L. Baker, Outcomes Assessment in Higher Education, p. 12


Outcome Assessment in Libraries

– User oriented rather than institution centered• Statements about what students will know/think/be

able to do as a result of our programs• Not statements about what the library should/could

do to bring about desired outcomes

– Measures changes in library users as a result of their contact with an academic library’s programs, resources and services

• What student knows, content, development of skills and abilities, and acquire attitudes and values


Key Concepts

• Accountability

– Examples

• Cost

• Return on investment

• Student outcomes

• Educational Quality (continuous quality improvement)

– Examples

• Student learning outcomes

• Research outcomes


Student Outcomes

• Student outcomes refer to aggregate statistics on groups of students

Graduation rates Retention ratesTransfer rates Employment rates for a graduating class)

Such outcomes are institutional outcomes and are used to compare institutional performance

These outcomes are outputs and reflect what the institution has accomplished

they do not reflect what (or how much) students learned they do not measure changes in students themselves due to their college

experience


Research Outcomes

• Analyze a problem statement in its full complexity• Identify and relate relevant literature to the

problem under investigation• Select or develop a theoretical framework

appropriate to solving the problem• Select an appropriate procedure (research design

and methodologies) that address study objectives, research questions, and hypotheses

• Adopt appropriate indicators of reliability and validity


Outcomes Assessment

• Often focuses on the changes (one related to learning) that result from:

• Completion of a program of study, including a major or minor

• Completion of a course

For either of above, factor in contact with library resources, programs, or services—the role of the library in enhancing the educational experience and learning


Levels of Outcomes

Course-based outcomes

Program (college) -level outcomes

Institutional-level outcomes


Learning is Define as Encompassing

1. Knowledge leading to understanding 2. Abilities3. Habits of mind4. Ways of knowing5. Attitudes6. Values7. Other dispositions that an institution and its programs

and services assert they develop

Source: Assessing for Learning, by Peggy L. Maki (2004)


The Assessment Cycle Views Planning in Two Contexts

InstitutionalMissionVisionValues

See next slide

Impact

Quality improvement


Assessment Cycle (adapted from Assessing for Learning, by Maki, p. 5)

Identify

outcomes

Interpret

evidence

Implement

changeReview outcomes

Gather evidence

MissionGoalsObjectives


Understanding the Full Experience(Matching with Mission)

Outcomes

Student outcomes

Student learning outcomes

Research outcomes

Outputs

Scholarly output

Etc.

Inputs as work load, resource allocation, and grants received


Alternative Perspective

Student learning outcomes&

Research outcomes

Student outcomesTeaching effectiveness

Conveyance of knowledge through research and scholarship


Documenting Student Learning Outcomes

Course Program Lifelonglearning


Key Questions (Related to Student Learning Outcomes)

How well do students – Transfer and apply concepts, principles, ways of

knowing, and problem solving across their major program of study?

– Integrate their core curriculum, general studies, or liberal studies into their major program or field of study?

– Develop understanding, behaviors, attitudes, values, and dispositions that the institution asserts it develops?

Source: Assessing for Learning, by Peggy L. Maki (2004)


Seven Questions

1. What should students learn? 2. How well are they learning it? 3. What do students know that they did not know before?4. What can they do that they could not do before? 5. What measures and procedures does the institution

utilize to determine that the answers to the above questions?

6. To what extent does the institution offer evidence that demonstrates its effectiveness to the public?

7. What does the institution plan to do with this evidence to improve outcomes, including the quality of the educational experience?


Some Focal Points

Communication skills: oral and written

Critical thinking

Information literacy (e.g., evaluation)

Problem solving

Quantitative reasoning ability

Setting up a good research study


Student Learning Outcomes*See Information Literacy Competency Standards for Higher Education (ACRL,

2000)

• Higher Ordered

– Problem solving

– Critical thinking

• See Bloom’s Taxonomy of Educational Objectives

• Lower Ordered

– Skills


Verb Choice for Outcomes

• Analyze• Apply• Demonstrate• Develop• Differentiate• Identify• Recognizes• Synthesize

• Etc.

• Not– Discusses– Understand– Etc.

– An Action Plan for Outcomes Assessment in Your Library (ALA, 2002), p. 72


Examples

• Outcome: “Students will think logically and critically in solving problems; be able to evaluate, critique, and apply the thinking of others; and reach an appropriate conclusion”

• Outcome: “Students will demonstrate leadership abilities”– Leadership abilities evidenced in learning contexts

include team building and shared decision-making, strategic planning, advocacy, and consensus building and collaboration


Information Literacy: An Area of Library Focus

• Association of College & Research Libraries, “Information Literacy Competency Standards for Higher Education” (2000)– Offers a basis to develop student learning and some

research outcomes

See also Middle States Commission and standardized form being developed at Kings’ College (Terry Mech)


Information Literacy: Shared Responsibilities for Learning

Information Literacy Skills Responsibility

Know: Determines the nature and extent of information needed

Starts with the faculty member; reinforced by librarians

Access: Efficiently and effectively accesses information sources

The librarian usually leaders, with faculty support

Evaluate Sources: Critically evaluates information sources

The librarian may lead initially; faculty make the ultimate determination from student’s work product or performance


Information Literacy (continued)

Information Literacy Skills Responsibility

Evaluate Content: Critically evaluates information content; considers impact on students’ prior knowledge, value system, and future direction in life

Faculty leads in classroom or other course context; students also may consult librarians, external subject experts, or peers

Use: Uses information found to accomplish a specific purpose

Faculty leads; librarians can

reinforce


Information Literacy (Continued)

Information Literacy Skills

Responsibility

Ethically/Legally: Understands the economic, legal, and social issues surrounding the acquisition and use of information

Faculty and librarians jointly and continuously

Source: Middle States Commission on Higher Education, Developing Research & Communication Skills (2003), p. 23


Institutional Effectiveness and Improvement

Outcomes (Impacts)

Outputs

InputsAnalysis/ Review/ Evaluation

Service Quality*

Satisfaction

*Not LibQUAL+


Methods of Evaluating Student Learning (Outcomes)

• Direct Methods– Embedded course

assessment (performance on assignments, etc.; minute papers

– Portfolio assessment

– Performance (internships, practicum, student teaching)

• Direct Methods– Professional jurors or

evaluators

– Testing (standardized or pre- and post- testing)

– Analysis of theses/ senior papers (content analysis, interview, or oral defense)


• Direct Methods– Think-aloud protocol

– Directed conversations

– Capstone course/ activity

– Minute paper

• Direct Methods– Videotape and

audiotape evaluation


Example (Kathleen Dunn, p. 28)

• Core CompetencyCore Competency: “locate and retrieve information, in all of its various formats using, when appropriate, technological tools.”

• ScenarioScenario: “You’ve been selected to lead a group of students on a four-week trip to another country. In preparation you need to find background information on the country’s history, current events, language, local customs, art, and music. How would you locate the information? What type of information would you expect to find?

• How can we assess the extent of success--to which scenario met and change--over time?


Methods of Evaluating Student Learning (Outcomes)

• Indirect Methods– Surveys (self-reports)

• Indirect Methods– Curriculum and

syllabus evaluation

– Exit interviews

– External reviewers

– Observation

– Self-assessment

– (Library) database statistics


Public learns to use technology

• Using database A (or search engine B), construct a search strategy to locate information that:

applies Boolean operators to narrow the scope of the search

applies basic and advanced search protocol to retrieve needed information


• Objective: to reduce the number of to reduce the number of retrievals (hits) per search on aggregate retrievals (hits) per search on aggregate and specialized databasesand specialized databases– Is this objective measurable? What types of Is this objective measurable? What types of

measures would you apply? measures would you apply? – Can this objective be taught? How?Can this objective be taught? How?– How would you measure if anything was How would you measure if anything was

learned?learned?– What will the results tell you?What will the results tell you?


SomeSome Relevant Readings

• Outcomes Assessment in Higher Education (Libraries Unlimited, 2004)—companion volume coming in January 2006

• An Action Plan for Outcomes Assessment in Your Library (ALA, 2002)

• The Journal of Academic Librarianship (Jan.-Mar. 2002); (Nov. 2002)

• Assessing for Learning, by Peggy Maki (Stylus, 2004)

• Textbooks from Middle States Commission

Questions

Documents

Outcomes Assessment A Sustained Effort to Improve the Quality of Student Learning