View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Laura Saunders 3
Why Take This Class Take 3 minutes to answer these questions…
What do you expect/ hope to learn from this class?
What do you want to be able to do after this course? Why?
Why is this field (evaluation research) important? Or is it?
Laura Saunders 4
Background Edward Suchman observed that “one of the
most appealing ideas of our century [the 20th century] is the notion that science can be put to work to provide solutions to social problems” Substitute “social sciences” for “science” Expand “social problems” to problems faced by
organizations
Laura Saunders 5
Background: Evaluation vs. Social Research Evaluation Research
Uses many of the same methods/ practices Typically takes place within an organization-
with its own culture and politics Requires understanding/ ability to work with
groups, personalities, teams, etc. Requires sensitivity to issues and concerns of
various interests involved.
Laura Saunders 6
What and Why According to Peter Hernon
The purpose of evaluation is “to collect data useful for documenting the quality and effectiveness of existing programs and services, and to use that information for service improvement, demonstrating best practices, or providing evidence of accountability.
According to William Trochim Evaluation is the systematic acquisition and assessment of
information to provide useful feedback about some object
Laura Saunders 7
Purpose and Potential Evaluation compares “what is” with “what
should (could) be”- element of judgment Evaluation must have a purpose, rather than
be an end in itself. There is no need to evaluate without the
potential for action. Evaluation must go beyond description to
understand relationships. Joseph Matthews
Laura Saunders 8
Contexts Evaluation research may generate conflict
Often begins with problem identification Tends to involve groups of people, personalities,
etc. Competing priorities. Purpose is improvement- often results in change.
Laura Saunders 9
What and Why Unfortunately, according to James G. Neal,
“decisions are routinely not supported by the evidence of well-designed investigations,”
“research in the field is poorly communicated, understood and applied.”
Neal, J.G. (2006). “The research and development imperative in the academic library: Path to the future.” portal: Libraries and the Academy, 6: 1.
Laura Saunders 10
Current Developments Increased attention to utilization of evaluation results
Focus on Evidence-Based Practice Technology affects evaluation research
Topics to cover Software to analyze data collected Presentation of findings (e.g., word-processing and
graphics capability) New techniques for data gathering (e.g., Web surveys)
New skills and competencies needed
Laura Saunders 12
Evaluation Can Be Applied To Collections Staff Facilities Services Products Information Technology
Laura Saunders 13
Some Topical Areas of Evaluation Collections
People find what they want/need Quality/value Size Use patterns
Online Systems Complaint analysis Ideal page design Navigation/location of needed
information Tracking eye movement Use patterns
Reference Service Accuracy Complaint/compliment analysis Satisfaction Service quality (e.g.,
approachability) Technical Service
Efficiency (costs and time) Time: staff time to
complete task Time: time spent waiting
between process activities
Quality Accuracy (e.g., typographical
errors in bibliographic records) Workflow analysis
Laura Saunders 14
Evaluation Can Be Formative
Providing feedback during the development or offering of a program or service
Summative After the program or
service is over
Laura Saunders 15
Formative Evaluation How can the program or service be improved
or made more efficient or effective? To what extent are the goals, objectives, and
“performance measures” realized?
Laura Saunders 16
Summative Evaluation Is program or service X worth continuing or
expanding? How effective is it? What conclusions can be drawn about its
effects or various components? Did it meet its goals, objectives, and
performance expectations?
Laura Saunders 18
Concept Measurements of Evaluation Extensiveness, or the amount of a service provided
in relation to the population served. This criterion is generally a measure of quantity rather than quality
Effectiveness Efficiency, or the appropriateness of resource
allocations Cost-effectiveness Cost-benefit Quality
Laura Saunders 19
Continuous Quality Improvement General Systems Model
Input (staff, budget, resources) Throughput (processes) Output (services, collections) Feedback to Input (evaluation)
All of above viewed within the context of the planning process
Laura Saunders 20
Uses of Evaluation Continue or discontinue (or improve) services
(programs) Improve, add, or drop practices and
procedures (strategies and techniques) Institute programs developed elsewhere Allocate resources among competing services
or programs Accept or reject a program approach or theory
Laura Saunders 21
Types of Evaluation Activities Program Planning: focuses on
“What is the extent and distribution of the target population?”
“Is the program or service designed in conformity with its intended goals?”
“Are the chances of successful implementation maximized?”
Program (Service) Monitoring: focuses on “Is the program or service reaching the persons,
households, or other target units to which it is addressed?” “Is the program or service providing the resources and
other benefits that were intended in the project design?”
Laura Saunders 22
Types of Evaluation Activities Impact assessment: focuses on
“Is the program or service effective in achieving its intended goals?” “Can the results of the program or service be explained by some
alternative process that does not include the program?” “Is the program or service having some effects that were
unintended?” Economic efficiency: focuses on
“What are the costs of delivering services and benefits to program participants?”
Does the program represent and efficient use of resources in comparison to alternative uses of the resources?”
Source: Rossi and Freeman, Systematic Evaluation
Laura Saunders 24
Evaluation Centers on “How” Questions
Library-Centric View
Combination View
Customer-Centric View
How much?How many?How economical?How prompt?
How reliable?How accurate?How valuable?
How well?How courteous?How responsive?How satisfied?
Joseph Matthews
Laura Saunders 25
Examples of Evaluation Questions Are the services of the Libraries appropriate
considering the needs of the university, its academic programs, the faculty, and students?
Are the services effectively delivered? Are there important expectations or interests
of the academic programs that the Libraries do not meet?
Laura Saunders 26
Examples Cont’d What are the specific strengths and weaknesses in
libraries services at XXX? Are there good lines of communication within the
Libraries? Is staff morale good? Are there adequate provisions for professional
development of staff? Is there adequate succession planning within a
framework of future organizational structures? Are there adequate and appropriate avenues for users
to communicate needs and desired services back to the Libraries?
Laura Saunders 27
Examples Cont’d Is appropriate and sufficient attention being paid to
anticipating future IT needs and trends? Are there healthy and productive relationships with
other academic service units on the campus? Are staff adequately trained in and familiar with new
technological directions? What will be the most important qualities sought in
the next director of libraries?
Laura Saunders 29
Barriers & Obstacles The belief that everything is fine in my library The library might apply evaluation inappropriately
or merely serve organizational or administrative self-interests
Justify a weak or bad program or service by deliberately selecting for evaluation only those aspects that “look good” on the surface. Appearance replaces reality
Cover up program or service failure or errors by avoiding objective appraisal. Vindication replaces verification
Laura Saunders 30
Barriers Cont’d “Torpedo” or destroy a program or service
regardless of its effectiveness. Politics replaces research
Treat evaluation as a “gesture” of objectivity or professionalism. Ritual replaces research
Delay needed action by pretending to seek the “facts.” Research replaces decision making
Perceived lack of resources to support the evaluation process
Etc.
Laura Saunders 31
Barriers Cont’d Lack of staff Space Budget Conflicting perceptions of library Limited staff skills Physical location Limited bibliographic control Apathy of clientele and stakeholders about library Poor organizational structure, planning, and decision
making
Laura Saunders 33
Overview of the Evaluation Process Phase One: Preparation
Establish library goals and objectives Train staff Understand evaluation Put in place team or however the study will be done Decide on the problem
Phase Two: Evaluation Research Set up and execute the study
Phase Three: Assessing and, as decided, applying the results