Library value and impact

Mary Dunne & Mairea Nelson

value impact demonstrating your worth

Value is ‘the benefit the user obtains from the use of information and the effect of that use’.

(Urquhart & Hepworth 1995, 33).

Poll R (2012) Can we quantify the library’s influence? Creating an ISO standard for impact assessment, Performance Measurement and Metrics, 13(2), 121-130.

Urqhuart C & Hepworth J (1995) The value to clinical decision making of information supplied. London: British Library Research and Development Department.

Value & Impact

Indicators: • Usefulness• Satisfaction - a personally perceived response to an

experience.

(Poll 2012) Impact is the effect of a service on its users (and others).

- may be viewed as an aspect of value.

ISO definitions

ISO (2014) Information and documentation — Methods and procedures for assessing the impact of libraries, ISO 16439. Switzerland: ISO.

.

input: contribution of resources in support of a library (e.g. funding, staff, collections, space, equipment)

process: set of interrelated or interacting activities which transforms inputs into outputs (e.g. cataloguing, lending, reference service)

output: products of library processes (e.g. titles catalogued, loans, downloads from the electronic collection, reference questions answered)

outcome: direct, pre-defined effect of the output related to goals and objectives of the library’s planning (e.g. number of users, user satisfaction levels)

impact: difference or change in an individual or group resulting from the contact with library services

value: importance that stakeholders (funding institutions, politicians, the public, users, staff) attach to libraries and which is related to the perception of actual or potential benefit

P

P

P

P

P

P

P

7 p’s of marketing

Process

Promotion

Product

Price

People

Place

Physical evidence

People

Customers Potential customers

Funders Library staff

People

• Who is my audience for the evaluation?How do I ensure I report results that are of value to them in a way that suits them?

• Who is my study population? • How do I ensure I gather information from

my priority customers? • How do I ensure I include measures that

demonstrate the value of librarians as well as the library?

Starr S (2013) Creating brand love for libraries: can we be a kind of paradise? J Med Lib Assoc 101(3), 168 – 170 Shore, E. (2013) The role of the library in the transformative higher education environment: or fitting our measures to our goals. Presented at the 10th Northumbria International Conference on Performance Measurement in Libraries and Information Services, University of York, 22-25 July, 2013

Product

• Are you going to measure value and/or impact?

• What do you want to evaluate – the whole library / specific resources and services – or both

• Do you have indicators or objectives that you want to evaluate?

• Are you going to include measures that enable to you increase value and impact?

PlaceWhere do you conduct your study:• email survey • web survey • focus groups • Interviews (phone or face-to-face)• observation• triangulation (combining approaches)

Dunn, K., Brewer, K., Gard Marshall, J. and Sollenberger, J. (2009) Measuring the value and impact of health sciences libraries: planning an update and replication of the Rochester study, Journal of the Medical Library Association, 97(4): 308-312.

Physical evidence• What is the best way of gathering that evidence -

quantitative or qualitative measures?

Survey questions: • Is each question necessary? • How will I use the results of each question? (does it show

satisfaction; tell you if something needs to change, and how, or provide context)

• Will questions make sense to respondents? (jargon free)• Do question and answer options match?

(Try reading the question and each answer option out loud)• How will I report the results of each question?

(Will you group categories. Will your categories make powerful statements in your final report)

• Which questions will I make mandatory?

Answer options

Survey tips

• Pilot the survey• Personalize the request, stressing the

importance of the survey and assuring confidentiality.

• Send at least one, and ideally two or even three, reminders.

• If you amend the questionnaire, keep it brief.• Consider the use of a financial incentive

such as a lottery draw.

Weightman, A., Urquhart, C., Spink, S. and Thomas, R. (2009) The value and impact of information provided through library services for patient care: developing guidance for best practice, Health Information and Libraries Journal, 26,63-71.Best practices for improving survey participation http://www.oracle.com/us/products/applications/best-practices-improve-survey-1583708.pdf Smart survey design by survey monkey http://s3.amazonaws.com/SurveyMonkeyFiles/SmartSurvey.pdf

Organisation goals NDC (library) goals Logic underpinning goal

HRB goal 4: Generate and synthesise evidence, and promote the application of knowledge to support decision-making by policymakers and relevant practitioners.

1. To be known as a reliable source of information on all aspects of the substance use situation in Ireland.

Providing a comprehensive and reliable resource will increase confidence and change attitudes to research evidence.

2. To deliver high quality information services and develop innovative and responsive knowledge resources.

Providing easy access to material through specific, tailor-made resources and a responsive query service will improve (impact on) knowledge acquisition.

3. To contribute to the shift towards research-based approaches to work and decision-making in substance use work.

Actively increasing levels of research literacy and promoting research translation into practice will assist people to change behaviour and develop the competencies required to enable evidence-based practice.

Impact – aligning to goals

Library taxonomy: the value of library and information services in hospitals and academic health services centers

Abels, EG, et al. (2002) Journal of the Medical Library Association, 90(3), 276-284.

Organizational mission concept Organizational goals LIS contributions

Clinical care Provide excellent clinical care

Support informed and timely clinical decision makingSupport the development of policies and procedures relating to clinical care

Promote clinical learning

Provide new knowledge and substantiate prior knowledge about clinical practice. Inform users about current developments in clinical practice

Manage operations

Meet accreditation standards

Maintain information required for responses to accrediting bodies

Reduce corporate risk

Disseminate information on best practicesIncrease corporate compliance (health care regulations and copyright)

Provide an organizational learning environment

Provide leadership in information management for the organization.Provide information about developments in information technologies and resourcesSupport professional development of staffProvide physical environment conducive to learning

Education Provide excellent educational programs

Enhance educational programsPromote academic excellencePromote satisfaction with quality of educational programs.

Performance indicators

Dalton, M. (2012) Key performance indicators in Irish hospital libraries: developing outcome-based metrics to support advocacy and service delivery, Evidence Based Library and Information Practice, 7A. Critical incident technique http://www.ucc.ie/hfrg/emmus/methods/cit.html

Following 13 interviews, our researcher produced a report of results under identified themes • better informed workers in the drugs sector• better informed policy • programme and project development• better informed interventions with service users• improved course educational development • better informed written outputs

Reporting impact

Impact type Associated change Examples

Affective change Change in attitudes, perceptions or confidence. 

Participants described how NDC staff had shared their expertise to help them refine their thoughts and build their confidence in accessing evidence.

Knowledge-based change

Know more about a subject and its key sources. 

Participants described how they were able to find “facts” about drug use on the NDC website for use with clients in one-to-one or group sessions.

Behavioural change 

Become more inquisitive, critical, broader perspective on issues.

In response to a study informed by the NDC, a participant’s organisation planned to develop alternative ways to deal with polydrug users.

Competence-based change

Better able to find evidence / do things more effectively.

After reading papers from the NDC, a project worker felt he could ask more insightful questions and was better equipped to inform users of risks.

Aligning results to impact types

Organisation goals

NDC (library) goals

Logic underpinning goal

Impact typeAssociated

changeExamples

HRB goal 4: Generate and synthesise evidence, and promote the application of knowledge to support decision-making by policymakers and relevant practitioners. 

1. To be known as a reliable source of information on all aspects of the substance use situation in Ireland.

Providing a comprehensive and reliable resource will increase confidence and change attitudes to research evidence.

Affective change

Change in attitudes, perceptions or confidence. 

Participants described how NDC staff had shared their expertise to help them refine their thoughts and build their confidence in accessing evidence.

2. To deliver high quality information services and develop innovative and responsive knowledge resources.

Providing easy access to material through specific, tailor-made resources and a responsive query service will improve (impact on) knowledge acquisition.

Knowledge-based change

Know more about a subject and its key sources. 

Participants described how they were able to find “facts” about drug use on the NDC website for use with clients in one-to-one or group sessions.

3. To contribute to the shift towards research-based approaches to work and decision-making in substance use work.

Actively increasing levels of research literacy and promoting research translation into practice will assist people to change behaviour and develop the competencies required to enable evidence-based practice

Behavioural change 

Competence-based change

Become more inquisitive, critical, broader perspective on issues.

Better able to find evidence / do things more effectively.

In response to a study informed by the NDC, a participant’s organisation planned to develop alternative ways to deal with polydrug users.

After reading papers from the NDC, a project worker felt he could ask more insightful questions and was better equipped to inform users of risks.

Aligning organisation goals – library goals - impacts

Impact Assessment Method

Pluye, P., Grad, R.M., Stephenson, R. and Dunikowski, L.G. (2005) A new impact assessment method to evaluate knowledge resources, AMIA 2005 Symposium Proceedings, 609-613. Information Impact Assessment Method - http://www.flintbox.com/public/project/1946

Ten item scale used to measure impact at McGill University

Price

Do you have a budget?

Costs – • recruiting an external researcher• purchase access to survey provider • staff time - how long will it take to

prepare, administer, analyse, write up and promote?

Promotion

How will you communicate your results?

• for customers• for health practitioners (?potential customers)• for funders• for other librarians

Process

What are the pieces of the puzzle that have to fit together and in what time-frame?

• Do you need to get approval from senior managers?

• When is the best time to conduct a study?

• Do you have a deadline e.g. your annual report?

• How long do you give your responders?

• Do you need a gantt chart to show timelines?

Conclusion

We have learned that there is no one-size-fits-all approach, but that there are many tools and resources to help us as librarians demonstrate the importance of our work.

VALUE

Validating A Librarian’s Unique Expertise

IMPACT

Impressing Managers & Practitioners About Change Transfer