Embed Size (px)
Citation preview
Demonstrating the ‘L’ factor: providing quality impact &
evaluation evidence
Virginia Power,Graduate Tutor/Researcher in Information Management & Science, University of the West of England, Bristol
Some statistics provided by Carol Parker, Barton Peveril College, Hampshire
Drawing on research from the UK, the US and Scandinavia, the LISU study shows a world in which libraries are actively engaged in gathering evidence to demonstrate their value – but challenges remain. Though a good deal of evidence is collected, much of it is evidence of activity rather than evidence of value and impact. Barr, S. (2012) How should academic libraries communicate their own value? Guardian Online http://www.theguardian.com/higher-education-network/blog/2012/aug/20/academic-libraries-value-research-teaching
Evolving Value for Libraries
Your motivation. Identify the key reasons why you want to engage in a proving and/or improving process, your expectations about the process, ‘windows of opportunity’, the available resources, leadership issues and the appropriate tools or methods.
Your vision. Set forth the organisation’s mission, values, and objectives. Identify its major activities.
Stakeholders. Know who is affected by or affects your organisation in all of its activities, whether intentionally or not.
Impact Map. Specify what the organisation does and how it does it; how it intends to achieve its social, environmental/economic mission as well as its financial sustainability; and how it plans to live up to its values.
Measuring the Library
Identifying how well a service fulfils its purpose and adds value to an organisation
Accountability and justification of financial spending
Identifying areas for improvement Monitoring progress against goals Comparing standards with other organisations Monitoring effects of past decisions Monitoring interactions with customers and
suppliers
Purposes of Performance Measurement
Have YOU read and understood the inspection framework for your institution?
‘Distance travelled’ is the description given to the change in ‘soft outcomes’, that is, those achievements that cannot be directly measured in the same way as ‘hard outcomes’ such as qualifications, earnings or employment status. Mostsimply, ‘distance travelled refers to the progress that a beneficiary makes towards employability or harder outcomes’. It is easy to measure an individual’s educational progress, for example, as this can be a number of qualifications or ajump from level 2 to level 3. It is much more difficult to measure progress in social skills or motivation, although these are as important for an individual’s life chances.
Soft Outcomes for Universal Learning (SOUL)
Are YOU able to demonstrate the distance travelled by the students that you work with in Library & Information Services?
The library needs to be able to identify and show progress
Identify both intended and unexpected outcomes.
Prioritise what is most important to measure Choose the ways in which you will see
change happening.
Indicators
What do you consider to be the most effective Key Performance Indicators?
Choose KPIs that align with service and organisational aims/objectives
Review KPI annually How will they be measured/monitored? Choose quality over quantity Adopt a range of methods to measure
performance How much time/money/tools do you have to
measure/monitor PI?
Choosing Performance Indicators
Make a plan. After prioritising the indicators, decide on methods for collecting information relating to the indicators.
Collect data. Collecting information can be done in a number of ways as outlined in your methodology or plan, as long as it can be stored and drawn upon usefully.
Analyse information. Seeing what the data you collect tells you can happen throughout the process, and can also be summed up at the end of a cycle to make some conclusions.
Planning
Objectives
You are trying to understandWhether, how and what impact occursWhat the outcome of this impact is for the people concernedWhat the broader impact of this evidence might be
Before you begin to collect evidence decideWhy you are seeking this evidenceWhat kind of information you will collectWho will collect itWhen it will be collectedHow to use the results What do you want to find out?
Keep it simple and focusedWhat are the learning outcomes of the information skills session?What new knowledge did students learn and how did this knowledge increase their understanding?How did using the library change attitudes towards libraries and learning?
What do you consider to be the most effective methods of user feedback?
I Already Do This….
Review the data you collect alreadyComment booksTeacher's evaluation sheetsLetters from usersFootfall countsUser surveysProject evaluation formsOther...
For each data source, ask the question: does it give evidence of impact?YesNoMaybe?
Discussion at meetings
Comments and suggestions
Feedback forms
Staff voice meetings
One-to-one sessions
Surveys
Boards of study
Annual survey
Liaison role
Course committees
Focus groups
0 2 4 6 8 10 12 14 16 18 20
Q.4 Feedback methods from internal users
Percentage of respondents
Feedback methods
Feedback Methods from Internal Users
Most Effective Method for Gathering User Feedback
Percentage of respondents
Notice boards 9.0
Student union 23.9
Social media 26.9
Student council 26.9
National surveys 29.9
Online feedback mechanisms 35.8
Organisational/institutional surveys 44.8
Formal complaints/feedback procedures 47.8
Focus groups 53.7
User surveys 82.1
Most Effective Methods for Gathering User Feedback
Simple Informal Methods
Library managers need to ensure that they recognise the concept of the internal customer within their organisation. This is because
“errors in the service provided within an organisation will eventually affect the product or service which reaches the external customer” (Slack et al, 2004, p. 723).
Measuring Internal Feedback
TICSI (no date) states that benchmarking encourages a service to “become open to new methods, ideas, processes and practices that improve effectiveness, efficiency and performance in customer service”
Further Education – CoLRIC Higher Education and Research – SCONUL
and National Student Survey
Benchmarking
Balanced scorecard CoLRIC – performance and impact indicators
/ peer accreditation scheme EFQM – European Foundation for QM Inspiring Learning for All SCONUL benchmarking LibQUAL+ Prove and Improve Comparison Chart Return on investment – ROI Calculator
Performance Measurement Tools
Balanced Scorecard
Business ProcessesStreamlining processesNew products/suppliers
Supply chain
Staff skills and expertiseTraining needs
Customer satisfactionMeeting customer needs
Cash flowCosts
Budgets
Balanced scorecard
Return on Investment
Olson (2002, p.19) states “information services are considered valuable when they are perceived to be equivalent to the funds spent on them”.
ROI Calculator
Library Performance Measurement in the UK and IrelandAssociation of Research Libraries, 2009
Library Performance Measurement
Sharing information. Make what you have learnt a part of a dialogue or conversation, offering different stakeholders the opportunity to learn from the process and feed their thoughts and opinions into the next round.
Learning & Action. A proving and/or improving process is a chance to do things better in the future. Make what you have learnt part of a plan for improvement –the necessary actions, changes and the resources needed to enable the desired changes.
Sharing
Informal Communication Methods
Service level agreement
Annual report
Quarterly report
Self assessment review
Business report
0 10 20 30 40 50 60
Q.7 How do you communicate the results of per-formance measurement?
Percentage of respondents
Methods of Com-munication
Communicating Performance Measurement
Performance measurement and impact assessment is time consuming and difficult to do.
Anecdotal evidence suggests that even when mechanisms are put in place to measure performance e.g. feedback and customer satisfaction - it isn't always forthcoming!
Services need to use a variety of methods to get good results.
Library Managers need a range of skills to support this.
What skills, knowledge & understanding do you consider to be the most important for librarians to develop in this field?
Communication Skills Marketing Skills Promotional Tools Systematic collection of evidence Analytical skills Social Informatics Understanding of big data
We need ….
The role of data informing decision-making New approaches to collecting, analysing
and using data Using analytics to develop new services and
improve the user experience The opportunities of library data as big data Role of library analytics uncovering new
opportunities to demonstrate impact and value
The future is bright….
Thank You – further questions?
Virginia Power,Graduate Tutor/Researcher in Information Management & Science, University of the West of England, Bristol [email protected]
