Creating value and maximising impact : issues, concepts and methods

Presentation to 2013 AWHILES Conference The proof of the pudding. Providing the evidence of our

value for money 26th & 27th June 2013, Wrexhamby

Dr Judith Broady-PrestonReader in Information Management

Institute Director of Postgraduate StudiesDepartment of Information Studies, Institute of Management,

Law and Information Science, Aberystwyth UniversityEmail: jbp@aber.ac.uk

(copyright retained)

Context and BackgroundNeed to demonstrate value and impact = not new• “ACRL has long been concerned with accountability [and]

assessment [since] the early 1980s” (2012, 4)

BUT - still searching for mechanisms to describe impact i.e. effectiveness accurately

• “impacts...are extremely difficult to measure” (Hernon & Altman, 2010, 50)

• Direct vs indirect – “holy grail”?

Efforts to date focused on demonstrating economic value or impact library service activity (outcome)

Need to assign economic value and measure more socially oriented outcomes or aspects library activity – the qualitative

Context and Background

Is economic impact sole criterion?

Need to assign economic value to the more socially oriented outcomes of library, museum and archive services

• Current methods “fail to fully capture the wider economic value gained from this activity” (ERS, 2011, p.35)

Need to “Capture social returns” (ERS, 2011)

• ACRL 2012 Report – need to demonstrate, assess and communicate impact academic libraries vis-à-vis student learning and achievement

Need to gather perceptions of value

• From a wide variety of stakeholders including current NON-users

• Use these design, deliver and measure service performance

Outline

Context and background : drivers for impact

Value

Complexity – what are we measuring?

ImpactConcepts and semantics

Definitions and perspectivesModels and methodologies

Where are we now?

Why is demonstrating value and impact so important?

The Global Economic downturn

Global retrenchment in library services

The Future of University Libraries: 2012 Midwinter Report:

– “continuously shrinking budgets are the “new normal” [sic] for university libraries“ (2012, 2)

AND

Research Information Network (RIN) and SCONUL 2010 Report:

– “After a decade of growth in budgets and services, academic librarians now expect a sustained period of cuts over the next three to five years” (Challenge…2010, 4)

Drivers for Impact• Economic recession = global

– Results in need to target service activity to achieve greatest impact

– Defining/determining essential (‘core) and additional (‘value-added’) services (Broady-Preston and Swain, 2012)

• Open Access

– Berlin Declaration (2003) creates possibility to increase markedly reach, and ergo impact academic research

• REF2014 – Research Excellence Framework

– NEW = not only academic impact but also wider, societal impact

• OA and Research Council funding – new business models?– Finch Report (UK) June 2012 – gold vs green?? Accepted in full by UK

Government 16 July 2012; RCUK Policy on Open Access and Guidance (updated 24 May 2013 http://www.rcuk.ac.uk/documents/documents/RCUKOpenAccessPolicy.pdf)

– Stakeholder perspective

Concepts

Two core constructs...

VALUE and IMPACT

in relation to

INFORMATION

in relation to

Information and Library SERVICES

What is value?Is this the same as impact or quality or satisfaction?

Perspectives:• organisation vs customer

• economic vs societal

Synonymous with:– Worth?– Fitness for purpose (quality)?– Exceptional/Perfection?– Utility

Relationship to values – morality, ethics, culture

Perceived value – whose perspective?

Intellectual capital and value/sRelationship between

• categorisations impact; value; and Intellectual Capital (IC)

• Conceptual links IC re: change, knowledge transfer and learning organisation (Kong, 2010)

• Deeper purpose IC to change individual behaviour and values

• Is IC suitable framework within which to place our understanding and evaluation models such as Balanced Scorecard (BSC) re: measuring progress towards fulfillment complex social objectives and purpose?

• “Public sector reforms often carry with them values consistent with value for money and competition, causing threats to …traditional qualities such as fulfilling social objectives [ergo] IC becomes a valid strategic management conceptual framework within the …nonproft context in the knowledge economy” (Kong, 2010, p.295).

Complexity: what are we measuring?Determining what should be measured, how, and why = fundamental

to our understanding of service impactSignificance

• Criteria employed

• Whose perspective? judgments of quality, worth and value

Performance measurement = judged in relation to basic purpose. Difficulties:

• In defining basic purpose in library context

• Measuring performance in relation to basic purpose viz: “in practice it is difficult to draw distinct lines between core

business and value added services. Views of service priorities change over time, in relation to external pressures and developments and are…subject to individual interpretation, depending on perspective (Broady-Preston & Swain, 2012, p. 117).

Complexity: what are we measuring?COMPLEXITY = key characteristic attempts to determine impact in the

information sector. Need to

• take account of complex array multiple stakeholders beyond management and customers

• anticipate and satisfy stakeholders’ perception of value and worth• acknowledge centrality of requirement that evaluating indirect impact

on third parties such as Government = crucial to continued success or even existence

Related issue = increasing use volunteer labour to deliver library servicesDifficulties

• evaluating performance = complicated when staff also customers or clients of the service

Central challenge = identify methodologies which enable libraries to demonstrate their activity results in measurable change re: social

objectives AND do so reliably and robustly

More than mere semantics?Terminology and relationships

• Impact– Variation + attempts to standardise (Poll + ISO 16439, 2012; 2013)

– Definitions:• “all changes resulting from an activity, project or

organisation...intended as well as unintended effects, negative as well as positive and long-term as well as short-term” (Arvidson 2009 adapted from Wainwright 2002)

• “An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life beyond academia” (REF2014)

• “Difference or change in an individual or group resulting from contact with library services NOTE the change can be tangible or intangible” (ISO cited in Poll 2012)

NB Centrality of Change

What is Impact?Differing perspectives

Organisation Customer Society

– Implies ‘measurement’ - the ‘Q’ debate i.e. Qualitative vs Quantitative

– BUT “not everything worth knowing can be counted precisely and reduced to a rank, percentage or ratio; many aspects of information service are intangible, and must be evaluated in other ways” (Hernon & Altman, 2010, 50)

– Relationship to purpose – aims and objectives (SMART) Organisational Perspective

– Measures need to be Meaningful/relevant and are often related to one another to give performance INDICATORS

What is Impact ? Comparative perspective - Benchmarking and Standards

• Comparisons with and to– other orgs - ‘league tables’

– external standards – accreditation – e.g. ISO (Poll, 2012; 2013) CSE (Broady-Preston & Lobo, 2011)

• Process– Gap identification

– Monitoring progress

• Encourages “reflective” and “evidence-based practice”• Multiple benchmarking standards exist; however

– “basic principles behind the present QM [Quality Management] systems…are actually rather similar. They all emphasise the continuing improvement of services…[and] all adopt a customer oriented focus” (Balague & Saarti, 2009, 227-8)

Types of Impact

Determining meaning = key to demonstrating or measuring its existence

Need to differentiate between:

1. Economic impact library services on a community

2. Economic value of social returns

3. Social impact = transformative aspect of library services

(Broady-Preston, 2012)

No. 3 = Most challenging!

Economic impact: models and theoriesEconomic Impact Toolkit (ERS, 2011)

– commissioned by the Archives Libraries and Museums Alliance UK (ALMA-UK)

– Critique contemporary methodologies in 4 categories• Multiplier analysis• Contingent valuation

– methodology of choice? (cf Poll, 2012) – Predictive vs actual

• Return On Investment (ROI) – Generic studies of University performance overall claim ROI amongst others =

demonstrably “inadequate” (Fryer et al , 2009; Zangoueinezhad and Moshabaki, 2011)

– critique of ROI by Jim Neal (March/April 2011)“Stop the Madness: The Insanity of ROI and the Need for New Qualitative Measures of Academic

Library Success”

• Economic valuation

Social Enterprise methodologiesOriginating in voluntary or non-profit sector of economy

• Often categorised as “third or tertiary sector”. Defined as

“An intermediate space between business and government where private energy can be deployed for public good” (BusinessDictionary.com, 2013)

• Closer resemblance to libraries, especially with increasing use of volunteers, multiple stakeholders and social objectives?

Methodologies and toolkitsSocial Enterprise Balanced Scorecard (BSC)

http://www.proveandimprove.org/new/tools/socialenterprise.php

• Adaptation of Kaplan & Norton’s original BSC• Designed to help social enterprises deliver “multiple bottom lines”• Clarifies and articulates complexity arising from multiple stakeholders• BUT – even modified BSC may focus on satisfying service funders

rather than recipients cf Kong, 2010.• No reports of use by LIS June 2013.

Social Enterprise Balanced Scorecard: Strategy map

Sample strategy map

Key: White circles reflect objectivesDotted arrows reflect cause and effect linkages

Social Return on Investment (SROI)Social Return on Investment (SROI)

• Outcomes – based tool quantifying social, environmental and economic value created by organisations

• Narrative of how organizations create and destroy value whilst making changes in the world, producing a ratio stating how much social value (in ££s) created per 1£ spent

• 6 step methodology Primer = http://sroi.london.edu/

Libraries for Life: Wales http://wales.gov.uk/topics/cultureandsport/museumsarchiveslibraries/cymal/researchandevidence/lflevaluation/?lang=en

• Welsh Government used SROI to evaluate impact of strategic development programme for library services (2011)

• Demonstrated Government's investment in library services had “significant impact” AND excellent value for money

• Politically = very popular – profile raising • BUT – limited value in evaluating large scale projects

Social/Societal Impact Methodologies

Customer Relationship Management (CRM)

Customer Value Discovery (CVD)Commonality and/or Core set of customer values? Do these change

over time? (McKnight, 2010)

Value, Values and Valuation - Values Scorecard“transcendent value” = holistic, recognising need to assess

additional dimensions – human, structural, and relational capital (Town, 2011; Town & Kyrillidou, 2013)

Impact: measuring activity + behaviour• JISC – Activity Data Programme

– “benefit from exploiting activity data” (2011: 2013) (https://repository.jisc.ac.uk/5010/1/JISC_Activity_Data_singlepages.pdf (2013))

• US and UK projects = statistical correlation between library service usage and student learning and achievement

• BUT – Statistical measures alone = limited perspective

– University of Nottingham – systematic reviews(Barr, August 2012); IMLS LibValue project (Fleming-May, Kyrillidou, Walker, 2013)

– Phase 2 Huddersfield University Library Impact Data Project“Seeks to deepen our understanding…by investigating additional data such as gender, age, ethnicity, declared disability, retention, VLE and reading list use…to identify predictors for student outcomes…to understand better how library activity relates to student attainment, including causal relationships” (Stone, Collins and Patmore, 2012)

Finally…• Provided an overview of SOME ideas and concepts in relation

to demonstrating the value and impact of information and library services

• NB – not definitive!

• Economic measures alone = limited perspective. However

– “promoting the value of the libraries in the community also through economic statements can be quite effective. Speaking in terms of dollars and cents seem to have a heavy impact on people, both politicians and ordinary citizens who may not register the value of library services otherwise” (Aabø, 2009, 323)

• Need to look beyond economic impact alone. “Money speaks” but is not the sole language in terms of demonstrating impact!

• Measuring true impact library service activity clearly extremely important in turbulent and unpredictable environment.

Finally…• In a global recession, service organisations need to focus on

continuous service improvement and the stakeholder perspective

• BUT – a complex activity. Demonstrating social impact may be difficult but we need to look beyond methodologies used currently, and evaluate those used successfully elsewhere.

• reinforce the message to staff that “ excellence in the provision of our services should always be at the forefront of our minds” (Professor John Lancaster, 2010).

• Active engagement and partnership with customers = imperative; not only to be valued but to be seen as vital and survive!

ANY QUESTIONS?

If you think of something later, please contact me – jbp@aber.ac.uk

THANK-YOU!