Redefining Measurement for Continuous Learninglogin.bersin.com/uploadedFiles/...for-continuous-learning-bersin.pdf · employee learning and development is wasted ... ASTD / ROI Institute,

Redefining Measurement for Continuous Learning

Todd Tauber

Vice President, Learning Research

Bersin by Deloitte, Deloitte Consulting LLP

April 1, 2014

How to demonstrate impact when

learning goes beyond L&D

Copyright © 2014 Deloitte Development LLC. All rights reserved. 1 IMPACT 2014 THE BUSINESS OF TALENT®

The importance of measuring impact

Persistent challenges to measuring impact

New complexities in measuring impact

Redefining measurement and evaluation

Contents

Todd Tauber

Vice President, Learning Research

Bersin by Deloitte, Deloitte Consulting LLP

The importance of measuring impact


Measuring impact is about demonstrating – and communicating – the value of L&D to the organization

If you don’t tell people about your success,

they probably won’t know about it.

Donald Trump,

The Art of the Deal


Learning is increasingly important

Source: Ocean Tomo LLC, Intangible Asset Value, 4/2011

Companies are increasingly relying on human capital

17%

32%

68%

80% 80%

83% 68% 32% 20% 20%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1970s 1980s 1990s 2000s 2010s

Sh

are

of

S&

P 5

00 V

alu

e Tangible assets

Intangible assets


Organizations invest a lot in L&D

Organizations expect a lot from L&D

0%

1%

2%

3%

4%

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

% o

f p

ayro

ll

Source: ASTD, 2013 State of the Industry, 10/2013;

Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014

$1,169 per learner


Three ingredients for “high-impact” learning

…and employee satisfaction isn’t one of them

Source: Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012

Business and

organizational impact

Alignment

Efficiency

Effectiveness


Learning isn’t working for many organizations

Source: Deloitte, Global Human Capital Trends, 3/2014;

Chief Learning Officer, Scrap Learning and Manager Engagement, 3/2011

of global HR executives

believe their organization

is “ready” with the right

people with the right skills

in the right places.

15%

of the knowledge and

skills people learn in

training are actually

applied on-the-job with

positive results

10%

of the money spent on

employee learning and

development is wasted –

irrelevant, redundant, not

absorbed, not applied.

50%

It’s not aligned It’s not effective It’s not efficient


L&D is often one of the first things to get cut when times are tough

Learning is essential, but L&D is vulnerable

Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014

7.0 6.2

5.3 5.2

4.2

5.9

0

1

2

3

4

5

6

7

8

9

10

2008 2009 2010 2011 2012 2013

L&

D s

taff

/ 1

,000 l

earn

ers


Would anyone miss you?

Source: Henley Business School, Big Data and HR, 7/2013

I told them not to send [the reports] to anyone…Guess what? At the

end of the month, hardly anyone had asked for their reports, so they

clearly weren’t getting any insights from them. [But] if we had asked

them if they wanted their reports, I’m sure they would have said yes.

The moral of the story is that overuse or misuse of data analytics…is

in danger of …diverting attention away from the real business issues.

former Chief Learning Officer,

manufacturing company

Persistent challenges to measuring impact


Three stubborn hurdles

Source: Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012

Where to start?

Who / what it’s for?

What methodologies to use?

People and relationships

Systems for capturing data

Tools for analyzing data

Skills

Funding

Leadership / management support

Culture

Frameworks

Infrastructure

What are the biggest challenges you have in measurement?


How CEOs measure learning success

The customer is always right

Source: ASTD / ROI Institute, Measuring What Matters: How CEOs View Learning Success, 12/2010

96%

74%

61%

82%

28% 22%

8% 4%

11%

78%

32%

53%

Impact ROI Application Cost efficiency Learning Reactions

We should measure this

We do measure this


Directional insights, not precise calculations

Source: CFO, Measured Response, 6/2011

Chief Learning Officer,

technology company

Where people get messed up is in thinking that they have to measure

individual training classes to make sure that participants absorbed

the information and are applying it to their jobs.

I'm not interested in that, and neither is our CFO. He wants to

measure whether managers are performing well and whether the

organization is productive.


More mature L&D groups dig deeper into talent and business outcomes

Business metrics, not busy metrics

Source: Bersin by Deloitte, Practical and Proven Approaches to Learning Measurement, 4/2012

0%

50%

100%

0% 50% 100%

Frequency

Va

lue

Learner satisfaction (Level 1)

Enrollments

Utilization

Hours delivered

Job impact (Level 3)

Business impact (Level 4)

Impact on business metrics

ROI Manager

satisfaction

Learning

Scores

(Level 2)

Costs

Business leaders CLOs Instructional designers

New complexities in measuring impact


Learning isn’t just programs and courses anymore

Continuous learning happens with and without L&D

Time

Kn

ow

led

ge

/ s

kill

s

Tra

inin

g e

vent

Traditional training

Continuous learning

Reinforcement via… • Coaching & mentoring

• Peer feedback

• Job aids / tools

• Online / mobile courses

• On-demand content

• Conferences

• Professional networks

• Social network

• Communities of practice

• …etc.


We learn

through

interaction,

everywhere,

all the time

Continuous learning activities can't be measured as simply as events, courses and programs

Mentoring

Books

Wikis

Conferences

e-learning

Video

Coaching Webinars

Websites

Podcasts

Job rotations

Manuals

Peers

Discussion boards

Lunch-and-learns

Chalk talks

Social networking

Whitepapers

Case studies

Following

Tagging

Micro-blogs

Virtual worlds

Stretch goals

Post-mortems

Performance support

Articles

Meetings

Onboarding

Communities-of-practice

Demonstrations

Debates

Role models

Job shadowing

Mistakes

Successes

Visual aids

Instructions

Consulting

Advice

Collaboration

Search

Lectures

After action reviews

Interviews

Feedback

Customer data

Observation Journaling

Role Playing Simulations

Goals Experiments

Labs

Play

Conversations Teaching

Presentations

Dialogue


What % of learning hours are delivered via…

More digital learning means lots more stuff to track

2005 2006 2007 2008 2009 2010 2011 2012 2013

Source: Bersin by Deloitte, 2014 Corporate Learning Factbook, 2/2014

Online self-study

Other things

Live instructor-led

Live virtual


There’s a lot more to collect and connect

Data is fragmenting way beyond the LMS

Learning systems HR / HCM systems

• Multiple LMSs

• vILT systems

• Collaboration platforms

• Learning portals

• ATS / assessment

• Core HRIS

• Workforce analytics

• Talent management systems

Business systems Other systems

• ERP

• Finance

• Business intelligence

• CRM / marketing

• Social media

• Web analytics

• Mobile apps

• MOOCs


2009

2014

Are your business leaders asking for quantitative measures of HR / L&D?

Growing demand for better insights

Source: KnowledgeAdvisors, The Business Drivers of Talent Analytics, 3/2014

In 2014

52% are saying yes

(and 29% expect

them to soon)

In 2009,

21% said yes


Which functions have strong analytics capabilities?

Expectations outpacing capabilities

15%

56%

58%

77%

81%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

HR

Marketing

Sales

Operations

Finance

Bersin by Deloitte, High-Impact Talent Analytics, 10/2013


Same old lack of know-how, systems and support

The more things change, the more they stay the same

For informal learning

For on-demand learning

For continuous, self-directed learning

New data

New data sources

New systems and tools

Changing demands

Increasing expectations

Different skills and capabilities

Culture

Frameworks

Infrastructure

Redefining measurement and evaluation


The basics still apply

Source: Bersin by Deloitte, Key Trends in the Measurement of Talent Development, 3/2012;

Bersin by Deloitte, Becoming a High-Impact Learning Organization, 8/2012

60% of L&D leaders we surveyed said their

companies don’t have a real plan for

measuring learning impact.

1.2x companies that regularly engage with

stakeholders are more likely to have

high-impact measurement capabilities.

2% 64% of L&D organizations spend 2% or

less of their budgets on measurement.

72% think they should spend more.

Dedicate resources

and manpower

Have a plan

Build relationships

with stakeholders


Rethink yesterday’s tools and practices

Kirkpatrick and Phillips methods are only the beginning

Source: Bersin by Deloitte, Key Trends in the Measurement of Talent Development, 3/2012

Too simplistic…

Does liking a course or

passing a test really

translate directly into

better performance or

business results?

Too hard…

There are no clear,

consistent definitions

and it’s rarely simple or

easy to isolate

learning’s contribution

(…or L&D’s).

And too backward

Finding out after

training that people

were not satisfied – or

that results didn’t

improve – doesn’t tell

you what to change.


Build (some) new capabilities

Utility / usefulness

Stakeholder satisfaction

Experience API (xAPI) Employee engagement

Learning records stores Readiness / promotions

Information technology Adaptive learning Employee retention

Finance and accounting Web analytics Time to proficiency

Marketing Social listening Activity / contributions

Business intelligence CRM systems Influence and feedback

New friends New tools New metrics


Adopt some new approaches

Good enough is good enough

Predictive analytics

Continuous monitoring


Marketers measure and optimize ad campaigns continuously

Find inspiration in finance and marketing

Source: Adapted from Harvard Business Review, A Better Way to Measure Your Ad Campaign, 3/2013

Market conditions • Season

• Consumer confidence

• Fuel prices

Marketing actions • TV advertisements

• Online ads

• Social media

Competitive activity • Product releases

• Pricing

• Promotions

• Search

• Online chatter

• Store visits

• Purchases

• Unit sales

• Revenues

• Margins

• Market share

• Share of voice

• Customer value

Response

Outcomes Analyze Optimize Allocate


…which (often) makes it easier

to gain access to resources

The payoff

And more

efficient operations

Better

alignment

More

effective learning

…which (often) leads to better

relationships with business leaders

…which (often) leads to more

credibility and opportunities

This publication contains general information only and Deloitte is not, by means of this publication, rendering accounting, business, financial, investment, legal, tax, or other professional advice or

services. This publication is not a substitute for such professional advice or services, nor should it be used as a basis for any decision or action that may affect your business. Before making any decision

or taking any action that may affect your business, you should consult a qualified professional advisor.

Deloitte shall not be responsible for any loss sustained by any person who relies on this publication.

As used in this document, "Deloitte" means Deloitte Consulting LLP, a subsidiary of Deloitte LLP. Please see www.deloitte.com/us/about for a detailed description of the legal structure of Deloitte LLP

and its subsidiaries. Certain services may not be available to attest clients under the rules and regulations of public accounting.

Copyright © 2014 Deloitte Development LLC. All rights reserved.

36 USC 220506

Member of Deloitte Touche Tohmatsu Limited

Documents

Redefining Measurement for Continuous Learninglogin.bersin.com/uploadedFiles/...for-continuous-learning-bersin.pdf · employee learning and development is wasted ... ASTD / ROI Institute,