Embed Size (px)
Citation preview
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 1/9
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 2/9
www.LTSGlobal.com
1-888- 877-9531
Elwood F. "Ed" Holton III, is CEO of Learning Transfer Solutions
Global LLC and Jones S. Davis Distinguished Professor of Human
Resource, Leadership and Organization Development at Louisiana
State University, USA. Dr. Holton has led efforts to create theLearning Transfer System Inventory and the Training Transfer
Solution system over the last 15 years. With over 200 articles and
17 books, he is widely considered to be an international expert on
human resource development and particularly learning transfer.
Contact him at [email protected]
“Dr. Ed Holton is one of the premier experts in learning transfer. Great to work with and extremely
professional.” November 29, 2010 Robin Kistler,
Director, LSU Executive Education, LSU - Stephenson Entrepreneurship Institute
“Ed Holton is one of, if not the, foremost experts in the area of transfer of training and perhaps HRD ingeneral. He has rather ingeniously used the fruits of his career-long research and experience andshaped it into the tools that companies can and should benefit from. Having personally worked withEd on research projects in this area I can definitively say that his solutions are meticulouslydeveloped and designed and boast rigorous theoretical framework (not something you encounterfrequently in HRD consulting). At the same time Ed’s tools and methods are designed for the real -world, demonstrating his exceptional ability to connect research and practice. Last but not least, he isa pleasure to work with, approachable, and down-to-earth and I have always walked away from aconversation with him feeling like I’ve learned something new.” November 21, 2010
Bogdan Yamkovenko, PhD
Organizational Development and Research Coordinator, The Shaw Group
“I regard Ed Holton as among the leading experts in the world on the subject of transfer of learning.Although he has written widely and is highly-regarded in the academic community, his unique gift isthe ability to convert ideas to practice and make a real difference in the effectiveness of learning
initiatives in organizations.” November 19, 2010
Tim Baldwin,
Eveleigh Professor of Business Leadership,Kelley School of Business
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 3/9
The mistake we make is that we use
these few people as justification for
the worth of the entire
training/learning effort. These
anecdotal "success stories" becomethe metric for success. Instead of
looking at the return on investment
from the whole class, we often
focus just on the few success
stories and conclude the whole
effort was successful.
The Mirage Of The Innovators and Early Adopters
In every training class there are at least a few people
who just LOVE what we are teaching and jump to
implement what they learn. And, oh how we love these
people! These are the students trainers live for and
keep us motivated from day to day. They are a joy and
they make our training lives rewarding and satisfying.
Unfortunately they are a bit of a mirage. The mistake we
make is that we use these few people as justification for
the worth of the entire training/learning effort. These
anecdotal "success stories" become the metric for
success. Instead of looking at the return on investment
from the whole class, we often focus just on the few
success stories and conclude the whole effort was
successful.
Let me explain why this is
shortsighted. There is a rich
body of research that explains
why we are always likely to
find these ambitious fewemployees who will transfer
what they learn. It is called
the diffusion of innovation
research. It turns out that it
doesn't matter whether we are
teaching uneducated farmers
to plow rows in their fields
differently or college-educated
professionals how to service
their customers better the rate of adoption follows a
remarkably similar pattern. Typically, 2.5% of the
people, called innovators, will jump immediately to try
something new just because they love new things.
Another 13.5% of the people, called early adopters, can
be persuaded to try new things with only reasonable
effort such as we might do in a training class. The
remainder of the people will be much slower to adop
new things.
Now if you do the math that means that we can fairly
easily expect to convince 15% of our trainees to use the
new methods we are teaching. Remember what our
estimate is of typical rates of learning transfer--10-30%-
Exactly in the range that is predicted by the diffusion
research!! And 2.5% of the people will do anything new
just because they like change!
So you see it is a mirage for us to focus just on these
innovators and early adopters. Fortunately the early
adopters can be influential in getting other people to
adopt the new learning but at the end of the day the
"fallacy of the few" means we have to look beyond the
15% who are likely to be success stories and find waysto reach the other 85% of trainees.
The solution is transfe
management. The first 15%
of trainees are the "low-
hanging fruit" of training. To
get the rest we have to pu
in place PROACTIVE
strategies that will catalyze
learning transfer.
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 4/9
www.LTSGlobal.com
1-888- 877-9531
Suppose you could DOUBLE the
performance improvement that
results from your training
programs. That's real money you
can make (or save) for your
organizations. Cutting costs by not
working on improving learning
transfer is NOT the right response.
Increase Learning Transfer Improvement To Help Your
Organization Through The Tough Times
Everywhere I look I see people and organizations trying
to do more with less in this weak economy. For
example, I am continuing to drive my car with 90,000
miles on it rather than buy a new one. We eat out at
restaurants less than we used to and generally try to
squeeze more out of our income each month. Every
organization I talk to is trying to do the same thing. They
are cutting budgets, reducing staff and generally trying to
squeeze all they can out of every dollar. I suspect you
are doing it too.
Trainers and human resource developers, I have the
perfect answer for you---improve your learning transfer!
I know, the typical answer is to reduce costs. But,
suppose you could DOUBLE the performance
improvement that results from your training programs.
That's real money you can make (or save) for your
organizations. Cutting costs by not working on
improving learning transfer is NOT the right response.
Let me relate it to my personal car situation. I just spent
some money on maintenance to extend the life of my car
with 90,000 miles on it. If I hadn't spent the maintenance
money, the car would have had major problems and I
would have to buy a new car. By investing a modest
amount now, I can get a much greater return out of the
car. In this case, SPENDING money was my best way
to SAVE money. Makes sense right?
Well then spending some money to increase learning
transfer also makes sense. Think about this--how much
more money would your organization make--or save--i
you doubled your learning transfer rate? And how much
would it cost you to increase transfer that much? O
course, with our TransferLogix system the answer is no
very much.
In the end, increasing transfer is your best investment
right now. Don't let tight budgets cause you to overlook
your best way to help your organization through the
weak economy. It's simple math---get more "bang fo
the buck!"
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 5/9
www.LTSGlobal.com
1-888- 877-9531
If CEOs decide to champion learning
transfer change, they will need HR
as a committed partner so they
tend to tread lightly.
Learning Transfer Change Needs A Champion
Every organizational change needs a champion. And,
improving learning transfer in an organization is a
change process. The question is: Who should be the
learning transfer change champion?
The conversation usually starts with the Human
Resources department. They seem to be the logical
choice, right? In some respects they are since they
usually control the learning initiatives. It seems to make
sense that they should therefore be the ones to "push"
transfer through the organization
and be made accountable for the
outcomes. The problem is that
about half of the outcomes areinfluenced by factors outside their
control; that is, in the work
environment. So they are put in the
uncomfortable position of trying to influence line
organizations to do things to help transfer. The reality is
many HR organizations simply don't feel empowered to
do it. Sometimes they aren't, and sometimes HR just
doesn't know how to be a business partner. Regardless,
they just don't feel comfortable working outside whatthey directly control--the learning events.
So why shouldn't line management (CEO, COO, etc.)
take the lead and champion the change? It would
certainly make sense because they are the ones
responsible for the business results from training,
right? Clearly they have the most at stake as to whethe
business results occur. Perhaps they should "pull
transfer through the organization. On the other hand
most departments don't like to have change imposed on
them and HR departments are no different. HR
departments are quite capable of subverting change
efforts they don't believe in. If CEOs decide to champion
learning transfer change, they will need HR as a
committed partner so they tend to tread lightly
And then there is the Chief Financial Officer. They are
responsible for monitoring the return on investment from
the organization's investments. Thus, wouldn't it make
sense for them to push to increase learning transfer so
that the ROI from learning increases? CFO's certainly
"get" the need for improving learning transfer and often
have a lot of power in organizations. On the other hand
how many HR departments do you know that have a
close relationship with their CFO? And how many CFO's
do you know that understand learning?
So what happens in
practice--nobody
champions learning
transfe
improvement! It simply
falls through the cracks because nobody feels
empowered to take the lead--and nothing changes. It sits
out there in the "gray area," belonging to everybody bu
nobody.
I argue that SOMEBODY must step up and become the
learning transfer change champion in each
organization. The answer to who it should be will likely
vary from organization to organization. It could be HR
the CEO, or the CFO--but somebody has to take the
lead.
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 6/9
www.LTSGlobal.com
1-888- 877-9531
When resources are plentiful it's easy to
get away with ignoring best practices in
learning transfer. In tight times like we
are in now, learning transfer is not
optional.
So decide today who it should be in your organization--
and get started on the learning transfer change journey!
It's Not Just More Of The Same!
From all over the world I am hearing the same message-
-we have to get more results from our training/learning
investments. In the down economy every dollar counts
and our customers (be they internal or external) are
challenging the training
profession to get more
performance results from
training.
While the desire to improve
training results is not new, I
am sensing a tipping point for our profession. The
pressure I am hearing from management is at a level I
have never heard in my almost 30 years in the
profession. What I am hearing are managers that are
simply fed up with training that doesn't produce results.
We (the training/learning profession) have to be veryhonest with ourselves. We are pretty good at getting
people to learn. We know a lot about learning and our
knowledge base about learning methods is pretty good.
But, our track record on getting performance results from
training is NOT good. We have been far too complacent
about allowing poor learning transfer rates to persist.
Fortunately our knowledge base about learning transfe
is also good--it's the adoption in practice that is lacking
When resources are plentiful it's easy to get away with
ignoring best practices in learning transfer. In tight times
like we are in now, learning transfer is not optional.
Smart training organizations are getting very serious
about improving results, which means they are getting
very serious about fixing their learning transfer problem
They are adopting best practices and looking at the mos
innovative products to economically fix their transfer
problems.
Smart training organizations are realizing that the only
solution is to change
the way they have
been doing business
More of the same
just won't deliver the
results in many
cases. The endresults will be cuts in
training resources if results aren't delivered.
Is your training organization one of the smart ones?
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 7/9
www.LTSGlobal.com
1-888- 877-9531
Utility analysis has been seen sometimes
as “too good to be true” but it really is a
carefully researched and effective
technique. Now TransferLogixTM from
Learning Transfer Solutions Global has
implemented this evidence-based
approach so that it is “doable” by every
organization for every training program.
I happen to be sitting on an airplane right now and have
spent the last two days engaging with the airline
industry. Despite my frustrations, I had time to step back
and admire how this very complex industry manages to
keep planes and people moving rather safely.
One of the things I notice is how well trained the
employees seem to be. As much as I hate it when they
"go by the book" in implementing a policy, I am at the
same time impressed that they are transferring their
learning.
Of course the advantage they have is that much of their
learning has life and death
implications--literally. Because the
stakes are so high the industry has
apparently learned to transfer learning
much better than in many other
industries.
So I wonder why? I wonder how wecan infuse the same sense of urgency
for learning transfer in other types of
learning. Don't all organizations have
mission-critical training with high stakes if it fails? In my
experience, yes--but I don't see them infusing the same
sense of urgency.
It looks simple to me--if the training you are doing is
mission critical, then you MUST insure learning transfer
And if it´s not mission critical, why are you doing it? Stop
the "nice to have but not critical training and use the
resources to get real results from mission-critical training
and you will be way ahead.
Yes, It Can Be Done Practically and Economically!
For as long as I have been in the HRD profession RO
has been the "holy grail" of training evaluation
Unfortunately, an effective AND practical method has
been elusive. Despite literally decades of promoting it
the fact is ROI is rarely calculated for training
investments.
Debate has
swirled fo
years abou
how best to
calculate
ROI
Evaluation
purists have
developed
effective
methods
but they are mostly impractical for all but the biggest
training initiatives. The methods promoted by purists
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 8/9
www.LTSGlobal.com
1-888- 877-9531
tend to be costly in either time and staff resources, or
money for consultants.
On the other side are "quick and dirty" methods which
lack validity but are easy to implement. These methods
have proved unsatisfying because the results just aren't
credible.
So do we just give up? Is ROI simply unattainable or
even ill-advised as some suggest? I say no, we can do
it and we should. The fact remains that ROI is the
universal language of organizations, particularly private
sector organizations. And, if we want resources spent
on training to be viewed as investments--which they are-
-then we HAVE to at least estimate ROI.
Fortunately there is a practical and economical way to
estimate ROI. Largely overlooked has been a technique
advocated for over 60 years by industrial-organizational
psychologists called utility analysis. The list ofresearchers advocating utility analysis reads like a
“Who’s Who” of I/O psychology with Wayne Cascio and
John Boudreau (2011) being the most current and well-
known advocates but with much of the research
foundation built by the renowned John Hunter and Frank
Schmidt . Thus, it has deep and well-researched roots
that have demonstrated its validity. Unfortunately it has
not been adopted by practitioners because they
mistakenly see the statistics involved as too complex.
Utility analysis has been seen sometimes as “too good
to be true” but it really is a carefully researched and
effective technique. While I would never argue that
these estimates are better than a carefully conducted
custom study, I (and others) do argue that the RO
metrics produced by utility analysis have a high level of
validity. Because they can be easily calculated fo
EVERY training program—which is not true of custom
studies—they are a superior approach to making sound
decisions about development interventions.
Now TransferLogixTM from Learning Transfer Solutions
Global has implemented this evidence-based approach
so that it is “doable” by every organization for every
training program. TransferLogixTM demystifies utility
analysis so ROI can truly be the benchmark for every
training program and become part of the daily lexicon for
human resource development.
Check it out.
This post may be heresy to my evaluator friends but
believe we have made training evaluation too complex
for everyday use.
I have spent my whole career championing the need to
evaluate results from learning. But traditional evaluationtexts make it too much like a scientific endeavor
probably because they are written by researchers.
8/3/2019 Evaluating Learning Transfer
http://slidepdf.com/reader/full/evaluating-learning-transfer 9/9
www.LTSGlobal.com
1-888- 877-9531
We will never get organizations to tak
measurement seriously if we continue
to make it too complex to implementOur approach is to find a balance
between validity and practicality. The
real test is whether your measuremen
methods lead you to the right decision
not whether they meet research qualit
standards.
We have to remember that the purpose of evaluation is
TO MAKE BETTER ORGANIZATIONAL DECISIONS. A
careful read of the decision making research clearly
shows that organizational decisions are rarely made
using perfect data. Rather, there is a concept called
"bounded rationality" which is everyday terms means the
data just has to be "good enough."
That's why I argue for measurement tools and processes
that are practical to implement, even if they fail the
research standard. We will never get organizations to
take measurement seriously if we continue to make it too
complex to implement.
Our approach is to find a balance between validity and
practicality. The real test is whether your measurement
methods lead you to the right decision, not whether they
meet research quality standards. If the data is "good
enough," then that's all you need.
Keep in mind that so called level one data is NOT valid
for good organizational decisions about learning. You
have to measure behavior change and business results.
Measurement is in itself an intervention that will improve
learning transfer.
So MEASURE, but don't over-invest. Use a practical
approach first and then if the decision is in doubt you
can always invest more to get better data. Often the
practical approach is plenty good enough. For example,
one evaluation I recently did showed learning yielded an
ROI estimate of 475%. Sure we could get a more
accurate answer but is the decision really in doubt. No.
Often, good enough is all you need.
