Evaluating Learning (English)

  • View

  • Download

Embed Size (px)

Text of Evaluating Learning (English)

  • 7/30/2019 Evaluating Learning (English)

    1/16D17. Evaluating Learning | 1

    D17. Evaluating Learning

    Education and Training should be evaluated as

    robustly as any activity which uses up precious

    resources and it is important that a orm o

    measurement is introduced so that not only is

    the quality o the learning experience measured

    in some way, but also the impact o the learning

    experience on the learner, their work and

    subsequently the patients cared or by their


    With this, more robust, model in mind it

    would be better to think o evaluation as a

    orm omeasurement of effectiveness

    a orm oassessment.

    In this section o the Good Practice Guide we

    will be looking at theories around evaluation

    o education and training and how it is possible

    to measure not only the quality o the learning

    experience, but also the impact on the learner

    and the extent to which they have used their

    new skills and knowledge. This approach fts with

    NHS employers growing desire to demonstrate

    outcomes and outputs or the resources invested.

    What is Evaluation?

    Many people think that evaluation of Education and Trainingis about checking that things were okay during the training.Where evaluation happens it is often characterised byhappy sheets which usually check with the learners thatthe conditions for learning were okay and how highly thelearners rate the facilitator and the training as a whole. In fact,this is only a part of what evaluation is about.

  • 7/30/2019 Evaluating Learning (English)

    2/162 | D17. Evaluating Learning

    Why evaluate education

    and training training?

    Evaluation is vital because it should enable

    an organisation to gauge whether:

    the teaching and acilitation techniques and

    methods used are worthwhile

    we are making the best use o our


    the trainer or acilitator requires


    the education and training unction/team/

    service is supporting the organisation to

    develop its sta and its services

    the education and training unction is

    contributing well to the recruitment and

    retention unctions o the organisation workers, and the services they provide are

    improving as a result o the investment in

    their development

    teaching and acilitation materials need to

    be improved

    there is a need or urther investment in

    education and training

    Evidence from trainers

    Evidence rom a signifcant number o people

    involved in education and training in the NHS

    indicates that organisations are aware o a need

    to evaluate but they oten struggle in getting

    to grips with actually doing it, or getting the

    inormation they want rom the evaluation.

    Methods of assessmentQuestionnaires are the most requently

    used method o evaluating or assessing the

    eectiveness o education and training and they

    can be very eective. The trick here is to ocus

    on the real issue. I the questionnaire approach

    is not working, dont necessarily abandon the

    method, just re-examine the quality o the

    questionnaire. Set out later in this section is a

    questionnaire which has been built by the AllWales Workorce Development and Education

    Forum which has been tested robustly.

  • 7/30/2019 Evaluating Learning (English)

    3/16D17. Evaluating Learning | 3

    Issues to consider when

    designing evaluation of

    education and training

    Evaluation should have a purpose. Think

    about why you are measuring and what

    inormation you want to get out o the

    process beore selecting a tool.

    What are the objectives and learning

    outcomes or the session? The evaluation

    should aim to measure whether you have

    achieved them. This is very important

    because it moves the education and training

    close to how accredited learning operates.

    Decide who is going to collect the

    inormation and how it is going to be

    collected and collated. The

    responses may be affected for example

    i the trainer is going to ask about the

    quality o a session, participants may not

    want to give negative responses.

    How oten is the evaluation going to take


    Is this going to be a routine activity ater

    each session?

    Is this something that will be done every

    so oten or a one o event or example

    ater the introduction o new methods

    or materials?

    Are you going to collect baseline data so

    that you have a yardstick or measurement?

    A general rule is that evaluation should

    be as simple and economical as possible

    (particular i evaluation is going to be

    undertaken on a routine basis).

    Tips for evaluating

    Evaluation can be difcult but it is essential

    because it helps the education and training

    unctions to improve and demonstrates the

    quality and impact o the service it provides

    to the organisation.

    Identifying users perceptions

    of knowledge and skillsOne method o identiying whether the training

    you have given has had an eect is to ask learners

    whether they eel their knowledge and skills have

    improved ollowing training. This requires a word

    o caution - research in this feld has shown that

    learners can over estimate or underestimate their

    skills. This may be due to the act that, quite

    legitimately, their confdence has increased. It is a

    worthwhile activity i there is verifcation available.

    This approach is better or knowledge acquisition

    than skills and even better is assessment or

    witness testimony, preerably rom the workers

    manager, to get an objective view.

    Assessing users skills

    Assessing participants skills is a good method

    o evaluating whether education and traininghas been eective, ie have participants learned

    what you intended to teach them as a result o

    the training have the learning outcomes or

    objectives been met?

    Assessing is the most eective method but it

    is also the most time consuming and learners

    can be intimidated by testing. Any orm

    o assessment should be unobtrusive and

    accompanied by reassurance.

    Using assessment combined with a questionnaire

    gives a highly robust method.

  • 7/30/2019 Evaluating Learning (English)

    4/164 | D17. Evaluating Learning

    The four levels of Kirkpatricks evaluation

    model essentially measure:

    reaction of learner what they thought and

    elt about the education and training

    learning the resulting increase in knowledge

    or capability

    behaviour extent o behaviour and capability

    improvement and implementation/application

    results the eects on the business or

    environment resulting rom the learners


    Theoretical Approaches

    There are numerous theories about evaluating learning but it isworth focusing on one or two of the most famous:

    Donald L Kirkpatricks Evaluation Model

    The four levels of learning evaluationKirkpatrick stated that there are our levels o learning evaluation and that there should be

    measurement o each o the our levels. All these measures are recommended or ull and

    meaningful evaluation o learning in organizations, although their application broadly increases in

    complexity, and usually cost, through the levels rom leveL 1-4.






  • 7/30/2019 Evaluating Learning (English)

    5/16D17. Evaluating Learning | 5

    Kirkpatricks four levels of training evaluation in detail

    This diagram illustrates how Kirkpatrick s theory can be applied to a process or evaluating learning.

    Start with the Learning Environment in the block on the let and work through the process.


    (Summative Evaluation)

    Revise System

    Working Environment

    (Formative Evaluation)

    Learning Environment

    (Formative Evaluation)

    Level 1Reaction


    Level 2Learning

    Level 4Impact How is the individual impacting onthe organisation? How the training worked?

    Continueand refne

    the learning

    Learning NeedsImprovement?



    Level 3Behaviour and Perormance

  • 7/30/2019 Evaluating Learning (English)

    6/166 | D17. Evaluating Learning

    Kirkpatricks Four Levels of Evaluation

    Since Kirkpatrick established his original model, other theorists (or example Jack Phillips), and indeed

    Kirkpatrick himsel, have reerred to a possible fth level, namely ROI (Return On Investment). ROI

    can easily be included in Kirkpatricks original ourth level, Results.

    Level evaluation

    type (what is



    description and


    examples of

    evaluation tools

    and methods

    relevance and

    ease of use

    1 Reaction Reaction evaluation is

    how the delegates elt

    about the training or

    learning experience

    eg., happy sheets,

    eedback orms;

    also verbal reaction,

    post-training surveys

    or questionnaires

    Quick and very easy

    to obtain

    Not expensive to

    gather or to analyse

    2 Learning Learning evaluation

    is the measurement

    o the increase in

    knowledge - beore

    and ater

    Typically assessments

    or tests beore and

    ater the training

    Interview or

    observation can also

    be used

    Relatively simple to

    set up; clear-cut or

    quantifable skills

    Less e