Introduction

Training requires time , energy and money. Therefore an organization needs to know whether the investment made in training is being effectively and is worth the effort. Top management is concerned with evaluation as a process by which the effectiveness of the organizations programs and operating procedures are demonstrated. Supervisors are concerned more with the “results” of training as compared by changes in workers on the job performance.

Evaluation is a planned process which provides specific information about a selected session, program for the purpose of determining value and or decision making. Related to training evaluation is concerned with providing information on the effectiveness of the training activity to decision-makers who will make decisions based on the information. There are various models that have been developed to describe the role of evaluation in the training process. It is important that evaluation be a planned or systematic effort that is built-in from the start of the training process.

“The reason for evaluating is to determine the effectiveness of a training program.” (Kirkpatrick, 1994, )

1.“To justify the existence of the training department by showing how it contributes to the organizations’ objectives and goals.”

5

2. “To decide whether to continue or discontinue training programs.”

3. “To gain information on how to improve future training programs.” (Kirkpatrick, 1994, )

7

Ascertaining Reactions Of the Participants

As the programme gets over, it is important to ascertain the participants views and observations about its various components both in terms of the contents and the training process. The evaluation can either focus on a session-by-session review or on the programme as a whole. The formal format that the trainers choose to adopt for eliciting end-term appraisal of the participants’ ranges from the relatively simple “highlight the best and worst aspects” to a detailed response through a properly devised questionnaire or instrument.

Areas for Ascertaining Reactions of the ParticipantsAn important decision that the training team takes prior to the programme is about the specific elements it would like to include in the evaluation exercise.

Objective•How far the programme objectives are realistic.•Clarity•Individual understanding and response•How far the training objectives are realized.

Trainers Performance•Effectiveness of the presentations.•Skill of the trainers to use the training methods•Trainers expertise in responding to the questions of the participants.•Whether the trainers attitude was supportive and helpful.•Their relationship with the training group.

Training Methods•Their appropriateness tot the programme and the modules.•The extent to which they facilitated learning.

Training Group•Its size and composition•Selection procedure•Level of the group and quality of its participation.

Time Schedule•Duration of the programme•Allocation of time to each module•Was the daily schedule hectic.•Sequencing the topics•General flow and momentum of the programme

Training Facilities•Suitability of the sitting arrangement •General ambience•Appropriateness and quality of the equipment used•General learning environment at the venue.

Physical Arrangements•Appropriateness of the accommodation•Arrangement for food etc.

Training Support Materials•Usefulness and quality of the materials•Their timely distribution.

Sources for Ascertaining Participants’ Reactions

•Structured Questionnaire•Tests•Open Forum•Personal interviews•Program committee meetings for ongoing evaluation of the contents and process of the programme.•Observations of the trainers•Comments of the participation

Purposes and Uses of Evaluation

The definition of evaluation implies 2 purposes:

1.Making decisions about improvements to be made in the training program itself.2.Making decisions about the value of the training program in terms of whether to continue to conduct the program.

Specifically, evaluation can be used:

a)To determine whether the training program is accomplishing its assigned objectives and if they were the “right” objectives.b)To Identify the strengths and weaknesses of training activities.c)To determine the cost/Benefit Ratio of the Training Program.d)To Establish a Data base which Organization Leaders can use to Demonstrate the Productivity and Efficiency of their operational Procedures.e)To Establish a Data Base which can assist Organization Managers in Making Decision.

Evaluation of Training

After the training program has been completed and the current training record filled in, the performance of trainees can be evaluated and the attention of the training staff drawn towards specific action for improvement. Thus evaluation of training should help to:•Isolate areas of difficulty and suggest strategies for overcoming them.•Modify unrealistic training targets in the light of actual group performance

• Determine whether motivation of trainees is of the required level e.g. financial incentives for greater output during training.

• Compare initial selection to actual performance rating

• Highlight causes of absenteeism and labour turnover during training

• Evaluate and modify instruction for future training programme

• Formalized management training programme have a vital role to play in the development of management skills.

18

“What is quality training?”

“How do you measure it?”

“How do you improve it?”

Evaluation can be four-pronged attempt.1.Evaluation of reaction 2.Evaluation learning3.Evaluation behavior 4.Evaluation results

21Jayadeva de Silva

Training Expectation MeasurementTrainee feedback scores on initial impression on to what extent training met their expectations of learning, skills and knowledge.

Tool: Survey Form

Training effectiveness MeasurementPost-training normalized feedback scores and its quarterly trends. Feedback from trainee Manager on visible incremental changes in trainee skills, service parameters and on-job behavior.

Tool: Quarterly Follow-on Survey from Managers

Student feedbac

k

Tier-1

Expectations

Post-Training On-Job Behavior

Survey

Tier-3

Effectiveness

Improvement in business indicators, revenue figures

or success parameters

Tier-4

Impact

Pre-Training Vs Post-Training

Assessment

Tier-2

Improvement

Training Improvement MeasurementAssessment of trainee’s training exposure and expertise gained on same set of tasks before and after training. Includes comparison of in-training & Post-training test performance, If applicable.

Tool: Pre-training and Post-training Survey Forms. Post-training performance tests.

Training Impact MeasurementImpact of training for improving revenues, enhancing business or other success factors driving the training needs.

Tool: Comparison of Baseline data and Quarterly business data, Impact Factor calculation sheets

Measuring Training Effectiveness and Impact

I.Prior to training•The number of people that say they need it during the needs assessment process.•The number of people that sign up for it.

II. At the end of training•The number of people that attend the session.•The number of people that paid and attended.•Customer satisfaction at the end of training•A measurable change in knowledge•Ability to solve a “mock” problem at the end of training• Willingness to try to use the skill at the end of training.

III. Delayed Impact (non-job)•Customer satisfaction at X weeks after the end of training•Customer satisfaction at X weeks after the training when customers know the actual costs of training•Retention of knowledge at X weeks after the end of training.•Ability to solve a “mock” problem at X weeks after the end of training.•Willingness to try the skill at X weeks after the training.•The three systems followed are: 1. 360 degree 2. Performance system 3. Janus performance

Management System

IV. On the job behavior change•Trained individuals that self-report that they changed their behavior , used the skill on the job after the training.•Trained individuals who’s managers report that they changed their behavior or used the skill after training.•Trained individuals that actually are observed to change their behavior or use the skill after the training.

V. On the job performance change•Trained individuals who self-report that their actual job performance changed as a result of their changed behaviour, skill.•Trained individuals who’s manager’s report that their actual job performance changed as a result of their changed behaviour/ skill.•Trained individuals whose manager’s report that their job performance changed either through improved performance appraisal scores or specific notations about the training on the performance appraisal forms.

• Trained individuals that have observable / measurable improvement in their actual job performance as a result of their changed behaviour.

• The performance of employees that are managed by individuals that went through the training.

• Departmental performance in departments with X % of employees that went through training ROI of return on training dollar spent.

(a) Other measures•CEO / top management knowledge of/ approval of/ or satisfaction with the training program.•Rank of training seminar in forced ranking by managers of what factors contributed most to productivity/ profitability improvement.•Number of referrals to the training by those who have previously attended the training•Additional number of people who were trained by those who have previously attended the training.•Popularity of the program compared to others.

(b) What to evaluateOne way to approach the issue of what to evaluate

is to identify what kind of information is needed. Dr. Kirtpatrick developed four levels of evaluation. Reaction

1. Reacting2. Learning3. Behaviour4. Results

29

Feedback score collected from each trainee at end of the class

Record general impression about the Training on a numerical scale of 1-5 or similar

Average score reflects how closely training met expectations of field and how close it is aligned with business needs identified earlier which drives the training.

Reaction may best be defined as how well the trainees liked a particular training program."

Reactions are typically measured at the end of training.

30

1. Design a questionnaire based on information obtained during the need assessment phase.

2. Design the instrument so that the response can be tabulated and quantified.

3. To obtain more honest opinions provide for the anonymity of the participants.

4. Provide space for opinions about items that are not covered in the questionnaire.

5. Pretest the questionnaire n a sample of participants to determine its completeness.

32

33

FEEDBACK FORMPlease take a few minutes to fill out this feedback form. Your feedback will help us strengthen the course delivery. Course Title -

Name of the facilitator -You may provide feedback of the program on the following criteria by putting a () mark in the space provided.4 – Exceeds Expectation; 3 – Meets Expectation; 2 – Needs Improvement; 1 – Unsatisfactory

44 33 22 11

Subject KnowledgeSubject Knowledge

Presentation StylePresentation Style

CommunicationCommunication

Examples, Cases, Simulation & ExercisesExamples, Cases, Simulation & Exercises

RelevanceRelevance

Supporting MaterialsSupporting Materials

Overall Evaluation Overall Evaluation

1. What did you like most in the workshop?2. What did you dislike in the workshop?3. What other changes would you suggest in case the same course is conducted in future? Learning & DevelopmentHuman Resources

Typically 'happy sheets' Feedback forms based on subjective

personal reaction to the training experience

Verbal reaction which can be noted and analyzed

Post-training surveys or questionnaires Subsequent verbal or written reports

given by delegates to managers back at their jobs

34

Collect Pre-training and post-training data on trainee’s skills before and after the training. A well drafted skill based survey can be used.

Compare improvements in exposure or expertise gained by the trainee as seen on post-training survey by comparing it with pre-training survey.

Conduct in-training skill tests as part of the qualification criteria to measure the improvement in the learned skills. 35

2. LearningThe trainer is concerned with measuring the learning of principles, facts, techniques and attitudes that were specified as training objectives. There are many different measures of learning performance including paper-and-pencil test, learning curves etc. The objectives determine the choice of the most appropriate measure. There are several guideposts used in establishing a procedure for measuring the amount of learning that takes place.

1.The learning of each participant should be measured so that quantitative results can be determined.2.A before-and-after approach should be use so that learning can be related to the program.3.Learning should be measured in an objective basis.4.A group not receiving training should be compared with the group receiving training.5.Where possible the evaluation results should be analyzed statistically. Where principles and facts are taught rather than skills, it is more difficult to evaluate learning

What principles, facts, and techniques were understood and absorbed by the participants?"

What the trainees know or can do can be measured during and at the end of training

38

39

40

WHAT?What knowledge was acquired? What skills were developed or enhanced? What attitudes were changed?

HOW?Tests before and after the training Interview or observation can be used before and

after training.Measurement and analysis is possible and easy

on a group scale Reliable, clear scoring and measurements need

to be established

41

Collect quarterly feedback from trainee’s manager on his observations on improvement of trainee’s on-job performance/ behavior or skills after attending training.

Analyze the trends in key on-job-performance parameters or indicators as seen on survey with respect to previous quarters.

Convert delta into normalized scores to indicate the value created by training in the work efficiency of the trainee

Ensure continual measurement of effectiveness after regular intervals to assess long term value of the training.

42

3. BehaviourThe term is used in reference to the measurement of job performance. There are several guideposts in evaluating training programs in terms of behaviour changes.1.A systematic appraisal should be made on-the-job performance on a before-and-after basis.2.The appraisal of performance should be made by one or more of the following groups: a. The person receiving the training. b. The persons supervisor. c. The persons subordinates. d. The persons peers.

3.A statistical analysis should be made to compare the performance before and after and to relate changes to the training program.4.The post training appraisal should be made three months or more after the training so that trainees have an opportunity to put into practice what they have learnt.5.A group not receiving the training should be used for comparison.

Changes in on-the-job behavior,Behavior changes are acquired in training and they then transfer (or don't transfer) to the work place.

What skills did the learner develop, that is, what new information is the learner using on the job?

45

When to evaluateHow often to evaluateHow to evaluate

47

What?Whether the trainee is able to transfer the

learning to the work environmentNew learning is demonstrated Whether the trainee is motivated

How?self-assessment can be useful, using

carefully designed criteria and measurements

cooperation and skill of observers, typically line-managers, are important factors, and difficult to control

Use of focus groups

48

Measure quarterly the business indicators of the trainee’s job or service parameters based on nature of job. Business indicators could be collected based on individual job or group responsible for the said function.

Ideally business indicator data before the training should be used as baseline.

Record the business parameters or governing service parameters on quarterly basis

Calculate qualitative or quantitative impact factor based on normalized delta. 49

4. ResultsEvaluations in this level are used to relate the results of the training program to organizational objectives. Some of the results that could be examined are costs, turnover, grievances and morale. Where the objectives of the training program are closely tied to specific organizational objectives effort should be made to show a link between the training and the changes called for in the organization objective.

Reduction of costs; Reduction of turnover and absenteeism; Reduction of grievances; increase in quality and quantity or

production; or Improved morale which, it is hoped, will

lead to some of the previously stated results.These factors are also measurable in the

workplace

51

52

Jayadeva de Silva 53

“The Four Levels represent a sequence of ways to evaluate (training) programs….As you move from one level to the next, the process becomes more difficult and time-consuming, but it also provides more valuable information.” (Kirkpatrick, 1994,)

Realize an opportunity to use the behavioral changes.

Make the decision to use the behavioral changes.

Decide whether or not to continue using the behavioral changes.

Desire to changeKnowledge of what to do and how to do it

Work in the right climateReward for (positive) change

The process of Evaluation---Measuring Change

One way of visualizing a process of evaluation is to set the four levels of evaluation against a backdrop of a training program. Evaluation should start before the training program is designed. The next step is to plan the evaluation, how is the data to be gathered. Who will perform the evaluation? What techniques will be used? When will the evaluation be conducted? The evaluation design needs to be used on the objectives of the program and the criteria to be measured. Once the plan is completed and the training program begins, the data collection phase is initiated.

Validity and Reliability

Validity is a concept that means the degree to which an evaluation technique or instrument measures what it was intended to measure. Its a measure of accuracy.

Reliability is the degree to which evaluation techniques and instruments measure a given characteristic consistently. In order to get a long term view of a training programs effectiveness, an instrument must measure reliably for each session of the training program.

Types of Evaluation Techniques and Instruments

The process of Training EvaluationA training programme, like design, development, and manufacture of a product passes through several stages.1. Pre-Training EvaluationThis is prior to the course and should cover an analyses of the expectations of the trainee and his superior officer. The existing level of knowledge and skills are reassessed. It is carried out through discussions, workshops etc.

2. Input and Delivery EvaluationThis is done concurrent to the training. It involves long term management programme. It is done individually or in group. Each topic, module is evaluated in terms of its content, presentation, relevance and applicability. Using a questionnaire, soon after the course, invariably on the last day the participants reaction is obtained which generally elicits information about the course inputs and the impressions about the course.

3.Post Training EvaluationThe objectives of training is to enhance individual effectiveness which helps in improving organizational performance. Thus the process of post-evaluation must have reference to job improvement plan. This cannot be done when the person is training , he needs to go back to his job and demonstrate his newly acquired knowledge and skills. After a lapse of 6-12 months the results obtained are used for evaluating the applicability of training.

Transfer of Learning

Individual GainsTo evaluate the effect of training on the individual after 6-12 months, the training persons stated the significant areas where the individual development had taken place. They were: 1. State of art knowledge on the subject2. Development of the analytical skills. 3. Refined interpersonal skills. 4. Innovation/ creativity. 5. Management of stress.

Transfer of learning to the organization

To evaluate whether in-house training has resulted in enhanced organization performance, trainees were requested to indicate the resultant benefits to public enterprises. The non-tangible benefits are: 1. Better management practices. 2. improved problem solving. 3. job improvement plans. 4. better systems.There is general evidence that systematic in-house training has contributed to individual growth in public enterprises, while the same level of organization improvement has not taken place because of organization climate, PE systems and Lack of motivation in PEs

Alternate approach in post-course evaluations

New approaches to post-course evaluation are being explored. One such recent development is monitoring the progress of the participant through post-course project work. In this approach participants at the time of nomination to the course in consultation with their departmental heads identify a proble, of practical value to the organization. During the program participants discuss, reformulate and identify approaches to its solution in the light of the knowledge and skill inputs in the training sessions. This way they related the learning to work situations and made innovating solutions.

Evaluation of in-house training institutes

In order to make training a continuous activity addressing itself to specific training needs of the organization, public enterprises have established large number of in-house training institutes/ Centers.There are 5 factors considered important for evaluating the institute and methods of assigning point rating to each.a) Faculty Resources: The number of qualified faculty members on full time basis at the centre, years of experience in training are considered.

b) Infrastructure: This covers all infrastructure facilities other than library, such as class rooms, syndicate rooms, hostel, recreation facilities etc.c) Library: This is the “brain” of any training institution/ Centre. Relevant books, journal, cases, audio, video etc.d) In-house Research / Consultancy: The in-house centers suffer from under exposure to research development and consultancy work. Such research/consultancy may be both internal or external.e) Financial Support: The annual budget allocation, unit cost per training are items under this factor.

67

Return on Impact (RoI) is a new approach which measures the difference training has created on those governing business/ service or revenue parameters which drives training needs.

increased output reduced absenteeism and

tardiness reduced cost of new hires reduced turnover increased number of

employee suggestions climate survey data (morale

and attitudes)

68

sales volume average sale size add-on sales close-to-call ratio ratio of new accounts

to old accounts number of items per

order

69

accuracy of orders size of orders number of transactions per

day adherence to credit

procedures number of lost customers amount of repeat business number of referrals number of complaints

70