EVALUATING TRAINING EFFECTIVENESS:AN INTEGRATED PERSPECTIVE IN MALAYSIA

UNIVERSITY OFSOUTH AUSTRALIA

12 JUL 2006

LIBRARY

Lim Guan ChongMaster of Business Administration (Finance)

International Graduate School of ManagementDivision of Business and Enterprise

University of South Australia

Submitted on this 5th of August in the year 2005 for the partial requirements of the degree ofDoctor of Business Administration

University of South Australia

DOCTOR OF BUSINESS ADMINISTRATION

PORTFOLIO SUBMISSION FORM

Name: Lim Guan Chong Student Id No: 0111487H

Dear Sir/Madam

To the best of my knowledge, the portfolio contains all of the candidate's own workcompleted under my supervision, and is worthy of examination.

I have approved for submission the portfolio that is being submitted forexamination.

Signed:

Dr Travis Kemp/Professor Dr Leo Ann Mean Date

Supported by:

Dr Ian WhyteChair: Doctoral Academic Review CommitteeInternational Graduate School of Business

ge)(2,/2$

Date

14

DBA Portfolio Declaration

I hereby declare that this paper submitted in partial fulfillment ofthe DBA degree is my own work and that all contributions from anyother persons or sources are properly and duly cited. I further declarethat it does not constitute any previous work whether published orotherwise. In making this declaration I understand and acknowledgeany breaches of the declaration constitute academic misconduct whichmay result in my expulsion from the program and/or exclusion from theaward of the degree.

Signature of candidate:

11

Lim Guan Chong Date:5th August 2005

TABLE OF CONTENTS

Portfolio Submission FormPortfolio DeclarationAcknowledgements

Overview 1

1 Research Paper 1 Methodological Issues 3

In Measuring Training Effectiveness1.1 Abstract 41.2 Introduction 4

1.3 Approaches to Training Evaluation 6

1.3.1 Discrepancy Evaluation Model 7

1.3.2 Transaction Model 10

1.3.3 Goal-Free Model 101.3.4 Systemic Evaluation 12

1.3.5 Quasi-Legal Approach 13

1.3.6 Art Criticism Model 13

1.3.7 Adversary Model 141.3.8 Contemporary Approaches Stufflebeam's 14

Improvement-Oriented Evaluation (CIPP)Model, 1971

1.3.9 Cervero's Continuing Education Evaluation, 1984 151.3.10 Kirkpatrick Model, 1959a, 1959b, 1960a, 1960b,

1976, 1979, 1994, 1996a, 1996b, 1998 16

1.4 Critical Review 221.5 Future Research 27

1.5.1 The Transfer Component 271.5.2 Evaluating Beyond the 4 Levels 281.5.3 Incorporating Competency-based Approach 29

into Training Evaluation1.5.4 Multi-Rater System in Training Evaluation 31

1.6 Conclusion 331.7 References for Paper One 34

2 Research Paper 2 Evaluating Training Effectiveness: 43An Empirical Study of Kirkpatrick Model ofEvaluation in the Malaysian Training Environment ForThe Manufacturing Sector

2.1 Abstract 442.2 Introduction 442.3 Training Practices in Malaysia 452.4 Practice of Evaluation in Training 462.5 Training Evaluation Practices In Malaysia 492.6 Methodology of Study 53

2.6.1 Questionnaire Construction 542.6.2 The Sample and Sampling 552.6.3 Questionnaire Responses 56

2.7 Findings and Discussion 562.8 Limitations of Study 632.9 Conclusion 64

2.10 References for Paper Two 652.11 Appendix A The Questionnaire for Research Paper Two 69

3 Research Paper 3 Multi-rater Feedback For Training 74And Development: An Integrated Perspective

3.1 Abstract 753.2 Introduction 753.3 The Use of Multi-rater Feedback 763.4 The Effectiveness Of Multi-rater Feedback For 79

Development3.5 The Effectiveness Of Multi-rater Feedback For 81

Appraisal3.6 The Variation Of Multi-rater Feedback Information 81

3.7 Multi-rater Feedback Practices In Malaysia 83

3.8 Integrating Multi-rater Feedback With Developmental 86Tool

3.9 Multi-rater Feedback: Process Consultation As A 87Developmental Tool

3.10 Micro Perspective Of Conversation Theory In Process 89Consultation

3.11 An Integrated Approach for Post Multi-rater Feedback 91Development

3.12 Conclusion 963.13 References for Paper Three 97

iv

Acknowledgements

I am sincerely grateful to my supervisors, Dr Travis Kemp and Professor Leo AnnMean, who have been so supportive, by taking their time to look through my papersand gave me tremendously useful feedback and suggestions.

First and foremost, thanks to my spouse, Linda Liew Mei Ling who acted as myresearch assistant and has put in her late nights and thoughtful moral supportthroughout this endeavor.

Special thanks are reserved for my friends who acted as my proof readers whonever let me produce less than the best I had to offer.

In particular, my sincerest thanks to my respondents, relatives, families and otherparties who have supported me along the way and helped me find the time tocomplete my thesis.

Finally, my utmost appreciation to University of South Australia, InternationalGraduate School of Management for their support and enthusiasm to achievingexcellence in education.

Lim Guan Chong

Overview

The majority of organizations realize that training must be a worthwhile effort; there must be

returns towards labour productivity after training. Evaluation is possibly the least developed

aspect of the training cycle. This research portfolio looks at the effectiveness of Kirkpatrick

Four-Levels of Evaluation with emphasis on the assessment of the methodology within the

training perspective.

Evaluating training is typically linked with measuring change and quantifying the degree of

change which leads to performance. Measuring gains in organization effectiveness that

resulted from training interventions is probably the most difficult task in training evaluation.

This research portfolio, as a partial fulfillment of the requirement of the degree of Doctor of

Business Administration, develops a series of ideas that expand on traditional approaches to

training evaluation. The research portfolio is divided into three papers.

Paper 1 critically reviews the methodological problems faced when adopting the evaluation

model developed by Donald L Kirkpatrick in 1959. A series of industrial research conducted

shows little application of this definite approach. The literature provides little understanding

about the transfer of the learning component when using Kirkpatrick model to determine

training effectiveness. Most current researchers find that future research on training

evaluation lies in the effectiveness of transfer of the skills learned. The objective of this

research portfolio through the anatomy of this classical theory is to effectively address the

weaknesses by re-focusing the issue of transfer of learning as a major key to unlock the

model's practicality and validity.

Paper 2 adopts a survey method to track the history, rationale, objectives, implementation and

evaluation of training initiatives in the Malaysian manufacturing sector. It utilizes the survey

research to triangulate reliable and convincing findings.

The research looks at the extensiveness of Kirkpatrick model as practised in the Malaysian

manufacturing sector. This paper reports the practice of Kirkpatrick's 4 levels of evaluation

and the effectiveness of this evaluation model within the Malaysian manufacturing sector.

1

Paper 3 is on the effective use of the multi-rater feedback system in providing multi-source

information and creating self awareness based on individual strength and weaknesses. One

underlying rationale to such system is their potential impact on the individual's self

awareness which is thought to enhance performance at the development stage.

This paper serves as a conceptual paper, which studies how multi-rater feedback could

effectively lead to a successful developmental process through process consultation in the

context of Malaysia training environment. Through the years, training evaluation culture in

Malaysia has not been properly developed. A comprehensive approach is necessary for

organizations to see the benefits of conducting pre training analysis. This should be followed

by an effective development plan so that a comprehensive training approach could be

instilled in the Malaysian environment.

The process consultant holds the key to effective development process by using a multi-rater

assessment as a pre-training gap analysis. Process consultation provides the opportunity to

'check and balance' the degree of learning and development activities through reflections,

problem solving capabilities and application of theory throughout the developmental process.

Good conversation was introduced as an intervention tool to complement double loop

learning during process consultation.

This portfolio systematically discusses the issue of training evaluation faced by the Malaysian

manufacturing sector. It is recommended that an integrated model approach comprising

preliminary and post assessment using multi-rater feedback, followed by a developmental

process using process consultation, complemented by good conversation as an intervention

tool, may serve as a rational balance between training financial outlays and development

outcome.

2

Research Paper I

METHODOLOGICAL ISSUES IN MEASURINGTHE TRAINING EFFECTIVENESS

Lim Guan ChongMaster of Business Administration (Finance)

University of Hull

International Graduate School of ManagementUniversity of South Australia

3

Methodological Issues in MeasuringTraining Effectiveness

Lim Guan ChongInternational Graduate School of Management

University of South Australia

1.1 Abstract

This literature review examines the effectiveness and the methodological issues related to

Kirkpatrick's four-level model of evaluation and its application to training. The paper first

measures the extent that the Kirkpatrick's evaluation model has been used by organizations to

measure learning outcomes, reactions towards development, transfer learning, change of

behavior and return of investment after training. Research was conducted to determine the

weaknesses of this model faced by most practitioners. An examination of this classical

theory was carried out to address the weaknesses of this model by re-focusing the issue of

transfer learning as a key to unlock the model's practicality and validity.

1.2 Introduction

Training evaluation is regarded as an important human resource development strategy.

However, there seems to be widespread agreement that systematic evaluation is the least well

carried out training activity. Chen and Rossi (1992) commented that evaluation knowledge

found in the literature has not been fully utilized in program evaluation. This reveals that

training evaluation has not been culturally embedded in most organizations. The first reason

could be that companies have no knowledge in conducting training evaluation. Secondly, the

available training evaluation models are not sufficient in providing a total approach for

effective training evaluation. This is further evidenced by a study on the benefits of training

in Britain, which revealed that 85 percent of British companies make no attempt to assess the

benefits gained from undertaking training (HMSO, 1989).

4

Since evaluation started in the area of education, most of the early definitions were in that

area. Tyler (1949) was the first researcher to define evaluation as a process of determining to

what extent the educational objectives are actually being realized by the curriculum and

instruction. The early researchers emphasized the need to look at attaining objectives as an

important process in determining the effectiveness of any programs. This was found in the

study by Steel (1970), who compared effectiveness of the program with its cost. Boyle and

Jahns (1970) defined evaluation as the determination of the extent to which the desired

objectives have been attained or the amount of movement that has been made in the desired

direction. Further study by Provus (1971) conceptualized the need to have a certain standard

of performance as an objective-based criterion to judge the success of the program. His

model made comparisons between this preset standard and what actually exists. Noe (2000)

defined evaluation by referring to training evaluation as the process of analyzing the

outcomes needed to determine if training was effective. However, Goldstein and Ford (2002),

were of the opinion that evaluation is a systematic collection of descriptive and judgemental

information necessary to make effective training decisions which are related to the selection,

adoption, value, and modification of various activities.

After many in-depth studies were conducted on training evaluation and the high cost-

effective expectation from training, the term evaluation has been given a broader perspective

in which it no longer focuses on achieving program objectives but mainly covers the

methodology element of evaluation (Brinkerhoff, 1988; Goldstein, 1986; Junaidah, 2001;

Shadish & Reichardt, 1987; Stufflebeam & Shinkfield, 1985). The basis of goal-based

process formed only part of the overall evaluation process, unlike in the past when

researchers used one preferred methodological principle to assess the degree to which

training had attained their goal. With the availability of a wider range of philosophical

principles and scientific methodologies, many social scientists emphasized scientific rigor in

their evaluation models, and this is reflected in their definition of the field (Junaidah, 2001).

The evaluation model of these social scientists involves primarily the application of scientific

methodologies to study the effectiveness of the programs. These evaluators emphasized the

importance of experimental designs (Goldstein & Ford, 2002), quantitative measures (Rossi

& Freeman 1993) and qualitative assessment (Wholey, Hatry & Newcomer, 1994).

Contemporary social scientists, Cascio (1989), Mathieu and Leonard (1987), Morrow, Jarrett

5

and Rupinski (1997), Tesoro (1998) even adopted utility analysis in evaluating the worthiness

and effectiveness of the programs.

In brief, the concept of evaluation consists of two distinct definitions; congruent and

contemporary definitions (Junaidah, 2001). The congruent definition is more concerned with

meeting the desired objectives. It is a process of collecting information, judging the worth or

value of the program and ensuring training objectives are met. The contemporary definition

of evaluation places emphasis on scientific investigation to facilitate decision-making.

Stufflebeam (1971) mentioned that evaluation is the process of delineating, obtaining and

providing useful information for judging decision alternatives. This can be seen from the

evolution of the early 70s models to the current contemporary evaluation models.

1.3 Approaches to Training Evaluation

Evaluation in its modern form has developed from attempts to improve the educational

process (Bramley, 1996). Evaluating the effectiveness of people became popular at about the

same time as scientific management, and school officials began to see the possibility of

applying these concepts to school improvement (Bramley, 1996). Tyler (1949) model is

generally considered an early prominent evaluation model which was planned to evaluate the

value of progressive high-school curricula with more conventional ones (Stufflebeam &

Shinkfield, 1985).

Tyler (1949) introduced the Basic Principles of Curriculum and Instruction, which is

organized around four main concerns:

What educational purposes should the organization seek to attain?

How to select learning experiences that are likely to be useful in achieving these

purposes?

How can the selected learning experiences be organized for effective instruction?

How can the effectiveness of these learning experiences be evaluated?

Tyler laid the foundation for an objective-based style of evaluation. Objectives were seen as

being critical because they were the source for planning, guiding the instruction and

6

preparing the test and measurement procedures. Tyler's objective-based evaluation model

concentrates on clearly stated objectives by changing the evaluation from appraisal of

students to appraisal of programs. He defined evaluation as assessing the degree of

attainment of the program objectives. Decisions made on any program had to be based on the

goal congruence between the objectives and the actual outcomes of the program (Stufflebeam

& Shinkfield, 1985).

1.3.1 Discrepancy Evaluation Model

The Discrepancy Evaluation Model, developed by Provus (1971) is used in situations where a

program is examined through its development stages with the understanding that each stage

(which Provus defines as design, installation, process, product and cost-benefit analysis) is

measured against a set of performance standards (objectives). The cost-benefit analysis

identifies the potential benefits of the training before it is carried out. The expected

behaviours which result from the training are agreed upon between the trainer and the

trainees. The analysis also establishes training objectives, which are defined as changes in

work behaviour and increased levels of organizational effectiveness (Bramley & Kitson,

1994). The program developers had certain performance standards in mind regarding how the

program should work and how to identify if it were working. The discrepancies that are

observed between the standards and the developed design are communicated back to the

relevant parties for review or further corrective action. A discrepancy evaluator's role is to

determine the gap between what is and what should be. This model helps the evaluators to

make decisions based on the difference between preset standards and what actually exists

(Boulmetis & Dutwin, 2000).

Provus's Discrepancy Evaluation Model can be considered an extension to Tyler's earlier

objective-based model where a set of performance standards must be derived to serve as the

objectives to which the evaluation of the program is based. Furthermore, the model may be

also viewed as having properties of both the formative and summative evaluation (Boulmetis

& Dutwin, 2000). The design stage comprises the needs analysis and program planning

stages; installation and process are parts of the implementation stage where formative

7

evaluation is done; and product and cost-benefit analysis stages comprises a summative

evaluation stage.

Formative evaluation focuses on the process criteria to provide further information to

understand the training system so that the intended objectives are achieved (Goldstein &

Ford, 2002). Brown, Werner, Johnson and Dunne (1999) note several potential benefits of

formative evaluation. The program could be assessed half way through to see whether it is

on track, effectively performed, and whether the activities are meeting the needs of the

training. The evaluator determines the extent to which the program is running as planned,

measures the program progress in attaining the stated goals, and provides recommendations

for improvement. The evaluation findings in these reports and the monitoring data could be

used to end a program in midstream (Goldstein & Ford, 2002). Unlike formative evaluation,

summative evaluation is fairly stable and does not allow adjustments during the program

cycle. Summative evaluation involves evaluating and determining whether the program has

experienced any unplanned effects. It helps organizational decision makers decide whether

to use the program again or improve it in some way. Campbell (1988) discriminates between

two types of summative evaluations; the first evaluation simply questions whether a

particular training program produces the expected outcome. The second evaluation compares

and investigates the benefits and viability of programmed instruction procedures. By

comparing the two evaluations, it was found that programmed instruction produces quicker

mastery of the subject, but the eventual level of learning retention is the same with either

technique (Campbell, 1988).

Provus Discrepancy Evaluation Model provides information for establishing measures of

training success by determining whether the actual content of the training material would

develop knowledge, skill and ability (KSA) and eventually lead to a successful job

performance. However, there are too many subjective issues that exist, especially on the

setting up of the performance criterion. The chosen criterion is based on the relevance of

three components: knowledge, skill and ability which are necessary to succeed in the training

and eventually on the job. Considering that modem approaches to assessing training

programs must be examined with a multitude of measures, including participant reactions,

learning, performance, and organizational objectives, it is necessary for training evaluators to

view the performance criteria as multidimensional (Goldstein & Ford, 2002). Training can

best be evaluated by examining many independent performance dimensions. However, the

8

relationship between measures of success should be closely scrutinized because the

inconsistencies that occur often provide important insights into training procedures

(Goldstein & Ford, 2002). Decisions and feedback processes depend on the availability of all

sources of information. There are many different dimensions in which the performance

criteria can vary. Issues like relevance and reliability of the criterion are important to consider

should one wish to adopt this discrepancy evaluation model. There are several considerations

in the evaluation of the performance criteria. These include acceptability to the organization,

networks and coalition that can be built between trainees and realistic measures (Goldstein &

Ford, 2002).

Responsive approaches used in the goal-free model are better evaluative approaches as there

is considerable variation in what the objectives of a program are thought to be. Responsive

approaches are a form of action research which involves the stakeholders in the data

collection process (Bramley, 1996). The intention is not to attribute causality, but to gain a

sense of the value of program from different perspectives. The term "responsive evaluation"

was first used by Stake (1977) to describe a strategy in which the evaluator is less concerned

with the objectives of the program than its effect in relation to the concerns of interested

parties, namely the stakeholders.

The responsive approach involves protracted negotiations with a wide range of stakeholders

in constructing the report. It is thus more likely to reflect their reality and be useful for them.

However, the underlying philosophy of responsive evaluation is different from the goal-based

approach. Evaluators are seen as subjective partners and the evaluation is based upon a joint-

collaborative effort which results in findings being constructed rather than revealed by the

investigation. Truth is a matter of consensus among informed parties. Facts have no meaning

except within some value framework. Phenomena can only be understood in the context in

which they are studied, generalization is not possible.

The suggested method intends to achieve progressive focus by giving more attention to

emerging issues rather than seeking the truth. Legge (1984) introduced a model similar to

goal free evaluation which evaluates planned organizational change. The evaluation is a joint,

collaborative process, which results in something more constructed than revealed by the

investigation. Legge (1984) suggests that instead of attempting evaluation as a thoroughly

9

monitored research, a contingency approach should be adopted. The contingency approach

is used to decide which approach is more appropriate or best matches the functional

requirements of the evaluation exercise. Campbell (1988) revealed that internal validity of

the scientific approach may not be so crucial. To increase internal validity, the legitimate

stakeholders should agree on the evaluation approach. The highlight on internal validity in

the scientific approach will frequently imply controlling key aspect of the context and many

organizational variables. This may lead to rather simplified information which clients find

difficult to use because it does not reflect their perception of organizational reality. Due to

this strong bipolarity between practitioners and academics, not many responsive evaluations

have been described in the training literature (Bramley, 1996).

1.3.2 Transaction Model

The Transaction Model developed by Stake (1977) affords a concentration of activity among

the evaluator, participants and the project staff (Madaus, Scriven & Stufflebeam, 1986). This

model combines monitoring with process evaluation through regular feedback sessions

between evaluator and staff. The evaluator uses a variety of observational and interview

techniques to obtain information and the findings will be shared with all the relevant parties

to improve the overall program. The evaluator participates and provides project activities.

Besides trying to obtain objectivity, the evaluators use subjectivity in the transaction model.

This model may have a goal-free or a goal-based orientation. Findings are shared with the

staff of all the projects in order to improve both individual and overall projects (Boulmetis &

Dutwin, 2000).

1.3.3 Goal-Free Model

Unlike early models, the goal-free model developed by Michael Scriven is a model that

involved methodological studies and processes (Popham, 1974). The evaluation model

examines how the program is performing and how the program could address the needs of the

client population. Program goals are not the criteria on which evaluation is based. However,

it is a data gathering process which studies actual happenings and evaluates the effectiveness

10

of the program meeting the client's needs. The evaluator has no preconceived notions

regarding the outcome of the program (as opposed to the goal-based model). Categories of

evaluation naturally emerge from the evaluator's actual observation. Once the data have been

collected, the evaluator attempts to draw conclusions about the impact of the program in

addressing the needs of the stakeholders.

However, this model has its weakness in terms of its subjective measures. There are some

preconceived notions that the evaluator must be an expert in his respective field and some say

no expertise is better (Rossi & Freeman, 1993). Some researchers said that an evaluator who

is not familiar with the nuances, ideologies and standards of a particular professional area

will presumably not be biased when observing and collecting data on the activities of a

program. They maintain, for example, that a person who is evaluating a program to train

dental assistants should not be a person trained in the dental profession. But other researchers

allege that a person who is not aware of the nuances, ideologies and standards of the dental

profession may miss a good deal of what is important to the evaluation. Both sides agree that

an evaluator must attempt to be an unbiased observer and be adept at observation and capable

of using multiple data collection methods (Wholey, Hatry 8z Newcomer, 1994). This is a

topic of debate among many experts. Scriven suggested using two goal-free evaluators, each

working independently to address the preconceived issues and reduce the possible biasness in

evaluation (Scriven, 1991).

A study by O'Leary (1972) illustrates the importance of considering other dimensions of the

criteria. She used a program of role-playing and group problem-solving sessions with hard-

core unemployed women. At the conclusion of the program, the trainees had developed

positive changes in attitude toward themselves. However, it also turned out that these

changes did not reflect the lack of positive attitudes toward their tedious and structured jobs.

These trainees apparently raised their levels of aspiration and subsequently sought

employment in a working setting consistent with their newly found expectations. It was

obvious that the trainees were leaving the job as well as experiencing positive changes in

attitude. However, there are many other cases in which the collection of a variety of criteria

related to the objectives is the only way to effectively evaluate the training program

(Goldstein & Ford, 2002). This has caused goal-based evaluation lost ground during the last

20 years because of the growing conviction that evaluation is actually a political process and

11

that the various values held in the society are not represented by an evaluative process which

implies that a high degree of consensus is possible (Bramley, 1996).

Further studies by Parlette and Hamilton (1977) rejected the classical evaluation system,

which focuses on objective reality, assumed to be equally relevant to all stakeholders in

acknowledging the diversity posed by different interest groups. They suggested the

"illuminative evaluation", with description and interpretation rather than with measurement

and prediction.

1.3.4 Systemic Evaluation

Systemic evaluation analyses the effectiveness of the whole system and enhances the

interfaces between the sub-systems in such a way as to increase the effectiveness of the

system. That is what the "system approach" sets out to do (Rossi & Freeman 1993). The most

comprehensive purpose of systemic evaluation is to find out to what extent training has

contributed to the business plans of various parts of the organization and consider whether the

projected benefits obtained outweigh the likely cost of training.

The main questions, which this strategy sets out to answer, are (Bramley, 1996):

Is the program reaching the target population?

Is it effective?

How much does it cost?

Is it cost effective?

These questions are used to derive facts about the evaluation by defining the size of the target

population and working out the proportion that have attended the training and not opinions of

whether useful learning has taken place. Effectiveness is difficult to measure as the word may

imply different meanings to different people. However, the model seems to measure quantity

rather than the quality of what is being done.

In the system analysis model, the evaluator looks at the program in a systematic manner,

studying the input, throughput and output (Rivlin, 1971).

12

Input are elements that come into the system (i.e. clients, staff, facilities and resources).

Throughput consists of things that occur as the program operates, for example, activities,

client performance, staff performance, and adequacy of resources such as money, people and

space. Output is the result of program-staff effectiveness, adequacy of activities etc. The

evaluator mainly examines the program efficiency in light of these categories.

1.3.5 Quasi-Legal Approach

Quasi-legal evaluation operates in a court of inquiry manner. Witnesses are called to testify

and tender evidence. Great care and attention is taken to hear a wide range of evidence

(opinions, values and beliefs) collected from the program. This approach is basically used to

evaluate social programs rather than formally evaluate training or development activities.

Quasi-legal evaluation was reported flawed by Porter and McKibbin (1988) in the area of

management education in the USA. The substantial information received from stakeholders

was analysed by a small group of professors from a business school. The students were

basically satisfied with the qualification which they have obtained and found course

worthwhile and useful. However, the researchers criticized that young graduates who attend

MBA courses have never worked in an organization and thus do not understand the sort of

issues, which should be the basic discussion material of MBA courses. A similar problem

arose with Constable and McCormick's (1987) report on the demand for and supply of

management education and training in the UK. The researchers found that judgement by

insufficiently impartial judges in the quasi-legal approach may be irrelevant, biased or

inconclusive (Bramley, 1996).

1.3.6 Art Criticism Model

In the Art Criticism Model developed by Eisner (1997), the evaluator is a qualified expert in

the nuances of the program and becomes the expert judge of the program's operation. The

success of this model depends heavily upon the evaluator's judgment. The intended outcome

may come in the form of critical reflection and/or improved standard. This model could be

13

used when a program wishes to conduct a critical review of its operation prior to applying for

funding or accreditation.

1.3.7 Adversary Model

In Owen's Adversary Model, the evaluator facilitates a jury that hears evidence from

individuals on particular program aspects (Madaus, Scriven & Stufflebeam, 1986). The jury

uses multiple criteria to "judge" evidence and make decisions on what have happened. This

model can be used when there are different views of what is actually happening in a program

such as arguments for and against program components.

1.3.8 Contemporary Approaches - Stufflebeam's Improvement-Oriented

Evaluation (CIPP) Model, 1971

Stufflebeam considers the most important purpose of evaluation is not to prove but to

improve (Stufflebeam & Shinkfields, 1985). The four basic types of evaluation in this model

are context (C), input (I), process (P) and product (P).

Context evaluation defines relevant environment and identifies training needs and

opportunities of specific problems. Input evaluation provides information to determine usage

of resources in the most efficient way to meet program objectives. The results of input

evaluation are often seen as policies, budgets, schedules, proposals and procedures. Process

evaluation provides feedback to individuals responsible for implementation. It is

accomplished through providing information for preplanned decisions during implementation

and describing what actually occurs. This includes reaction sheets, rating scales and content

analysis. Ultimately, product evaluation measures and interprets the attainment of program

goals. Contemporary approaches could take place both during and after the program with the

aim to improve program evaluation by expanding the scope of evaluation through its four

basic types of evaluation (Madaus, Scriven & Stufflebeam, 1986).

The CIPP model was conceptualized as a result of attempts to evaluate projects that had been

funded through the Elementary and Secondary Act of 1956 (Stufflebeam, 1983). To conduct

CIPP model evaluation, the evaluator needs to design preliminary plans and deal with a wide

14

range of choices pertaining to evaluation. This requires collaboration between clients and

evaluators as a primary source for identifying the interest of the various stakeholders.

1.3.9 Cervero's Continuing Education Evaluation, 1984

In Cervero's book titled "Effective continuing education for professionals" he suggested

seven categories of evaluation questions organized around seven criteria to determine

whether the programs were worthwhile (Cervero, 1988). The seven criteria are (a) program

design and implementation, (b) learner participation, (c) learner satisfaction, (d) learner

knowledge skills and attitudes, (e) application of learning after the program, (t) impact of

application of learning and (g) program characteristic associated with outcomes.

Program design and implementation is concerned with what was planned, what was actually

implemented and the congruence between the two. Factors such as the activities of learners

and instructors and the adequacy of the physical environment for facilitating learning are

common questions which are asked in this category.

Learner participation has both quantitative and qualitative dimensions. The quantitative

dimension deals with evaluative questions that are most commonly asked in any formal

program. The data is not used to infer answers in the other categories. Qualitative data is

collected in an anecdotal fashion by unobtrusively observing the proceedings of the

educational activities.

Learner satisfaction is concerned with the participants' reaction and is collected according to

various dimensions, such as content, educational process, instructor's performance, physical

environment and cost.

Learner knowledge, skills and attitudes focus on changes in the learner's cognitive,

psychomotor and affective goals. Normally, the evaluator will adopt a pen and paper test to

judge the effectiveness of these categories.

Application of learning addresses the degree of skill transfer to the actual work place. The

impact of application of learning focuses on the second-order effects, which means the

transfer and impact on the public (Cervero, 1988).

15

Program characteristics are associated with the outcome of the program. There are two kinds

of evaluative questions: the implementation questions and the outcome questions.

Implementation questions are useful for determining what happened before and during the

program. Outcome questions are useful for determining what occurred as a result of the

program.

The seven categories in this model are not viewed as a hierarchy (Junaidah, 2001). Cervero's

ideas have several antecedents in the evaluation literature. His framework was influenced by

Kirkpatrick's (1959) and Tyler's (1949) models. It is considered to be a comprehensive

model as it covers all the stages involved in starting from the program design stage to the

outcome stage. However, this model evaluation may be viewed as being too tedious to

implement due to its complexity. The author is too immersed in getting facts of the entire

process and ignores the efficiency of the whole evaluation process. This makes the model

more summative than formative in nature.

1.3.10 The Kirkpatrick Model, 1959a, 1959b, 1960a, 1960b, 1976, 1979,

1994, 1996a, 1996b, 1998

One of the most widely used model for classifying the levels of evaluation, used by Barclays

Bank PLC, Reeves in 1996 and others, was developed by Kirkpatrick. His model looks at

four levels of evaluation, from the basic reaction of the participants to the training and its

impact to the organizational. The intermediary levels examine what people learned from the

training and whether learning has affected their behaviour on the job. Level one (Level 1)

concerns itself with the most immediate reaction of participants and is easily measured by

simple questionnaires after the training. Level two (Level 2) is harder to measure and is

concerned with measuring what people understood and how they were able to demonstrate

their learning in the work environment. Level two (Level 2) can be measured by pen and

paper tests or through job simulations. Level three (Level 3) looks at the changes in people's

behaviour towards the job. For example, after a writing skills course, did the individual make

fewer grammatical and spelling errors and were their memos easier to understand? Level

four (Level 4) measures the "result" gained from the training. It focuses on the impact of the

training on the organization rather than the individual.

16

Kirkpatrick (1959) developed this coherent evaluation model by producing what was thought

to be a hierarchy system of evaluations which indicates effectiveness through:-

Level 1 (Reaction)

Level 2 (Learning)

Level 3 (Behaviour)

Level 4 (Results)

Kirkpatrick's (1994) Training Evaluation Model

Reaction How did the participants react to the training?

Learning What information and skills were gained?

Behaviour How have participants transferred knowledge and skills to their jobs?

Results What effect has training had on the organization and the achievement

of its objectives? (Timely and quality performance appraisals are

corporate goal)

Kirkpatrick was the first researcher to develop a coherent evaluation strategy by producing

what was thought to be a hierarchy of evaluations, which would indicate benefit (Plant &

Ryan, 1994).

Level 1: Reaction Evaluation

Kirkpatrick proposed the use of a post course evaluation form to quantify the reactions of

trainees. Evaluation at this level is associated with the terms "happiness sheet" or "smile

sheet" because reaction information is usually obtained through a participatory

questionnaire administered near or at the end of a training program (Smith, 1990).

Studies on evaluation mechanisms have shown that such evaluation sheets are not held in

high esteem, despite their general use by trainers of many organizations and in institutions

of higher learning (Bramley 1996; Clegg, 1987; Love, 1991; Rae, 1986;). Clegg (1987)

found that training evaluation was conducted for 75 percent of training programs done in

17

organizations. A study by Dawson (1993) found that Level 1 evaluation sheets were

ubiquitous.

Level 2 Learning Evaluation

The learning level is concerned with measuring the learning principles, facts, techniques

and skills presented in a program (Kirkpatrick, 1994). Tyler (2002) found that 32

percent of companies in America have carried out post-training evaluation on Level 2.

Another research conducted by Mathews, Ueno, Kekale, Repka, Pereira and Silva (2001)

on 450 companies in UK, Portugal and Finland which focused on training quality and

training evaluation showed that 40 percent of UK companies, 31 percent of Finland

companies and 51 percent of Portugal companies conduct formal assessment on learning

of the principles, facts, skills and attitudes which were specified as training objectives.

This level evaluates the knowledge, skills development and attitudinal changes that have

taken place. Examination of both knowledge and attitudinal outcomes is important to

increase coverage of training impacts because the pattern of change can vary between the

pre-test and post-test (Basadur, Graen & Scandura, 1986; Kraiger, Ford & Salas, 1993).

Researchers either assessed change before and after a program (Basadur et al., 1986;

Bretz & Thompsett, 1992), or they look merely at the post-training attainment score

(Davis & Mount, 1984; Warr & Bunce, 1995). Measures of learning should be objective,

with quantifiable indicators of how new requirements are understood and absorbed. This

data is used to confirm that participant learning has occurred as a result of the training

initiative (Phillips & Stone, 2002).

Level 3 Behavioural Evaluation

Job performance after training is referred to as behavioural by Kirkpatrick (1959, 1976)

and transfer by Alliger, Tannenbaum, Bennett, Traver and Shotland (1997). Level 3

evaluates the extent to which the "transfer" of knowledge, skills and attitudes has

18

occurred. Tyler (2002) reported that only 9 percent of America industries have carried

out post training evaluation at this level. The focal point is on performance at work after

a program. It is essential to record before and after performance but sometimes self-

report are obtained if information are unavailable to an evaluator (Wexley & Baldwin,

1986). It determines the extent of change in behaviour that has taken place and how this

behaviour would be transferred to the workplace. It further encourages one to take into

account the possible factors in the job environment that could prevent the application of

the newly learned knowledge and skills since a positive climate is important for

transferring.

Level 4 Results Evaluation

The evaluation of a particular training program becomes more complex as one progress

through every level of Kirkpatrick model. Results can be defined as the final results that

occurred because the participants attended the training program. This includes increased

production, improved quality, increased sales and productivity, higher profits and return

on investment. Level 4 evaluation observes changes in the performance criteria (i.e. key

results area) of organizational effectiveness. This level anticipates the gains the

organization can expect from a training event. This level of evaluation is made more

difficult as organization often demand that the explanation be given in financial terms

with measurable quantifiers (Redshaw, 2001).

For the past 30 years since Kirkpatrick's first idea was published in 1959, much debate had

been recorded on this model. Despite criticism, Kirkpatrick model is still the most generally

accepted by academics (Blanchard & Thacker, 1999; Dionne, 1996; Kirkpatrick, 1996a;

1996b; 1998; Phillips, 1991). However, research conducted in the United States has

suggested that US organizations generally have not adopted all of Kirkpatrick's 4-level

evaluation (Geber, 1995; Holton, 1996). This is especially true for the last two, more

difficult, levels of Kirkpatrick's hierarchy (Geber, 1995). In a survey of training in the USA,

Geber (1995) reported that for companies with 100 or more employees, only 62 percent

assessed behavioural change. Geber's (1995) results also indicated that only 47 percent of

US companies assess the impact of training on organizational outcomes. This poses a good

19

research question about the model's methodology and it forms the basis for epistemological

studies around the methodology.

Kirkpatrick's work has received a great deal of attention within the field of training

evaluation (Alliger & Janek, 1989; Blanchard & Thacker, 1999; Campion & Campion, 1987;

Connolly, 1988; Dionne, 1996; Geber, 1995; Hamblin, 1974; Holton, 1996; Kirkpatrick,

1959; 1960; 1976; 1979; 1994; 1996a; Newstrom, 1978; Phillips, 1991). His concept calls

for four levels of evaluation namely reaction, learning, behaviour and results. His four levels

of training effectiveness stimulated a number of supportive and conflicting models of varying

levels of sophistication (Alliger & Janek, 1989; Campion & Campion, 1987). There are

models and methods that incorporate financial analyses of training impact (Swanson &

Holton, 1999). However, Warr, Allan and Birdi (1999) conducted a longitudinal study of the

first three levels of training evaluation. The study correlated the following: relationships

between evaluation levels, individual and organizational predictors of each level and the

differential predictions of attainment vs change score. The study showed that immediate and

delayed learning were predicted by the trainee's motivation, confidence and use of learning

strategies. The researchers highlighted that it is preferable to measure training outcomes in

terms of change from pre-test to post-test, rather than merely through attainment (post-test)

scores (Warr, Allan & Birdi, 1999).

A review of the most popular procedures used by US companies to evaluate their training

programs showed that over half (52 percent) use assessments about participants' satisfaction

with the training. 17 percent assessed application of the trained skills to the job and 13

percent evaluated changes in organizational performance following the training. 5 percent

tested for skill acquisition immediately after training while 13 percent of American

companies carried out no systematic evaluation of their training programs (Mann &

Robertson, 1996). Many of these procedures reflect Kirkpatrick's four levels of reactions,

learning, behaviour and results of which will be further discussed.

More than 50 evaluation models available use the framework of Kirkpatrick model (Phillips,

1991). Currently, majority of the employee training is evaluated at Level 1. Evaluation at

Level 1 is associated with the terms smile sheet or happiness sheet, because reaction

information is usually obtained through a participatory questionnaire administered near the

end or at the end of a training program (Smith, 1990). The specific indication of the smile

20

sheet or happiness sheet is enjoyment of the training, perceptions of its usefulness and its

perceived difficulty (Warr & Bunce, 1995).

Phillips and Stone (2002) enhanced the popularity of the Kirkpatrick model by inserting the

fifth level into the existing 4-level model, though he further argued the inadequacy of this

model in capturing the return on investment aspect of the training outcome. Phillips and

Stone's (2002) 5-level evaluation model was seen as an extension of Kirkpatrick's 4-level

evaluation model as different companies have their own definition of pay offs to measure the

training results. Return on investment compares the training's monetary benefits with the

cost of the training, so that the true value of the training to the organization can be assessed.

Converting data to monetary values is the first phase in putting training initiatives on the

same level as other investments that organizations make (Phillips, 2002). It cannot be used to

cover other variables that may affect the results (i.e. culture, productivity, etc). Kirkpatrick

(1994) refuted this idea by claiming that there are many ways to measure training results.

This raises the question whether training evaluation be varied only as a measure of financial

benefits? Lewis and Thornhill (1994) are of the opinion that there should be 5 levels of

evaluation measuring the training effects on the department (i.e. Level 4) and its effects on

the whole organization (i.e. Level 5). Lewis and Thornhill (1994) emphasized the need to

look at the value and the organization cultures as the variables to measure training

effectiveness.

In recent times others have tried to make the system easier to deal with. Warr et al. (1999)

came up with the context, input, reaction and outcome (CIRO) evaluation system with the

context part going someway towards front-loading the evaluation and partly towards

mirroring Kirkpatrick model. Dyer (1994) proposed an evaluation system that suits all

organizations, irrespective of size or diversity of operation. It is a system that is relatively

easy to come to terms with and can be implemented at all the hierarchical stages of an

organization. It fits the individual and it fits the whole organization. The system puts

Kirkpatrick's evaluation system against a mirror. The benefits of using Kirkpatrick's Mirror

should be self-evident to anyone involved in management. Application of the paradigm

allows the individual to become more business focused, and if adopted universally should

provide efficient and effective training throughout any organization (Dyer, 1994).

21

A different model was used in a study by Shireman (1991) on the evaluation of a hospital

based health education program. The study adopted the CIPP model in examining the type of

evaluation which was being conducted in the hospital. A structured questionnaire was sent to

a stratified random sample of 160 hospitals of four different sizes in four mid-western states.

The result showed that 48 percent of the respondents reported that product evaluations were

usually done and less than 25 percent reported that other types (i.e. context, input, process) of

evaluations were done. The product evaluation is outcome-based and quite similar to

Kirkpatrick's end process evaluation. Both types of evaluations require appropriate data

collection activities.

Kirkpatrick model was used by most researchers as an initial framework of evaluation model

generation. This paper addresses the methodological issues surrounding the taxonomy of

Kirkpatrick model as an area for epistemological study. The theoretical an empirical

literature of Kirkpatrick model will be critically evaluated and further research opportunities

will be outlined.

1.4 Critical Review

Phillips (1991) concluded that out of more than 50 evaluation models available, the

evaluation framework that most training practitioners used is the Kirkpatrick model. Though

the model seemed to be weathered well, it has also limited our thinking on training evaluation

and possibly hindered our ability to conduct meaningful training evaluation (Bernthal, 1995).

More than ever, training evaluation must demonstrate improved performance and financial

results. But in reality, according to Garavaglia (1993), training evaluation often assessed

whether the immediate objectives have been met; specifically, how many items were

answered correctly on the post-test. Some based their evaluation only on trainee reaction; the

first level of Kirkpatrick model developed in 1959 (Brinkerhoff, 1988). Such information

gave organization no basis for making strategic business decisions (Davidove & Schroeder,

1992). Most practitioners are familiar with Kirkpatrick's 4-level evaluation model but many

never seemed to get beyond Levels 1 and 2 (Regalbutto, 1992). Numerous organizations have

adopted the model presented by Kirkpatrick to suit their own situations; the solution seems to

cause the growth of generic models (Dyer, 1994).

22

Kirkpatrick called for a definite approach to the evaluation model. All 4 levels must be

measured to ensure effectiveness of the whole evaluation system since each level provides

different kinds of evidence.

This view was supported by Hamblin (1974), who suggested that reaction leads to learning

and learning leads to change in behaviour, which subsequently leads to changes in the

organization. He further stated that each level can be broken at any link and having positive

reaction is necessary to create positive learning. According to Bramley and Kitson (1994),

there is not much evidence to support this linkage. Further research carried out by Alliger

and Janek (1989) found only 12 articles which attempted to correlate the various levels

advocated by Kirkpatrick. Although there are problems in external validity with such a small

data, the tentative conclusion was that there was no relationship between reaction and the

other three levels of evaluation criteria. A correlation study, which was run on these four

levels of evaluation showed insignificant results. A literature search based on Kirkpatrick's

name, yielded 55 articles but only 8 described evaluation results and none described

correlations between levels (Toplis, 1993). This concluded that good reactions did not

predict learning, behaviour or results.

A series of industrial surveys conducted in the last 30 years show little application of all 4

levels of Kirkpatrick model. Surveys conducted since 1970 showed that most industrial

trainers rely on student reaction, fewer on test learning and almost none on test application

and benefit (Brandenburg, 1982; Plant & Ryan 1994; Raphael & Wagner, 1972). In the last

20 years, a number of writers claimed to have performed a full Kirkpatrick evaluation;

however, the linkages described in connecting the training event with the outcome are

subjective and tenuous (Salinger & Deming, 1982; Sauter 1980).

A survey conducted by the Bureau of National Affairs and American Society of Training and

Development (ASTD) in 1969 using questionnaires indicated that most of the companies

conducted Level 1 evaluation and unsystematic approaches to Level 2 evaluation (Raphael &

Wagner, 1972). The survey indicated that problems of evaluation at higher levels were

mainly due to a lack of understanding of the approach used. Kirkpatrick model seems to

offer a one-size fits all solution to measure training effectiveness. However, there has been

little contribution and reliability of this model despite great industrial emphasis in this area.

23

Kirkpatrick model focuses mainly on immediate outcome rather than the process leading to

the results. The following questions were never successfully addressed. In fact the

improvement of these processes is the main forces of effectiveness (Murk, Barrett &

Atchade, 2000).

How well a person's motivation level affects the learning behaviour

The degree of superiors' support after the training

The extent to which training interventions was appropriate for meeting needs

Longer-term effects of the training, the pay-off in determining a course's overall

impact and cost-effectiveness

The conduciveness of the training environment

An empirical study by Warr, Allan and Birdi (1999) showed that external processes like

increasing confidence and motivation levels of trainees as well as use of certain learning

strategies are important contributing factors towards training effectiveness. A 2-day training

course was studied on 23 occasions over a 7-month period in the Institute of Work

Psychology, UK. Technicians who attended the training courses which involved operating

electronic tools were asked to complete a knowledge test questionnaire on arrival and at the

end of the course. A follow up questionnaire was mailed to the trainees one month later.

More than 70 percent of the respondents returned the questionnaire. The questionnaire was

designed to capture what the researches defined as third factors (i.e. confidence, perception,

motivation, learning strategies, age, etc). The results showed a non-significant correlation

between reactions towards the course and job behaviour. Perceptions of course difficulty

were significantly negatively associated with frequency of use of equipment. Correlation

between levels two and three evaluation were small. Learning scores and changes in those

score - Level 2 were strongly predicted by trainee's specific reactions to the course, but those

reactions were not significantly associated with later job behaviour - Level 3 (Warr, Allan &

Birdi, 1999).

Alliger et al. (1989) carried out a meta-analysis of studies where reaction measures had been

related to measures of learning (11 studies) and changes in behaviour (9 studies). They found

that positive reactions did not predict learning gains better than negative ones (the average

24

correlation between reactions and amount of learning was .02 nor were they any better at

predicting changes in behaviour after the program was .07).

Bramley and Kitson (1994) asserted that measuring learning is problematic because designing

a reliable measuring instrument is difficult and the necessary skills are often not available.

Grove and Ostroff (1990) pointed out that training directors often do not possess the essential

skills to conduct training evaluation. This could be part of the reason why companies are

reluctant to evaluate their training effectiveness.

Though Kirkpatrick's traditional assessment methods were widely used on Level 1 and 2

evaluations, the benefits of collecting data at each level are unclear. This uncertainty may

result in organization failing to evaluate training completely or selecting forms of evaluation

that may not be reliable. Inadequacy in Kirkpatrick model on each level forces one to look

for other possible measures. Therefore, one may argue that to make Kirkpatrick model

definite, a more detailed assessment method must be conducted at each level to ensure

practicality, validity and applicability (Mann & Robertson, 1996).

Mann and Robertson (1996) undertook to investigate the utility of various methods used in

evaluating training programs. Twenty-nine subjects were selected from a three-day training

seminar for the European National Run in Geneva, Switzerland. The seminar was a computer

training event (on e-mail and the Internet) for youth workers, and trainees were asked to

complete training evaluation forms before and after the training program and by post one

month later. Sixteen people returned this final questionnaire. Each questionnaire contained

three sets of questions designed to measure knowledge, attitudes and self-efficacy. The

results showed doubt over the value of the data received from reaction and learning levels.

Recommendations were made based on the following findings:-

Measuring learning (Level 2) as a method of evaluating training effectiveness is

important. The study showed that not all of what is learned immediately after training

is retained one month later. This denotes that the practitioner should be aware of the

short-term training effectiveness.

25

To ensure a more realistic evaluation at Level 2, one must be prudent of the pre and

post course evaluation method proposed by Kirkpatrick. The time frame for learning

to take place was never specified. An appropriate measuring model is necessary to

determine the extent of learning has taken place. In another words, Kirkpatrick model

lacks longitudinal considerations.

Measuring changes in learning through data collection as prescribed by Kirkpatrick

(absolute term) gained no value in predicting how well a person can perform the skills

attained from the training after a one-month period.

A positive attitude does not show any relevance on how well a person can perform a

trained task after a month. Reaction evaluation that shows positive attitude attained

have no direct linkage to performance.

However, individual self-efficacy did not decrease over time. Empirical studies

shown that self-efficacy correlates with actual performance (Kraiger, Ford & Salas

1993). One might look at the possibility of measuring self- efficacy instead of

reaction evaluation. In another words, self-efficacy offers more tangible results as

compared to reaction evaluation.

The reasons for Kirkpatrick failure in Level 3 and Level 4 evaluation was due to lack of a

defined framework and specific tools that are appropriate for measuring transfer of learning

since its first introduction 40 years ago. It is necessary, at the most basic level, to have a body

of case studies from which the generalizations can be drawn and thus hypotheses formed.

However, this body of information has not been published (Bramley & Kitson, 1994).

The issue here is whether or not the knowledge taught during training is being transferred or

demonstrated by the trainees on the job. The transfer component of training evaluation was

examined by Olsen (1998) in a study conducted in 1996. Transfer is evidence of whether

what has been learned is actually being used on the job for which it was intended.

The survey asked questions regarding how Kirkpatrick's 4-level evaluation were performed,

what percentage of payroll was spent on training, how much training was actually transferred

26

to the job and what specific items would enhance the level of transfer. A content analysis

was carried out on the 138 survey comments received on how the respondents made estimates

of the percentage of transfer value they reported. Follow up interviews were also undertaken

to provide additional clarification on responses and record impressions and opinions about

the data collection. The results showed that the percentage of transfer depended on the types

of training. Technical training showed the best rate of transfer, soft skills (interpersonal) do

not transfer as readily and are not easily observed. Transfer is not so readily apparent in the

effective work areas (Olsen, 1998).

Bramley (1996) offered an explanation why evaluation is not being carried out at the

behaviour and result levels. Traditionally most trainers use individual and educational

models of training process. The process has its limitation as emphasis is on encouraging

individuals to learn something rather than to find uses (if any) for the learning.

1.5 Future Research

Bramley and Kitson (1994) argued that the problems of evaluation at Levels 3 and 4 were not

well understood because not enough evaluation of this kind has been carried out. This is due

to the fact that effective measurement methods for Levels 3 and 4 are not available and the

amount of work in setting up the criteria for measuring these two levels is time consuming. It

is apparent that the incompleteness of Kirkpatrick model lies in its Levels 3 and 4 of

evaluation.

1.5.1 The Transfer Component

The transfer component is a potential area for future research. Transfer of training can be

defined as 'the application of knowledge, skills and attitudes learned from training on the job

and subsequent maintenance of them over a certain period of time (Baldwin & Ford, 1988;

Xiao, 1996). This process does not appear to have received much attention since most

organizations were apparently looking primarily at Levels 1 and 2 evaluations. Early studies

lacked theoretical framework to guide these investigations (Baldwin & Ford, 1988).

27

A survey conducted by Cheng and Ho (1998) revealed that there were inconsistent findings

on the variables that promised positive training transfer. The main intention of further

research is to develop common variables that are critical to different training and transfer

situations, including the establishment of common scales or instruments that can be used in

different research settings.

The current approach which uses variables such as individual ability, motivation and

environmental favourability has shown a profound effect on training transfer research (Noe &

Schmidt, 1986). However, this approach raises the question of application. This is because

individual differences (e.g. self efficacy and locus of control) are expected to extent

considerable influence on transfer outcome (Cheng & Ho, 1998).

A longitudinal study would be a better way of measuring the effectiveness of transfer

learning. It is argued that trainees who show similar levels of transfer performance after a

short period of training, may differ substantially in the long run (Kraiger & Ford, 1993).

Therefore, another major aspect of transfer research is to examine the level of newly acquired

knowledge, skills or behaviour retained in the transfer settings after a longer period of time.

For example, research should record the changes in terms of levels of skill proficiency as a

function of time after training.

1.5.2 Evaluating Beyond the 4 Levels

In considering the above studies, an effective evaluation should measure beyond the aspect of

reaction, learning, behaviour and results. Lewis and Thornhill (1994) suggested that an

effective training evaluation needs to be integrated and matched to the culture of the

organization. This integrated culturally related approach is advocated because it would be

able to minimize the risk of not meeting the objectives of carrying out training at the input

stages as well as evaluating reactions and impact at the outcome stage. This brings more

strategic approaches in identifying and prioritizing training needs, in relation to

organizational objectives.

28

29

To justify the training evaluation results, we may consider Brinkerhoff s (1987) criticism on

Kirkpatrick model, which only concentrates on the outcome of training. This is further

supported by Bernthal (1995) who found necessary to look for a broader linkage between

training and the organization context. Bernthal (1995) introduced the training-impact tree

method in measuring organization context. This is done by listing the barriers of training and

the factors that facilitate training next to their associated values and practices which are

aligned with the organization objectives.

Although Kirkpatrick model focuses on the attainment of tangible outcomes, it is important to

note that the question of measuring intangible outcomes that are related to training

effectiveness must not be ignored. Kirkpatrick (1994) revisited his 4-level evaluation model

and states that as long as the evidence collected is beyond a reasonable doubt, one should be

satisfied with the evidence. Perhaps an experienced training practitioner may want to explore

the possibility of interacting the absolute 4-level evaluation model with other process models.

As a result of this, the gap that exists in short and long term measures of training evaluation

may be minimized. Future research may be built upon deriving the integrated model that

would complement both absolute and process evaluation on training effectiveness.

1.5.3 Incorporating Competence-based Approach into TrainingEvaluation

The aim of future research is to develop a comprehensive training evaluation by

incorporating the absolute Kirkpatrick model with the competence-based process. The

competence-based assessment system could be used in collecting sufficient evidence to

determine whether individuals are performing competently in their jobs.

Strebler, Robinson and Heron (1997) classified two different meanings of the term

competency namely expressed as behaviours that an individual needs to perform a job and as

minimum standards of performance. The term competency has been used to refer to the

meaning expressed as behaviours and performance standards. Competence-based assessment

is helpful to provide a behaviourist framework for learning in training evaluation. A

behaviourist approach to learning provides simpler tasks for the trainer and clarity of

outcome for the learner (Hoffmann, 1999). Another definition of competencies is the quality

of outcome which may be used to evaluate gains in productivity or efficiency in the

workplace as a result of training (Strebler et al., 1997).

Further research by Sternberg and Kolligian (1990) defined competency as the underlying

attributes of a person such as their knowledge, skills or abilities. The use of this definition

created a focus on the required inputs of individual in order for them to produce competent

performances. This is aligned with the traditional training evaluation approach of measuring

knowledge, skills and abilities of a person after training. Rowe (1995) suggested that

competence-based assessment which looks at evaluating the whole process of learning should

consist of:-

Objective: The trainer should exhibit clear learning objectives and methods for

obtaining those objectives.

Evidence: Evidence must be provided to indicate competent performance.

Observation: An assessor looks out for competent performance.

Peers' Comments are obtained from work colleagues, peers.

Comments: and customers.

The key point is that a competence-based model supplements knowledge-based

achievements. Programs will be designed by permitting competence-based models to build

on knowledge-based achievement. In this way knowledge supports work, learning supports

skill and theory supports practice (Rowe, 1995).

The competence-based method would be able to assess whether knowledge and skills learned

are being effectively applied in the workplace and whether the trainee can now be described

as competent after completion of a training program.

This integrated model could also be used prior to designing a training program in order to

establish development needs and to determine training program content.

30

1.5.4 Multi-Rater Feedback System in Training Evaluation

There does not appear to be a distinct individual who founded or invented this process and

according to Moses, Hollenbeck and Sorcher (1993), the term multi-rater feedback is

misleading as it suggests a newly discovered concept, whereas they argue that perceptions of

people have been available as long as there have been people to observe them.

Nowack (1993) presents a useful summary of some of the reasons for the increased use of

multi-rater feedback in organizations:

The need for a cost-effective alternative to assessment centers;

The increasing availability of assessment software capable of summarizing data from

multiple sources into customized feedback reports;

The need for continuous measurement of improvement efforts;

The need for job-related feedback for employees affected by career plateauing; and

The need to maximize employee potential in the face of technological change,

competitive challenges and increased workforce diversity.

From the organizational perspective, multi-rater feedback can be used solely for

developmental purposes. Romano (1994) and Atwater et al. (1993) found that the most

common use is in the area of training and development. The overall net effect of training and

development should enhance organizational performance.

From the individual perspective, the feedback is invaluable because it comes from numerous

sources, providing multiple perspectives and opinions. Each opinion and perspective may

provide relevant yet different feedback (Atwater et. al, 1993; Hazucha et. al, 1993; Tornow,

1993). This form of feedback can increase the reliability, fairness and acceptance of the data

by the person being rated (London, Wohlers & Gallagher, 1990). This occurs because the

feedback is received from multiple sources and not just from one ratee.

One of the advantages of using multi-rater feedback is that it provides the opportunity for

individuals who are being assessed to compare their self perceptions against the perceptions

of others regarding their behaviour (Rosti & Shipper, 1998).

31

The difference in perspective between the rater and the ratee is not treated as an error but is a

source of information which can enhance personal learning. Ratees can learn from the

discrepancy between self rating and the rating of others.

The use of multi-rater feedback provides a natural method for both enhancing learning of the

participants and improving the evaluation process. Feedback is seen as a critical element in

affecting change (Bennis, Benne & Chin, 1969). Multi-rater feedback could be used to serve

as an unfreezing process in Lewin's (1948) model of change. This would enhance the ratee's

learning by creating doubts on the ratee's current performance standard and provides an

opportunity for prospective development. Most training evaluation models emphasize the

absolute outcome of training. However, multi-rater feedback involves the change process

where the resultant behaviour involved reinforcement of past performance and also provides

an opening for future learning. Thus, collecting multi-rater feedback before and after training

will enhance learning and provide at least part of the data needed to evaluate training.

Moses et al. (1993) provides the following criticism of multi-rater feedback:

It relies on generalized traits as there is a limited or non-existent frame of reference

for making rater/observer judgments.

It is based on an individual's memory, which can often be incomplete descriptions of

past performance.

The observer may be unable to interpret behaviours

It relies on the instrument designers' scoring system, factor analysis or data collection

methods to interpret the information for the participant.

The main argument of Moses et al. (1993) is that multi-rater feedback is based on other

people's observations and that such observations are often incomplete descriptions of past

performance because the observer does not know what to look for. The unresolved issue is

what behaviours to study. Multi-rater feedback has been taken to identify the behaviour of

effective management. There is lack of sufficient definitional detail to study managerial

proficiency or the effectiveness of training (Morrison & McCall, 1978; Schriesheim & Kerr,

1977). Yulk (1994) argued that further refinement of these constructs is needed by identifying

32

specific skills which make up each construct. Hence, development of construct and its

validity is important prior to training.

Multi-rater feedback has been found to be widely used in managerial and leadership

development programs (Cacioppe R., 1998; Cacioppe & Albrecht, 2000; Garavan, Morley &

Flynn, 1997; McCauley & Moxley, 1996; Thach, E.C., 2002). However, its usage in other

fields needs further research and exploration. This is further supported by Rosti and Shipper

(1998) in their study on the impact of training in a management development program based

on multi-rater feedback.

1.6 Conclusion

It is widely acknowledged that the Kirkpatrick evaluation model has been providing the most

basic thoughts on training evaluation throughout this decade. However, there seems to be

incomplete application of Kirkpatrick's 4-level evaluation model being carried out by the

industries. No significant success has been identified from the use of 4-level evaluation

model by the majority of organizations that have conducted training evaluations.

Based on this literature review, it may be concluded that Kirkpatrick model has not reached a

stage of clarity for in-depth training evaluation to be carried out. His model would provide

training managers with the idea of what is training evaluation on a systematic approach

however the aspect of training measurement method was not well explored or detailed.

While training has been conceptualized as a continually evolving process, the existing

literature appears to have failed to provide adequate strategies for organizations wanting to

evaluate the immediate, as well as the long-term, effectiveness and value of their training

efforts.

At face value, the literature shows that the full Kirkpatrick evaluation strategy is being widely

applied; however, more detailed analysis found that none were able to demonstrate Level 4

evaluation and of those who claimed evaluation at Levels 2 or 3, none were able to

demonstrate a systematic approach to the problem.

33

Arguably the dilemma in adopting the Kirkpatrick's taxonomy as a comprehensive and

integrated approach to evaluation lies in both the qualitative and quantitative attempts that

may or may not provide good phenomenological studies. Further analysis of the method

shows considerable confusion as to what is, or is not, a valid indicator for evaluation. Clearly,

there has been little change in terms of level of confidence towards the reliability of training

evaluation, notwithstanding greater emphasis on this key organizational development

process.

The weaknesses of Kirkpatrick model have brought opportunity for future research in

incorporating competencies and multi-rater feedback approach into the long-term evaluation

of training.

These weaknesses have also opened up opportunities for further research in the transfer

learning especially in the studies of its longitudinal and application effect.

1.7 References for Paper One

Alliger, G.M. & Janek, E.A. 1989, 'Kirkpatrick's levels of training criteria: thirty yearslater', Personnel Psychology, vol. 42, pp. 331-342.

Alliger, G.M., Tannenbaum, S.I., Bennett, W., Traver, H. & Shotland, A. 1997, 'A meta-analysis of the relations among training criteria', Personnel Psychology, vol. 50, pp.341-358.

Atwater, L., Roush, P. & Fishthal, A. 1993, The Impact of Upward Feedback on Self andFollower Ratings of Leaders, Centre for Creative Leadership, New York.

Baldwin, T.T. & Ford, J.K. 1988, 'Transfer of training: a review and directions for futureresearch', Personnel Psychology, vol. 41, pp. 63-105.

Basadur, M., Graen, G.B. & Scandura, T.A. 1986, 'Training effects on attitudes towarddivergent thinking among manufacturing engineers', Journal of Applied Psychology,vol. 71, pp. 612-617.

Bennis, W.G., Benne, K.D. & Chin, R.1969, The Planning of Change, 2nd edn, Holt,Rinehart & Winston, New York.

Bernthal, P.R. 1995, 'Education that goes the distance', Training and Development, vol. 49,no. 9, pp. 41.

34

Blanchard, P.N. & Thacker, J.W. 1999, Effective Training, Systems, Strategies andPractices, Prentice Hall Publisher, New Jersey.

Blanchard, P.N., Thacker, J.W. & Way, S.A. 2000, 'Training evaluation: perspectives andevidence from Canada', International Journal of Training and Development, vol. 4,no.4, pp. 295-303.

Boulmetis, J. & Dutwin, P. 2000, The ABCs of Evaluation: Timeless Techniques forProgram and Project Managers, Jossey-Bass Publisher, San Francisco.

Boyle, P.G. & Jahns, I. 1970, 'Program development and evaluation' in Handbook of adulteducation, eds Smith, R.M., Aker, G.F. & Kidd, J.E., Macmillan Company, NewYork, pp. 70.

Bramley, P. & Kitson, B. 1994, 'Evaluating training against business criteria', Journal ofEuropean Industrial Training, vol. 18, no.1, pp. 10-14.

Bramley, P. 1996, Evaluating Training Effectiveness, McGraw-Hill, Maidenhead and NewYork.

Brandenburg, D. 1982, 'Training evaluation: what is the current status?' Training andDevelopment Journal, pp. 14-19.

Bretz, R.D. & Thompsett, R.E. 1992, 'Comparing traditional and integrative learningmethods in organizational training programs', Journal of Applied Psychology, vol. 77,pp. 941-951.

Brinkerhoff, R. 0. 1987, Achieving results from training, Jossey-Bass Publisher, SanFrancisco.

Brinkerhoff, R.O. 1988, 'An integral evaluation model for human resource development',Training and Development Journal, vol. 42, no. 2, pp. 66-68.

Brown, K.G., Werner, M.N., Johnson, L.A. & Dunne, J.T. 1999, Formative evaluation inIndustrial/Organization Psychology: further attempts to broaden training evaluation,presented at a symposium on training evaluation: advances and new directions forresearch and practice, Society of Industrial and Organizational Psychology, Atlanta.

Cacioppe, R. 1998, 'An integrated model and approach for the design of effectiveleadership development programs', Leadership and Organization DevelopmentJournal, vol. 19, no. 1, pp. 44-53.

Cacioppe, R. & Albrecht, S. 2000, 'Using 360-degree feedback and the integral model todevelop leadership and management skills', Leadership and OrganizationDevelopment Journal, vol. 21, no. 8, pp. 390-404.

Campbell, J.P. 1988, Training Design for Performance Improvement, in Productivity inOrganizations, eds Campbell, J.P. & Campbell, R.J., Jossey-Bass Publisher, SanFrancisco.

35

Cascio, W.F. 1989, Using utility analysis to assess training outcomes, in Training andDevelopment in Organizations, ed. I.L. Goldstein, Jossey-Bass, San Francisco.

Cervero, R.M. 1988, Effective Continuing Education for Professionals, Jossey-BassPublisher, San Francisco.

Campion, M.A. & Campion, J.E. 1987, 'Evaluation of an interview skills trainingprogram in a natural field setting', Personnel Psychology, vol. 40, no. 4, pp. 675-91.

Chen, H.T. & Rossi, P.H. 1992, Using Theory to Improve Program and PolicyEvaluations, Greenwood Press, Westport, CT.

Cheng, E. & Ho, D. 1998, 'The effects of some attitudinal and organizational factors ontransfer outcome', Journal of Managerial Psychology, vol. 13, no. 5/6, pp. 309-317.

Clegg, W.H. 1987, 'Management training evaluation: an update', Training and DevelopmentJournal, vol. 41, no. 2, pp. 65-71.

Connolly, M.S. 1988, 'Integrating evaluation, design and implementation', Training andDevelopment Journal, vol. 42, no. 2, pp.20-23.

Constable, J. & McCormick, R. 1987, The Making of British Managers, BIM, CBI, London.

Davidove, A.E. & Schroeder, P.A. 1992, 'Demonstrating ROI of training' Training andDevelopment Journal, vol. 46, no. 8, pp. 70-71.

Davis, B.L. & Mount, M.K. 1984, 'Effectiveness of performance appraisal training usingcomputer assisted instruction and behaviour modeling', Personnel Psychology, vol.37, pp. 439-452.

Dawson, R.P. 1993, Model of evaluations of equal opportunities training in localgovernment with special reference to women, unpublished PhD thesis, South BankUniversity, London.

Dionne, P. 1996, 'The evaluation of training activities: a complex issue involvingdifferent stakes', Human Resource Development Quarterly, vol. 7, pp. 279-86.

Dyer, S. 1994, `Kirkpatrick's mirror', Journal of European Industrial Training, vol. 18,no. 5, pp 31-32.

Eisner, E.W. 1997, The Enlightened Eye: Qualitative Inquiry and the Enhancement ofEducational Practice, 2nd edn., Merrill, New York.

Garavaglia, L.P. 1993, 'How to ensure transfer of training', Training & DevelopmentJournal, vol. 47, no. 10, pp. 63-68.

Garavan, T.N., Morley, M. & Flynn, M. 1997, '360-degree feedback: its role in employeedevelopment', Journal of Management Development, vol. 16, no.2, pp. 134-147.

36

Geber, B. 1995, 'Does your training make a difference? Prove it!', Training andDevelopment Journal, vol. 3, pp. 27-34.

Goldstein, L.I. 1986, Training in Organizations: Needs Assessment, Development andEducation, Cole Publishing Company, California.

Goldstein, L.I. & Ford, J.K. 2002, Training in Organizations: Needs Assessment,Development and Evaluation, Thomson Learning, Wadsworth, Canada.

Grove, E.A. & Ostroff, C. 1990, Program evaluation, in Developing Human Resources, edsWexley, K. & Hinnicks, J., BNA Books, Washington D.C.

Hamblin, A.C. 1974, Evaluation and Control of Training, McGraw-Hill Publisher, NewYork.

Hazucha, J.F., Hezlett, S.A. & Schneider, R.J. 1993, 'The impact of 360-degree feedback onmanagement skills development', Human Resource Management, vol. 32, pp. 325-351.

HMSO 1989, Training in Britain: A Study of Funding, Activity and Attitudes, HerMajesty's Stationery Office, London.

Hoffmann, T. 1999, 'The meanings of competency', Journal of European IndustrialTraining, vol. 23, no. 6, pp. 275-285.

Holton, E.F. III 1996, 'The flawed four-level evaluation model', Human ResourceDevelopment Quarterly, vol. 7, pp. 5-21.

Junaidah, H. 2001, 'Training evaluation: clients' roles', Journal of European IndustrialTraining, vol. 25, no. 7, pp. 374-379.

Kirkpatrick, D.L. 1959a, 'Techniques for evaluating training programs: part 1 - reaction',Journal of American Society for Training and Developing, vol. 13, pp. 3-9.

Kirkpatrick, D.L. 1959b, 'Techniques for evaluating training programs: part 2 - learning',Journal of American Society for Training and Developing, vol. 13, no. 12, pp. 21-26.

Kirkpatrick, D.L. 1960a, 'Techniques for evaluating training programs: part 3- behaviour',Journal of American Society for Training and Developing, vol. 14, no. 1, pp. 13-18.

Kirkpatrick, D.L. 1960b, 'Techniques for evaluating training programs: part 4 - results',Journal of American Society for Training and Developing, vol. 14, no. 2, pp. 28-32.

Kirkpatrick, D.L. 1976, Evaluation of Training, Training and Development Handbook: Aguide to human resource development, 2nd edn, Craig, R.L.O., McGraw-HillPublisher, New York.

Kirkpatrick, D.L. 1979, 'Techniques for evaluating training programs', Training andDevelopment Journal, vol. 33, pp. 78-92.

37

Kirkpatrick, D.L. 1994, Evaluating Training Programs: The Four Levels, Berrett-KoehlerPublishers, San Francisco.

Kirkpatrick, D.L. 1996a, 'Great ideas revisited', Training and Development Journal,vol. January, pp. 54-59.

Kirkpatrick, D.L. 1996b, 'Invited reaction: reaction to Holton article', Human ResourceDevelopment Quarterly, vol. 7, pp. 23-24.

Kirkpatrick, D.L. 1998, Evaluating Training Programs: The Four Levels, Berrett-Koehler Publishers, San Francisco.

Kraiger, K., Ford, J.K. & Salas, E. 1993, 'Application of cognitive, skill-based and affectivetheories of learning outcomes to new methods of training evaluations', Journal ofApplied Psychology, vol. 78, no. 2, pp. 311-328.

Legge, K. 1984, Evaluating Planned Organizational Change, Academic Press, London.

Lewin, K. 1948, Resolving social conflicts, Harper & Bros Publishers, New York, NY.

Lewis, P. & Thornhill, A. 1994, 'The evaluation of training an organizational cultureapproach', Journal of European Industrial Training, vol. 18, no. 8, pp. 25-32.

London, M., Wholers, A.J. & Gallagher, P. 1990, '360-degree feedback surveys: a source offeedback to guide management development', Journal of Management Development,vol. 9, pp. 17-31.

Love, A.J. 1991, Internal Evaluation: Building Organizations From Within, SagePublication, California, CA.

Madaus, G.F., Scriven, M.S. & Stufflebeam, D.L. 1986, Evaluation Models: Viewpointson Educational and Human Services Evaluation, Kluwer-Nijhoff Publishing, Boston.

Mann, S. & Robertson, I. T. 1996, 'What should training evaluation evaluate?'Journal of European Industrial Training, vol. 20, no. 9, pp. 14-20.

Mathieu, J.E. & Leonard, R.L. Jr. 1987, 'Applying utility concepts to a training program insupervisory skills: a time-based approach', Academy of Management Journal, vol.30, pp. 316-335.

Mathews, B.P., Ueno, A., Kekale, T., Repka, M., Pereira, Z.L. & Silva, G. 2001, 'Qualitytraining: needs and evaluation-findings from a European survey, Total QualityManagement, vol. 12, no. 4, pp. 483-490.

McCauley, C.D. & Moxley, R.S. Jr. 1996, Developmental 360: How Feedback Can MakeManagers More Effective, Jossey-Bass Publisher, San Francisco.

Morrison, A.M. & McCall, J.D. 1978, Feedback to Managers: A ComprehensiveReview of Twenty-four Instruments, Centre for Creative Leadership, Greensboro, NC.

38

Morrow, C.C., Jarrett, M.Q. & Rupinski, M.T. 1997, 'An investigation of the effect andeconomic utility of corporate-wide training', Personnel Psychology, vol. 50, pp. 91-119.

Moses, J., Hollenbeck, G.P. & Sorcher, M. 1993, 'Other people's expectations', HumanResource Management, vol. 32, Summer Fall.

Murk, P., Barrett, A. & Atchade, P. 2000, 'Diagnostic techniques for training andeducation: strategies for marketing and economic development', Journal ofWorkplace Learning, vol. 12, no. 7, pp. 296-306.

Noe, R.A. & Schmitt, N. 1986, 'The influence of trainee attitudes on trainingeffectiveness: test of a model', Personnel Psychology, vol. 39, pp. 497-523.

Noe, R.A. 2000, Employee Training and Development, McGraw-Hill Publisher, New York.

Nowack, K. 1993, '360-degree feedback: the whole story', Training and DevelopmentJournal, vol. 47, no. 1, pp. 69-73.

Newstrom, J.W. 1978, 'The problem of incomplete evaluation of training', Training andDevelopment Journal, vol. 32, no. 11, pp. 22-24.

O'Leary, V.E. 1972, 'The Hawthorne effect in reverse: effects of training and practice onindividual and group performance', Journal of Applied Psychology, vol. 56, pp. 491-494.

Olsen, J. H. Jr. 1998, 'The evaluation and enhancement of training transfer', InternationalJournal of Training and Development, vol. 2, no. 1, pp. 61-75.

Parlette, M. & Hamilton, D. 1977, 'Evaluation as a new approach to the study of innovativeprogrammes', in Beyond the Numbers Game, eds Hamilton, D. et al., Macmillan,London.

Phillips, J.J. 1991, Handbook of Training Evaluation and Measurement Methods, GulfPublishing Company, Houston, TX.

Phillips, J.J. 2002, Return on Investment in Training and Performance ImprovementPrograms, 2nd edn, Butterworth-Heinemann, Woburn, MA.

Phillips, J.J. & Stone, R.D. 2002, How to Measure Training Results, A Practical Guide toTracking the Six Key Indicators, McGraw-Hill Publisher, New York.

Plant, R.A. & Ryan, R.J.1994, 'Who is evaluating training?', Journal of European IndustrialTraining, vol. 18, no. 5, pp. 27-30.

Popham, W. J. 1974, Evaluation in Education: Current Applications, Berkeley,McCutchan, California.

Porter, L., & McKibbin, L. 1988, Future of Management Education and Development DriftOr Thrust Into the 21' Century?, McGraw-Hill Publisher, New York.

39

Provus, M. 1971, Discrepancy Evaluation, Berkeley, McCutchan, California.

Rae, L. 1986, How to Measure Training Effectiveness, Gower Publications, Aldershot,London.

Raphael, M. & Wagner, E. 1972, 'Training surveys surveyed', Training and DevelopmentJournal, vol. 26, pp. 10-14.

Redshaw, B. 2001, 'Evaluating organizational effectiveness', Measuring BusinessExcellence, vol. 5, no. 1, pp. 16-18.

Regalbutto, G.A. 1992, 'Targeting the bottom line', Training and Development Journal, vol.46, no. 4, pp. 29-32.

Rivlin, A.M. 1971, Systematic Thinking for Social Action, Brookings Institution,Washington.

Romano, C. 1994, 'Conquering the fear of feedback', Human Resource Focus, vol. 71, no. 3.

Rossi, P.H. & Freeman, H.E. 1993, Evaluation.. A Systematic Approach, 5th edn, SagePublication, California.

Rosti, R.T. Jr. & Shipper, F. 1998, 'A study of the impact of training in amanagement development program based on 360 feedback', Journal of ManagerialPsychology, vol. 13, no.1/2, pp. 77-89.

Rowe, C. 1995, 'Incorporating competence into the long term evaluation of training anddevelopment', Industrial Commercial Training, vol. 27, no.2, pp. 3-9.

Salinger, R. & Deming, R. 1982, 'Practical strategies for evaluating education', Training andDevelopment Journal, vol. 4, pp. 20-29.

Sauter, J. 1980, 'Purchasing public sector executive development', Training andDevelopment Journal, vol. 34, no. 4, pp. 92-98.

Schriesheim, C.A. & Kerr, S. 1977, 'Theories and measurement of leadership: a criticalappraisal of present and future directions', in Leadership: The Cutting Edge, edsHunt, J.G. & Larson L.L., Southern Illinois University Press, Carbondale, IL.

Scriven, M. 1991, Evaluation Thesaurus, Sage Publication, Newbury Park, California.

Shadish, W. R. & Epstein, R. 1987, 'Patterns of program evaluation practice amongmembers of the evaluation research society and evaluation network', EvaluationReview, vol. 11, no. 5, pp. 555-590.

Shadish, W.R. & Reichardt, C.S. 1987, 'Evaluation studies', Evaluation Review, vol. 12, pp.13-30.

40

Shireman, J.A.R. 1991, Utilization of program evaluation for decision making regardinghospital based patient/client focused health education programs, doctoral dissertation,University of Iowa, dissertation abstracts international, 52/12A, AA C9212928.

Smith, A.J. 1990, 'Evaluation of management training subjectivity and the individual',Journal of European Individual Training, vol. 14, no. 1, pp. 12-15.

Stake, R. 1977, 'Responsive evaluation', in Beyond the Number Game, eds Hamilton, D.,Jenkins, D., King, C., MacDonald, B. & Parlett, H.M., Macmillan, London.

Steel, S. 1970, 'Program evaluation: a broader definition', Journal of Extension, vol. 13, pp.13-20.

Sternberg, R. & Kolligian, J. Jr. 1990, Competence Considered, Yale University Press, NewHeaven, CT.

Strebler, M., Robinson, D. & Heron, P. 1997, 'Getting the best out of yourcompetencies', Institute of Employment Studies, University of Sussex, Brighton.

Stufflebeam, D.L. 1971, Education Evaluation: Decision Making, by the PDK national studycommittee on education, Itasca, III: F.E. Peacock Publisher Inc, Boston.

Stufflebeam, D.L. 1983, 'The CIPP model for program evaluation', in Evaluation Models,eds Madaus, G.F., Scriven, M.S. & Stufflebeam, D.L., Kluwer-Nijhoff Publishing,Boston, pp. 117-141.

Stufflebeam, D.L. & Shrinkfield, J.A. 1985, Systematic evaluation, Kluwer NijhoffPublishing, Boston.

Swanson, R.A. & Holton, E.F. 1999, Results: How to Assess Performance, Learning AndPerceptions in Organizations, Berrett-Koehler Publishers, San Francisco.

Tesoro, F. 1998, 'Implementing an ROI measurement process at Dell Computer',Performance Improvement Quarterly, vol. 11, pp. 103-114.

Thach, E.C. 2002, 'The impact of executive coaching and 360-feedback on leadershipeffectiveness', Leadership and Organization Development Journal, vol. 23, no. 4,pp. 205-214.

Toplis, J. 1993, 'Training evaluation reflections on the first steps', European WorkOrganization Psychology, vol. 2, no. 2, pp. 146-152.

Tornow, W.W. 1993, 'Perceptions or reality, is multiple-perspective measurement ameans or an end?', Human Resource Management, vol. 32. no. 2 & 3, pp. 209-408.

Tyler, R.W. 1949, Basic Principle of Curriculum and Instruction, University of ChicagoPress, Chicago.

Tyler, R.W. 2002, 'Evaluating evaluations', Human Resource Magazine, vol. June, pp. 85-93.

41

Warr, P. & Bunce, K. 1995, 'Employee age and voluntary development activity',International Journal of Training and Development, vol. 2, pp. 190-204.

Warr, P., Allan, C. & Birdi, K. 1999, 'Predicting three levels of training outcome', Journalof Occupational and Organizational Psychology, vol. 72, pp. 351-375.

Wexley, K.N. & Baldwin, T.T. 1986, 'Post-training strategies for facilitating positivetransfer: an empirical exploration', Personnel Psychology, vol. 29, pp. 503-520.

Wholey, J.S., Hatry, H.P. & Newcomer, K.E. 1994, Handbook of Practical ProgramEvaluation, Jossey-Bass Publisher, San Francisco.

Xiao, J. 1996, 'The relationship between organizational factors and the transfer oftraining in the electronics industry in Shenzhen, China', Human ResourceDevelopment Quarterly, vol. 7, no. 1, pp. 55-73.

Yulk, G.A. 1994, Leadership in Organizations, 2nd edn, Englewood Cliffs, Prentice HallPublisher, New Jersey.

42

43

Research Paper 2

EVALUATING TRAINING EFFECTIVENESS:AN EMPIRICAL STUDY OF KIRKPATRICK

MODEL OF EVALUATION IN THEMALAYSIAN TRAINING ENVIRONMENT FOR

THE MANUFACTURING SECTOR

Lim Guan ChongMaster of Business Administration (Finance)

University of Hull

International Graduate School of ManagementUniversity of South Australia

Evaluating Training Effectiveness: An EmpiricalStudy of Kirkpatrick Model Of Evaluation in the

Malaysian Training Environment for theManufacturing Sector

Lim Guan ChongInternational Graduate School of Management

University of South Australia

2.1 Abstract

This research adopted an empirical approach to track the history, rationale, objectives and the

implementation of training evaluation initiatives in Malaysia's manufacturing sector. Since

the establishment of the Human Resource Development Fund, training activities in Malaysia

have increased. The majority of Malaysian organizations that conduct training are doubtful

about how training activities could add value to the organization performance and justify

their training investment. This research provides an understanding of training evaluation

culture within the Malaysian manufacturing sector and the effectiveness of this Kirkpatrick's

4-level evaluation model as applied to the Malaysian manufacturing sector.

2.2 Introduction

The Malaysian government is committed towards education, training and human resource

development. The government recognizes the importance of human resource development in

its quest for achieving a fully developed nation status. This commitment has translated into

the establishment and growth of the training practice in the country.

Being the sole provider of training previously, the government has adopted the policy of

involving private enterprises in all aspects of training. Training needs have become crucial

and vital to the development of capital-intensive and value added industries. Apart from

44

involving enterprise to make training more market-driven, there is a need for enterprise to

share the burden of training. In the Seventh Malaysia Plan, the private sector was expected to

play a more active role in upgrading the qualification and skill of its workers (Junaidah,

2001).

2.3 Training Practices in Malaysia

Training activities within Malaysian companies are behind countries like Singapore, Japan

and Korea. Training activities in Malaysia are mainly conducted by large multinational

companies. The International Labour Organization's study in 1997 showed that Malaysia is

in the 12th position in terms of providing in-company training (Junaidah, 2001).

The Malaysian government passed a new Act of Parliament entitled Human Resources

Development Act in 1992, to encourage and stimulate the private sector to introduce training

and development for its employees (HRDC, 1992). The objective of this Act is to set aside

accumulated funds to promote training activities within the organization. Under this Act,

companies with more than 50 employees will have to contribute 1 percent of their total staff's

monthly salary to the Ministry of Human Resources through the Human Resources

Development Council (HRDC). The fund is known as the Human Resources Development

Fund (HRDF), was launched in January 1993. The government set up the HRDC to manage

this fund by identifying the systematic training needs and approving relevant training

programs required by organizations. The levy is partially refunded under special schemes

known as Training Aid Scheme and Approved Training Program (ATP) Scheme to the

respective organizations once the training program is completed. The policy lays down the

parameters for a Human Resource oriented development strategy that is designed to mobilize

national effort to increase technological capabilities and competitiveness as well as create

highly skilled, productive, disciplined and efficient workforce. This strategy would aid

Malaysia's transition into an industrialized economy. Private sector companies are also

expected to enhance their training activities by utilizing the HRDF and participating in skill

development programs run by the state governments (MEPU, 1996). Since the establishment

of the HRDC, how has the Malaysian manufacturing sector gained from the training

conducted? With information on how training benefit organizations, it would help the

45

Malaysian government to chart the progress and expected time frame needed for Malaysia to

transform into an industrialized economy.

The need to develop a highly trained workforce is evident from the increase of more than 200

management consulting and training institutions, professional associations and management

schools operating in Malaysia (Arthur Anderson & Co, 1991). The number of employees

who return to formal education and training has increased consistently since 1972 (Ahmad,

1998). The government set up the National Institute of Public Administration Malaysia

(INTAN) which is responsible for training government employees in administration and

management (Junaidah, 2001).

There are some real difficulties in assessing the full extent of skill development for

government training in Malaysia even after conducting evaluation (Mirza & Juhary, 1995).

Firstly, much of skill development takes place in the private sectors. Most skills even those

involving advanced manual skills are acquired on the job. Secondly, skill development

during employment tends to be demand-driven (Pillai, 1994). Workers gain experience on

the job and upgrade their skills when they are exposed to a higher skill level. A study by

Pillai and Othman (1994) showed that the budget for training and education in Malaysia has

increased by 40 percent. Company emphasis has been on improving the quality of training to

help develop competent labour force that improves the competitiveness of the industrial

sector in Malaysian. This new demand will force employers to further develop employee

competencies. Saiyadain (1995) found that as many as 82.6 percent of organizations

sponsored their managers for training, and on average these organizations spent 4.65 percent

of the managerial payroll on training managers. This shows that the number of knowledge

workers and new knowledge-based opportunities is expected to increase dramatically in the

next few years.

2.4 The Practice of Evaluation in Training

Although the methodology of evaluating training effectiveness may look fair, it could make it

difficult to express rational criticism. A survey by Wagel (1977) found that 75 percent of

companies have no formal method for evaluating training effectiveness. In a subsequent

46

survey by Easterby-Smith (1985), the result showed that out of 15 organizations with 320

300,000 employees, only one conducted some form of evaluation on a regular basis which

was a post-course questionnaire. According to Rowe (1992), although every training manual

gives lip service to evaluation, it is notoriously difficult to carry out effectively. The

extensive survey by Plant and Ryan (1994) served to further underline the lack of widespread

sophistication in evaluation. They point to budget cutting and economies pressures as being

possible explanations. A recent study by Blanchard, Thacker and Way (2000) on 202

organizations in Canada reported that more than half of the organizations are not

comprehensively evaluating their training.

According to Carnevale and Schulz (1990), the American Society for Training and

Development (ASTD) research indicated that the most popular reasons for evaluation are to

gather information to help decision makers improve the training process and facilitate

participants' job performance. This explains why the outcome-based Kirkpatrick model is so

popularly used. Evaluation also helps measure the degree of improvement in application and

assesses how well the learner achieves the established goals (Attkinsson, Sorenson,

Hargreaves & Hororwitz, 1978).

For the past 30 years the Kirkpatrick model had been considered the most prominent training

evaluation model (Bernthal, 1995). Phillips (1991) concluded that, out of more than 50

evaluation models available, the evaluation framework that most training practitioners use is

the Kirkpatrick model. It is easy to find firms that practice training evaluation. However,

most firms only conduct post course evaluation using Kirkpatrick's Level 1 evaluation.

Another important purpose for training evaluation is to meet the accountability requirements

of funding groups or clients (Rossi & Freeman, 1993). The demand for accountability has

been the major impetus for program evaluation since 1980s. Fiscal constraints have

increased the competition of companies' activities for available dollars and raised the

question of value for money from their activities (Ruthman & Mowbray, 1983).

Training evaluation is more than a set of empirical methods governed solely by the standards

of social science. Judgments on the quality of program evaluation must also be based on

criteria that are meaningful both to immediate users and the larger system in which the

program is embedded (Corday & Lipsey, 1986).

47

Phillips (1991) stated that when it comes to training evaluation, there still appears to be more

talk than action. In many organizations, training evaluation is either ignored or approached in

an unsystematic manner. Previous literature (Davidove & Schroeder, 1992; Shelton &

Alliger, 1993; Smith, 1990) demonstrated that training evaluation is unsystematic and based

on simple means. Gutek (1988) stated that there was little or no demand on the part of the

organization to seriously evaluate a training program. Most organizations evaluate their

training programs by emphasizing one or more levels of Kirkpatrick model (Chen & Rossi,

1992). The researchers, however, commented that evaluation knowledge found in the

literature is not being fully utilized in evaluation practices.

Admittedly it is difficult to completely ascertain a training program's effectiveness. What

works at a particular time at a particular training location with a group of participants may not

necessarily work as well when transferred to another time, setting and group (Junaidah,

2001).

Bramley and Kitson (1994) asserted that measuring learning is problematic because it is

difficult to design a reliable measuring instrument. There are also few people who possess

the necessary skills to evaluate training however these skills are often not available. Grove

and Ostroff (1990) mentioned that training directors often do not possess the necessary skills

to conduct training evaluation. However, Bramley (1996) mentioned that the lack of training

evaluation skills could be due to the methodological weakness embedded within the

Kirkpatrick model of evaluation.

In addition to the unavailability of a reliable measuring instrument, Barron (1996)

commented that why management does not demand evaluation because the management

believes that training will be reflected in an employee's work performance. The research by

Smith and Piper (1990) supported this view and showed that trainers openly said, "We do just

what we are asked to do deliver training. We do not do what we are not asked to do

improve human performance in the workplace". Smith and Piper (1990) also mentioned this

as one of the reasons for providing training but not evaluation. The research found that their

clients did not request for an evaluation. This could be the reason why training providers do

not evaluate their products.

48

A research by the ASTD in 1990 showed that most companies now conduct some form of

evaluation of their training programs. Practitioners tend to use different methodology and

approaches. In examining evaluation methods in business-education partnerships, Erickson

(1991) found that there is little standardization in the methodology. Shadish and Epstein

(1987) conducted a study to look at program evaluations among members of the Evaluation

Research Society and Evaluation Network. They found that practitioners had different

methodologies as well as different assumptions about evaluation. In their study, three patterns

of practices emerged from the evaluation practices which they labeled the academic pattern,

decision-driven pattern and the outcome pattern.

Heneman and Schurab (1986) stated that the evaluation of training programs is considered

different compared to the theory and models in the literature. Many authors commented that

once participants leave the training setting, program providers seldom attempt to determine

the effect of their program. Indeed, the word evaluation raises all sorts of emotional defense

reactions. Such response indicates a low level of commitment among training professionals

toward evaluation. Most of the time, the practices are informal, unsystematic and based on

one popular model. However in the study by Junaidah (2001) on Malaysian training

evaluation practices, it was found that evaluation was moderately formal, comprehensive and

systematic but could be further improved. Nevertheless, it is uncertain whether this so-called

comprehensive approach to training evaluation is within the taxonomy of the Kirkpatrick

framework. Currently, there is little literature on the evaluation system within the Malaysian

context.

2.5 Training Evaluation Practices in Malaysia

Validation of training effectiveness and benefits of training and development programs have

gained importance in public and private sectors in Malaysia. The Malaysian government

places great emphasis on program evaluation and appointed two federal agencies to be

responsible for evaluation. They are the National Institute of Evaluation and the Evaluation

Unit at the Prime Minister's Department. This unit is responsible for evaluating special

governmental projects and programs (Maimunah, 1990). Another evaluating body is the

Publication and Consultancy Bureau which carries out evaluation for government training.

49

There are three types of evaluation process currently being practiced in the agency. The

formal training evaluation uses standard evaluation questionnaires and oral evaluation in the

form of informal discussions, while the informal evaluation conducted during training

(Junaidah, 2001).

The reasons why Malaysian organizations do not evaluate training may lie in the inability to

develop relevant measuring tools or the difficulty in determining which performance

outcomes are attributed to training.

The rise in the awareness of training evaluation during the Malaysian economic downturn in

1997 has increased the pressure for organizations to justify the investment cost placed on

training (Junaidah, 2001). Organizations realized that training must be a worthwhile effort

and this raises the need for measuring training effectiveness. Evaluating training

effectiveness does not seem to be the culture of most organizations in Malaysia. Thousands

of training programs have been conducted in Malaysia since the rise of HRDF, (Mirza &

Juhary, 1995). However, effectiveness in terms of productivity, skills improvement, increase

in performance standards and return on investment is still unknown. Training should be

evaluated to learn the weaknesses of the training program. The selection criteria for

evaluation should be able to find out the improvement in the participants' work performance.

The need for greater quality management during the economic downturn forced Malaysian

companies to upgrade their current version of International Standard Organization (ISO) to

ISO 9001:2000 which emphasized on documenting the training evaluation process.

Companies that pursued this latest version of ISO are required to justify their training efforts

and money spent by linking skill development with the quality philosophy of the company.

As organizations pursue the latest version of ISO, evaluating training ranks high among top

management as a means of justifying training investment (Junaidah, 2001). The opportunity

cost of foregoing training commitment has become extremely high. More than ever, training

evaluation must demonstrate improved performance and financial results. As the investment

spent on training is costly, it is understandable why top managers wish to see value for

money and demand justification for training cost. Training providers need to show clients that

they are getting good returns on their investment in training. This demand for accountability

had been the major impetus for training in the past few years (Junaidah, 2001).

50

Most organizations in Malaysia have sufficient training facilities. Most managers are

sponsored to attend training programs on production, general management and human

resources management for an average of 2 days (Mirza & Juhary, 1995). On average

organizations spend 4.65 percent of the managerial payroll on training (Saiyadain, 1995).

The measurement of training effectiveness varies from organization to organization. A few

organizations have developed systematic plans to follow up on training. The top

management's attitude towards training has been identified as a critical factor in effective

operationalization of training (Mirza & Juhary, 1995). In organizations where the top and

middle management have been perceived to be supportive, training seems to have contributed

to the overall growth. But how far the evaluation process has been conducted to prove the

growth is still questionable. In order to improve the overall effectiveness of training, all

organizations should undertake training evaluation effectively. As mentioned by Brinkerhoff

(1988), training needs to adopt evaluations and measuring systems that can improve the

feedback mechanism in order to build their response capacity. A system of pre course

evaluation followed by post course evaluation may help in setting relevant expectations for

improvement.

A serious gap in the Malaysian training context is the insufficient information on the number,

nature and content of training facilities in the country. The skill-level at which the output

would fit into the labour market is not known while the syllabus, duration and quality of

training vary from one agency to another. This is due to the lack of collaboration and

consultation between industry and training institution. The quality of training is not up to the

mark. Trainees have theoretical knowledge but little practical experience (Pillai, 1994).

There has been limited study on training evaluation practices in Malaysia. A training

evaluation research by Shamsuddin (1995) was on the contextual factors associated with

evaluation practices of selected adult and continuing education providers in Malaysia.

According to him even though the management directed an evaluation to be conducted, it was

only for a narrow purpose. It was used to demonstrate program success by showing how

good was the training and how many people received the training which is merely Level 1

evaluation. The wider purpose of program evaluation such as measuring the acquired

learning (Level 2), program impact (Level 3) and cost effectiveness (Level 4) was not the

management priority. According to Shamsuddin (1995), the clients were not aggressive

stakeholders who cared and demanded accountability from the training providers. Their

51

behaviour and characteristics did not push the training provider to examine the real effect of

the programs in terms of learning gain and program effectiveness.

Besides Shamsuddin's (1995) study, four other studies conducted locally included the

element of evaluation practice. The first study by Hamid, Mohd, Muhamad and Ismail

(1987) asked 235 organizations if management education in Malaysia significantly provides

candidates with a set of skills. Organizations found that 67.6 percent of management

programs offered by local universities and colleges are too theoretical. Out of 121

respondents, 60.3 percent indicated that training is important while the rest felt the contrary.

This study focused on reaction evaluation (Level 1) to study the participants' satisfaction

level towards the overall programs. Another study conducted by Asma (1994) examined the

design of training practices of four training providers in Malaysia and found that the

evaluation practiced by the trainers do not conform to any theory and most of the evaluations

used were ad hoc and informal.

Mirza and Juhary (1995) conducted a study on local and multinational organizations and

found that in the majority of these organizations even if managers who return from training

may write a report, no formal systematic mechanism exists to assess how well they are

utilizing their training in the organizations. The research further found that participants were

only encouraged to apply learning at work but do not take the effort to find out what caused

the change. The result indicates that the behaviour towards measuring training effectiveness

is not popularly practiced. Organizations feel that if learning does not take place, it would

show in the next appraisal report. Participants who have learned something should have

applied it and therefore not necessary to track changes in performance.

Mirza and Juhary's (1995) study also revealed that most organizations in Malaysia evaluate

training effectiveness on a superficial level. Some encourage their managers to try out new

ideas while others do not show the same kind of support. Unfortunately for most companies,

measuring training effectiveness may not be practiced organization wide. This is because

measuring training effectiveness has never been a policy in most organizations. Lack of

support by most department heads is deterring most organizations from carrying out post-

training evaluation. Most organizations felt that if they had a more supportive top

management they could have established systems for measuring training effectiveness.

52

The most recent study was by Junaidah (2001) on training evaluation practices by training

institutions in Malaysia. The study showed moderately formal training evaluation practices

by Malaysian training practitioners. However, the researcher was uncertain whether these

training practitioners applied the taxonomy of Kirkpatrick model in training evaluation

practices.

Generally, training evaluation practices in Malaysia are either not done or if done, do not

follow any theory suggested in the literature. There is a paucity of detailed evidence of direct

causal links between investment in training and the resultant return in the form of increased

performance. Brandenburg (1982) suggested that part of the reason training practitioners

tended not to conduct evaluation or if they did, they relied heavily on soft information

evaluation methods and did not disseminate the results widely. Pauzi (1985) felt that part of

the problem lies in the attitude of the top management who do not show full commitment to

the evaluation process.

A further study is needed to study current training evaluation practices in Malaysia and to

understand updates of this practice. It is important to understand training effectiveness in

Malaysia as it is worthwhile to analyze the training evaluation process which has undergone

in the country. This study would contribute to the existing body of knowledge as current

information on training evaluation is inadequate. Since a large number of professional

associations, private consultants and management schools in universities are organizing

training programs in Malaysia, the results of the study would indicate areas where training

evaluation could be practiced for different training programs.

2.6 Methodology of Study

Most recent surveys of training and evaluation practices in Malaysia were conducted by

Hamid et al. (1987), Asma (1994), Mirza and Juhary (1995), Shamsuddin (1995) and

Junaidah (2001). The dearth of published materials on training and development activities of

managers in Malaysia has prompted this study.

53

This explorative study was conducted to understand the evaluation culture and the

extensiveness of training evaluation practices in Malaysia. The lack of baseline information

prevented the evaluation of transfer learning. This prompted the use of empirical approach in

this study. The study evaluates the perceptual effects on both management and non-

management levels of training programs in the manufacturing sector. This survey asked the

level of training evaluation performed, the percentage of payroll spent on training, the

impediments to training and the percentage of training transferred to the job. Follow up

interviews were also undertaken to provide additional clarification and interpretation on

responses and enabled impressions and opinions about the data to be recorded accurately.

2.6.1 Questionnaire Construction

A comprehensive survey of the literature was done to find out the degree of training

evaluation being conducted by training practitioners in Malaysia. The survey questions asked

the degree that training evaluation practices were conducted in Malaysia based on

Kirkpatrick's 4-level of evaluation (Kirkpatrick, 1959a, 1959b, 1960a, 1960b, 1976, 1979).

Examples of questions are:-

Reaction How did the participants react to the training?

Learning What information and skills were gained?

Behavior How have participants transferred knowledge and skills to their jobs?

Results What effect has training had on the organization and achievement of

its objectives?

The instrument was designed primarily based on the published work of Blanchard, Thacker

and Way (2000) with modification based on the Malaysian training environment. The

modifications from Blanchard et al. questionnaire include rephrasing and simplifying

question structure to suit local linguistic understanding. Words which were ambiguous or

misunderstood were replaced. These modifications were applied in order to encourage a more

accurate response. Care was taken to ensure that simple and clear questions were used to

54

seek information on significant areas of training evaluation activity in Malaysia. The

questionnaire can be found in Table 4.

The questionnaire is made of 34 questions. There are 8 questions in Level 1, 5 in Level 2, 13

in Level 3 and 8 in Level 4. Level 3 was constructed with the most questions as it asked

about practices for measuring transfer learning. Practitioners could use a variety of

assessment to measure transfer learning hence the survey questions require detailed practices

undertaken by practitioners.

The questions in the questionnaires were randomly sorted to avoid biasness caused by the

order of the questions. The survey questions used a 5-point Liken scale to permit good scale

discrimination.

A panel of experts which consisted of training professionals from the Malaysia Institute of

Management was used to evaluate the items in the questionnaire. Extensive pilot testing was

undertaken by the training professionals to ensure that the questions were easily understood.

The internal consistency was determined using the Cronbach alpha method. The Cronbach

alpha coefficient is 0.8458.

2.6.2 The Sample and Sampling

To improve the effectiveness and efficiency in terms of time and resources, a purposeful

sampling technique was employed. The sample was manufacturing based companies found

in the HRDC Directory. The HRDC Directory listed approximately 5000 organizations but

only 40 percent from the listing are manufacturing based companies. The questionnaires

were sent to 2000 manufacturing based companies with more than 50 employees. The

questionnaires were posted between December 2003 and January 2004. The questionnaires

were addressed to the Personnel and Human Resources Managers of the organizations. A

self-addressed stamped envelope was enclosed to maintain anonymity on the return of the

completed questionnaires through the postal service.

55

2.6.3 Questionnaire Response

The questionnaires were posted to 2000 of manufacturing organizations in Malaysia found in

the HRDC Directory. The appeal highlighted the focus of the study, i.e. training evaluation

activities that relate to the benefits of training.

Of the 2000 questionnaires posted 94 were returned with a note that the organizations were

closed down or had moved to a new address. This reduces the original samples of 2000 to

1906. Reminder notes were sent out three weeks after the first posting in order to encourage

greater response rate. However there were only 109 completed questionnaires returned.

The overall lack of organizational response can be attributed to a variety of causes: low

interest, lack of time to respond, current restructuring of the organization, unavailable contact

person, and outdated addresses.

2.7 Findings and Discussion

Data was analysed using SPSS for XP Windows (Version 13). Statistical significance was

accepted at the 0.05 level of confidence. A total of 5.5 percent of the questionnaires were

returned. Part 1 of the questionnaire gathered information on the background of the

companies. It was found that out of the 109 companies, 46 percent are multinational

companies while 54 percent are Malaysian companies. Part 2 of the questionnaire gathered

information on the organization's commitment to training. The results are shown in Table 1.

56

Table 1. Commitment to Training

A total of 41.3 percent of organizations agreed that a training needs analysis was conducted

prior to conducting any training program. The rest of the organizations conduct training to

meet the needs of the organization such as low productivity or a morale problem, reaction to

a crisis and frequently not coordinated with other functions of the organization. The lack of

baseline information prevents evaluation and no meaningful comparison of the participant's

performance before and after training can occur.

The results indicate that 64.3 percent of organizations organized technical training. A large

number of organizations felt the need to upgrade the technical competence of their employees

in the areas of quality, productivity, product training, IT training, accounting system and job

related training. Of all the organizations interviewed, 65 percent reported that they have

57

Commitment to Training Statistics (n =109)

Does your organization conduct training programsfor employee development

Does your organization conduct training needs analysis beforeconducting any training programs

What type of training is conducted by your organizationManagement

e.g. Leadership, supervisory, managing change,communication, human relations and interpersonal skills

Organization Specifice.g. training programs related to policies, values, cultures,goals and objectives of the whole organization

Technicale.g. quality, productivity, product training, IT training,accounting system and job related training

Personal Improvemente.g. motivation, time management, self development,managing self, presentation skills and businesscommunication skills

Others

Yes = 100 percentNo = 0 percent

Yes = 41.3 percentNo = 58.7 percent

Multinational = 39Malaysian companies = 6

45.9 percent

18.9 percent

64.3 percent

24.2 percent

0 percent

extended their range of products during the last two years and 88 percent had made changes

to machinery and equipment.

Management training was ranked the second at 45.9 percent followed by personal

development at 24.2 percent. One fifth of the organizations are also concerned with

management training. Many feel that skills such as leadership, supervision, managing

change, communication, human relations and interpersonal skills are needed for management

development. Although organization specific training is an emerging area, only about 18.9

percent of the organizations feel the need to impart training in this field.

Table 2 shows the level of evaluation conducted on management and non-management

training by the organization.

Table 2. Training Evaluation Practices in Organization

Training Evaluation Practices in Organization

Level 1 reaction evaluation

Level 2 learning evaluation

Level 3 behavioural evaluation

Level 4 results evaluation

No training evaluation practices

Statistics (n =109)

35 percent

25 percent

16.5 percent

11 percent

12.5 percent

The results indicate that out of 109 companies, 35 percent of organizations conducted Level 1

evaluation by measuring the participant's reactions towards the training program while 25

percent of the organizations conducted Level 2 evaluation by measuring the participant's

degree of learning as the result of the training initiatives. Only 16.5 percent of organizations

conducted Level 3 evaluation by measuring the changes in the participant's behaviour

towards the job after each training program. However, 11 percent of organizations

quantified the results of training and calculated its return on investment in training which is

classified as level 4 evaluation. The remaining 12.5 percent of organizations have never

conducted training evaluation after each training program. The results indicate that more

58

than half of the organizations do not evaluate their training at the behavioural or the results

levels. The reason for this is that sometimes training function is seen as an isolated and

peripheral function, which is not truly integrated into the job setting (Olsen, 1998).

The means and standard deviations of the four levels of training evaluation for all 109

companies are shown in Table 3.

Table 3: Means and Standard Deviations of the Four Levels of Training Evaluation

Note: Likert scale: where 5 = strongly agree; 4 = agree; 3 = neutral; 2 = disagree; 1 = strongly disagree

The majority of the organizations agree that they conduct Level 1 evaluation after each

training program. The average for Level 1 evaluation is 3.63 which suggest that the majority

of organizations conduct Level 1 evaluation. The average score for Level 2 evaluation is

3.41 which indicate that some companies conduct Level 2 evaluation selectively and the

majority is done on technical training. The average for Level 3 evaluation is 3.26. The result

indicates that the degree of measuring behavioral changes in the job after training is not that

popular among these manufacturing organizations. This could be due to the unavailability of

specific tools to measure the subjective changes in behavior. The average score for level 4

evaluation is 2.99 indicating that the majority of these manufacturing organizations do not

conduct result evaluation. The result was further confirmed by an interview which

mentioned that the benefits of training are not easily measured in quantitative terms and most

benefits cannot be measured immediately.

The means and standard deviations of the 34 questions in the instrument for all 109

companies are shown in Table 4.

59

Level Mean + SD

Level 1 3.63 ±0.62

Level 2 3.41 + 0.62

Level 3 3.26 + 0.63

Level 4 2.99 + 0.68

Table 4: Means and Standard Deviations of 34 Questions in the Instrument

60

L evel 1

- eactR ion1. Departmental heads conducted collective opinions from

MeanScore

SD

Evaluation participants with regards to the training program conducted 4.12 0.6892. Evaluate perceptions of participants on key benefits and value

arising from training3.03 0.934

3. Conduct training environmental audit to track participantssatisfaction after training

4.03 0.724

4. Focus on perception of trainees towards the training program. 4.38 0.862

5. Measure trainers competency and credibility after eachtraining program

2.74 0.908

6. Most training programs conduct post course reactionevaluation after training.

3.89 0.715

7. Always make an effort to ask participants whether they enjoyattending the training programs

4.20 0.815

8. Measure the accuracy of the training program in addressingthe exact requirement of the job

2.67 1.021

Level 2

- Learning

1. Allow participants to write down what they have learnedwhich might be useful for their work

3.69 0.641

Evaluation 2. Conduct pen and paper test for measuring the amount ofknowledge gained from a training program

4.28 0.703

3. Administer a test before and after training with regards to theknowledge gained from a training program

3.41 0.912

4. Identify the principles, facts and techniques learned byparticipants

2.98 1.090

5. Participants were asked if there were any barriers preventingthem from using what they have learned

2.69 0.932

Level 3- Behavioral

1. Develop performance-based tests as part of the trainingevaluation

2.89 0.909

Evaluation 2. Assess the level of transfer of learning to the job 3.04 0.994

3. Measure the success rate of participants performing each itemlearned

3.23 0.089

4. Define an action plan for participants and evaluate theimplementation success rate

3.43 0.745

5. Identify specific skill improvement as a result of a trainingprogram

3.93 1.079

6. Measure positive changes in personnel efficiency andeffectiveness after training

3.77 0.931

7. Measure the behavior changes resulting from the trainingprogram

3.51 1.099

8. Organize the trainer's follow up session to track theparticipant's behavioral change after training

3.28 1.141

9. Use observation techniques to monitor changes of behaviorand attitudes resulting from the training program

2.62 1.062

10. Conduct work performance evaluation in the workplace aftertraining

2.71 0.703

11. Observing and documenting the practice of knowledge andskills learned by the trainee into the workplace.

3.32 0.773

12. Assess the increase in knowledge and skills as well as attitudechange of trainees

2.84 0.842

13. Conduct a preview session with your trainee to specify theexpected objectives to achieve from the training

3.79 0.952

Note: Likert scale: where 5 = strongly agree; 4 = agree; 3 = neutral; 2 = disagree; 1 = strongly disagree

The results indicate that Level 1 evaluation (reaction) seems to be the most significant

training evaluation practice. A high mean score of 4.38 indicates that the majority of

Malaysian manufacturing companies focus on the perception of trainees towards the training

program. Managers do play an active role in conducting Level 1 evaluation by collecting

opinions from participants with regards to the training program conducted. Measuring the

accuracy of a training program in addressing the exact requirement of the job is the least

practiced and is indicated by a low mean score of 2.67.

The practice of pre and post pen and paper test after a training program is most popularly

practiced by these manufacturing companies and is shown in the mean score of 4.28. The

lowest mean score for Level 2 evaluation was 2.69 indicates that organizations seldom ask

participants if there were any barriers which prevented them from using what they have

learned.

Level 3 evaluation is modestly practiced by manufacturing companies in Malaysia. The

highest mean score of 3.93 indicates that the majority of these manufacturing companies

identified specific skill improvement as a result of a training program. The use of

observation techniques to monitor changes of attitude and behaviour as a result of the training

program shows the lowest mean score of 2.62.

The apparent lack of practice in Level 4 evaluation (result) is probably due to the effort and

potential complexities involved which entails much more work. This is reflected in the

61

Level 4- Results

Evaluation

1. Measure the level of productivity before and after a trainingprogram

2.56 0.721

2. Link effectiveness of training to financial benefit 2.91 0.668

3. Conduct cost-benefit analysis on training programs conducted 3.10 0.711

4. Measuring the worthiness of attending training in terms ofcost and time away from work

3.35 0.823

5. Measure the tangible cost in terms of reduced cost andimproved quality after training

2.82 0.913

6. Calculate the cost of training and its impact towardsorganization improvements

2.71 0.793

7. Compare the cost of training program with benefits obtainedfrom it

3.24 0.894

8. Finding evidence of direct links between training investmentand returns from training

3.18 0.615

survey result which indicates low interest in conducting cost-benefit analysis of training by

these organizations. Measuring the worthiness of attending training in terms of cost and time

away from work showed a mean score of 3.35. This is regarded as one of the most popular

practice of Level 4 evaluation by these organizations. Calculating the costs of training and its

impact towards organization improvements showed the lowest mean score of 2.71.

Independent t-tests were used to test for significant difference in the four levels of training

evaluation conducted by multinational and Malaysian companies. It was found that there

were significant differences between training evaluation at Level 1, Level 2, Level 3 and

Level 4 between multinational companies (N=50) and Malaysian companies (N=59) at p <

0.05. See Table 5.

Table 5: Summary oft-tests of the four levels of training for multinational companiesand Malaysian companies

*p <0.05

The results indicate that the majority of multinational companies operating in Malaysia have

a clearer objective of what ought to be done and have enshrined this in mission statements on

training. These multinational companies provide training and development for all employees

in all areas of operations with expensive investment and serious attempts to produce a

competent and quality workforce. The results show that multinational companies judge

training effectiveness as their immediate reaction to training evaluation. These multinational

companies applied formal and systematic procedures and processes to assess training

effectiveness as compared to Malaysian companies.

The results show that the majority of Malaysian companies did not conduct Level 3 and

Level 4 evaluations. Most Malaysian companies seem to lack the formal mechanism to

62

Level 1 Level 2 Level 3 Level 4Company (Mean + SD) (Mean + SD) (Mean + SD) (Mean + SD)

Multinational 3.78 + 0.52 3.69 + 0.56 3.63 + 0.48 3.50 + 0.53

Malaysian 3.49 + 0.67 3.17 + 0.57 2.94 + 0.56 2.56 + 0.46

t-value 2.635 * 4.758 * 6.794 * 9.838 *

assess training effectiveness. The results of the t-tests were further confirmed by interviews

which suggested relatively mild commitment of top management to training and some

resistance by middle management to the function of training in Malaysian companies.

Training seems to be a low priority area and training evaluation is conducted on an ad hoc

basis. Part of this could be because identifying individual performance improvement after

training is regarded as a tedious and lengthy process.

Only six Malaysian companies have conducted training needs analysis prior to conducting

training. The result was further confirmed by interview which mentioned at times managers

send employees to training programs just to fill the quota. These employees are not the

intended participants of the training program and would return without learning much.

Since they do not have much commitment for learning after training, it does not permit Level

2, Level 3 or Level 4 evaluation to take place. This trend is shown in less Malaysian

companies practising Level 2, Level 3 and Level 4 evaluation as compared to multinational

companies.

2.8 Limitations of Study

The number of respondents was relatively low. Even though the majority of manufacturing

companies that have more than 50 employees registered with the Human Resource

Development Council, the actual number of organizations that actively participate in training

and development is rather low.

Out of the 2000 manufacturing based organizations listed in the EIRDC Directory, less than

40 percent of them conducted at least one training program in a year (HRDC, 2003). Details

of companies that do not participate in training were not disclosed by HRDC. The reason is

because HRDC does not want training providers to seek for organizations with high unused

funds. The survey was decided to send to all 2000 manufacturing based organizations as the

details of the organizations that do not conduct training program were not known. Hence, the

majority of these organizations that do not conduct training could not answer the

questionnaire.

63

The success of the study depends on the willingness of respondents to cooperate. Some may

not see the value in participation while others may view the topic as sensitive or irrelevant.

Despite reminder notes were sent out three weeks after the first posting to encourage greater

response rate. A comparison between respondents and non-respondents would have been

helpful. Unfortunately, data were not available for making such comparisons in this study.

2.9 Conclusion

Kirkpatrick model has been considered one of the most prominent models of evaluation

practised in Malaysia. The application of the 4-level of evaluation in Malaysia is not well

adopted. This study reveals that training evaluation carried out by most organizations in

Malaysia is mainly to judge trainees' reactions. A culture of fill in one of this before you go

typically pervades in training evaluation. Most organizations lack the formal and systematic

mechanisms to assess training effectiveness. Many companies remain blissfully unaware of

how much they spend on training, whether it is effective or not. Indeed, even the use of

expensive external trainers does not appear to trigger detailed evaluations.

The majority of Malaysian organizations show little or no interest in conducting training

evaluation and have even less interest in the results of evaluation as method of evaluating

effectiveness. Some find evaluation difficult as it is almost impossible to determine which

participant efforts are attributable to training and which are not.

Although Kirkpatrick model of evaluation serves as an outcome of training, most

practitioners do not know what evaluation criteria to look for. The confusion of the actual

outcome possibly hindered the ability to conduct Level 3 and Level 4 evaluations

meaningfully.

Hence this research gap shows the opportunity to examine specific outcome required from

training and the transfer component of training in detail. This study will determine what

strategies might be most helpful in maximizing the transfer learning and constructing an

appropriate model for evaluation.

64

2,10 References for Paper Two

Ahmad, R.H. 1998, 'Educational development and reformation in Malaysia: past, presentand future' Journal of Educational Administration, vol. 36, no. 5, pp. 462-475.

Attkinson, C. C., Sorenson, J.E., Hargreaves, W.A. & Hororwitz, M.J. 1978,Evaluation of Human Service Programs, Academic Press, London.

Arthur Anderson & Co. 1991, Professional Services in Malaysia, Arthur Anderson & Co.,Kuala Lumpur, Malaysia.

Asma, A. 1994, Training design development: The practice of four development agencies inMalaysia, Unpublished Ph. D. dissertation, University Pertanian Malaysia, Serdang.

Barron, T. 1996, 'A new wave in training funding', Training and DevelopmentJournal, vol. 50, no. 5, pp. 28-32.

Bernthal, P. R. 1995, 'Evaluation that goes the distance', Training and Development',vol. 49, no. 9, pp. 41-49.

Blanchard, P.N., Thacker, J.W. & Way, S.A. 2000, 'Training evaluation: perspectives andevidence from Canada', International Journal of Training and Development, vol. 4,no.4, pp. 295-303.

Bramley, P. 1996, Evaluating Training Effectiveness, McGraw-Hill, Maidenhead and NewYork.

Bramley, P. & Kitson, B. 1994, 'Evaluating training against business criteria', Journal ofEuropean Industrial Training, vol. 18, no.1, pp. 10-14.

Brandenburg, D.C. 1982, 'Training evaluation: what's the current status', Training andDevelopment Journal, vol. 36, pp. 28-29.

Brinkerhoff, O.R. 1988, 'An integrated evaluation model of HRD', Training andDevelopment Journal, vol. 42, no. 2, pp. 66-8.

Carnevale, A. P. & Schulz, E.R. 1990, 'Return on investment: according to training',Training and Development Journal, vol. 44, no. 7, pp. 1-32.

Chen, H.T. & Rossi, P.H. 1992, Using Theory to Improve Program andPolicy Evaluations, Greenwood Press, Westport, CT.

Davidove, A.E. & Schroeder, P.A. 1992, 'Demonstrating ROI of training',Training and Development Journal, vol. 46, no. 8, pp. 70-71.

Erickson, M.R.C. 1991, Business-education partnerships: a study of evaluationmethods, Doctorial dissertation, the George Washington University, dissertationabstracts international, vol. 52/07A, AAC9133008.

65

Easterby-Smith, M. 1985, 'Training course evaluation from an end to a means',Personnel Management, vol. April, pp. 25-27.

Gutek, S.P. 1988,'Training program evaluation: an investigation of perceptions andpractice in non-manufacturing business organizations', doctoral dissertation, WesternMichigan University, Kalamazoo, MI, dissertation abstracts international, vol. 49/05a,AA C8811388.

Groove, E.A. & Ostroff, C. 1990, 'Program evaluation', in Developing Human Resource, edsWexley, K. & Himicks, J., BNA Books, Washington D.C.

Heneman, H.G. & Schurab, D.P. 1986, Human Resource Management, Irwin, Illinois.

Hamblin, A.C. 1974, Evaluation and Control of Training, McGraw-Hill, New York.

Hamid, A.A., Mohd, S., Muhamad, A.H. & Ismail, Z. 1987, Management Education inMalaysia, in Developing managers in Asia, eds Tan Jing Hee & You Poh Seng,Addison-Wesley, Singapore.

Human Resource Development Council 1992, Human Resource Development Act 1992,Ministry of Human Resource, Kuala Lumpur, Malaysia.

Human Resource Development Council 2003, Ministry of Human Resource, Kuala Lumpur,Malaysia.

Junaidah, H. 2001, 'Training evaluation: clients' role', Journal of European IndustrialTraining, vol. 25, no. 7, pp. 374-379.

Kirkpatrick, D.L. 1959a, 'Techniques for evaluating training programs: part 1 - reaction',Journal of American Society for Training and Developing, vol. 13, pp. 3-9,

Kirkpatrick, D.L. 1959b, 'Techniques for evaluating training programs: part 2 - learning',Journal of American Society for Training and Developing, vol. 13, no. 12, pp. 21-26.

Kirkpatrick, D.L. 1960a, 'Techniques for evaluating training programs: part 3- behaviour',Journal of American Society for Training and Developing, vol. 14, no. 1, pp. 13-18.

Kirkpatrick, D.L. 1960b, 'Techniques for evaluating training programs: part 4 - results',Journal of American Society for Training and Developing, vol. 14, no. 2, pp. 28-32.

Kirkpatrick, D.L. 1976, Evaluation of Training, Training and Development Handbook: Aguide to human resource development, 2nd edn, Craig, R.L.O., McGraw-HillPublisher, New York.

Kirkpatrick, D.L. 1979, 'Techniques for evaluating training programs', Training andDevelopment Journal, vol. 33, pp. 78-92.

Malaysia Economic Planning Unit 1996, Seventh Malaysia Plan: 1996-2000, GovernmentPrinter, Kuala Lumpur, Malaysia.

66

Maimunah, I. 1990, Extension: Implication to Community Development, 2nd ed, DewanBahasa and Pustaka, Kuala Lumpur, Malaysia

Mirza, S.S. & Juhary, H.A. 1995, Managerial training and development in Malaysia,Malaysian Institute of Management, Malaysia.

Olsen, J.H. 1998, 'The evaluation and enhancement of training transfer', InternationalJournal of Training and Development, vol. 2, no. 1, pp 61-75.

Pauzi, M. 1985, 'Training nuisance, 12th ARTDO International Conference', Petaling Jaya,Malaysia, 22-27 July.

Phillips, J.J. 1991, Handbook of Training Evaluation and Measurement Methods, GulfPublishing Company, Houston, TX.

Pillai, P. 1994, Industrial Training in Malaysia: Challenge and Response, ISIS Publication,Setiakawan Printers Sdn Bhd, Malaysia.

Pillai, P. & Othman, R. 1994, 'Learning to work, working to learn', Institute of Strategic andInstitutional Studies, Kuala Lumpur.

Plant, R.A. & Ryan, R.J.1994, 'Who is evaluating training?', Journal of European IndustrialTraining, vol. 18, no. 5, pp. 27-30.

Rowe, C. 1992, 'How useful was it? The problem of evaluating in-house training programs',Industrial and Commercial Training, vol. 24, no. 7, pp. 14-18.

Rossi, P.H. & Freeman, H.E. 1993, Evaluation: A Systematic Approach, 5th edn, SagePublications, California.

Ruthman, L. & Mowbray, G. 1983, Understanding Program Evaluation, Sage Publication,London.

Saiyadain, M.S. 1995, 'Perceptions of sponsoring managers, training organizations, and topmanagement attitude toward training', Malaysian Management Review, vol. 30, no. 4,pp. 69-74.

Shadish, W.R. & Epstein, R. 1987 'Patterns of program evaluation practice among membersof the evaluation research society and evaluation network', Evaluation Review, vol.11, no. 5, pp. 555-590.

Shamsuddin, A. 1995, 'Contextual factors associated with evaluation practices of selectedadult and continuing education providers in Malaysia', unpublished PhD dissertation;University of Georgia, Athens, G.A.

Shelton, S. & Alliger, G. 1993, 'Who's afraid of level of evaluation?', Training andDevelopment Journal, vol. 47, no. 6, pp. 43-46.

Smith, A. 1990, 'Evaluation of management training subjectivity and the individual', Journalof European Industrial Training, vol. 14, no. 1, pp. 12-15.

67

Smith, A.J. & Piper, J.A. 1990, 'The tailor-made training maze: a practitioner's guide toevaluation', Journal of European Industrial Training, vol. 14, no. 8, pp. 2-24.

Wagel, H.W. 1977, 'Evaluating management development and training programmes',Personnel Management, vol. 54, no. 4.

68

2.11 Appendix A The Questionnaire for Research Paper Two

This survey is about .... grainituj gAlat.t.k. 9...N.4 4. aZatatt44.

Enormous resources of time, money, and energy are invested in every imaginable kind oftraining and development program. Little effort is invested in discovering the how well thoseprocess work, how they might be improved or, indeed, if they work at all. It is important fororganization that uses training and development activities to seek practical ways ofevaluating those activities.

With greater emphasis by the Ministry of Human Resources since the enactment of HumanResources Development Act, 1992, there is a need to improve the effectiveness of trainingactivities in Malaysia in order to achieve greater productivity among the workforce.However, effective evaluation requires the examination of training outcomes at several levelsof evaluation. This research study is designed to study to what extent the Malaysianmanufacturing sectors have carried out training evaluation and how these organizations havebenefited from the training event. The information you provide will help us betterunderstand the quality and effectiveness of training evaluation system that has so far beingcarried out within the Malaysian context.

Because you are the one who can give us a correct picture of how you experience conductingtraining evaluation, I wish to invite you to participate in this research study. The resultswill be presented in an aggregate and untraceable manner.

If you have any enquiry about this research or the questionnaire, feel free to contact me, LimGuan Chong, at No. 54, Jalan SS2167, 47300 Petaling Jaya, Selangor Darul Ehsan, or my cellphone 019-4781553, or my e-mail cipdlgc@tm.net.my. You can also contact my supervisors,to verify this survey and my doctoral candidateship: Dr. Travis Kemp (e-mail:travis@teleran.com.au) or Professor Dr. Leo Ann Mean (e-mail: drleo@itd.com.my).

Part 1: Tell us about your Organization

Name of organization:

Type of company:

Multinational

Malaysian companies

Nature of business:

Manufacturing

Service

Others, please specify

69

Part 2: Commitment to Training

Do your organization conduct training program (in house training program, public programand on-the-job training) for employees development?

Yes

No

Do your organization conduct training needs analysis before conducting any trainingprograms?

Yes

No

What types of training programs conducted by your organization?

Management (e.g. leadership, supervisory, managing change, human relation andinterpersonal skills, communication)

Organizational specific (e.g: training programs related to whole organization policies, values,culture, goals and objectives)

Technical (e.g. quality, productivity, product training, IT training, accounting system and jobrelated training)

Personal Improvement (e.g. motivation, time management, self development, managing self,presentation skills and business communication skills)

Others. Please specify

Part 3: Training Evaluation Practices

Instructions: Please indicate your agreement and disagreement that truly represents the practice inyour organization on a scale of 5 (strongly agree), 4 (agree), 3 (neutral), 2 (disagree) to 1(strongly disagree), to express your view.

Training Evaluation Practices

1 Most training programs conductpost course reaction evaluationafter training

2 Always make an effort to askparticipants whether they enjoyattending the training programs

70

StronglyAgree

Agree Neutral Disagree StronglyDisagree

5

5

4

4

3

3

2

2

1

1

Training Evaluation Practices

3 Departmental heads conductedcollective opinions from participantswith regards to the training programconducted.

4 Participants were asked if therewere any barriers preventing themfrom using what they have learned

5 Allow participants to write down whatthey have learned which might be usefulfor their work

6 Define an action plan for participantsand evaluate the implementationsuccess rate

7 Conduct pen and paper test formeasuring the amount of knowledgegained from a training program

8 Administer a test before and aftertraining with regards to the knowledgegained from a training program.

9 Develop performance-based testsas part of the training evaluation

10 Identify specific skill improvementas a result of a training program

11 Measure positive changes in personnelpersonnel efficiency and effectivenessafter training

12 Measure the behaviour changesresulting from the training program

13 Conduct a preview session with yourtrainee to specify the expectedobjectives to achieve from the training

14 Organize the trainer's follow upsession to track the participant'sbehavioural change after training

15 Measuring the worthiness ofattending training in terms of costand time away from work

71

StronglyAgree

Agree Neutral Disagree StronglyDisagree

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

Training Evaluation Practices

16 Use observation techniques tomonitor changes of behaviour andattitudes resulting from the trainingprogram.

17 Measure the level of productivitybefore and after a training program

18 Link effectiveness of training tofinancial benefit

19 Conduct cost-benefit analysis ontraining programs conducted

20 Evaluate perceptions of participantson key benefits and value arisingfrom training

21 Identify the principles, facts andtechniques learned by participants

22 Measure the tangible cost in termsof reduced costs and improvedquality after training

23 Measure the accuracy of the trainingprogram in addressing the exactrequirement of the job

24 Measure the success rate ofparticipants performing eachitem learned

25 Conduct training environmentalaudit to track participantssatisfaction after training

26 Measure productivity improvementafter each training

27 Calculate the cost of training and itsimpact towards organizationimprovement

28 Conduct work performanceevaluation in the workplaceafter training

29 Measure focus on perceptionof trainees towards thetraining program

72

StronglyAgree

Agree Neutral Disagree StronglyDisagree

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

Training Evaluation Practices

30 Assess the increase in knowledgeand skills as well as attitude changeof trainees

31 Compare the cost of training programwith benefits obtained from it

32 Observing and documenting thepractice of knowledge and skillslearned by the trainee into the workplace.

33 Assess the level of transfer oflearning to the job

34 Measure trainers competency andcredibility after each training program

35 Finding evidence of direct linksbetween training investment andreturns from training.

Thank you for your participation!

73

StronglyAgree

Agree Neutral Disagree StronglyDisagree

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

5 4 3 2 1

Research Paper 3

MULTI-RATER FEEDBACK FOR TRAININGAND DEVELOPMENT: AN INTEGRATED

PERSPECTIVE

Lim Guan ChongMaster of Business Administration (Finance)

University of Hull

International Graduate School of ManagementUniversity of South Australia

74

Multi-Rater Feedback For Training AndDevelopment: An Integrated Perspective

Lim Guan ChongInternational Graduate School of Management

University of South Australia

3.1 Abstract

This paper looks at the difference between success and failure of multi-rater feedback in

enhancing employee self awareness and encouraging them to engage in development

programs. Multi-rater feedback is basically used as an unfreezing process in which

employees are motivated to rethink their behaviour and its impact on others. Multi-rater

feedback provides employees with good data from multiple perspectives, encouraging

openness to listen and accept self weaknesses for development. A comprehensive integral

model encompassing process consultation and good conversation is used to facilitate

effective development after multi-rater feedback. Process consultation provides a description

of prescriptive approach to help the employees recognize and accept responsibility for the

difference in perception. The flexibility of process consultation should be enhanced by

integrating good conversation to promote ideal communication and interaction between the

process consultant and employees which will eventually build trust for open learning and

development. Post multi-rater feedback is introduced to assess the degree of performance

improvement which resulted from the development program.

3.2 Introduction

Multi-rater or 360-degree feedback has gained wide acceptance and usage to support

development of leadership and management skills (Cacioppe & Albrecht, 2000). The multi-

rater feedback process provides a comprehensive feedback collected from people around

ratees, in the workplace. One underlying rationale to such system is its potential impact on

75

the target ratee's self awareness: increasing self awareness is thought to enhance development

(Ashford, 1993; Mount, Judge, Scullen, Sytsma & Hezlett 1998). According to Van Veslor

et al. (1993), the number of multi-rater feedback instruments has increased significantly in

the past 15 years. It is estimated that American companies spent $152 million in 1992 on this

form of feedback for development (Hoffman, 1995). Multi-rater feedback was first

introduced to the UK in the early 1990s, and has spread quickly across public and private

sector organizations (Fletcher & Baldry, 2000). The spread is based on the perceived benefits

of fairer and greater accuracy in representing a performance, which creates development and

learning potential that can consequently motivate changes in behaviour. In review of 20

organizations responding to the delivery of multi-rater feedback, London and Smither (1995)

found that 40 percent of the respondents always linked multi-rater feedback to specific

developmental activity. According to Moses, Hollenbeck and Sorcer (1993), there does not

appear to be a distinct individual who founded or invented the multi-rater feedback process.

They argued that the term multi-rater feedback has been mistaken to be a newly discovered

concept, as perceptions of people have been available as long as there were people around to

observe them.

3.3 The Use of Multi-rater Feedback

The implementation of multi-rater feedback varies among organizations. The widespread

adoption of multi-rater feedback and other multi-source feedback is based on the perceived

benefits of fairer and greater accuracy in representing performance because it offers a more

rounded assessment of the individual, not just the top-down perspective of conventional

appraisal. It is an empowering mechanism, which allows subordinates to exert some

influence over the way they are managed. The same is true for peers, who can think back and

improve a colleague's role perform as a team member. Moreover, the multi-rater feedback

system provides a natural method for both enhancing learning and improving performance.

As the complexity of job function increases in the workplace, it is crucial for employees to

receive feedback from a variety of constituencies and not only the traditional superior-

subordinate appraisal approach. This feedback facilitates self awareness by enabling

participants to compare their own perceptions of their skills and personal style with the

perceptions of important observers in their work environment. Multi-rater systems are

76

assumed to improve performance by increasing selfawareness through diversified

information from multi-rater feedback (Borman, 1997). Ratees who receive feedback or

appraisal on their performance from a variety of sources will be responsible to improve

current job performance through continuing to add value to the organization needs and be

prepared for the future.

Continuing development has been the key priority for most organizations in keeping the

workforce updated with on-going technological changes. Measuring and improving worker

performance has become increasingly important for organizations to stay competitive.

According to Nowack (1993), the increased use of multi-rater feedback in organizations was

mainly due to the increasing need for continuous measurement of improvement efforts; the

need for job-related feedback for employees affected by career plateauing; and the need to

maximize employee potential in the face of technological changes, competitive challenges

and increased workforce diversity.

Multi-rater feedback has also been seen to increase reliability, fairness and acceptance of the

data by the person being rated (London, Wohlers & Gallagher, 1990). This is because

feedback is received from multiple sources and not just from one. A study conducted in an

American company showed that only 3.9 percent of staff felt that feedback should come

solely from the superior, while 94.8 percent felt that feedback should come from both

superior and co-workers (Cacioppe & Albrecht, 2000). This result indicates that there is an

on-going trend in American companies to have performance feedback from multiple sources

despite the fact that there may always be variation in terms of perceptual differences between

the self and others on the feedback results. Although multi-rater feedback system provides

the ratee with greater information as a base for development, Tornow and London (1998)

suggested that multiple feedback sources require balancing, as multiple sources may

potentially offer conflicting viewpoints to the ratee. However, the differences in perspective

between the rater and the ratee should not be treated as an assessment error. This is further

supported by Ashford (1993) who found that multi-rater feedback is important to the ratee

because the information could further stimulate the ratee's cognitive reactions that would

likely give impact to subsequent behavioral changes. A multiple feedback system is a source

of information, which can enhance personal learning by providing the opportunity to ratees

who are being assessed to compare their self-perceptions against the perceptions of others

77

regarding their behaviour. Multi-rater feedback is simply a set of performance-related

information which is essential for learning and development.

Bennis, Benne and Chin (1969) is of the view that multi-rater feedback is a critical element in

affecting change in performance evaluation. According to Zemke and Zemke (1995), adults

undertake learning experiences when they see a need to acquire a new or different skill or

knowledge. Multi-rater feedback provides the opportunity for open communication between

rater and ratee to discuss on ratee's past behaviour and weaknesses, encouraging openness to

hearing and accepting feedback. Such feedback and open communication is instrumental for

an unfreezing process which ratee is motivated to rethink back his or her weaknesses and

strengths (Shipper & John, 1992). McCauley and Moxley (1996) also viewed multi-rater

feedback as an instrument in an unfreezing process, in which ratees will have a chance to re-

think their previous and current behaviour based on the discrepancies of the results between

self and others and how their weaknesses would create impact on others. Hence, conducting

multi-rater feedback before and after training provides an avenue for the training provider to

evaluate performance changes. This shows that the feedback received can be used both as

reinforcement of past learning and also an opportunity for future learning (Rosti & Shipper,

1998).

Evidence from various settings has demonstrated an association between self awareness and

performance outcome (Fletcher, 1997). It has been found that high self awareness is related

with high performance ratings in various aspects (Atwater, Ostroff, Yammarino & Fleenor,

1998; Bass & Yammarino, 1991; Furnham & Stringfield, 1994). Nasby (1989) is of the

opinion that ratees with high self awareness are more able to integrate various feedbacks into

self-perception in order to reach a higher performance outcome. Ashford (1984) found that

people with low self awareness are more likely to ignore or discount feedback about them

and will have a negative attitude towards work. This indicates that a highly self-aware ratee

is likely to exert self-positivism to accept feedback and show self-motivation for

improvement (Fletcher & Bailey, 2003).

However, according to Greguras, Ford and Brutus (2003), further research needs to be

conducted to investigate the effectiveness of multi-rater feedback systems in increasing one's

self awareness. This will lead to eventual improvement in one's performance as the ratee

may react differently to different source of information they received. Information obtained

78

from different sources would have different affects on the ratee's self awareness. If the

feedback proves to be true, how would the ratee react to this perceptual reality. Conway and

Huffcutt (1997) commented that although different raters present different information, not

much research has explored how the ratee attends to, integrates, and uses the information

from the various raters. Although multi-rater feedback system increases one's self awareness

and leads to further improvement, a further specific mechanism needs to be included

(Hazucha, Hezlett & Schneider, 1993; Reilly, Smither & Jasilopoulos, 1996; Walker &

Smither, 1999). The specific mechanism refers to identification of appropriate personality

traits, skills or competency needed by the ratees; establishing an appropriate feedback rating

approach and selection of relevant raters. This proposed mechanism must be embedded

within the feedback system prior to the implementation. This will increase ratee readiness in

accepting the multi-rater feedback and lead to individual self awareness for further

improvement. Many studies have been conducted to discover specific interventions of the

multi-rater feedback mechanism.

3.4 The Effectiveness of Multi-rater Feedback for Development

Fletcher and Bailey (2003) found that multi-rater feedback provides the opportunity for the

ratee and raters to agree on the level of competence that is needed. Church (1997) also

supports this view and suggested that multi-rater feedback provides both the rater and ratee

with the opportunity to agree on the development needs ofa required performance standard,

competency and skills necessary for the ratee. Both the rater and ratee would have an

opportunity to clarify their respective expectations in order to develop a psychological

contract or agreement. The ratee would be more focus on what is needed by people working

around them and raters would have a better understanding of the ratee's strengths and

weaknesses. In certain multi-rater feedback systems this is known as a gap analysis process

and other multi-rater literature refer to it as congruence-d (Warr & Bourne, 1999). Edward

(1993, 1994) stated that d is the score difference score between the ratee and other raters.

However, Fletcher and Bailey (2003) commented that telling a ratee d score is of no use

unless the rater can provide specific and meaningful information to reduce the gap between

ratee's and raters' scores. Congruence-d is obtained by subtracting the average score of the

other raters from the self-rating for each feedback questionnaire item, and dividing that value

79

with the standard deviation of raters and ratee's scores (Warr & Bourne, 1999). The level of

self awareness is signified by the d score. If the d score is equal to 0, this signifies complete

agreement between the self and the others rating on all items. Disagreement between ratee's

and raters' ratings generally showed low correlations between-source ratings. This showed

that different rater sources actually provide different information (Conway and Huffcutt,

1997; Harris and Schaubroeck, 1988). Ashford (1993) and Brutus, London and Martineau

1999) conducted studies on relative impact of different rater sources showed raters have

different implications on development of the ratee. The studies discovered that subordinate

ratings had the largest impact on goal selection, followed by peers and superior. This shows

that the selection of information from different rater source is important for ratees to decide

which rater source is most qualified and which feedback is important for further improvement

(Kluger & Denisi, 2000). Mount (1984) supports the validity of subordinates rating and

indicated that the majority of ratees show approval for subordinate ratings for developmental

purposes (Bernardin, Dahmus & Redmon, 1993; Facteau et al., 1997, 1998; London et al.,

1990; McEvoy, 1990).

Further study by Greguras, Ford and Brutus (2003) on 213 managers using a policy capturing

design that allowed factors (i.e. lead others, general administrative performance, building

working relationship and overall performance) to be manipulated. The study showed that

superior ratings would be weighted more heavily than peer or subordinate ratings for the

ability to lead others, general administrative performance, building working relationship and

overall performance. Ratees will attend more to peer ratings than subordinate ratings for

general administration of roles and responsibilities because peers are more likely to

understand the ratee's duties, which are similar to their own. However, a study by Atwater,

Roush and Fischthel (1995) showed that the ratee attend more to subordinate ratings as

compared to peer ratings for the ability to lead others as subordinates have first-hand

experience with the ratee's leadership behaviour. Selection of feedback information is tied

closely to the ratee's perception. Needless to say, ratee development success is closely related

to the ratee's perception of the source of information. User should consider whether multi-

rater is best used for development or only provides different dimensions of reference for the

ratees.

80

3.5 The Effectiveness of Multi-rater Feedback for Appraisal

There are debates on the use of multi-rater feedback for appraisal and development (Bracken,

Dalton, Jaka, McCauley & Pollman, 1997). According to London et al. (1990) and Antonioni

(1994), respondents will answer questions differently if it is for appraisal purposes. A study

by London and Smither (1995) showed that 40 percent of people who provided multi-rater

feedback ratings said they would have altered those ratings if the company planned to use

them for evaluation or appraisal. According to McEvoy and Buller (1987), ratees view the

process as most useful when uses for development as apposed to appraisal. London and

Beatty (1993) found evidence to support this. They reported that 34 percent of the

respondents in their study would rate their superior differently if the feedback were shared

with their superior. Hence, there is still an element of fear for individuals to appraise their

superiors honestly. Further studies should be carried out to determine the capability of

multi-rater feedback that is used in performance appraisal. Few researchers agree that multi-

rater is useful solely for developmental purposes as it is also widely used in managerial and

leadership development programs (Cacioppe, 1998; Cacioppe and Albrecht, 2000; Garavan,

Morley & Flynn, 1997; McCauley & Moxley, 1996; Thach, 2002). O'Reilly (1994)

suggested that when multi-rater feedback is used for development purposes, scores from

raters do not vary much. However, this was not the case for formal performance appraisals.

3.6 The Variation of Multi-rater Feedback Information

A study by Kluger and DeNisi (1996) on the effectiveness of multi-rater feedback

interventions showed that only one-third actually yielded positive improvements in

performance. There is an urgent need to take a closer look at the effectiveness of multi-rater

feedback in performance development. Feedback is invaluable to ratee as it comes from

multiple sources, and provides multiple perspectives. Each opinion or perspective may

provide relevant yet different feedback for the ratee to focus upon (Atwater & Yammarion,

1993; Hazucha et al., 1993; Tornow, 1993). Ghorpade (2000) commented that having more

information does not necessarily mean a higher accuracy rate and information provided by

just a superior does not mean it is not impartial. If the source does not have an opportunity to

observe the ratee's behaviour, or does not recognize the requirements of a particular

81

performance dimension, feedback from the source may be inaccurate for the ratee's

development. Therefore the quality of ratings from different sources for a particular

dimension should be assessed (Kluger & DeNisi, 1996). London and Smither (1995) stated

that ratings provided by different raters are likely to be inconsistent because it may create

much confusion and disagreement on the results and may not increase future development.

According to Moses et al. (1993) multi-rater feedback relies solely on the instrument scoring

system or data collection methods to interpret the information for ratees. Moses et al. (1993)

argued that multi-rater feedback is based on people's observations and the observer may not

know what behaviour to look for. If the primary purpose of multi-rater feedback is to

identify developmental opportunities, then a set of competent performance behaviours has to

be identified and communicated to all raters prior to the process. This would enable the rater

to understand the required habits, behaviors or styles so that a proper and fair judgment

towards the ratee's performance is ensured. The rater's feedback is important and may have

an impact on the ratee's subsequent developmental priorities.

The rater's feedback such as perception bias, cultural issues and gender should also be given

special attention (Cacioppe & Albrecht, 2000). An example of perception bias is a man will

show better leadership than a woman. A study of three organizations, with a total of over

20,000 employees, showed that there was a positive correlation between performance and age

until the age of 45 (Cacioppe & Albrecht, 2000). The study indicates that raters are likely to

stereotype younger ratees as performing better than older ratees, or older ratees may have

better experience compared to younger ratees. However, by looking at the cultural

dimension, Leslie, Gryskiewicz and Dalton (1998) argued that multi-rater feedback might not

necessarily be well accepted by cultures in certain countries. Some cultures do not subscribe

to the same notion that feedback is valuable and can guide manager development. For

instance, cultures such as the French may place more value on lineage or social class than

developing managers. Different cultures may find it a shock to be asked personal

information regarding their superiors. American managers find difficult to get those that

report directly to them to give negative feedback (Wilson et al., 1996). Another example is

Asian value of face-saving where a request for information needed in a multi-rater feedback

may come across as offensive (Wilson et al., 1996). An organization that wishes to conduct

multi-rater feedback needs to take a closer look at the age, culture and genders of the raters or

ratee as these may affect the effectiveness of multi-rater feedback process.

82

Honey and Mumford (1982) reflected that in the event of self assessment, most managers are

poor reflectors. They prefer to charge on with new ideas rather than look backwards and

reflect on how things might have gone better. However, according to Waldman, Atwater and

Antonian (1998) individuals who rated themselves higher are likely to have higher self-

esteem and self-concept. Disagreement over the result could be a threat to the ratee's self

esteem and weaken their motivation for further development. Special caution need to be

taken in designing multi-rater feedback in order to minimize the potential of ratee being

pessimistic and to ensure that ratee's self-image is converted to productive behavioural

change (Wood, Allen, Pillenger & Kahn, 1999). The feedback process should be designed as

a tool to ensure effective interpretation of information received from multi-rater feedback to

stimulate individual and organization improvement in attaining strategic business objectives

(Heisler, 1996). Information from multi-rater feedback is mainly used for developing people

but increasingly, it is being used for strategic planning in training and development (Romano,

1994; Atwater et al., 1993). A research conducted with 48,000 participants indicated that

multi-rater feedback could successfully contribute to the effectiveness of training and

development (Cacioppe & Albrecht, 2000).

3.7 Multi-rater Feedback Practices in Malaysia

In the Malaysian training environment, multi-rater feedback could be used as one of the

assessment models for training and development. Al imo-Metcalf (1998) commented that

multi-rater feedback should only be used in the context of assessment for development.

Payne (1998) supported the view that multi-rater feedback could be a potentially powerful

and even dangerous tool. Therefore it should be confined to the developmental arena and

used by people who know what they are doing.

Training evaluation and assessment practices in Malaysia are still considered at an

elementary stage. A study conducted by Zakaria and Rodzhan (1993) on 94 manufacturing

and service organizations in Malaysia found that only 44 percent of respondent organizations

conducted formal training. Of those who conducted formal training, 23 per cent did not

conduct any training needs assessment. The main reason was lack of expertise to perform

83

assessment. Among these respondents, the main source of information for training needs

assessment was the problems faced by their organizations. This evidence shows the lack of

attention given to transference of skills in training evaluation and feedback. Therefore, it is

wise to establish and instill the right approach to training and development as jobs today are

increasingly complex, and the traditional method of having a superior rate a subordinate

performance is inadequate in giving quality information to improve performance and skills.

The training culture in Malaysia has been indirectly influenced by multinational companies

operating in Malaysia. This is supported by a survey by Wan Aziz (1994) showed that the

majority of multinational companies operating in Malaysia brought in training culture. A

survey by Zakaria and Rodzhan (1993) on 108 manufacturing companies, suggested that

about 67 percent of the multinational companies interviewed conducted general and specific

training programs for all levels of staff. These multinational companies in Malaysia need to

conduct training because they require highly-skilled manpower who are able to operate new

and sophisticated machinery or research product improvement. A study by Wan Aziz (1994)

on 120 companies showed that 55.6 percent of Malaysian-owned companies conduct training.

This shows that Malaysian-owned companies are emulating the training culture of

multinational companies in order to cope with a challenging environment. A research by

Junaidah (1999) showed that Malaysian companies feel discouraged when undertaking

training, as they are not able to mark the progress of development after training. The main

reason may to lie in their inability to see the tangible benefits of training (Saiyadain &

Juhary, 1995). The majority of Malaysian companies conduct training needs on a general

basis. Zakaria and Rodzhan (1993) found that only 16 per cent of Malaysian companies

indicated that their training needs assessment was based on the strategic plan of the

organization. This indicates a lack of strategic orientation in the way training was conducted

in Malaysian companies. Components of training and development in an organization need

to cohere with one another in supporting organization strategy.

During the pre-training stage, a needs assessment is crucial in identifying relevant skills

needed by the individual to contribute to the strategic objectives of the company. Multi-rater

feedback would compliment the needs analysis by providing ratee with multi-source

feedback for further development. The ratee will be given an opportunity to understand their

strengths and weaknesses from a different source and focus on reducing weaknesses and

maintaining strengths. In Malaysia training needs assessment are not conducted by measuring

84

85

individual skill deficiency but through general perceptions of a few top executives in thewhole department or organization. Organizations see training as an organizational need

rather than an individual need. Mirza and Juhary (1995) found that training organizations in

Malaysia offered training programs that were relevant to the needs of the organizations and

were too theoretical, one-shot with no follow up and not interactive. Organizations have

neither the professional competence nor the resources to identify training needs and mountrelevant training programs. Mirza and Juhary (1995) indicated that the stated flaws of

training could be attributed to the partial training culture brought by multinational companies

in Malaysia. They commented that assessment is difficult; it is almost impossible to

determine which employee weakness can be addressed by training. The culture of

conducting training evaluation among Malaysian companies was simply not popular orencouraging.

According to June and Rozhan (2000), given no proper pre and post training evaluation, the

organization will be constrained in its ability to link training with strategic objectives. It

would be difficult for the training and development to have a meaningful impact on

organizational effectiveness. Their study also provided the argument that multi-rater

assessment is not practiced by Malaysian-owned companies for development. If Malaysian

companies wish to conduct complete training and proper assessment, it is wise to use multi-

rater feedback on the training needs assessment so that organizations would be more focused

in the development process and able to measure its effectiveness.

Shipper and John (1992), found that multi-source information may be a mechanism for open

communication among diverse groups to establish proper psychological contract and clarify

expectations. This is supported by Luthans and Farner's (2002) study using the Kirkpatrick

(1994) training evaluation framework integrated with multi-rater feedback on 409 expatriateworkers from 49 multinational companies on whether transfer learning on the job was well

received. The mentioned training evaluation framework may be applied to local managerswho worked in multinational companies in Malaysia who are not clear of the cultures broughtin by expatriates and the expectation of their foreign counterparts within the company.

Therefore, multi-rater feedback, which has been described as needs analysis process, will

clarify ratee's expectations with the people working around them (Fletcher & Bailey, 2003).

Instilling multi-rater feedback as part of the pre-training needs analysis in Malaysian can

bring practical benefits for the organization by focusing on a particular behaviour or key

competency that is necessary for employee development. Employees will also have the

chance to audit self-perception against others through this self awareness mechanism, which

will result in higher work performance (Atwater et al., 1998; Bass & Yammarino, 1991;

Fumham & Stringhfield, 1994). The information received from the multi-rater feedback

would impact on the targeted individual's selfawareness and lead to the achievement of

agreed developmental needs (Fletcher & Bailey, 2003). Indeed, research has confirmed that

the use of multi-rater feedback is one of the best methods to promote ratees' self awareness of

their strengths and skill deficiencies (Hagberg, 1996; Rosti & Shipper, 1998; Shipper &

Dillard, 2000). Multi-rater feedback has been defined as an information gathering process

from relevant observers and is linked to specific business needs or objectives. Therefore, a

multi-rater feedback refers to the practice of providing an employee with perceptions of his or

her performance competencies from numerous sources (Cacioppe & Albrecht, 2000). By

reviewing different perceptions of their performance competencies, ratees can confirm their

strengths as well as identify their blind spots, habits, behaviours and styles, which may have

an adverse impact on others and their developmental priorities. This process helps a ratee to

focus on and develop performance competencies through a well-structured development

process. Waldman et al. (1998) were concerned about the lack of research examining the

effectiveness of multi-rater feedback on the performance developmental cycle.

3.8 Integrating Multi-rater Feedback with Development Tool

Organizations need to look at development as a continuous process by incorporating

development model in the multi-rater feedback system. If the purpose of having multi-rater

feedback is not clear and not integrated with the developmental systems, it will come across

like a trend. This is shown by Judge and Cowell (1997) using executive coaching as a

development process after conducting multi-rater assessment. The study showed that the

combination of multi-rater feedback coupled with individual coaching as a developmental

process increased leadership development effectiveness by 60 percent. This was based on the

direct report and peer post-survey feedback. Another study by Heisler (1996a) used a

comprehensive combined model which integrated multi-rater feedback with the leadership

86

and management skills development process. This approach was applied to a sample of 304

superiors and more than 1000 subordinates. The result showed that the ratees felt an increase

of ownership towards their personal and professional development. Ratees reported

improved communication and interaction with their superiors, peers and subordinates

(Heisler, 1996a). Effective communication and interaction between raters and ratee will

reduce possible multi-rater feedback drawbacks.

The developmental process of multi-rater feedback involves a great deal of cognitive

complexity and acknowledgement of the validity and legitimacy of the feedback. It also

requires balancing multiple or conflicting perspectives and balancing a sense of self with the

larger context and role requirements. There should be some mechanism to address the

discrepancy between the ratee's and rater's feedback in order to make it into a coherent

developmental tool. The identified discrepancies can be used to assist ratees in developing

their personal action plan for development. Research is needed to clarify and validate the

most effective concept design to develop ratee after multi-rater feedback.

3.9 Multi-rater Feedback: Process Consultation as a Development Tool

Process consultation is an ideal support tool for development (Schein, 1997). The process

consultation session should be conducted after multi-rater feedback so that it turns out to be a

very positive experience, regardless of discrepancies in the results. Process consultation is an

ongoing development system approach that has skilled third party (process consultant) work

with ratees and helping them learn about their competency gap from the multi-rater feedback

process. The process consultant should emphasize on the ratee's strengths and improve on

what the ratee does best, not what he does worst. In spite of this, if different world-view

arises between the process consultant and the ratee (client), the process consultant may use

non-directive techniques in order to help the client recognize and accept responsibility for the

deficiency (Hall, Otaza & Hollenbeck, 1999; Judge & Cowell 1997; Thach & Heinselman,

1999).

Process consultation is based on the idea that ownership of the issues of concern remain with

the ratee, who has actively participated in defining the key issues resulting from multi-rater

87

feedback and formulating a solution that is culturally appropriate (Schein, 1987). The role of

the consultant revolves around facilitation and engaging in a helpful relationship with the

ratee, rather than simply being a provider of expertise. The process consultant's role is more

nondirective and questioning as he or she gets the groups to solve their own problems

(French & Bell, 1999). This approach increases the likelihood ofconfronting the most

pressing issues and helps the ratee benefits from problem-solving skills needed for ongoing

organizational change.

Schein (1987) commented that process consultation is not one single thing the process

consultant does but are paramount goals the process consultant helps the ratees (client)

achieve, change and resolve key issues ofconcern through different interventions. Although

information on the stages of change (Lovelady, 1989) and the focus of intervention

(Fagenson & Burke, 1990) are important, it reveals too little about the specific activities that

process consultants engage in, and the skills they need to accomplish them successfully.

Schein (1987) concluded that process consultants make interventions in the following order:

agenda setting, feedback of observations or other data, counseling and coaching, and

structural suggestions if any. During the process consultation, ratee (client) who wishes to

change their traditional practices and behaviors need to be given the opportunity to reflect on

a wide range of meaningful feedback. Without reflection, it is just lip service to change

ratee's behaviour or performance. Process consultants will also be given the opportunity to

reflect their feelings, thoughts and perceptions on ratee's development. Through the

reflection process, process consultants will be able to evaluate the degree of reaction and

learning of the ratee. This is supported by Kolb's (1984) learning theory which states that an

individual will learn effectively if he or she is able to reflect on the feedback received.

It is important to take a closer look at a process consultant's actual intervention role which

involves intertwining events, issues, thoughts, emotions and human interactions. Schein

(1987) and Weisbord (1988) showed appreciation for the complex role and behavioural

repertoire required by the process consultant. Most research does not distinguish between the

different settings and contexts for consultancy practice (Chapman, 1998). The question arises

on whether the process consultant engages in different activities in their work within the

organization. If so, what particular skills are required for them to successfully develop a

ratee (client). This question is of interest to many people including the process consultants

themselves. Chapman (1998) asserted that a successful facilitation process requires building

88

89

emotional ties between the process consultant and client through good communication and

interpersonal skills. According to Kirkpatrick (1959), adults must be motivated to learn.

Hence through effective communication and interpersonal interactions, development of

psychological contract and emotional ties between both parties will motivate them to

participate in the development plan (Wolfe & Kolb, 1984).

3.10 Micro Perspective of Conversation Theory in Process Consultation

Pask's (1975) work in developing a human learning system through conversation theory may

be used to enhance facilitation between the process consultant and client. Conversation

theory is a framework for intervention analysis called the conversations model developed by

Ledington in 1989. The elements of the framework are individuals, groups or organizations

that formed. The framework is used to manage an intervention and the intervener is free to

construct a social group or community. Conversation is a means of knowledge acquisition

and is a process in gaining self-understanding and mutual understanding. It is also a way to

achieve predetermined objectives by using specific strategies (Navarro, 2001). The specific

strategies could be used in a fair manner by trying to genuinely convince the other

participants in a good conversation session.

The conversation model would be able to guide process consultants on how to go through

various intervention strategies so that the client is stimulated to tell his or her story with

minimal disruption of either the process or content. This can be done if every learning

conversation is followed by reflections by both the process consultant and client. Pask's

(1976a) conversation theory mentioned that reflection would bring about a desired emergent

behaviour which shows what the participants have learned and achieved, how the participants

have interacted interpersonally, and what the participants need to learn in the future.

Pask's (1976a) conversation theory contemplated the phenomenon of human learning as the

result of an emergent process of conversation such as linguistic interaction based on

conscious, conceptual resonance between several P-individuals. These P-individuals can be

distinct points of view within a biological individual, different biological individuals or even

specific groups of them. P-individual is an effective participant in a conversation, which

connect with many P-individuals. He suggested the existence of a close relationship between

these two aspects P-individuals as perspectives and P-individuals as participants.

According to Pask's (1976a) conversation theory, process of communication can also be

considered as a P-individual: a strict conversation is a prototypical P-individual. There are

three different conceptual aspects coexisting in the idea of a P-individual: the concept of a

cognitive perspective, the concept of a participant (in a conversation) and that of a whole

conversation. The conversation is a P-individual, and so are the participants who converse

with each other (Pask, 1961).

Hence, good conversation is an important concept derives from Pask's conversation theory

that forms the basis for effective process consultation by encouraging process consultant and

his client to reproduce new behaviour through mutual information transfer and network of

concepts. Pask (1975) adopted a few alphabetical equations to explain his conversation

process by phrasing A (the process consultant) is conscious with B (the client) and

committing themselves to some dependency or relationship T (the agreed course of action).

The commitment of A and B to T is sought because this supposedly leads to desired

outcomes. In an analogous manner, the performance of a client in a conversation will

potentially involve the whole personality and not just the epistemic resource. Pask (1975) did

not consider information and conversation as a pre-selection of interaction but as a

consequence of the emergence of new realities when a given system interacts with other

systems. This emergence is due to the synchronization effect of the two systems.

The creation of the learning context requires selfawareness as well as a social context for

intentional interaction (Black & Mendenhall, 1990). The learning context can be facilitated

by developing good conversation between the process consultant and his or her client. Good

conversation creates a form of conversation between the process consultant and client where

norms of discourse are developed consensually, values and assumptions can be surfaced and

tested, and all voices can be heard (Schuurman & Veermans, 2001). Through good

conversation the process consultant can enhance transfer learning and proceed with the

development process for his or her client.

Good conversation creates a cycle of effective transaction between the process consultant and

client who come into conversation and learn from each other. Research on stereotyping has

found an association between the level of self-acceptance a client feels and the tendency to

90

stereotype or accept others (Adorno, Frenkel-Brunswik, Levinson & Stanford, 1950; Rubin,

1967). Therefore, we expect that as clients increasingly accept themselves, they are more

able to let go of their prejudices and stereotypes of the process consultant. When clients are

fully free to speak, and feel they are genuinely being heard, the affirmation they experience

enhances self-acceptance. This enables them to listen more completely, allows for the

synergistic cycle of being heard and experience increased self-acceptance (Rogers, 1970).

Through good conversation, the possibility of stereotyping will be minimized to allow the

process consultant plays his/her role more effectively.

3.11 An Integrated Approach for Post Multi-rater Feedback Development

Post multi-rater feedback development start with a contact client or known as the ratee with

whom the process consultant meets concerning his or her performance deficiencies.

Whether or not that client admits to owing the performance deficiencies that is to be worked

on in the event of development, the process consultant would not want to be prematurely

perceived as an expert. The process consultant would want the client to feel helped after a

few meetings. The client should feel that every conversation is helpful especially during

early interactions.

According to Schein (1997), the process consultant and client have something to learn from

each other during the development process. He came up with eight general principles to

improve the flexibility of process consultation. They are: always be helpful, always deal with

reality, access your ignorance, everything you do is an intervention, it is the client who owns

the problem, go with the flow, be prepared for surprises and learn from them, share the

problem. These eight general principles govern the process consultant's roles and

relationship with the client. Chapman (1998) said that a process consultant should adopt

flexible consulting roles. Some clients may need a mentor and adviser on general

management matters as much as they require a facilitator and project manager. He further

commented that good process consultants help to identify the real issues and challenges

facing the organization as well as discuss a tailor-made process for constructive change.

91

Although Schein's eight general principles were used to enhance flexibility of the process

consultant's role, it does not mention how an effective dialogue session could be established

between the process consultant and his/her client. Vygotsky (1978) mentioned that

psychological contract could be established through effective dialogue and it will help the

process consultant and client reach higher levels of understanding. The establishment ofpsychological contract would be an opportunity for the process consultant and client to learn

important new things about a situation when they explore it together.

According to Schein (1997), the client owns the problem and has to live with the

consequences of the problem and the solution. Therefore the consultant must not withdraw

any problems away from the client because the client is the best person to understand and

appreciate what would be the next best steps. Involvement of client depends on their

willingness to openly discuss issues they are facing and the trust they give the consultant.

Sometimes the client hides the real problem because he or she is testing the consultant to

determine whether the relationship is characterized by sufficient trust to reveal what may be

very intimate and personal information. Trust building therefore requires greater 'good

conversation' between both parties to explore their commitment and intention.

Learning about good conversations and adjusting our responses to different individuals,

groups and issues appropriately, can have a dramatic impact on outcomes for individuals,

teams and whole organizations. Any significant human learning is not just cognitive

information-processing but also moral and aesthetic co-construction of parts of our life-world

(Boyd, 2001). The process consultant and client may have conflicting views and feelings if

both parties hold strongly to their beliefs or worldview. The resolution to this conflicting

belief or worldview demands that new realities be generated through synchronisation of

perceptual differences (Navarro, 2001). Reflections provide an avenue for both parties to

understand each other's views and learn from each other's differences. The process

consultant may use reflections to help his or her client to focus on one behavioral change they

would like to make as a result of their experience: "What do you want to work on, and are

you willing to make a commitment to change?" This gives their reflection an action

component which is often beneficial.

Learning occurs in two forms: single-loop and double-loop (Argyris, 1994). Single-loop

learning asks a one-dimensional question to elicit a one-dimensional answer. Double-loop

92

learning takes an additional step, or more often than not, several additional steps. It turns the

question back on the questioner. It asks what the media calls follow-up. A double loop

process might also ask why the current setting was chosen in the first place. Because double-

loop learning depends on questioning one's own assumptions and behaviour, this apparently

benevolent strategy is actually anti-learning (Argyris, 1994). Admittedly, being considerate

and positive can contribute to the solution of single-loop problems for example cutting costs.

But it will never help people figure out why they lived with problems for years, why they

covered up, why they were so good at pointing to the responsibility of others and so slow to

focus on their own. The notion of good conversation expands the phenomenon of ideal

speech to include ideal listening and promote interaction. For ideal listening to occur, the

individuals must feel secure and accepting enough of themselves to be open to new

possibilities. Enhanced self-acceptance can contribute to the possibility of valuing the

diversity of others. Thus, the responsibility of the process consultant includes nurturing

clients' self-acceptance and inspiring a sense of personal power among people around them.

This framework promotes a deep commitment to empathetic interaction between process

consultant and client to construct a shared reality as a common setting for development

pathway (Navarro, 2001).

In good conversation theory, learning is approached from an inside-out perspective based on

personal experience (Hunt, 1987). Under the process consultation practice, individual

personal experience is required to be reflected on. Reflection pinpoints and dramatizes what

individuals have learned and achieved, how individuals have interacted interpersonally, and

what individuals need to learn in the future. Through valuing each person's individual

experience, the uniqueness of every person is assumed and considered a resource. With the

more typical outside-in approach to learning, the dissimilarity of each person is considered a

problem to be solved (Hunt, 1987). The point of departure for learning in 'good

conversation' is not only the presumption that each individual is different, but that diversity is

an inherent resource. In this consensual and self-reflective process, as more and more diverse

reflections become fully heard within the group, the values and perspectives of each member

influence others, and the process of mutual socialization evolves.

The process consultant must become the reflective practitioner cum learner besides helping

the individual benefit from the double loop processes (Argyris & Schon, 1978). They must

diagnose the issues and take action to improve the practice by involving themselves directly

93

and fully, preparing to investigate such experiences from as many different perspectives as

possible and patterning their observations into meanings through reflection. Documentation

of the agreed proposed course of action was not mentioned by Schein (1997). In the absence

of documented agreement between the process consultant and client, commitment in fulfilling

the course of action is unlikely to happen. Schuurman and Veermans (2001) derived two

classes of consequences from conversation theory: the weak consequence and the strong

consequence. The weak consequence stresses observation, the strong consequence stresses

control. The weak consequence concentrates on record keeping of the experimental subject,

closed conversation and topic of exchanges. The strong consequence notes the records of

agreements derived from the outcome of negotiations between two parties. Both these

consequences will bring about total commitment between the process consultant and the

client. The documentation process that records the reflections made between two parties

during the learning process will become an obligated implication to be fulfilled. Besides this,

the record keeping will also provide both parties with a flow of progression towards their

development goals.

Pask (1961, 1965, 1975a, 1975b, 1976a, 1976b) introduced both the object-language and

meta-language to explain the required exchanges during the learning process. He stressed the

need for researchers to distinguish object-language and meta-language with any of the

learning interfaces (Schuurman & Veermans, 2001). The object-language comprises a system

of expression (i.e. sentences during conversation) belonging to the object of study. These

sentences should be internal expressions of the object, that is to reflect properties of the

object and these expressions should conform to well define rules (in this case, the

developmental pathway undertaken by the process consultant and the client). Within the

meta-language, a new object-language can be proposed. If the new object-language fits the

purpose (i.e. learning objective) better than the original object-language then it can be

replaced. However, this is only possible if the process consultant and/or the client knows

what to replace. Keeping apart object-language and meta-language allows revisions to be

tracked. This is a crucial prerequisite for systematic inquiry (De Zeeuw, 1995). Record

keeping process holds a very important key in making this a success (Schuurman &

Veermans, 2001). Pask (1965) considered interaction between object-language and meta-

language pivotal in learning and human performance in general. Pask observed object-

language and meta-language interactions so as to study how conversations are punctuated by

agreements (including agreements to disagreements). According to Pask, researcher needs to

94

keep proper record of the interaction between object-language and meta-language and to

mark all agreements. The agreements serve as controlled conversation of true hard data. He

argued that psychological experiments start with basic meta-language interactions: the

experimenter and experimental subject have to agree on their respective roles (Schuuman &

Veermans, 2001). The meta-language interactions that Pask strongly advocated should serve

as the whole basis of process consultation sessions where emergent behaviour for learning is

likely to happen.

However, there are a few drawbacks to the conversation theory where the theory itself

actually eliminates some basic traits of the human mind, human interaction and ignores other

aspects of human reality (Navarro, 2001). The other factors which prevent a straightforward

application of conversation theory to the study of social realities are strong dependence on

Pask's theory and not the study of real social life, specifically of human interactions. To

address the weakness of conversation theory in a real social environment and natural

conversation situation, the strength of the theory depends on its ability to bring into sharp

focus on specific aspect of the world. Massaro and Cowan (1993) suggested that in building

a community of good conversation, people are required to put themselves in the shoes of

others and to empathise if they are to arrive at consensually developed norms. Through

empathy for others, they can begin to understand, bring life, feelings and even accommodate

for the consequences of each other's norms distinct from themselves. It has to be an attempt

to truly see the world as the other sees it, understand the real life situation of the other and

adopt other's perspectives and values (Massaro & Cowan, 1993). One assumption of good

conversation is its essential dynamic quality and process, resisting a tendency to control for

predictability. This form of conversation implicitly and explicitly sets the conditions for

valuing individuals or organization through the integration of affective and cognitive modes

of experience and learning (Argyris, 1994). In this initial phase of the process, the process

consultant's role is particularly important. As part of the norm creation process, the process

consultant needs to be continually modeling a respectful and inclusive approach throughout

to foster a safe, receptive space for the conversation to unfold (Argyris, 1994).

95

3.12 Conclusion

Although research on multi-rater feedback assessment indicates that different rater sources

provide different information, multi-rater feedback technique is still useful at the preliminary

stage to provide information or create self awareness on individual strength, weaknesses or

blind spots. One underlying rationale to such systems is their potential impact on the target

individual's self awareness which increasing self awareness is thought to enhance

performance. This paper provides a concept on how multi-rater feedback can lead to a

successful developmental process through process consultation in Malaysia. Through the

years, training evaluation culture in Malaysia has not been properly practiced, hence it is

recommended that a proper approach be used to enable organizations to see the benefits of

holding pre training needs analysis and effective development approach so that a

comprehensive training and effective development approach could be instilled in the

Malaysian environment. Hence, the process consultant holds the key to effective

development process using multi-rater assessment as a pre-training gap analysis.

Process consultation provides the opportunity to check and balance the degree of learning and

development activities through reflection, problem solving capabilities and application of

theories throughout the developmental process. The flexibility of process consultation should

be enhanced by integrating conversation theory using good conversation and documentation

of pre-agreed commitment of action known as reflection. This will promote ideal

communication and interaction between the process consultation and client which will

eventually build trust for open learning and development. Good conversation is an important

intervention tool that has potential for applying effective human communication, decision

making, and policy making in the development process through single loop and double-loop

learning.

Multi-rater feedback approach also gathers information from various sources, in order to

evaluate the level of transfer learning of an individual at the end of the development stage of

process consultation. It is recommended that an integrated and comprehensive model

comprising preliminary multi-rater feedback assessment, followed by developmental process

using process consultation and good conversation in an effort to facilitate transfer learning to

the organization.

96

3.13 References for Paper Three

Adorno, T.W., Frenkel-Brunswik, E., Levinson, D.J. & Stanford, R.N. 1950, Theauthoritarian personality, Harper and Brother, New York.

Alimo-Metcalf, B. 1998, 'Editorial 360-degree assessment and feedback',Professional Forum, vol. 6, no. 1, pp. 16-18.

Antonioni, D. 1994, 'Designing an effective 360-degree appraisal feedback system',Personnel Psychology, vol. 47, pp. 349-356.

Argyris, C. & Schon, D. 1978, Organization Learning: A Theory in Action Perspective,Addison-Wesley, Reading, MA.

Argyris, C. 1994,'Good conversation that blocks learning', Managerial Excellence, HarvardBusiness Review, vol. 15, pp. 303-317.

Ashford, S.J. 1984, 'Self-assessments in organizations: a literature review and integrativemodel', Research in Organizational Behaviour, vol. 11, pp. 133-174.

Ashford, S.J. 1993, 'The feedback environment an exploratory study of cue use', Journal ofOrganizational Behaviour, vol. 14, pp. 201-224.

Atwater, L.E. & Yammarino, F.J. 1993, 'Personal attributes as predictors of superiors' andsubordinates' perceptions of military academy leadership', Human Relations, vol. 46,pp. 645-668.

Atwater, L.E., Ostroff, C.M., Yammarino, F.I. & Fleenor, I.W. 1998, 'Self-other agreement:does it really matter?' Personnel Psychology, vol. 51, no. 3, pp. 577-598.

Atwater, L.E., Roush, P. & Fischthal, A. 1995, 'The influence of upward feedback on self-and follower ratings of leadership', Personnel Psychology, vol. 48, pp. 35-49.

Bass, B.M. & Yammarino, F.I. 1991, 'Congruence of self and others' leadership ratings ofnaval offices for understanding successful performance', Applied Psychology: AnInternational Review, vol. 40, no. 4, pp. 437-454.

Bennis, W.G., Benne, K.D. & Chin, R. 1969, The Planning of Change, 2nd edn, Holt,Rinehart and Winston, New York, NY.

Bernardin, H.J., Dahmus, S.A. & Redmon, G. 1993, 'Attitudes of first line supervisorstowards subordinate appraisals', Human Resource Management, vol. 32, pp. 315-324.

Black, J.S. & Mendenhall, M. 1990, 'Cross-cultural training effectiveness: a review and atheoretical framework for future research', Academy of Management Review, vol. 15,no. pp. 113-136.

97

Borman, W.C. 1997, '360-degree ratings: an analysis of assumptions and research agendafor evaluating their validity', Human Resource Management Review, vol. 7, pp. 315-324.

Boyd, G. 2001, 'Reflections on the conversation theory of Gordan Pask', Kybernetes, vol.30, no. 5/6, pp. 560-570.

Bracken, D.W., Dalton, M.A., Jako, R., McCauley, C.D. & Pollman, V.A. 1997, Should360-degree Feedback Be Used Only for Developmental Purposes? Greensboro, NC:Center for Creative Leadership.

Brutus, S., London, M. & Martineau, J. 1999, 'The impact of 360-degree feedback onplanning for career development', Journal of Management Development, vol. 18, pp.676-693.

Cacioppe, R. 1998, 'An integrated model and approach for the design of effectiveleadership development programs', Leadership and Organization DevelopmentJournal, vol. 9, no. 1, pp. 44-53.

Cacioppe, R. & Albrecht, S. 2000, 'Using 360-degree feedback and the integral model todevelop leadership and management skills', Leadership and OrganizationDevelopment Journal, vol. 21, no. 8, pp. 390-404.

Cacioppe, R. & Albrecht, S. 2000, 'Differing perceptions of managers: behavioursusing the holon leadership-management model', in Parry, K. edn. forth coming.

Chapman, J. 1998, `Do process consultants need different skills when working withnonprofits?', Leadership and Organization Development Journal, vol. 19, no. 4, pp.211-215.

Church, A.H. 1997, 'Managerial self awareness in high performing individuals inorganizations', Journal of Applied Psychology, vol. 82, pp. 281-292.

Conway, J.M. & Huffcutt, A.I. 1997, 'Psychometric properties of multi-source performanceratings: a meta-analysis of subordinate, supervisor, peer and self-ratings', HumanPerformance, vol. 10, pp. 331-360.

Dezeeuw, G. 1995, 'Values, science and the quest for demarcation', System Research, pp.15-24.

Edwards, J.R. 1993, 'Problems with the use of profile similarity indices in the study ofcongruence in organizational research', Personnel Psychology, vol. 46, pp. 641-65.

Edwards, J.R. 1994, 'The study of congruence in organizational behaviour research: critiqueand proposed alternative, Organizational Behaviour And Human Decision Processes,vol. 58, pp. 51-100.

98

Facteau, J.D., Facteau, C.I., McGonigle, T.P. & Fredholm, R.I. 1997, Characteristics offeedback and managers' reactions in multi-source appraisal systems, paper presentedat the 12th annual conference of the Society of Industrial and OrganizationalPsychology, St. Louis, MO.

Fagenson, E. & Burke, W. 1990, 'Organization development practitioners' activities andinterventions in organizations during the 1980s', Journal of Applied BehaviouralScience, vol. 26, no. 3, pp. 285-297.

Fletcher, C. 1997, 'Self awareness: a neglected attribute in selection and assessment?',International Journal Of Selection And Assessment, vol. 5, no. 3, pp. 183-187.

Fletcher, C. & Baldry, C. 2000, 'A study of individual differences and self awareness in thecontext of multi-source feedback', Journal Of Occupational And OrganizationalPsychology, vol. 73, pp. 303-319.

Fletcher, C. & Bailey, C. 2003, 'Assessing selfawareness: some issues and methods',Journal of Managerial Psychology, vol. 18, no. 5, pp. 395-404.

French & Bell 1999, Organization Development: Behavioural Science Interventions forOrganization Improvement, 6th edn, Prentice-Hall Publisher, New Jersey.

Furnham, A. & Stringfield, P. 1994, 'Correlates of self and subordinate ratings ofmanagerialpractices as a correlate of supervisor evaluation', Journal of Occupational andOrganizational Psychology, vol. 67, no. 1, pp. 57-67.

Garavan, T.N., Morley, M. & Flynn, M. 1997, '360-degree feedback: its role in employeedevelopment', Journal of Management Development, vol. 16, no.2, pp. 134-147.

Ghorpade, J. 2000, 'Managing five paradoxes of 360-degree feedback', Academy ofManagement Executive, vol. 14, pp. 140-50.

Greguras, G.J., Ford, J.M. & Brutus, S. 2003, 'Manager's attention to multi-source feedback',Journal of Management Development, vol. 22, no. 4, pp. 345-361

Hagberg, R. 1996, 'Identify and help executives in trouble', Human Resource Magazine,vol. 41, no. 8, pp. 88-92.

Hall, D., Otazo, K. & Hollenbeck, G. 1999, 'Behind closed doors: what really happens inexecutive coaching', Organizational Dynamics, vol. 27, no. 3, pp. 39-58.

Harris, M.M. & Schaubroeck, J. 1988, 'A meta-analysis of self-supervisor, self-peer andpeer-supervisor ratings', Personnel Psychology, vol. 41, pp. 43-62.

Hazucha, J. Fr., Hezlett, S.A. & Schneider, R.J. 1993, 'The impact of 360-degree feedback onmanagement skills development', Human Resource Management, vol. 32, pp. 325-351.

Heisler, W.J. 1996a, '360-degree feedback: an integrated perspective', Career DevelopmentInternational, vol. 1, no. 3, pp. 20-23.

99

Hoffman, R. 1995, 'Ten reasons you should be using 360-degree feedback', HumanResource Management Magazine, vol. 40, no. 4, pp. 82-86.

Honey, P. & Mumford, A. 1982, Manual ofLearning Styles, Honey Publication,Maidenhead.

Hunt, D.E. 1987, Beginning With Ourselves, Brookliine Books, Cambridge, MA.

Judge, W. & Cowell, J. 1997, 'The brave new world of executive coaching', BusinessHorizons, vol. 40, no. 4, pp. 71.

Junaidah, H. 1999, Training Management: A Malaysian Perspective, Prentice-HallPublisher, Pearson Education, Malaysia.

June, M.L. P. & Rodzhan, 0. 2000, 'Management training and development practices ofMalaysian organizations', Journal of the Malaysian Institute of Management,Malaysian Management Review, vol. 35, no. 2, pp. 77-85.

Kirkpatrick, D.L. 1959a, 'Techniques for evaluating training programs: part 1 - reaction',Journal of American Society for Training and Developing, vol. 13, pp. 3-9,

Kirkpatrick, D.L. 1959b, 'Techniques for evaluating training programs: part 2 - learning',Journal of American Society for Training and Developing, vol. 13, no. 12, pp. 21-26.

Kirkpatrick, D.L. 1994, Evaluating Training Programs The Four Levels, Berrett-KoehlerPublishers, San Francisco.

Kluger, A.N. & Denisi, A.D. 1996, 'The effects of feedback interventions on performance:historical review, a meta-analysis and a preliminary feedback intervention theory',Psychological Bulletin, vol. 119, pp. 254-284.

Kluger, A.N. & Denisi, A.D. 2000, 'Feedback effectiveness: can 360-degree appraisals beimproved?', Academy of Management Executive, vol. 14, pp. 129-139.

Kolb, D. 1984, Experimental Learning, Prentice-Hall Publisher, New Jersey.

Leslie, J., Gryskiewicz, N. & Dalton, M. 1998, 'Understanding cultural influences on the360-degree feedback process', in Maximizing the Value of 360-degree Feedback: AProcess for Successful Individual and Organization Development, eds Tornow, W. &London, M., Jossey-Bass, San Francisco, pp. 196-216.

London, M. & Beatty, R.W. 1993, '360-degree-feedback as a competitive advantage', HumanResource Management, vol. 2-3, pp. 353-372.

London, M. & Smither, J.W. 1995, 'Can multi-source feedback change perceptions of goalaccomplishment, self evaluations and performance related outcomes? Theory-basedapplications and directions for research', Personnel Psychology, vol. 48, pp. 803-839.

100

London, M., Wholers, A.J., & Gallagher, P. 1990, '360-degree feedback surveys: a source offeedback to guide management development', Journal of Management Development,vol. 9, pp. 17-31.

Lovelady, L. 1989, 'The process of organization development: a reformulated model of thechange process, Part 1', Management Decision, vol. 27, no. 4, pp. 143-154.

Luthans, K.W. & Farner, S. 2002, 'Expatriate development: the use of 360-degree feedback',Journal of Management Development, vol. 21, no. 10, pp. 780-793.

Massaro, D.W. & Cowan, N. 1993, 'Information processing models: microscopes of themind', Annual Review and Psychology, vol. 44, pp. 383-425.

McCauley, C.D. & Moxley, R.S. Jr. 1996, Development 360: how Feedback Can MakeManagers More Effective, Jossey-Bass Publisher, San Francisco.

McEvoy, G.M. 1990, 'Public sector managers' reactions to appraisals by subordinates',Public Personnel Management, vol. 19, pp. 201-212.

McEvoy, G. M. & Buller, P.F. 1987, 'User acceptance of peer appraisals in an industrialsetting', Personnel Psychology, vol. 40, pp. 785-797.

Mirza, S.S. & Juhary, H.A. 1995, Managerial training and development in Malaysia,Malaysian Institute of Management, Malaysia.

Mount, M. K. 1984, 'Psychometric properties of subordinate ratings of managerialperformance', Personnel Psychology, vol. 37, pp. 687-701.

Mount, M.K., Judge, T.A., Scullen, S.E., Sytsma, M.R. & Hezlett, S.A. 1998, 'Trait, rater,and level effects in 360-degree performance ratings', Personnel Psychology, vol. 51,pp. 557-576.

Moses, J., Hollenbeck, G. P. & Sorcer, M. 1993, 'Other people's expectations', HumanResource Management, vol. 32, Summer Fall.

Nasby, W. 1989, 'Private self-consciousness, self awareness and the reliability of self-reports', Journal of Personality and Social Psychology, vol. 56, no. 6, pp. 950-957.

Navarro, P. 2001, The Limits of Social Conversation, Kybernetes, MCB University Press,vol. 30, no. 5/6, pp. 771-788.

Nowack, K. 1993, '360-degree feedback: the whole story', Training and DevelopmentJournal, vol. 47, no. 1, pp. 69-73.

O'Reilly, B. 1994, '360-degree feedback can change your life', Fortune Magazine, vol. 130,no. 8, pp. 93-97.

Pask, G. 1961, An Approach to Cybernetics, Hutchinson, London.

Pask, G. 1965, Inleiding tot de Cybernetica, Het Spectrum, Utrecht.

101

Pask, G. 1975a, The Cybernetics Of Human LearningAnd Performance', Hutchinson,London.

Pask, G. 1975b, Conversation, Cognition and Learning:A Cybernetic Theory andMethodology, Elsevier, Amsterdam.

Pask, G. 1976a, Conversation Theory: Applications In Education And Epistemology,Elsevier, Amsterdam and New York.

Pask, G. 1976b, Revisions in the foundations of cybernetics and general systems theory as aresult of research in education, epistemology and innovation (mostly in man-machinesystems), proceedings of the 8th International Congress on Cybernetics, Namur, vol.6, no. 11, September, pp. 83-109.

Payne, T. 1988, 'Editorial 360-degree assessment and feedback', International Journal ofSelection and Assessment, vol. 6, no. 1.

Reilly, R.R., Smither, J.W. & Vasilopoulos, NJ. 1996, 'A longitudinal study of upwardfeedback', Personnel Psychology, vol. 49, pp. 599-612.

Rogers, C.R. 1970, Encounter Groups, Harper and Row, New York.

Romano, C. 1994, 'Conquering the fear of feedback', Human Resource Focus, vol. 71, no. 3.

Rosti, R.T., Jr & Shipper, F. 1998, 'A study of the impact of training in a managementdevelopment program based on 360-degree feedback', Journal of ManagerialPsychology, vol. 13, pp. 77-89.

Rubin, I. 1967, 'The reduction of prejudice through laboratory training', Journal of AppliedBehavioural Science, vol. 3, no. 1.

Saiyadain, M.S. & Juhary, A. 1995, 'Managerial training and development in Malaysia',Journal of the Malaysia Institute of Management, Management Review, vol. 5, pp.23-36.

Schein, E.H. 1987, Process Consultation, vol. 2, Addison-Wesley, MA.

Schein, E.H. 1997, 'The concept of 'client' from a process consultation perspective', Journalof Organizational Change Management, vol. 10, no. 3, pp. 202-216.

Schuurman, J.G. & Veermans, K. 2001, 'Conversation and research', Kybernetes, vol. 30, no.7/8, pp. 881-890.

Shipper, F. & Dillard, J.E. Jr. 2000, 'A study of impending derailment and recovery ofmiddle managers across career stages', Human Resource Management, vol. 39, no. 4,pp. 331-345.

102

Shipper, F. & John, J. 1992, 'Employees' feedback: its use for management development andthe results in a government organization', in Fargher, J.S. edn., Proceedings ofSymposium on Productivity and Quality Improvement with a Focus on Government,Industrial Engineering and Management Press, Washington, DC.

Thach, E.C. 2002, 'The impact of executive coaching and 360-degree feedback on leadershipeffectiveness', Leadership and Organization Development Journal, vol. 23, no. 4, pp.205-214.

Thach, I. & Heinselman, T. 1999, 'Executive coaching defined', Training and DevelopmentJournal, vol. 53, pp. 34-39.

Tornow, W.W. 1993, 'Perceptions or reality: is multi-perspective measurement a means oran end?' Human Resource Management, vol. 32, no. 2 and 3, pp. 221-230.

Tornow, W.W. & London, M. 1998, Maximizing the Value of 360-Degree Feedback: AProcess for Successful Individual and Organizational Development, Jossey-BassPublisher, San Francisco.

Van Veslor, E., Taylor, S. & Leslie, J.B. 1993, 'An examination of the relationship amongself-perception accuracy, self awareness, gender and leaders' effectiveness', HumanResource Management, vol. 32, summer fall, no. 2/3, pp. 249-263.

Vygotsky, L.S. 1978, Mind in Society, Harvard University Press, Cambridge.

Waldman, D.A., Atwater, L.E. & Antonian, D. 1998, 'Has 360-degree feedback goneamok?', Academy of Management Executive, vol. 12, no. 2, pp. 86-94.

Walker, A.G., & Smither, J. W. 1999, 'A five-year study of upward feedback: what managersdo with their results matters', Personnel Psychology, vol. 52, pp. 393-423.

Wan Aziz, W.A. 1994, 'Transnational corporations and human resource development',Personnel Review, vol. 23, no. 5, pp. 50-69.

Warr, P. & Bourne, A. 1999, 'Factors influencing tow types of congruence and similarity asrelated to interpersonal evaluation in manager-subordinate dyads', Academy ofManagement Journal, vol. 23, pp. 320-30.

Weisbord, M. 1988, 'Towards a new practice theory of OD: notes on sharpshooting andmoviemaking', in Research in Organizational Change and Development, edsPasmore, W. & Woodman, R., JAI Press, Greenwich, CT, vol. 2, pp. 59-96.

Wilson, M.S., Hoppe, M.H., & Sayles, R.S. 1996, Managing Across Cultures: A LearningFramework, Centre for Creative Leadership, Greensboro, NC.

Wolfe, D.M. & Kolb, D.A. 1984, 'Career development, personal development andexperiential learning', in Organization Psychology: Readings on Human Behavioursin Organizations, 4th edn, Prentice-Hall, NJ.

103

Wood, R., Allen, T., Pillenger, T. & Kahn, N. 1999, '360-degree feedback: theory, researchand practice', in Human Resource Strategies: An Applied Approach, edsTravaglione,T. & Marshall, V., McGraw-Hill, Sydney.

Zakaria, I. & Rodzhan, 0. 1993, Human resource development practice in the manufacturingsector in Malaysia: an empirical assessment, Paper Presented at the Seminar onHuman Resource Management, Faculty of Business Management, UniversityKebangsaan Malaysia.

Zemke, R. & Zemke, S. 1995, 'Adult learning: what do we know for sure?', Training andDevelopment Journal, vol. 32, pp. 31-37.

104