Performance Level EVALUATION Methods Used in MAN DEV Studies From 1986 to 2000

7/29/2019 Performance Level EVALUATION Methods Used in MAN DEV Studies From 1986 to 2000

1/21

http://hrd.sagepub.com

Human Resource Development Review

DOI: 10.1177/1534484302011005

2002; 1; 91Human Resource Development ReviewDoris B. Collins

Performance-Level Evaluation Methods Used in Management Development Studies from 1986 to 2

http://hrd.sagepub.com/cgi/content/abstract/1/1/91The online version of this article can be found at:

Published by:

http://www.sagepublications.com

On behalf of:

Academy of Human Resource Development

can be found at:Human Resource Development ReviewAdditional services and information for

http://hrd.sagepub.com/cgi/alertsEmail Alerts:

http://hrd.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://hrd.sagepub.com/cgi/content/refs/1/1/91SAGE Journals Online and HighWire Press platforms):

(this article cites 15 articles hosted on theCitations

2002 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.by Tomislav Bunjevac on August 22, 2008http://hrd.sagepub.comDownloaded from
http://www.ahrd.org/http://hrd.sagepub.com/cgi/alertshttp://hrd.sagepub.com/cgi/alertshttp://hrd.sagepub.com/subscriptionshttp://hrd.sagepub.com/subscriptionshttp://hrd.sagepub.com/subscriptionshttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsPermissions.navhttp://hrd.sagepub.com/cgi/content/refs/1/1/91http://hrd.sagepub.com/cgi/content/refs/1/1/91http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/cgi/content/refs/1/1/91http://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsReprints.navhttp://hrd.sagepub.com/subscriptionshttp://hrd.sagepub.com/cgi/alertshttp://www.ahrd.org/


2/21

Human Resource Development Review / March 2002

Collins / EVALUATION OF MANAGEMENT PROGRAMS

Performance-Level Evaluation Methods

Used in Management DevelopmentStudies From 1986 to 2000

DORIS B. COLLINS

Louisiana State University

A revi ew of the eva lua tio n lit era ture sho ws tha t the re has bee n a resur-gence of interest in the evaluation of management development programsby human resource development (HRD) professionals. However, thereremains a question of how well evaluation models measure the achieve-ment of intended outcomes, especially where the program goal is theenhancement of organizational performance. This article analyzes man-

agement development interventions from 1986 to 2000 through a resultsassessment lens to determine common evaluation characteristics in orga-nizational performance improvement efforts. The implication of factorssuch as timing of evaluation, evaluation methods, management develop-ment intervention categories, instrumentation, and measurement of orga-nizational performance are discussed in this manuscript. Findings revealthat HRD professionals must take the leadership in the development ofevaluation activities, especially those that test the long-term impact of theintervention on the organization.

Evaluation of management development programs is not a new phenome-

non. Kirkpatricks model for evaluation of training and development has

been used for 40 years to measure training effectiveness. Yet research indi-

cates that organizations are spending little time evaluating the effectivenessof their management development programs (Sogunro, 1997). It appears

that many corporations take for granted that management development

efforts willresult in improved management skills without valid datato prove

the return on investment.

A review of evaluation literature shows that there has been a resurgence

of interest in the evaluation of management development programs by

human resource development (HRD) professionals (Alliger, Tannenbaum,

Bennett, Traver, & Shotland, 1997; Dionne, 1996; Holton, 1996; Moller &

Mallin, 1996). However, there remains a question of whether existing evalu-

ation models are adequate to measure whether the program achieves its

intended outcome (Sogunro, 1997), especially where the goal of the pro-

gram is to enhance organizational performance. Nevertheless, annual bud-

Human Resource Development Review Vol. 1, No. 1 March 2002 91-1102002 Sage Publications

http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/


3/21

gets for management development programs are expected to grow through-

out the next decade as companies recognize the shortage of talented

managers, the importance of developing bench strength, and the need to

widen perspectives in order to compete globally (Gibler, Carter, & Gold-smith, 2000, p. xii). More than ever, organizations are concerned about the

management inadequacies of their employees and are committing to educa-

tion and training that deepen the skills, perspectives, and competencies of

their managers. And, more than ever, human resource departments must jus-

tify their training programs and compete for limited financial resources

(Dionne, 1996; Gordon, 1987).

Manageme nt devel opme nt is defined for this research as every form of

growth or stage of development in the life cycle that promotes, encourages,

and assists the expansion of knowledge and expertise required to optimize

ones management potential and performance (Brungardt, 1996). Manage-

ment development studies in this research include not only formal training

but also a full range of experiences that are defined by McCauley, Moxley,

and Van Velsor (1998) to include action learning, mentoring, job assign-ments, on-the-job experiences, and feedback-intensive programs.

In a study by Collins (2000), only 16 out of 54 management development

studies from 1986 to 2000 had organization-level performance (Rummler &

Brache, 1995) as the outcome variable. This leads us to wonder why there

are there so few studies that measure the effectiveness of training programs

contributing to organizational performance.

This article looks at management development interventions through a

results assessment lens (Swanson & Holton, 1999) to determine common

evaluationcharacteristics among thestudies. This is done in an effort to pro-

vide some theoretical undergirding for future research in determining

underlying reasons organizations are spending so little time evaluating the

effectiveness of their management development programs. It is hoped that

the findings will influence HRD professionals or corporate managers to

measure performance at the organization level (Rummler & Brache, 1995)

or inspire organizations to evaluate programs when they might not have a

tendency to do so. This article also addresses the measurement of organiza-

tional performance improvement efforts.

Evaluation as a Critical Issue

Organizations are committing to management development programs to

deepen the skills and competencies of their managers. Evaluation is one of

the critical components of these programs, but it is rarely practiced in the

field (Holton, 1996).

For this study, evaluation is defined as the systematic collection of data

regarding the success of management development programs (Goldstein,

92 Human Resource Development Review / March 2002



4/21

1986). It occurs when specified outcome measures are conceptually related

to intended learning objectives (Kraiger, Ford, & Salas, 1993). Evaluation is

normally conducted to answer the question of whether training objectives

were achieved or the accomplishments of training objectives resulted inenhanced performance of the individual on the job (Phillips, 1997). It

enables trainers to continuously monitor their programs and to identify

points of intervention for program improvement. All evaluations of man-

agement development programs start with learning and, basically, whether

trainees have learned the material covered in the training program (Camp-

bell, 1988). The analysis of the studies in this article will go a step further

with the purpose of the evaluation process and examine assessment of orga-

nizational performance improvement.

The results assessment system (Swanson & Holton, 1999) is the evalua-

tion model used as the lens for this analysis. The results assessment system

defines outcomes broadly to relate to the results of changes in leadership

style in top management, worker satisfaction, work teams, or organizational

change (Lynham & Swanson, 1997). The results assessment system pro-vides three outcomes levels, each subdivided into two outcomes categories.

Performance-level outcomes are subdivided into system or financial results,

whereas learning-level outcomes are subdivided into results of expertise or

knowledge. A perception level is also included in the model to evaluate per-

ceptions of participants and stakeholders, but it is not a focus of this

research.

The performance-system level involves mission-related outputs in the

form of goods or services, having value to the customer, that are related to

the core organizational, work, process, and group or individual contributor

to the organization. The performance-financial level measures the conver-

sion of outputs of goods or services attributable to the intervention in terms

of money and financial interpretation. The learning-expertise level mea-

sures whether human behaviors have effective results and optimal effi-

ciency, acquired through study and experience within a specialized domain.

The learning-knowledge level involves mental achievement acquired

through study and experience. Outcomes with a performance result, either

system or financial, are typically outcomes at either the organization or

group level (Rummler & Brache, 1995) and are analyzed further in this

study.

Measurement of OrganizationalPerformance Improvement Efforts

Assessing organizational performance results requires that mission-

related performance outcomes be carefully specified and connected to the

mission of the system (Swanson & Holton, 1999). Swanson and Holton

Collins / EVALUATION OF MANAGEMENT PROGRAMS 93



5/21

(1999) defined system as units of mission-related outputs in form of goods/

services that have value to the customer and that are related to the core orga-

nizational, work process, and group/individual contributors in the organiza-

tion (p. 14). Performance outcomes canbe at thewhole-system level (orga-nization) or be work process within the system (subsystems, or at the work

group (team) or individual level. According to Swanson and Holton (1999),

Every interventionshould lead to systems outcome(s) at some point (p. 69).

They defined outcomes as measures of effectiveness or efficiency relative

to core outputs of the system, subsystem, process, or individual (p. 231).

Examples of outcome measures include financial indicators, such as profit

or amount of goods sold.

Method

This analysis is a critical review of management development studies

from 1986 to 2000 that involve managers, leaders, and/or executives and

have either a system- or financial-level outcome. Both computerand manualsearches were used to locate studies. Keywords used in computer searches

with UNCOVER and ABI Inform included evaluation, assessment, out-

comes, impact, effectivenes, and influence in combination with the follow-

ing subject areas: leadership development, managerial training, manage-

ment training, management development, executive development,

leadership education, leadership, management education, and management

skills.

A manual search for published and unpublished management develop-

ment studies included the scanning of bibliographies; reviewing journal

indexes such as theJournal of Appl ied Psychology and Acad emy of Manage-

ment Journal; reviewing all leadership studies cited in The Impact of Lead-

ership by Clark, Clark, and Campbell (1992); reviewing proceedings from

the annual conferences of the Academy of Human Resource Development;

andby personally contacting presenters at the2000 annualconferenceof the

Society of Industrial and Organizational Psychology. Journal articles

included in this analysis appeared in the Journal of Applied Psychology,

Journal of Organizational Beha vior, and Public Personnel Management.

Approximately 1,000 titles, abstracts, and/or articles were located. From

this critical review of management development literature, an initial group

of 60 management development studies, which occurred since Burke and

Days (1986) meta-analysis of the effectiveness of managerial training, had

both performance and learning outcomes. Of those studies, 18 (36%) had

performance outcomes, which is the focus of this research, and therefore

compose the group of studies included in this analysis. The following cate-

gories were captured for each study: management development interven-

tion, thetraining content area, andthe intended outcome of themanagement




6/21

development experience. The management level of the individuals in the

sample, sample size, type of management development intervention, mea-

surement instrument(s), and evaluation outcome level also were recorded.

Detailed descriptions of the content areas and intervention categories areprovided in the following two sections.

To validate the coding scheme, each article was reviewed a second time

and the coding choice was compared with the first review to ensure similar-

ity. Two additional raters who were Ph.D. candidates in human resource

development independently coded a subsample of 25% of the studies.

Ratersdisagreed in onecoding element, andresolution wasbased on discus-

sion among raters.

Content Areas

Competency areas featured in the high-performance leadership compe-

tency model (Holton & Naquin, 2000) provided a basis for categorizing

studies into training content areas. The high-performance leadership com-petency model was used because it provides the only known definition of

management development outcomes that includes improving performance

as a core dependent variable or an explicit outcome of management. By

using this model, content areas more appropriately reflect leadership in

todays high-performance organizations.

Organizational-level performance outcomes of team management (Baker,

Walsh, & Marjerison, 2000) and strategic leadership competencies (Collins,

Lowe, & Arnett, 2000), which enable high-performance leaders to lead strategi-

cally in an environment of continuous change, were incorporated into the con-

tent areas of this study. Baker et al. (2000) described team management as the

development and leadership of strong, effective organizational teams and the

promotion of responsibility for team performance. Strategic leadership includes

transforming culture and values of the organization, implementing change, and

promoting continuous organizational improvement (Collins et al., 2000). Defi-

nitions of content areas in this analysis are as follows:

Employee development (ED): nurturing relationships that help employees reachtheir highest potential (Wilson, Boudreaux, & Edwards, 2000).

Humanresource systems (HRS): linkage of the right employee to theright job, andhuman resource policies and procedures to job activities. Evaluation of the organi-zations policiesand procedures in relationto employee performance, and develop-mentof leaders whose values closelymatch theorganization (Wilson et al.,2000).

Strategic leadership (SL): transforming the culture and values of the organization,institutionalizing change, and promoting continuous organizational improvement.Developing collaborative efforts and communicating strategy and performanceinformation to relevant subsystems (Collins et al., 2000).

Supportive environment (SE): building organizational systems (personnel, tech-nology, and control) with an open atmosphere and adequate resources, whichenables effective performance (Wilson et al., 2000).




7/21

Team management (TM): developing and leading strong, effective organizationalteams (Baker et al., 2000).

Intervention Categories

Because all management development interventions are not structured, for-

mal training programs, intervention categories for this study vary greatly and

have been loosely interpreted. Managementdevelopment interventions are cate-

gorized according to the full range of leadership development experiences as

described by McCauley et al. (1998). Experiences found in the 18 studies of this

analysis fit into the following four intervention categories:

Developmental relationships (DR): experiences in work settings in which anotherindividual influences the managers personal development such as one-on-onementoring or coaching.

Formal training programs (FT): structured training programs designed to developthe individual manager.

Jobassignment (JA):assignment to an entire job, as redesigninga systemor part ofa job, or serving on a temporary task force. Structured experiences (SX): Group activities that include goal-directed, live-

action, and task-based interactions such as leaderless group discussions, simula-tions, and targeted exercises.

Results and Discussion

Management development literature from 1986 through 2000 was

reviewed to determine trends in measuring organizational-level outcomes

of management development programs. Eighteen studies were located for

this research, and all occurred in the time span from 1991 to 1999 (see Table

1). No management development studies with organizational outcomes

were found in the literature from 1986 to 1990. Results of the research and

discussion of timing of the measurement activities, type of intervention,

intervention category, measurement instruments, and the measurement of

organizational performance are discussed in more depth in this article.

Timing of Evaluation

Over the years, most management development evaluations merely mea-

sured trainees reactions (Holton, 1996), with the evaluations completed at

the end of the program or at a short time period thereafter. This analysis pro-

vides conclusive evidence that testing of organizational performance as a

result of the management development intervention requires a greater com-

mitment of time than testing at Swanson and Holtons (1999) learning-

knowledge and learning-expertise levels.

An important finding is that the study by Syvantek and DeShon (1992)

evaluated the impact of the intervention over a 17-year time span. Also of


(text continues on p. 101)



8/21

97

97

TABLE 1: Management Development Studies From 1986 to 2000 With Performance Outcomes

Sample InterventionType of Measurement

Author(s) Sample Description Size Category Intervention Instrument

Employee development

Barling, Weber, and Bank branch managers 20 FT Transformational Organization Com

Kelloway (1996) Multifactor Leadership mitment Questio

Questionnaire (MLQ); naire; personal

leadership training and loan and credit

individual booster card sales

SessionsHuman resource systems

Avolio and Howell Senior executives in 237 JA Transformational Rotters Locus of

(1992) Canadian financial behavior of leader Control; Jackson

institutions personality

inventory; MLQ

Strategic leadership

Fullagar (1992) College presidents 637 JA Presidential success University revenu

presidential

succession

Glynn and Slepian Chief executive officer 61 JA Organizational name Organization nam

(1992) (CEO), chair, or presi- change change versus

dent of computer and leadership chang

financial service firms

Koene, Pennings, and Store managers in large 1,229 JA Organizational Hofstede construc

2002SAGEP

bli

ti

Alliht

d

N

tf

il

th

i

dditib

ti

byTomislavBunjevaconAugust22,2008


Downloadedfrom
http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/http://hrd.sagepub.com/


9/21

9898

Schreuder (1992) retail firm leadership style and scale measu

(Netherlands) satisfaction cultu

and performance

and motivation;

Basss leader cha

ma scale; Belgia

Dutch languageinstrument

Lohmann (1992) Business leaders in 358 JA Internationally competi- Survey

Japanese firms in tive leadership style

Hawaii

Moxnes and Eilertsen First-line managers in 320 FT Management training Argyriss learning

(1991) Norways largest program model I and II

private industry

Murphy and Settich Leaders at Catholic 6 JA Sharing of core values Interviews of pres

(1992) higher education by leaders dents and organi

institution tional members;

survey

Sashkin, Rosenbach, Private industry 8 FT Leadership training Leadership Behav

Deal, and Peterson business leaders program Questionnaire;

(1992) Leadership Desc

tion Questionnai

TABLE 1 Continued



2002SAGEP

bli

ti

Alliht

d

N

tf

il

th

i

dditib

ti



Downloadedfrom


10/21

99

Spoth (1992) Vice presidents, SBU, 7 JA Change in executive Repertory grid tec

and human resource ideology nique measuring

managers in Fortune ideology

500 companies

Syvantek and DeShon CEO of automobile 64 JA Leadership style Pre-post

(1992) manufacturing comparisons

company

Watad and Ospina Midlevel managers in Managerial training Managerial trainin

(1999) large nonprofit hospital 17 FT program individual and

group interviews

Westcott (1994) Management in 200 SX Applied behavior Analysis of improelectric and gas management training quality; probable

company opportunities for

quality solutions

Williams, Greene, and Top-level police 100 FT Strategic emphasis Department-wide

Bergman (1992) managers programs for managers survey instrumen

development of

strategic plan;

ascending to new

position of

leadership

Supportive environment Business-unit and 78 JA Implementation of MLQ Form 10;

Howell and Avolio corporate managers transformational Rotters Locus o

(1993) in large financial leadership style Control; measur

institution support for innovtion and risk-tak

scales

2002SAGEP

bli

ti

Alliht

d

N

tf

il

th

i

dditib

ti



Downloadedfrom


11/21

100 TABLE 1 Continued



Uhl-Bien and Graen Professionals in public 233 DR Leader-follower Project Leader-

(1992) sector service relationship; Member Exchan

scale; Team-Mem

Exchange scale

Team management

Penwell (1992) Leaders participating 309 SX Work groups SYMLOG (Syste

in teleconferences atic Multiple Obvations in Group

Riechmann (1992) Provost, vice president, 5 SX Formation of high- Self-report on tea

dean, associate dean, performance teams experience

department heads on

teams in higher

education setting

Note: FT = formal training programs; JA = job assignment; SX = structured experiences; DR = developmental relationships.

2002SAGEP

bli

ti

Alliht

d

N

tf

il

th

i

dditib

ti



Downloadedfrom


12/21

interest, the study by Spoth (1992) covers a 14-year time span. It is a signifi-

cant revelation that these two studies involve large corporations, one being

Chrysler, and the other Fortune 500 companies. Time frames of the other

evaluation studies varied from 5 years (Fullagar, 1992; Williams, Greene, &

Bergman, 1992), 4 years (Sashkin, Rosenbach, Deal, & Peterson, 1992), 3

years (Moxnes & Eilertsen, 1991; Uhl-Bien & Graen, 1992; Watad &

Ospina, 1999;Westcott, 1994), 2 years (Barling, Weber, & Kelloway, 1996),and 1 year (Avolio & Howell, 1992; Glynn & Slepian, 1992; Howell &

Avolio, 1993). It is important to note that nine studies (50%) were measured

in a time frame of less than 5 years. Also, some studies did not record a time

frame for the evaluation process (Koene, Pennings, & Schreuder, 1992;

Lohmann, 1992; Murphy & Settich, 1992; Penwell, 1992; Riechmann,

1992; see Figure 1).

When evaluating organizational performance outcomes, the researcher

must make a long-term commitment to observe the pay-off for the manage-

ment development intervention because organizational changes often take

many years to become evident. For example, some studies require the accu-

mulation of a minimum of 3 years of data to perform an evaluation that

determines whether a change in performance has occurred as a result of the

intervention. Career changes by HRD professionals, top management, ortrainees will likely affect the evaluation process during those years.


1 2 3 4 5 6-1011-15

15+Years

0

1

2

3

4

5

Numberofstudies

FIGURE 1: Time Period for Evaluation

Note: Information was unavailable for five studies.



13/21

Organizations cannot for the most part make a long-term time commit-

ment for evaluation of management development activities. In todays

world, work is fluid, organizations are flatter, organizational structures fre-

quently change, and people make more lateral movements (Byham, 1999).Onewouldassume that because time andresources arelimited, most organi-

zations provide training primarily for entry-level employees, and for that

reason one could likewise assume that more evaluations of training and

development interventions would primarily occur at those lower levels. In

addition, managers would be less likely to evaluate interventions affecting

their own performance for fear that the end result would not reflect posi-

tively on their performance. Nevertheless, with the implementation of any

plan, measuringoutcomes and comparing actual results to planned results is

essential.

Content Areas

Twelve studies (66%) focused on strategic leadership as the primary con-tent area for interventions. This is not surprising as a strategic leadership

intervention most likely occurs because of the need of the organization to

change as a result of globalization to meet the demands of a competitive

environment. Organizational change often involves a change in culture,

which typically occursnaturally over a periodof time as new beliefs andval-

ues are instilled in the organization. In addition, where there is a succession

of executives, the new leader must build credibility before becoming a role

model for employees, and this sometimes can take several years.

Two studies (11%) focused on team management and two (11%) focused

on a supportive environment as the content area for the intervention. The

remaining studies (n = 2, 11%) fell in theemployee development andhuman

resource systems categories (see Table 2).

Methods of Evaluation

Quantitative or qualitative method. The ideal assessment for organizational

performance incorporates both quantitative and qualitative measurement meth-

ods. Both qualitative and quantitative methods are needed to learn more about

the nature and application of transformational leadership, including how it can

be developed and used to create cultures in which both people and performance

are valued.

It is interesting to note that five studies (30%) reviewed for this report

exhibita mixture of quantitative and qualitative assessment methods (Koene

et al., 1992; Murphy & Settich, 1992; Riechmann, 1992; Sashkin et al.,

1992; Watad & Ospina, 1999). Nevertheless, the most common method of

assessment in this research was quantitative (n = 12, 66%). Quantitative

methods used in this study include instruments such as Rotters Locus of




14/21

Control, Jacksons personality inventory, the Multifactor Leadership Ques-

tionnaire, Hofstede construct, and Basss leader charisma scale and mea-

surements such as credit card sales.Quantitative methods are needed along with qualitative methods to deter-

mine more about the nature of both leadership and culture because no ques-

tionnaire alone (quantitative method) can possibly assess culture (Schein,

1990). This indicates that changes in performance based on transforma-

tional leadership should be evaluated with both qualitative and quantitative

methods to produce the best evaluation results.

Through qualitative research we can learn how to help leaders develop

and create or change cultures. Qualitative research can help define the

important variables and constructs and determine what is worth measuring,

at which point it becomes appropriate to turn to quantitative methods of

measurement. This analysis shows that 6%, one study (Riechmann, 1992),

used qualitative research methods only. Interviews were the qualitative

method used.

Measurement of informal training. Management development now includes

substantially more informal developmental programs than 15 years ago. How-

ever, informal training methods are difficult to measure. The focus of manage-

ment development has shifted from formal training programs to coaching,

action learning, self-development processes, mentoring, and peer-related learn-

ing activities (Garavan, Barnicle, & OSuilleabhain, 1999; McCauley et al.,

1998). One study (Uhl-Bien & Graen, 1992) used developmental relationships,

or strong leader-follower relationships, as the management development

intervention.

Pretest-posttest design. Four studies (22%) included in this report used a

pretest-posttest statistical design, or random assignment to experimental and con-

trol groups(Barlinget al., 1996; Moxnes& Eilertsen, 1991; Sashkin et al., 1992;

Syvantek & DeShon, 1992; see Table 3).


TABLE 2: Management Development Content Areas/Outcome Summary

Outcome Measurement

Performance Learning Content Area Number of Studies System Financial Exper tise Knowledge

Employee development 1a 0 1 1 0

Human resource systems 1a 1 0 1 0

Strategic leadership 12a 10 3 0 0

Supportive environment 2 2 0 0 0

Team management 2 2 0 0 0

Total 18 15 4 2 0

a. One study has multiple outcome dimensions.



15/21

Intervention Categories

Job assignments (JA) were the most common management developmentinterventions in this analysis (n = 9, 50%). Transformational leadership

changes in ideology, implementation of shared values, and succession of

presidents were the most common examples of job assignment experiences

that measured the job assignment intervention category. It is also interesting

that two studies in the job assignment intervention category had multiple

outcomes, one with financial and system (Koene et al., 1992) and one with

system and expertise (Avolio & Howell, 1992).

Formal training programs (FT) were found in five studies (27%) in this

analysis. A strategic emphasis in executive training programs was found

most often in the formal training interventions that measured organizational

performance. Of the remaining studies, three studies (17%) incorporated

structured experiences (SX) (Penwell, 1992; Riechmann, 1992; Westcott,

1994), and one study (5%) had developmental relationships (DR) as theintervention (Uhl-Bien & Graen, 1992).

The breakdown by intervention category is not surprising as transforma-

tional leadership, encompassed in this study in the job assignment (JA)

intervention category, basically changes strategic direction at the organiza-

tional level. According to Friedman (2000), Management always matters,

but in this more complex and fast-paced system, management and strategic

vision matter a lot more (p. 231). Therefore, it is important to align the goals

for management development programs with the strategic vision of the orga-

nization and to train managers in their new roles in strategic management.

Instrumentation

Multiple constructs were used to evaluate the effectiveness of manage-ment development programs in 14 cases (78%) in this study. Four studies, or


TABLE 3: Management Development Interventions/Outcome Summary

Outcome Measurement

Performance Learning Intervention Category Number of Studies System Financial Expertise Knowledge

Developmental relationships 1 1 0 0 0

Formal training 5a

4 1 1 0

Job assignments 9b

7 3 1 0

Structured experiences 3 3 0 0 0

Total 18 15 4 2 0

a. One study has multiple outcome dimensions.

b. Two studies have multiple outcome dimensions.



16/21

22%, used oneinstrumentto measure effectiveness of theprogram(Glynn &

Slepian, 1992; Lohmann, 1992; Riechmann, 1992; Spoth, 1992). Quantita-

tive methods were used most often with 17 (94%) studies using instruments

developed outside of the organization, such as Argyris Learning Model,Repertory Grid Technique, Hofstede construct, Rotters Locus of Control,

Jacksons personality inventory, Multifactor Leadership Questionnaire,

Bass leader charisma scale, Leadership Behavior Questionnaire, Leader-

ship Description Questionnaire, and Project Leader-Member Exchange

Scale. Quantitative methods measuring group performance often used the

Systematic Multiple Observation of Groups instrument. It is important to

note that the only qualitative study in this analysis (Riechmann, 1992) used

interviews as the instrumentation method.

Measurement of Organizational Performance

The best practice in evaluation processes is to test and analyze relation-

ship and linkages between performance measures at different levels of anal-ysis. For instance, how do outcomes at the process, critical subsystem, and

individual levels affect important mission-level outcomes? Of the studies

reviewed, 2 (11%) intentionally evaluated the management development

experience at both the performance and learning levels (Avolio & Howell,

1992; Barling et al., 1996). All other studies (n = 16, 89%) completed man-

agement development evaluations on performance-level data (or level 4

data) only. Of those 16 evaluations on the performance level, 13 studies

(81%) evaluated system performance only, 2 (12%) evaluated financial per-

formance only, and 1 (7%)evaluated both system and financial performance

at the organizational level (Koene et al., 1992).

Performance-level evaluation is normally considered to be at the system

level, but a system outcome can be in subsystems such as a functional unit,

work group, or team within the organization (Swanson & Holton, 1999).

The results of this study indicate that performance-level evaluations occur

in subsystems. Six studies (33%) measured performance of business units,

teams, or groups, which are considered subsystems within an organization

(Avolio & Howell, 1992; Howell & Avolio, 1993; Penwell, 1992; Riech-

mann, 1992; Spoth, 1992; Uhl-Bien & Graen, 1992). However, research

indicates that there is a problem in linking leadership, in terms of individual

behavior and characteristics, to a system-level variable as organizational

culture (Sashkin et al., 1992).

Recommendations for Future Research

HRD must make huge strides in the development of evaluation activities,

especially those that test the long-term impact of the intervention on the

organization. Many studies use the measurement of individual learning out-




17/21

comes to reflect performance at the organizational level. However, there is

no research that explicitly justifies that learning at the individual level trans-

lates to organizational performance (Bates, 1999).

Future research must be done on the outcomes of management develop-ment programs at the organizational level. Some topics that should be

researched further are as follows.

HRD professionals must take the lead in addressing the lack of performance-

level evaluation methods. A limited number of management development pro-

grams with organizational variables as the outcome are reported in the literature.

Obviously, standard evaluation methods as known today are not enough for

organizations to measure organizational-level performance improvement.

Because organizations are facing a multitude of outcome-based demands on

their timeand resources, the development of an evaluation instrument should not

be left up to the management of the organization. HRD should take the lead in

combining evaluation theory with performance-based management develop-

ment theory to create the appropriate system for measurement of organizational-

level performance improvement.

This research indicates that the primary content area of performance-

level evaluations is in the area of strategic leadership. The nature and appli-

cation of transformational leadership, including how it can be developed

and used to create cultures in which both people and performance are val-

ued, cannot possibly be measured by questionnaires alone. Therefore, HRD

should develop both qualitative and quantitative evaluation methods to pro-

duce the best results for assessing changes as a result of strategic develop-

ment interventions.

HRD must take the initiative to link management development programs to

organizational strategy. Sometimes management training programs failbecause

they have no connection to real life in the company (Berry, 1990; Carlisle &

Henrie, 1993) or fail to add value to corporate strategy (Swanson & Holton,

1999). Traditionally, training and development systems are relegated to nar-

rowly defined support roles, in which individualsare trained around current job-

based deficiencies or predicted knowledge and skill needs (Olian et al., 1998).

For results to occur, the intervention must be linked with organizational goals

and have utility or pay-off to the organization (Swanson & Holton, 1999). HRD

should take the lead by strategically aligning training and development systems

that advance and sustain the organizations competitive position in its market.

From this study, we become more aware that job assignment interven-

tions train managers in their new roles in strategic management and more

closely tie the intervention to real-life experiences on the job. HRD should

continue to provide job assignment experiences linked to the organizational

strategy as preferred management development experiences. However,

developmental relationships, mentoring, and coaching experiences should

be included as well.




18/21

Because the content area of the majority of studies with performance-

level outcomes is strategic development, HRD professionals must change

from the Kirkpatrick (1996) reaction paradigm of evaluation to one that

measures effectiveness of strategic development initiatives. Therefore, it isthe responsibility of HRD to create and use a model that has the appropriate

constructs, which can be used as a diagnostic tool for determining the criti-

cal strategic influences to be measured along w ith outcomes. Because con-

tinuous change is a core value of high-performance organizations (Collins

et al., 2000), HRD practitioners shouldconcentrate on providingmore man-

agement developmentprogramswith a strategic leadership focus for organi-

zations to compete successfully in a global environment.

Conclusion

The evaluation task is daunting. Research reported in the literature on the

effectiveness of management development programs is sparse, partially

because of ineffective performance-level evaluation methods. Outstandingpractices in management developmentindicatethat high-performance orga-

nizations always assess the impact of their leadership development pro-

cesses (Collins et al., 2000; Fulmer & Wagner, 1999). Program assessment

is seen as a way to generate buy-in and focus on the current objectives.

HRD professionals should become the leaders in developing measure-

ment methods for organizational-level outcomes so that organizations

develop the bench strength to compete globally. However, the challenge is

huge for HRD because, for the evaluation process to be meaningful for the

organization, the assessments type and intensity depend on the objectives

of the management development effort and the organizations culture.

Therefore, evaluation methods must be specific, but yet broad enough, to

satisfy the evaluation needs of all organizations while providing methods to

conduct empirical research on outcomes of management development

programs.

References

Note: Management development studies included in this analysis are marked with an asterisk.

Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H., & Shotland, A. (1997). A meta-analysis

of the relations among training criteria. Personnel Psychology, 50, 341-359.

*Avolio, B. J., & Howell, J. M. (1992). The impact of leadership behavior and leader-follower per-

sonality match on satisfactionand unit performance. In K. E. Clark,M. B. Clark,& D. P. Camp-

bell (Eds.), Impact of leadership (pp. 225-236). Greensboro, NC: Center for Creative

Leadership.

Baker, D. E., Walsh, M. B., & Marjerison, L. (2000). High-performance leadership at the process

level. In E. F. Holton & S. S. Naquin (Eds.), Developing high-performance leadership compe-

tency. San Francisco: Berrett-Koehler.




19/21

*Barling, J., Weber, T., & Kelloway,E. K. (1996). Effects of transformational leadershiptraining on

attitudinal and financialoutcomes: A field experiment.Journal of Applied Psychology, 81, 827-

832.

Bates, R. A. (1999). Measuring performance improvement. Advances in Developing Human

Resources, 1, 47-67.Berry, J. K. (1990, August). Linking management development to business strategies. Training and

Development, pp. 20-21.

Brungardt, C. (1996). The making of leaders: A review of the research in leadership development

and education. The Journal of Leadership Studies, 3(3), 81-95.

Burke, M. J., & Day, R. R. (1986). A cumulative study of the effectiveness of managerial training.

Journal of Applied Psychology, 71, 242-245.

Byham, W. C. (1999). Grooming next-millennium leaders. HR Magazine, 44(2), 46-51.

Campbell, J. P. (1988). Training design for productivity improvement. In J. P. Campbell & R. J.

Campbell (Eds.), Productivity in organizations (pp. 171-215). San Francisco: Jossey-Bass.

Carlisle,K. E.,& Henrie, D. (1993, August).Are youdoinghigh-impactHR? Training andDevelop-

ment, pp. 47-53.

Clark, K. E., Clark, M. B., & Campbell, D. P. (1992). The impact of leadership. Greensboro, NC:

Center for Creative Leadership.

Collins, D. B. (2000). Organizational performance: The future focus of leadership development.

Journal of Leadership Studies, 7(4), 43-54.Collins, D. B., Lowe, J. S., & Arnett, C. R. (2000). High-performance leadership at the organiza-

tional level. In E. F. Holton & S. S. Naquin (Eds.), Developing high-performance leadership

competency. San Francisco: Berrett-Koehler.

Dionne, P. (1996). The evaluation of training activities: A complex issue involving different stakes.

Human Resource Development Quarterly, 7, 279-286.

Friedman, T. L. (2000). The Lexus and the olive tree. New York: Anchor.

*Fullagar, P. K. (1992). The impact of institutionalfinancialperformance on executive successionin

higher education. In K. E. Clark, M. B. Clark, & D. P. Campbell (Eds.), Impact of leadership

(pp. 495-504). Greensboro, NC: Center for Creative Leadership.

Fulmer,R. M., & Wagner, S. (1999). Leadership: Lessons fromthe best.Training andDevelopment,

53(3), 28-33.

Garavan, T. N., Barnicle, B., & OSuilleabhain, F. (1999). Management development: Contempo-

rary trends, issues and strategies. Journal of European Industrial Training, 23(14-15), 191.

Gibler,D., Carter,L., & Goldsmith, M. (2000).Best practices in leadership development handbook.

San Francisco: Jossey-Bass.

*Glynn, M. A.,& Slepian, J. (1992). Leadersand transitions: Therole of leadership in corporate name

change. InK. E. Clark, M.B. Clark, & D. P. Campbell (Eds.),Impact of leadership (pp. 305-312).

Greensboro, NC: Center for Creative Leadership.

Goldstein, I. L. (1986). Training in organizations: Needs assessment, design, and evaluation.

Monterey, CA: Brooks-Cole.

Gordon, J. (1987, June). Romancing the bottom line. Training, pp. 31-42.

Holton, E. F., III. (1996). The flawed four-level evaluation model. Human Resource Development

Quarterly, 7(1), 5-21.

Holton, E. F., III, & Naquin, S. S. (2000). Developing high-performance leadership competency.

San Francisco: Berrett-Koehler.

*Howell, J. M., & Avolio, B. J. (1993). Transformational leadership,transactional leadership, locus

of control, and support for innovation: Key predictors of consolidated business-unit perfor-

mance. Journal of Applied Psychology, 78, 891-902.

Kirkpatrick, D. L. (1996). Invited reaction: Reaction to Holton article. Human Resources Develop-

ment Quarterly, 7, 5-15.




20/21

*Koene, B.A.S., Pennings, J. M., & Schreuder, H. (1992). Leadership, organizational culture, and

organizationaloutcomes. InK. E.Clark, M.B. Clark,& D.P.Campbell(Eds.),Impact of leader-

ship (pp. 215-224). Greensboro, NC: Center for Creative Leadership.

Kraiger,K., Ford, J., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories

of learning outcomes to new methods of training evaluation.Journal of Applied Psychology, 75,311-328.

*Lohmann,D. (1992). The impact of leadershipon corporatesuccess: A comparative analysis of the

Americanand Japanese experience.In K.E. Clark,M. B.Clark, & D.P.Campbell (Eds.),Impact

of leadership (pp. 59-80). Greensboro, NC: Center for Creative Leadership.

Lynham, S.A., & Swanson,R. A.(1997).The developmentand evaluationof a modelof responsible

leadershipfor performance: Beginning the journey. Proceedings of the 1997Annual Conference

for the Academy of Human Resource Development, pp. 570-577.

McCauley, C. D., Moxley, R. S., & Van Velsor, E. (1998). The center for creative leadership hand-

book of leadership development. San Francisco: Jossey-Bass.

Moller, L., & Mallin, P. (1996). Evaluation practices of instructional designers and organizational

supports and barriers. Performance Improvement Quarterly, 9(4), 82-92.

*Moxnes,P.,& Eilertsen,D. (1991). The influenceof managementtraining uponorganizational cli-

mate: An exploratory study. Journal of Organizational Behavior, 12, 399-411.

*Murphy, J. P., & Settich, J. F. (1992). Taking the measure of leadership impact: How leaders share

valuesin Catholichigher education. InK. E.Clark, M.B. Clark,& D.P.Campbell (Eds.),Impactof leadership (pp. 365-378). Greensboro, NC: Center for Creative Leadership.

Olian, J. D., Durham, C. C., Kristoff, A. L., Brown, K. G., Pierce, R. M., & Kunder, L. (1998).

Designing management training and development for competitive advantage: Lessons from the

best. Human Resource Planning, 21(1), 20-33.

*Penwell,L. W. (1992). The impact of leaders on group and intergroup dynamics in teleconferences

between interdependent work teams. In K. E. Clark, M. B. Clark, & D. P. Campbell (Eds.),

Impact of leadership (pp. 477-486). Greensboro, NC: Center for Creative Leadership.

Phillips, J. J. (1997). Handbook of training evaluation and measurement methods . Houston, TX:

Gulf.

*Riechmann, D. (1992). High-involvement, high-performance teams in higher education: The

impactof leadership. In K. E. Clark, M. B. Clark,& D. P. Campbell (Eds.),Impact of leadership


Rummler, G. A., & Brache, A. P. (1995). Improving performance: How to manage the white space

on the organization chart. San Francisco: Jossey Bass.

*Sashkin, M., Rosenbach, W. E., Deal, T. E., & Peterson, K. D. (1992). Assessing transformational

leadership anditsimpact.In K.E. Clark,M. B.Clark, & D.P.Campbell (Eds.),Impact of leader-

ship (pp. 131-148). Greensboro, NC: Center for Creative Leadership.

Schein, E. H. (1990, Summer). Organization development and the study of organizational culture.

Academy of Management OD Newsletter, pp. 3-5.

Sogunro, O. A. (1997). Impact of training on leadership development. Evaluation Review, 21, 731-

737.

*Spoth, J. (1992). The impact of executive ideology on structural change. In K. E. Clark, M. B.

Clark, & D. P. Campbell(Eds.),Impact of leadership (pp.281-292). Greensboro, NC:Center for

Creative Leadership.

*Syvantek, D. J., & DeShon, R. P. (1992). Leaders and organizational outcomes in established

industries: An analysis of Lee Iacocca and the American automobile industry. In K. E. Clark,

M. B. Clark, & D. P. Campbell (Eds.), Impact of leadership (pp. 293-304). Greensboro, NC:

Center for Creative Leadership.

Swanson, R. A.,& Holton, E. F. (1999).Results: How to assess performance, learning, and percep-

tions in organizations. San Francisco: Berrett-Koehler.




21/21

*Uhl-Bien, M.,& Graen,G. B. (1992). An empiricaltest of theleadership-making model in profes-

sionalproject teams.In K. E. Clark,M. B. Clark, & D. P. Campbell (Eds.),Impact of leadership


*Watad, M., & Ospina, S. (1999). Integrated managerial training: A program for strategic manage-

ment development. Public Personnel Management, 28, 185-196.*Westcott, R. (1994).In action: Measuring return on investment(Vol. 1). Alexandria,VA:American

Society for Training and Development.

*Williams, W. L., Greene, J. R., & Bergman, W. T. (1992). Strategic leadership in a big-city police

department: The Philadelphiastory. In K. E. Clark, M. B. Clark, & D. P. Campbell(Eds.),Impact

of leadership (pp. 107-118). Greensboro, NC: Center for Creative Leadership.

Wilson, L. S., Boudreaux, M. A., & Edwards, M. (2000). High-performance leadership at the indi-

vidual level. In E. F. Holton & S. S. Naquin (Eds.), Developing high-performance leadership

competency. San Francisco: Berret-Koehler.

Dori s B. Colli ns is pursui ng her Ph.D. in human resource develo pment at

Louis iana S tate Unive rsity with research i nterest s in the a reas of managerial

leadership development and organizational performance. She has more than

30 years experience in college and university administration and is currently

serving as the associate vice chancellor for student life and academic ser-

vices at Louisiana State U niversity. She is a nationally recognized consultant

in college student affairs administration, has participated in national

teleconferences, and has chaired national committees and task forces on

issues related to services and programs on college campuses.


Documents

Performance Level EVALUATION Methods Used in MAN DEV Studies From 1986 to 2000