4932328 Evaluating Management Development

8/10/2019 4932328 Evaluating Management Development

1/29

Working Paper 1

EVALUATING MANAGEMENT DEVELOPMENT

JOHN KENWORTHYDBA 17

(HUMAN RESOURCE MANAGEMENT)

31 Mount Sinai View

Singapore 276818

Tel: +65 62235638

Fax: +65 6223563

1


2/29


3/29

Working Paper 1

4eter%ine" w!et!er t!e per(or%an'e inter#ention a'!ie#e$ it" goal" ontrolling,

0""e""e" t!e #alue o( t!e per(or%an'e inter#ention again"t organi9ational re"ult"

Pro#ing,

$enti(ie" (uture '!ange" or i%pro#e%ent" to t!e inter#ention %pro#ing,

0''or$ing to u""ell& w!i'!e#er e#aluation %o$el i" '!o"en to (ollow ;t!e e#aluation "!oul$

(o'u" on !ow e((e'ti#e t!e per(or%an'e inter#ention wa" at %eeting t!e organi9ation


4/29


5/29

Working Paper 1

$i"'o#ere$ t!at onl- eig!t 'o%peten'ie" 'oul$ 'on"i"tentl- pre$i't per(or%an'e in an-

exe'uti#e wit! 8> per'ent a''ura'-)

T!e (ir"t "'ienti(i'all- #ali$ "et o( "'ale$ 'o%peten'ie" B 'o%peten'ie" t!at !a#e "et" o(

*e!a#iour" or$ere$ into le#el" o( "op!i"ti'ation or 'o%plexit- B were $e#elope$ *-

a-HM'Der Spen'er an$ Spen'er& 13,) T!e 'o%peten'ie" (oun$ to *e t!e %o"t 'riti'al (or

e((e'ti#e %anager" in'lu$e:

0'!ie#e%ent Crientation

4e#eloping Ct!er"

4ire'ti#ene""

%pa't an$ n(luen'e

nterper"onal In$er"tan$ing

Crgani"ational 0warene""

Tea% @ea$er"!ip

T!i" "et o( '!ara'teri"ti'"& or in$i#i$ual 'o%peten'ie"& t!at a %anager *ring" to t!e ?o* nee$

to %at'! well to t!e ?o* or a$$itional e((ort %a- *e ne'e""ar- to 'arr- out t!e ?o* or t!e

%anager %a- not *e a*le to u"e 'ertain %anagerial "t-le" e((e'ti#el-) T!e"e are in turn

a((e'te$ *- t!e organi"ational 'li%ate an$ t!e a'tual reuire%ent o( t!e ?o*)

Managerial e((e'ti#ene"" i" t!e 'o%*ination o( t!e"e (our 'riti'al (a'tor"& Crgani"ational

li%ate& Managerial St-le"& Jo* euire%ent" an$ t!e n$i#i$ual o%peten'ie" t!at t!e

%anager *ring" to t!e ?o*) e$$in 17>, point" out t!at K%anagerial e((e'ti#ene""K i" not a

ualit- *ut a "tate%ent a*out t!e relation"!ip *etween !i" *e!a#iour an$ "o%e ta"k "ituation)

0''or$ing to e$$in& it i" t!ere(ore not po""i*le to $i"'o#er a" (a't t!e ualitie" o(

e((e'ti#ene"" w!i'! woul$ t!en *e o( "el(/e#i$ent #alue)

(& !owe#er t!e a-HM'Der 'o%peten'ie" are a*le to pre$i't per(or%an'e in an- exe'uti#e to

8> per'ent a''ura'- an$ t!e"e 'o%peten'ie" are Atraina*le< t!en an- training progra%%e

$e"igne$ to $e#elop %anagerial e((e'ti#ene"" in an- role 'an *e e#aluate$ *- %ean" o(

a""e""ing t!e '!ange" in *e!a#iour o( t!e parti'ipant t!at $e%on"trate" t!e 'o%peten'-)

5


6/29

Working Paper 1

@earning out'o%e"

Mo"t re"ear'!er" 'on'eptuali9e learning a" a %ulti$i%en"ional 'on"tru't) T!ere i"

'on"i$era*le 'o%%onalit- a'ro"" $i((erent atte%pt" to 'la""i(- t-pe" o( learning out'o%e")

Graiger et al)& 13, "-nt!e"i"e$ t!e work o( Lagne& 18,& 0n$er"on& 182, an$ ot!er"propo"ing *roa$ *a"e$ 'ategorie" o( learning out'o%e": "kill/*a"e$& 'ogniti#e an$ a((e'ti#e

out'o%e")

Skill/*a"e$ learning out'o%e" a$$re"" te'!ni'al or %otor "kill") 0 nu%*er o( ga%e/*a"e$

in"tru'tional progra%%e" !a#e *een u"e$ (or pra'ti'e an$ $rill o( te'!ni'al "kill") For

exa%ple& Lop!er et al)& 1, in#e"tigate$ t!e e((e'ti#ene"" o( t!e u"e o( an a#iation

'o%puter ga%e *- %ilitar- trainee" on "u*"euent te"t (lig!t")

ogniti#e learning out'o%e" in'lu$e t!ree "u*'ategorie" o(:

4e'larati#e knowle$ge) T!e learner i" t-pi'all- reuire$ to repro$u'e or re'ogni"e

"o%e ite% o( in(or%ation) For exa%ple& W!ite& 18, $e%on"trate$ t!at "tu$ent"

w!o pla-e$ a 'o%puter ga%e (o'u"ing on ewtonian prin'iple" were a*le to %ore

a''uratel- an"wer ue"tion" on %otion an$ (or'e t!an t!o"e w!o $i$ not pla- t!e

ga%e)

Pro'e$ural knowle$ge) euire" a $e%on"tration o( t!e a*ilit- to appl- knowle$ge&

general rule"& or "kill" to a "pe'i(i' 'a"e) For exa%ple& W!ite!all an$ M'4onal$&

13, (oun$ t!at "tu$ent" u"ing a #aria*le pa-o(( ele'troni'" ga%e a'!ie#e$ !ig!er

"'ore" on ele'troni'" trou*le"!ooting ta"k" t!an "tu$ent" w!o re'ei#e$ "tan$ar$ $rill

an$ pra'ti'e)

Strategi' knowle$ge) euire" appli'ation o( learne$ prin'iple" to $i((erent 'ontext" or

$eri#ing new prin'iple" (or general or no#el "ituation" re(erre$ to *- ot!er"& a"

on"tru'ti#i"t @earning 4e$e& 17,,& Woo$ an$ Stewart& 187,& (or exa%ple& (oun$

t!at u"e o( a 'o%puter ga%e to i%pro#e pra'ti'al rea"oning "kill" o( "tu$ent" le$ to

i%pro#e%ent" in 'riti'al t!inking)

0((e'ti#e learning out'o%e" re(er" to attitu$e") n'lu$e" (eeling" o( 'on(i$en'e& "el(/e((i'a'-&

attitu$e"& pre(eren'e" an$ $i"po"ition") So%e re"ear'! !a" "!own t!at ga%e" 'an in(luen'e

attitu$e"& (or exa%ple Wie*e an$ Martin& 1, an$ T!o%a" et al)& 17,)

6


7/29

Working Paper 1

Leneral approa'!e" to e#aluation AS'!ool", 'ontra"t" responsivee#aluation wit! t!e

preor$inate approa'! o( experi%ental re"ear'!) T!e latter reuire" t!e $e"ign to *e 'learl-"pe'i(ie$ an$ $eter%ine$ *e(ore e#aluation *egin"& it %ake" u"e o( Ao*?e'ti#e< %ea"ure"&

e#aluate" t!e"e again"t 'riteria $eter%ine$ *- progra%%e "ta((& an$ pro$u'e" report" in t!e

(or% o( re"ear'!/t-pe report") n 'ontra"t& re"pon"i#e e#aluation i" 'on'erne$ wit!

progra%%e a'ti#itie" rat!er t!an intention"& an$ take" a''ount o( $i((erent #alue per"pe'ti#e")

n a$$ition& Stake "tre""e" t!e i%portan'e o( atte%pting to re"pon$ to t!e au$ien'e


16/29

Working Paper 1

EPERIENCE WITH TRAINING PROGRAMME EVALUATION MODELS

Man- re"ear'!er" %ea"uring t!e e((e't" o( training !a#e looke$ at one or %ore o( t!e

out'o%e" i$enti(ie$ *- Girkpatri'k 15H6>& 1,: rea'tion"& learning& *e!a#iour& an$

re"ult")

#aluation o( Trainee rea'tion" to learning !a" -iel$e$ %ixe$ re"ult" u""/(t an$ Pre"kill&

2>>1,& -et it i" o(ten t!e onl- (or%al e#aluation o( a training progra%%e an$ relie$ upon to

a""e"" learning an$ *e!a#iour '!ange) t %a- *e rea"ona*le to a""u%e t!at en?o-%ent i" a

pre'ur"or to learning& an$ t!at i( a trainee en?o-" t!e training& t!e- are likel- to learn *ut "u'!

an a""u%ption i" not "upporte$ *- a %eta/anal-ti' "tu$- 'o%*ining t!e re"ult" o( "e#eral ot!er

"tu$ie" 0lliger et al)& 17,)

Girkpatri'k>1,)

Training tran"(er i" $e(ine$ a" appl-ing t!e knowle$ge& "kill"& an$ attitu$e" a'uire$ $uring

training to t!e work "etting) Mo"t organi"ation" are genuinel- intere"te$ in t!i" a"pe't o( t!e

e((e'ti#ene"" o( training e#ent" an$ progra%%e" B -et t!e pau'it- o( re"ear'! $e$i'ate$ to

tran"(er o( training 'ontra$i't" t!e i%portan'e o( t!e i""ue") So%e re"ear'! in t!i" area !a"

(o'u""e$ on 'o%pari"on o( alternati#e 'on$ition" to training tran"(er) 0 t-pi'al $e"ign o( "u'!

re"ear'! 'o%pare" group" w!o re'ei#e $i((erent training %et!o$" an$Hor a A'ontrol group< t!at

re'ei#e" no training) Su'! "tu$ie" $o not all u"e t!e "a%e re"ear'! $e"ign an$ t!e re"ult" are

in'on"i"tent) owe#er& t!e re"ear'! $oe" in$i'ate t!at "o%e (or% o( po"t/training tran"(er

"trateg- (a'ilitate" training tran"(er u""/(t an$ Pre"kill& 2>>1,)

#aluating e"ult" o( training in ter%" o( *u"ine"" re"ult"& (inan'ial re"ult" an$ return on

in#e"t%ent C, i" %u'! $i"'u""e$ in t!e popular literature B %o"t o((ering ane'$otal

e#i$en'e or 'on?e'ture a*out t!e ne'e""it- o( e#aluating training


17/29

Working Paper 1

te'!niue" t!at 'oul$ *e applie$ to e#aluating training w!il"t re'ogni"ing t!at t!ere are

'o%%on rea"on" w!- "u'! (inan'ial anal-"e" are rarel- 'on$u'te$ or reporte$:

t i" $i((i'ult to uanti(- or e"ta*li"! a %onetar- #alue (or %an- a"pe't" o( training

o u"a*le 'o"t/*ene(it tool i" rea$il- a#aila*le

T!e ti%e lag *etween training an$ re"ult" i" pro*le%ati'

Training %anager" an$ trainer" are not (a%iliar wit! (inan'ial %o$el")

@ittle progre"" appear" to !a#e *een %a$e in t!i" area "in'e Mo"ier wrote)

CHOOSING AN EVALUATION STYLE

EPERIENCE WITH TRAINING PROGRAMME EVALUATION MODELS

e#iewing t!e !i"tor- an$ $e#elop%ent o( training e#aluation re"ear'! "!ow" t!at t!ere are

%an- #aria*le" t!at ulti%atel- a((e't !ow trainee" learn an$ tran"(er t!eir learning in t!e

workpla'e) u""/(t an$ Pre"kill "ugge"t t!at a 'o%pre!en"i#e e#aluation o( learning&

per(or%an'e & an$ '!ange woul$ in'lu$e t!e repre"entation o( a 'on"i$era*le nu%*er o(

#aria*le" u""/(t an$ Pre"kill& 2>>1,) Su'! an approa'!& w!il"t lau$a*le in ter%" o( puri"t

a'a$e%i' re"ear'!& i" likel- to 'au"e anot!er pro*le%& t!at o( 'olle'ting $ata to $e%on"trate

t!e a((e't" an$ e((e't" o( "o %an- in$epen$ent #aria*le" an$ (a'tor") T!u"& we nee$ to

re'ogni"e t!at t!ere i" a tra$e o(( *etween t!e 'o"t an$ $uration o( a re"ear'! $e"ign an$

in'rea"ing t!e ualit- o( t!e in(or%ation w!i'! it generate" Warr et al)& 17>,)

a%*lin 17, point" out t!at a 'on"i$era*le a%ount o( e#aluation re"ear'! !a" *een $one)

T!i" re"ear'! !a" *een 'arrie$ out wit! a great #ariet- o( (o'al t!eorie"& u"uall- looking (or

'on"i"tent relation"!ip" *etween e$u'ational %et!o$" an$ learning out'o%e"& u"ing a #ariet-

o( o*"er#ational %et!o$" *ut wit! a (airl- 'on"i"tent an$ "ta*le *a'kgroun$ t!eor-) owe#er&

t!e un$erl-ing t!eor- !a" *een taken (ro% *e!a#iourali"t p"-'!olog- "u%%e$ up a" t!e

Apatient< / !ere t!e e""ential #iew i" t!at t!e "u*?e't patient, $oe" *e!a#iour or re"pon"e, i" a

'on"euen'e o( w!at !a" *een $one to !i% treat%ent or "ti%ulu",)

0not!er approa'! a''or$ing to Durgo-ne Durgo-ne an$ ooper& 175, w!i'! re"ear'!er"

appear to take to a#oi$ 'on(ronting #alue i""ue" i" to !ol$ t!at all #alue ue"tion" 'an

ulti%atel- *e re$u'e$ to ue"tion" o( (a't) T!i" u"uall- take" t!e (or% o( regar$ing t!e ualit-

o( K%anagerial e((e'ti#ene""K a" a per"onal ualit- w!i'! i" potentiall- o*?e'ti#el- %ea"ura*le&

an$ t!ere(ore a ualit-& t!e po""e""ion o( w!i'! 'oul$ *e a""e""e$ a" a %atter o( (a't) n

17


18/29

Working Paper 1

pra'ti'e t!i" approa'! !a" appeare$ elu"i#e) Sea"!ore et al& Sea"!ore et al)& 16>, (elt t!at t!e

exi"ten'e o( t!e 'on'ept o( K%anagerial e((e'ti#ene""K a" a unitar- trait 'oul$ *e 'on(ir%e$& i(

t!e- (oun$ !ig! inter'orrelation" *etween (i#e i%portant a"pe't" o( %anagerial *e!a#iour:

!ig! o#erall per(or%an'e rating"& !ig! pro$u'ti#it-& low a''i$ent re'or$& low a*"enteei"% an$

(ew error"

T!ere are no (or'e$ rule" a*out t!e "t-le o( e#aluation u"e$ (or a parti'ular e#aluati#e

purpo"e) owe#er& a"ter*-/S%it!& 1, "ugge"t" t!at "tu$ie" ai%e$ at (ul(illing t!e

purpo"e o( provingwill ten$ to *e lo'ate$ towar$" t!e Are"ear'!< en$ o( t!e $i%en"ion& an$

"tu$ie" ai%e$ at improving will ten$ to *e lo'ate$ near t!e Aprag%ati'< en$) Cn t!e

%et!o$ologi'al $i%en"ion t!ere %a- *e %ore 'on'ern wit! provingat t!e A"'ienti(i'< en$& an$

learningat t!e A'on"tru'ti#i"t< en$) ,

ISSUES O* EVALUATION *OR SIMULATIONS

Margaret Lre$ler 16, note" t!at a %a?or $e"ign weakne"" o( %o"t "tu$ie" e#aluating

"i%ulation *a"e$ training an$ $e#elop%ent i" t!at t!e- are 'o%pare$ to regular 'la""roo%

in"tru'tion) owe#er& t!e in"tru'tional goal" (or ea'! 'an $i((er) Si%ilarl-& %an- "tu$ie" "!ow

%ea"ure%ent pro*le%" in t!e nature o( t!e po"t/te"t" u"e$)

T!e pro*le%" !ig!lig!te$ a*o#e regar$ing t!e '!oi'e o( e#aluation %et!o$olog- are (urt!er

'o%poun$e$ *- t!e la'k o( well/$e"igne$ re"ear'! "tu$ie" in t!e $e#elop%ent an$ u"e o(

ga%e" an$ "i%ulation" Lre$ler& 16, B %u'! o( t!e pu*li"!e$ literature 'on"i"t" o(

ane'$otal report" an$ te"ti%onial" pro#i$ing "ket'!- $e"'ription" o( t!e ga%e or "i%ulation

an$ report onl- on per'ei#e$ "tu$ent rea'tion")

Mo"t o( t!e re"ear'!& a" note$ *- Pier(- 177, i" (lawe$ *- *a"i' weakne""e" in *ot! $e"ign

an$ %ea"ure%ent) So%e "tu$ie" i%ple%ente$ ga%e" or "i%ulation" t!at were *rie( treat%ent"

o( le"" t!an > %inute" an$ a""e""e$ e((e't" week" later) nter#ening in"tru'tion in t!e"e 'a"e"&

!owe#er& 'onta%inate$ t!e re"ult")

DO SIMULATIONS RE+UIRE A DI**ERENT ASSESSMENT O*

E**ECTIVENESS,

M'kenna 16, 'ite" %an- re(eren'e" to paper" 'riti'i"ing t!e e#aluation o( learning

e((e'ti#ene"" u"ing ntera'ti#e Multi%e$ia an$ %an- re"ear'!er" a"k (or %ore re"ear'! lark

18


19/29

Working Paper 1

an$ raig& 12, ee#e"& 13, to uantitati#el- e#aluate t!e real *ene(it" o( "i%ulation" an$

ga%e"/*a"e$ learning)

T!e ue"tion w!et!er Si%ulation *a"e$ learning reuire" a new an$ $i((erent a""e""%ent

te'!niue" *e-on$ t!o"e in u" e to$a- re%ain" relati#el- unexplore$) 0lt!oug! t!ere !a#e

*een "o%e atte%pt" at 'on"tru'ting t!eoreti'al (ra%ework" (or t!e e#aluation o( #irtual worl$"

o"e& 15,& W!itelo'k et al)& 16,& #er- (ew working exa%ple" or report" on t!e pra'ti'al

u"e o( t!e"e (ra%ework" exi"t")

W!itelo'k et al) 16, argue t!at e((e'ti#e e#aluation %et!o$" nee$ to *e e"ta*li"!e$ to

$i"'o#er i( 'on'eptual learning take" pla'e in #irtual en#iron%ent") n pra'ti'e& !owe#er& t!e

a""e""%ent o( #irtual en#iron%ent" an$ "i%ulation" !a" *een (o'u"e$ pri%aril- on it"

u"e(ulne"" (or training an$ le"" on it" e((i'a'- (or "upporting learning e"pe'iall- in $o%ain"

wit! a !ig! 'on'eptual an$ "o'ial 'ontent)

o$ge" 18, "ugge"t" t!at "i%ulation" e"pe'iall- Virtual ealit-, pro#e %o"t #alua*le in

?o* training w!ere !an$"/on pra'ti'e i" e""ential *ut a'tual euip%ent 'annot o(ten *e u"e$)

Lreeno et al) 13, "ugge"t t!at knowle$ge o( !ow to per(or% a ta"k i" e%*e$$e$ in t!e

'ontextual en#iron%ent) 4ru'k%an an$ D?ork 1, "ugge"t t!at onl- w!en a ta"k i" learne$

in it" inten$e$ per(or%an'e "ituation" 'an t!e learne$ "kill" *e u"e$ in t!o"e "ituation")

Sin'e t!e earl- $a-" o( "i%ulation an$ ga%ing a" a %et!o$ to tea'!& t!ere !a#e *een 'all" (or

!ar$ e#i$en'e t!at "upport t!e tea'!ing e((e'ti#ene"" o( "i%ulation" 0n$er"on an$ @awton&

17,) T!eir paper "tate" t!at

despite the etensive literature! it remains difficult! if not impossible! to support

objectivel" even the most fundamental claims for the efficac" of games as a teaching

pedagog". #here is little relativel" little hard evidence that simulations produce

learning or that the" are superior to other methodologies.$

T!e- go on to re#iew t!e rea"on" a" *eing tra'ea*le to t!e "ele'tion o( $epen$ent #aria*le"

an$ to t!e la'k o( rigour wit! w!i'! in#e"tigation" !a#e *een 'on$u'te$)

1


20/29

Working Paper 1

HOW TO EVALUATE E**ECTIVENESS O* SIMULATIONS

Cne o( t!e %a?or pro*le%" o( "i%ulation" i" !ow to ;e#aluate t!e training e((e'ti#ene"" o( a

"i%ulationQ= Fein"tein an$ annon& 2>>2, 'iting a-" an$ Singer& 18,) 0lt!oug! (or %ore

t!an > -ear"& re"ear'!er" !a#e lau$e$ t!e *ene(it" o( "i%ulation Wol(e an$ rookall& 18,er- (ew o( t!e"e 'lai%" are "upporte$ wit! "u*"tantial re"ear'! Mile" et al)& 186, Dutler et

al)& 188, Wol(e& 181& Wol(e& 185& Wol(e& 1>,

Man- o( t!e a*o#e 'ite$ aut!or" attri*ute t!e la'k o( progre"" in "i%ulation e#aluation to

poorl- $e"igne$ "tu$ie" an$ t!e $i((i'ultie" in!erent in 'reating an a''epta*le %et!o$olog- o(

e#aluation)

T!ere are a nu%*er o( e%piri'al "tu$ie" t!at !a#e exa%ine$ t!e e((e't" o( ga%e/*a"e$

in"tru'tional progra%" on learning) For exa%ple *ot! i''i& et al) 16, an$ W!ite!all an$

M'4onal$ 13, (oun$ t!at in"tru'tion in'orporating ga%e (eature" le$ to i%pro#e$

learning) T!e rationale (or t!e"e po"iti#e re"ult" #arie$& gi#en t!e $i((erent (a'tor" exa%ine$

in t!e"e "tu$ie") W!ite!all an$ M'4onal$ argue$ t!at t!e in'orporation o( a #aria*le pa-o((

"'!e$ule wit!in t!e "i%ulation le$ to in'rea"e$ ri"k taking a%ong "tu$ent"& re"ulting in

greater per"i"ten'e an$ i%pro#e$ per(or%an'e) i''i& et al) propo"e$ t!at in"tru'tion

in'orporating ga%e (eature" en!an'e$ "tu$ent %oti#ation& lea$ing to greater attention to

training 'ontent an$ greater retention)

owe#er& alt!oug! ane'$otal e#i$en'e "ugge"t" t!at "tu$ent" "ee% to pre(er ga%e" o#er ot!er&

%ore tra$itional %et!o$" o( in"tru'tion& re#iew !a#e reporte$ %ixe$ re"ult" regar$ing t!e

training e((e'ti#ene"" o( ga%e" an$ "i%ulation") Pier(- 177, e#aluate$ t!e re"ult" o( 22

"i%ulation/*a"e$ training ga%e e((e'ti#ene"" "tu$ie" to $eter%ine pattern" o( training

e((e'ti#ene"") 21 "tu$ie" are reporte$ on !a#ing 'o%para*le a""e""%ent") T!ree o( t!e "tu$ie"

reporte$ re"ult" (a#ouring t!e e((e'ti#ene"" o( ga%e"& t!ree "tu$ie" reporte$ re"ult" (a#ouring

t!e e((e'ti#ene"" o( 'on#entional tea'!ing) T!e re%aining 15 "tu$ie" reporte$ no "igni(i'ant

$i((eren'e") 11 o( t!e "tu$ie" te"te$ (or retention o( learning& eig!t o( t!e"e in$i'ate$ t!at

retention wa" "uperior (or ga%e"& t!e re%aining 3 -iel$e$ no "igni(i'ant re"ult) 7 o( 8 o( t!e

"tu$ie" a""e""ing "tu$ent pre(eren'e (or training ga%e" o#er 'la""roo% in"tru'tion reporte$

greater intere"t in "i%ulation ga%e a'ti#itie" o#er 'on#entional tea'!ing %et!o$") More

re'entl-& 4ru'k%an 15, 'on'lu$e$ t!at ga%e" "ee% to *e e((e'ti#e in en!an'ing

2>


21/29

Working Paper 1

%oti#ation an$ in'rea"ing "tu$ent intere"t in "u*?e't %atter& -et t!e extent to w!i'! t!i"

tran"late" into %ore e((e'ti#e learning i" le"" 'lear)

THE PROBLEM O* SIMULATION VALIDATION

Spe'i(i'all- wit! regar$ to "i%ulation *a"e$ training progra%%e" *ut al"o appl-ing to all

training $eli#er- %et!o$" per!ap" wit! $i((erent ter%inolog-,& t!i" "e'tion re#iew" t!e

literature on "i%ulation e#aluation $e#eloping a 'o!erent (ra%ework (or pur"uing t!e

e#aluation pro*le%) T!ree pro%inent 'on"tru't" appear in t!e literature: (i$elit-& #eri(i'ation&

an$ #ali$ation) Vali$ation o( a "i%ulation a" a %et!o$ o( tea'!ing an$ t!at t!e "i%ulation

progra%%e a" a training inter#ention pro$u'e" or !elp" to pro$u'e, learning an$ tran"(er o(

learning are t!e i%portant 'riteria Fein"tein an$ annon& 2>>2, -et (i$elit- an$ #eri(i'ation

are ea"ier to e#aluate *ut not ne'e""aril- to %ea"ure o*?e'ti#el-, an$ o(ten $i"tra't e#aluator"

(ro% t!e %ore tri'k- i""ue o( #ali$ation)

Si%ulation Fi$elit-

Fi$elit- i" t!e le#el o( reali"% t!at a "i%ulation pre"ent" to a learner) a-" an$ Singer 18,

$e"'ri*e (i$elit- a" ;!ow "i%ilar a training "ituation %u"t *e& relati#e to t!e operational

"ituation& in or$er to train %o"t e((i'ientl-=) Fi$elit- (o'u"e" on t!e euip%ent t!at i" u"e$ to

"i%ulate a parti'ular learning en#iron%ent)

n %ore "op!i"ti'ate$ te'!nologi'all-, "i%ulation" t!at u"e #irtual realit- (or exa%ple& t!e

'on"tru't o( (i$elit- !a" an a$$itional $i%en"ion& t!at o( pre"en'e t!e $egree to w!i'! an

in$i#i$ual *elie#e" t!at t!e- are i%%er"e$ wit!in t!e #irtual worl$ or "i%ulation, Daile- an$

Wit%er& 1, Wit%er an$ Singer& 1, 4e$e& 17, Sal9%an et al)& 1,)

T!e $egree o( (i$elit- or pre"en'e in a learning en#iron%ent i" a $i((i'ult ele%ent to %ea"ure

Wit%er an$ Singer& 1,) Mu'! re"ear'! $uring t!e 6>


22/29

Working Paper 1

Si%ulation Veri(i'ation

Veri(i'ation i" t!e pro'e"" o( a""e""ing t!at a %o$el i" operating a" inten$e$) Veri(i'ation ;i" a

pro'e"" $e"igne$ to "ee i( we !a#e *uilt t!e %o$el rig!t= Peg$en et al)& 15,) 4uring t!e

pro'e""& "i%ulation $e#eloper" nee$ to te"t an$ $e*ug error" in t!e "i%ulation now u"uall-"o(tware error", t!roug! a "erie" o( alp!a an$ *eta te"t" un$er $i((erent 'on$ition"& #eri(-ing

t!at t!e %o$el work" a" inten$e$) C(ten& $e#eloper" are $i"tra'te$ *- t!i" pro'e"" pro$u'ing

w!at appear to *e *rilliant %o$el" t!at work A'orre'tl-< *ut wit! no appre'iation o( t!e

e$u'ational e((e't an$ !en'e t!eir #ali$it-) n t!i" "en"e& #eri(i'ation 'an *e a trap&

notwit!"tan$ing it" 'riti'al "tatu" a" a ne'e""ar- 'on$ition o( #ali$it- Fein"tein an$ annon&

2>>2,)

Vali$ation

Duil$ing on t!e work o( Fein"tein an$ annon& annon an$ Durn" an$ ot!er"& t!ere are t!ree

%ain ue"tion" *e-on$ t!o"e o( t!e general i""ue" o( e#aluation note$ a*o#e) ,

1) 0re we %ea"uring t!e Arig!t< t!ingO Vali$it- o( t!e 'on"tru't"

2) 4oe" t!e "i%ulation pro#i$e t!e opportunit- to learn t!e Arig!t< t!ingO Veri(i'ation o(

t!e "i%ulation an$ t!e appropriate (i$elit- (or t!e 'ontent an$ au$ien'e)

3) 4oe" t!e %et!o$ o( u"ing a "i%ulation $eli#er t!e learningO Vali$it- a" a %et!o$ o(

$e#eloping)

#aluation o( "i%ulation" (or learning out'o%e"

0n anal-"i" o( t!e re#iew o( "i%ulation literature i$enti(ie" learning out'o%e" in"tru'tor"

a$opt a" t!e- "tri#e to e$u'ate *u"ine"" "tu$ent") T!e"e learning out'o%e" !a#e *een

a$#an'e$ a" targeting t!e "kill" an$ knowle$ge nee$e$ *- pra'ti'ing %anager") n parti'ular&

t!e "our'e" are Tea'!& 18, Tea'! an$ Lio#a!i& 188, Mile" an$ an$olp!& 185, Mile"et al)& 186, 0n$er"on& 182, 0n$er"on an$ @awton& 17, ) Si%ulation re"ear'!er" !a#e

"pe'ulate$ t!at t!e %et!o$ i" an e((e'ti#e pe$agog- (or a'!ie#ing %an- o( t!e"e out'o%e")

*elow i$enti(ie" t!e"e learning out'o%e" w!ere

PR %ea"ure%ent o( learner per'eption" o( learning out'o%e"

CR an o*?e'ti#e %ea"ure%ent o( learning out'o%e"

t i" 'lear (ro% t!e ta*le a*o#e t!at t!e #a"t %a?orit- o( e#aluation" !a#e relie$ on t!e

learner


23/29

Working Paper 1

%ea"ure" to !elp un$er"tan$ i( "i%ulation" are an e((e'ti#e %et!o$ (or people to learn

*u"ine"" %anage%ent "kill")

C"#-lu'!"#

T!i" paper 'o%%en'e$ wit! t!ree ke- ue"tion":

1) W!- e#aluateO

2) W!at to e#aluateO an$

3) ow to e#aluateO

Why evaluate?

0n- $eli*erate learning inter#ention 'o"t" an$ organi"ation an$ in$i#i$ual"< ti%e an$ %one-)

T!e #alue pla'e$ on t!i" %a- #ar- (ro% per"on to per"on& !owe#er& t!ere i" a #alue& !en'e it

i" wort! en"uring t!at t!e #alue o( t!e out'o%e i" greater t!an or eual to t!e #alue o( t!e

input)

What to evaluate?

W!at to e#aluate i" %ore $i((i'ult to an"wer) n *u"ine"" an$ e"pe'iall- %anage%ent&

*e!a#ioural 'o%peten'ie" a" learning out'o%e" are in'rea"ingl- re'ogni"e$ a" a #ali$ an$

u"e(ul %ea"ure%ent)

How to evaluate?

ow to e#aluate po"e" "igni(i'ant pro*le%" (or t!e re"ear'!er) T!e training 'o%%unit- i"

%o"t (a%iliar wit! Girkpatri'kE a""e""%ent) owe#er& t!e re"ear'!er nee$" to *e aware o(

Wi%er


24/29

Working Paper 1

$e#eloping %anagerial 'o%peten'ie"& t!i" re"ear'!er i" intere"te$ in Apro#ing< rat!er t!an

learning, t!e e((e'ti#ene"" o( t!e $eli#er- %et!o$ an$ in Apro#ing< rat!er t!an i%pro#ing, t!e

$e#elop%ent inter#ention) T!e experi%ental re"ear'! approa'! to e#aluation appear" to *e t!e

%o"t o*?e'ti#e an$ "'ienti(i' in approa'! (or t!i" purpo"e)

e"ear'!er in#ol#e%ent an$ *ia"

T!ere are "igni(i'ant pro*le%" to 'on"i$er& t!o"e o(. t!e re"ear'!er" in#ol#e%ent B w!et!er

$ire't or not& t!e (a't o( %ea"uring parti'ipant" in a progra%%e %a- !a#e a $ire't i%pa't on

t!e a'!ie#e%ent o( t!e learning out'o%e") owe#er& i( t!i" i" t!e "a%e wit! all $eli#er-

%et!o$" *eing e#aluate$& t!e 'o%pari"on %ea"ure" on t!e "a%e *a"i")

Sa%ple "i9e

T!e next an$ #er- "igni(i'ant i""ue i" t!at o( "a%ple "i9e) t i" re'ogni"e$ t!at in or$er to

a'!ie#e "tati"ti'al #ali$it-& t!e "a%ple "i9e will nee$ to *e "igni(i'antl- large B to t!e or$er o(

1>> or %ore parti'ipant")

ontrol group an$ ot!er in(luen'e"

""ue" o( t!e "uita*ilit- o( t!e 'ontrol group nee$ to *e 'on"i$ere$ a" well) t 'an *e expe'te$

t!at in$i#i$ual" will rea't $i((erentl- to t!e "a%e training inter#ention) Su'! a"pe't" o( t!e

in$i#i$ual a" t!eir pre(erre$ learning "t-le& t!eir e$u'ational !i"tor-& per!ap" gen$er& per!ap"

ra'e& per!ap" 'ultural !eritage& t!eir age& et') %a- all pla- a part in w!et!er t!e-& a" an

in$i#i$ual& learn an$ '!ange *e!a#iour a" a re"ult or a" a 'on"euen'e o( a parti'ular learning

inter#ention)

T!e $e"ign o( a "uita*le e#aluation re"ear'! %et!o$ will nee$ to 'on"i$er all o( t!e"e i""ue")

2


25/29

Working Paper 1

Di*liograp!-

0@SS& S) M) 188, Fi$elit- in t!e 4e"ign o( n"tru'tional Si%ulation")%ournal of

Computer-&ased Instruction!15,>/7)

0@@L& L) M)& T0D0IM& S)& DT& W)& T0V& ) SCT@04& 0)

17, 0 Meta/0nal-"i" o( t!e elation" a%ong Training riteria)Personnel Ps"cholog"!5>,31/358)

04SC& J) ) 182, 0'ui"ition o( ogniti#e Skill")Ps"cholog" 'eview!8,36/>6)

04SC& P) ) @0WTC& @) 17, 4e%on"trating t!e @earning ((e'ti#ene"" o(

Si%ulation": W!ere We are an$ W!ere We ee$ to Lo)(evelopments in &usiness )imulation

* +periential +ercises!2,68/73)

D0@& J) WTM& D) 1, Pro'ee$ing" o( u%an Fa'tor" rgono%i'" So'iet-)

D4L0M& G) 17, Pro#ing t!e ((e'ti#ene"" o( Training)Industrial and Commercial

#raining!2,88/1)

DGCFF& ) C) 188, 0n ntegrate$ #aluation Mo$el (or 4) #raining *

(evelopment!2,66/68)

DGCFF& ) C) 18,,chieving 'esults from #raining! San Fran'i"'o& Jo"e-/Da"")DILC& J) CCP& ) @) 175, #aluation Met!o$olog-)%ournal of

ccupational Ps"cholog"!8,53/62)

DILC& J) SL& ) 177, #aluation o( Training an$ $u'ation: Mi'ro an$

Ma'ro Per"pe'ti#e")%ournal of +uropean Industrial #raining!1,17/21)

DIT@& ) J)& M0GI@S& P) M) ST0L& 4) ) 188, W!ere 0re WeO 0n 0nal-"i"

o( t!e Met!o$" an$ Fo'u" o( t!e e"ear'! on Si%ulation La%ing) )imulation * ames!1,

3/26)

0MPD@@& 4) T) ST0@& J) ) 166,+perimental and /uasi-+perimental (esign

for 'esearch! !i'ago& an$/M'all-)

0MPD@@& J) P)& 4ITT& M) 4)& @0W@& ) ) WG& G) ) 17>,

0anagerial &ehaviour! Performance and +ffectiveness! Mai$en!ea$& M'Lraw/ill)

@0G& ) 0L& T) 12, e"ear'! an$ T!eor- on Multi/Me$ia @earning ((e't")

L040& M) $),Interactive Learning +nvironments1 2uman 3actors and #echnical

Consideration on (esign Issues. Derlin& Springer/Verlag)

C@@S& 4) D) n Pre"", Per(or%an'e/@e#el #aluation Met!o$" I"e$ in Manage%ent

4e#elop%ent Stu$ie" (ro% 186/2>>>)2uman 'esource (evelopment /uarterl")

44& ) 17, T!e #olution o( on"tru'ti#i"t @earning n#iron%ent")+ducational

#echnolog"!52,5/6>)

4IGM0& 4) 15, T!e $u'ational ((e'ti#ene"" o( ntera'ti#e La%e")

CCG0@@& 4) 00& G) $"), )imulation and aming ,cross (isciplines and

Cultures4 I),, at a 5atershed. T!ou"an$ Cak"& 0)& Sage)4IGM0& 4) DJCG& 0) 1,Learning! 'emembering! &elieving4 +nhancing

2uman Performance! Wa"!ington 4& ational 0'a$e%- Pre"")

0STD/SMT& M) 18>, T!e #aluation o( Manage%ent an$ 4e#elop%ent: an

C#er#iew)Personnel 'eview!1>,28/36)

0STD/SMT& M) 1,+valuating 0anagement (evelopment! #raining and

+ducation! 0l$er"!ot& Lower)

0STD/SMT& M) 0STC& 4) J) @) 175, I"ing epertor- Lri$ Te'!niue to

#aluate Manage%ent Training)Personnel 'eview!,15/21)

0STD/SMT& M)& TCP& ) @CW& 0) 11,0anagement 'esearch4 ,n

Introduction! @on$on& Sage)

FST& 0) ) 0C& ) M) 2>>2, on"tru't" o( Si%ulation #aluation))imulation and aming!33,25&>)

25


26/29

Working Paper 1

F@ST04& W) J) 17, ualitati#e Met!o$": 0 ee$e$ Per"pe'ti#e in #aluation e"ear'!)

CCG& T) 4) 04T& ) S) $"), /ualitative and /uantitative 0ethods in

+valuation 'esearch. De#erl- ill"& Sage)

L0L& ) M) 18, @earning out'o%e" an$ t!eir e((e't": I"e(ul 'ategorie" o( !u%an

per(or%an'e),merican Ps"chologist!3,377/385)

LCP& 4)& W@& M) D0GT& T) 1, Tran"(er o( "kill (ro% a 'o%puter ga%etrainer to (lig!t)2uman 3actors!36,387/>5)

L4@& M) ) 16, $u'ational La%e" an$ Si%ulation": 0 Te'!nolog- in "ear'! o( a

e"ear'!, Para$ig%) JC0SS& 4) ) $),2andbook of 'esearch for +ducational

Communications and #echnolog".ew ork& Si%on S'!u"ter Ma'%illan)

LC& J)& SMT& 4) MCC& J) $"), 13, #ransfer on #rial4 Intelligence!

cognition and instruction!orwoo$ J& 0*lex)

LID0& ) L) @C@& ) S) 18,3ourth eneration +valuation! @on$on& Sage)

0MD@& 0) ) 17,+valuation and Control of #raining! Mai$en!ea$& M'Lraw ill)

0S& ) T) SL& M) J) 18, )imulation fidelit" in training s"stems design4

&ridging the gap between realit" and training.!ew ork& Springer/Verlag)

SS@L& P) 166, )trateg" of +valuation 'esearch in the 3ield of )upervisor" and0anagement #raining! 0n""en& Van Lor'u%)

C4LS& M) 18, Virtual ealit- in Training) Computer raphics 5orld.

C@TC& ) F)& 16, T!e Flawe$ Four/@e#el #aluation Mo$el)2uman 'esource

(evelopment /uarterl"!7,5/21)

JGS& 4)& SMMCS& ) W0@G& ) 181, T!ou ature are %- Lo$$e"")

aturali"ti' nuir- in $u'ational #aluation) Cambridge %ournal of +ducation!11,16/8)

G@& S) P) 2>>1, 0 Few T!oug!t" on xe'uti#e o%peten'- on#ergen'e) Center for

/ualit" of 0anagement %ournal!1>,67/71)

GGP0TG& 4) 15H6>, Te'!niue" (or e#aluating training progra%": Part" 1 to )

%ournal of the ,merican )ociet" for #raining and (evelopment!o#e%*er& 4e'e%*er&

Januar- an$ Fe*ruar-)

GGP0TG& 4) @) 1,+valuating #raining Programs4 #he 3our Levels! San

Fran'i"'o& Derret/Goe!ler)

GGP0TG& 4) @) 18, #aluating Training Progra%": #i$en'e #") Proo()

GGP0TG& 4) @) $),,nother Look at +valuating #raining Programs. 0lexan$ria& V0&

0ST4)

G0L& G)& FC4& J) G) S0@0S& ) 13, 0ppli'ation o( 'ogniti#e& "kill/*a"e$& an$

a((e'ti#e t!eorie" o( learning out'o%e" to new %et!o$" o( training e#aluation)%ournal of

,pplied Ps"cholog"!78,311/328)

M04C0@4/CSS& M) 173, De!a#ioural C*?e'ti#e": 0 riti'al e#iew)Instructional

)cience!2,1/52)M@@@04& 4) ) 173, Te"ting (or o%peten'e at!er T!an ntelligen'e),merican

Ps"chologist!28,1/1)

MG0& S) 16, #aluating MM: ""ue" (or e"ear'!er") !arle" Stuart Ini#er"it-)

M@S& ) ) 04C@P& W) 0) 185, #he rganisation ame4 , )imulation!

Llen#iew& l& S'ott& Fore"%an an$ o%pan-)

M@S& W) L)& DLLS& W) 4) SIDT& J) ) 186, Stu$ent" Per'eption" o( Skill

0'ui"ition T!roug! a"e" an$ a Leneral Manage%ent Si%ulation: 0 o%pari"on)

)imulation * ames!17,7/2)

MCS& ) ) 1>, Finan'ial 0nal-"i": T!e Met!o$" an$ T!eir 0ppli'ation to %plo-ee

Training)2uman 'esource (evelopment /uarterl"!1,5/63)

P0TTC& M) ) 178, 6tilization-3ocussed +valuation! De#erl- ill"& Sage)

26


27/29

Working Paper 1

PL4& ) 4)& S0C& ) ) S04CWSG& ) P) 15,Introduction to )imulation

using )I0,7! ig!t"town& J& M'Lraw/ill)

PF& 4) 0) 177, o%parati#e Si%ulation La%e e"ear'!: Stu%*ling Dlo'k" an$

Stepping"tone") )imulation and aming!8,255/68)

0G0M& ) 173, e'ent T!oug!t" on #aluation)Industrial and Commercial #raining!

5,5/61)44& W) J) 17>,0anagerial +ffectiveness! @on$on& M'Lraw ill)

VS& T) 13, e"ear'! Support (or ntera'ti#e Multi%e$ia: xi"ting Foun$ation" an$

ew 4ire'tion") @0TM& )& W@@0MSC& J) 4SC@0TT& @)

$"),Interactive 0ultimedia. @on$on& Gogan Page)

& G)& S0@0S& ) 0C/DCWS& J) 0) 16, 4o o%puter/*a"e$ La%e"

Fa'ilitate Gnowle$ge 0'ui"ition an$ etentionO0ilitar" Ps"cholog"!8,25/3>7)

CS& ) 15, 0""e""ing @earning in V: Towar$" 4e#eloping a Para$ig%) T@)

ISS/FT& 4) PSG@@& ) 2>>1,+valuation in rganizations

, )"stematic ,pproach to +nhancing Learning! Performance! and Change! a%*ri$ge& M0)&

Per"eu" Pu*li"!ing)

ISS@@& S) 1, #aluating Per(or%an'e nter#ention")Info-line)S0@M0& M) )& 44& )& )& D) @) & J) 1, 0 Mo$el (or In$er"tan$ing ow

Virtual ealit- 0i$" o%plex on'eptual @earning)Presence4 #eleoperators and 8irtual

+nvironments!8,23/316)

SV& M) 172, Pro" an$ on" a*out Loal/Free #aluation)+valuation Comment!3,1/

)

S0SC& S) )& 4G& D) P) LCLCPCI@IS& D) S) 16>, elation"!ip" 0%ong

riteria o( ((e'ti#e Jo* Per(or%an'e)%ournal of ,pplied Ps"cholog"!,15/2>2)

SP& @) M) SP& S) 13, Competence at 5ork4 0odels for )uperior

Performance!ew ork& Jo!n Wile- Son")

ST0G& ) ) 18>, e"pon"i#e #aluation) Ini#er"it- o( llinoi")

ST0& G)& MC0T& ) G4& ) 18, u%an Fa'tor i""ue" in Virtual

n#iron%ent")Presence4 #eleoperators and 8irtual +nvironments!7,327/351)

T0& )& 4) LCV0& L) $"), 188, #he 'ole of +periential Learning and

)imulation in #eaching 0anagement )kills)

T0& ) 4) $), 18, 6sing 3orecast ,ccurac" as a 0easure of )uccess in &usiness

)imulations)

TCM0S& )& 0@@& J) S0T@@& @) 17, I"ing an ntera'ti#e o%puter La%e to

n'rea"e Skill an$ Sel(/e((i'a'- egar$ing Sa(er Sex egotiation: Fiel$ Te"t e"ult")2ealth

+ducation * behavior!2,71/86)

W0& P) D)& D4& M) W) 0G0M& ) 17>,+valuation of 0anagement #raining!

0l$er"!ot& Lower)WT& D) 18, 4e"igning o%puter La%e" to elp P!-"i'" Stu$ent" In$er"tan$

ewtonK" @aw" o( Motion) Cognition and Instruction!1,6/1>8)

WT0@@& D) M4C0@4& D) 13, %pro#ing @earning Per"i"ten'e o( Militar-

Per"onnel *- n!an'ing Moti#ation in a Te'!ni'al Training Progra%) )imulation * aming!

2,2/313)

WT@CG& 4)& D0& P) C@@04& S) 16, Pro'ee$ing") 4TCS& )

$),+uropean Conference on ,I in +ducation. @i"*on Portugal& oli*ri $ition")

WD& J) ) M0T& ) J) 1, T!e %pa't o( a o%puter/*a"e$ 0$#enture La%e on

0'!ie#e%ent an$ 0ttitu$e" in Leograp!-)%ournal of Computing in Childhood +ducation!5,

61/71)

WM& S) 2>>2, T!e 4ark Si$e o( 36>/$egree) #raining * (evelopment,37/2)

27


28/29

Working Paper 1

WTM& D) SL& M) J) 1, Mea"uring %%er"ion in Virtual n#iron%ent") 0

Te'!ni'al eport 1>1)

WC@F& J) 181, e"ear'! on t!e @earning ((e'ti#ene"" o( Du"ine"" Si%ulation La%e": 0

re#iew o( t!e "tate o( t!e "'ien'e)(evelopments in &usiness )imulation * +periential

+ercises!8,72)

WC@F& J) 185, T!e Tea'!ing ((e'ti#ene"" o( La%e" in ollegiate Du"ine"" our"e": 0173/183 Ip$ate) )imulation * ames!16,251/288)

WC@F& J) $), 1>, #he uide to +periential Learning!ew ork& i'!ol")

WC@F& J) CCG0@@& 4) 18, 4e#eloping a S'ienti(i' Gnowle$ge o(

Si%ulationHLa%ing) )imulation * aming!2,7/1)

WCC4& @) ) STW0T& P) W) 187, %pro#e%ent o( Pra'ti'al ea"oning Skill" wit! a

o%puter La%e)%ournal of Computer-&ased Instruction!1,/53)

CIL& M) 2>>2, lari(-ing o%peten'- an$ o%peten'e)2enle" 5orking Paper.

enle-& enle- Manage%ent ollege)

28


29/29

Working Paper 1

Jo!n Genwort!-

0*out t!e aut!or

Jo!n i" t!e 4ire'tor o( enle- Manage%ent $u'ation 0"ia Pa'i(i', Pte) @t$)& a w!oll-

owne$ "u*"i$iar- o( t!e enle- Manage%ent ollege in Singapore& w!ere !e i" re"pon"i*le

(or %arketing an$ "upporting t!e enle- MD0 *- 4i"tan'e @earning in Singapore) e !a"

li#e$ in Singapore "in'e 17 an$ 'on"i$er" t!e i"lan$ 'it- "tate !i" !o%e) Jo!n i" al"o

4ire'tor o( orporate $ge 0"ia& a 'o%pan- "pe'iali9ing in $eli#ering 'o%puter/*a"e$

"i%ulation an$ experiential training progra%%e")

Jo!n !a" a DS in otel an$ atering Stu$ie" (ro% Man'!e"ter Pol-te'!ni' an$ an MD0 in

Te'!nolog- Manage%ent (ro% t!e Cpen Ini#er"it-) i" re"ear'! an$ *u"ine"" intere"t" are in

t!e u"e o( 'o%puter/*a"e$ "i%ulation" to $e#elop %anagerial 'o%peten'e) e i" an a'ti#e

%e%*er o( 0DS@ 0""o'iation (or Du"ine"" Si%ulation" an$ xperiential @earning, an$

0T 0""o'iation (or $u'ational o%%uni'ation" an$ Te'!nolog-,)

Jo!n i" a %e%*er o( 4D0 17& 'o%%en'ing !i" 4D0 in 2>>1)

Documents

4932328 Evaluating Management Development