The Kirkpatrick ModelThe Kirkpatrick Modelorganizational changeorganizational change

Richard Lambert, Ph.D. Richard Lambert, Ph.D.

ReviewReview

Did the trainees learn anything from Did the trainees learn anything from the training?the training?

Did they change behavior once back Did they change behavior once back on the job?on the job?

In education, we too often jump In education, we too often jump right to test scores without asking right to test scores without asking these questions.these questions.

Impacting Real ChangeImpacting Real Change

The better an educational The better an educational intervention is, the less likely you intervention is, the less likely you are to see test scores change are to see test scores change during the life of the evaluation.during the life of the evaluation.

What do I mean?What do I mean? Is this always true?Is this always true?

Impacting Real ChangeImpacting Real Change

Real change in education:Real change in education:

Teacher behavior.Teacher behavior.Building climate / culture.Building climate / culture.System climate / culture / policy.System climate / culture / policy.State / Federal educational policy.State / Federal educational policy.

Which of these can happen quickly?Which of these can happen quickly?

The LevelsThe Levels The four levels from Kirkpatrick’s The four levels from Kirkpatrick’s

modelmodel Level I – ReactionLevel I – Reaction

– How well did the attendees like the training?How well did the attendees like the training? Level II – LearningLevel II – Learning

– What concepts, facts, or skills were learned?What concepts, facts, or skills were learned? Level III – BehaviorLevel III – Behavior

– What changes in behavior or attitude did the What changes in behavior or attitude did the attendees make once back on the job?attendees make once back on the job?

Level IV – ResultsLevel IV – Results– What were the tangible impacts of the What were the tangible impacts of the

training program on the organization?training program on the organization?

The Results LevelThe Results LevelWith the CIPPI, we do not jump to Impact With the CIPPI, we do not jump to Impact

evaluation without first understanding evaluation without first understanding Context, Input, Process, and Product.Context, Input, Process, and Product.

Similarly, with the Kirkpatrick model, we Similarly, with the Kirkpatrick model, we do not jump to Results without first do not jump to Results without first asking about Reaction to training, the asking about Reaction to training, the Learning that took place, and if any Learning that took place, and if any Behavior was actually changed on the Behavior was actually changed on the job.job.

The Results LevelThe Results LevelThe ideal approach to establishing the The ideal approach to establishing the

long term impact of training, is to plan long term impact of training, is to plan from the beginning of the training to from the beginning of the training to conduct full Product and Impact conduct full Product and Impact evaluations after looking at the other evaluations after looking at the other stages of the model.stages of the model.

However, it is very difficult to attribute However, it is very difficult to attribute organizational change directly to organizational change directly to training.training.

The Results LevelThe Results LevelMany factors can contribute to a specific Many factors can contribute to a specific

organizational result.organizational result.

Even with Random Assignment and a Even with Random Assignment and a Control Group – Organizational unit Control Group – Organizational unit randomization.randomization.

Many things can happen in the Many things can happen in the organization by the time you get to this organization by the time you get to this phase.phase.

The Results LevelThe Results LevelTherefore, the evaluator has to Therefore, the evaluator has to

determine if Impact evaluation is still determine if Impact evaluation is still possible before proceeding.possible before proceeding.

The following focal questions will The following focal questions will help the evaluator determine if this help the evaluator determine if this level of evaluation can be pursued. level of evaluation can be pursued.

The Results LevelThe Results LevelWere any of the objectives of the training Were any of the objectives of the training

related to a direct impact on related to a direct impact on organizational change (e.g. cost savings, organizational change (e.g. cost savings, productivity increase, quality productivity increase, quality improvement, etc.)?improvement, etc.)?

Are there any specific outcomes that are Are there any specific outcomes that are both regularly tracked by the organization both regularly tracked by the organization and could be expected to show the and could be expected to show the influence of the training if it was influence of the training if it was successful?successful?

The Results LevelThe Results LevelHave the individuals who attended Have the individuals who attended

the training influenced the behavior the training influenced the behavior of those who did not attend the of those who did not attend the training?training?

Contamination of the control group.Contamination of the control group.

Have they been asked to train or Have they been asked to train or mentor colleagues?mentor colleagues?

The Results LevelThe Results LevelHave administrators identified the Have administrators identified the

work of trainees as exemplary, or as work of trainees as exemplary, or as a model for others? a model for others?

Have the behaviors that are Have the behaviors that are expected of those who have expected of those who have completed the training been completed the training been institutionalized as the policy of the institutionalized as the policy of the organization?organization?

The Results LevelThe Results LevelHow supportive of the training are How supportive of the training are

the Program administrators?the Program administrators?

Has the organization continued to Has the organization continued to send trainees to the training?send trainees to the training?

Have those who completed the Have those who completed the training recommended it to training recommended it to colleagues?colleagues?

SummarySummary If your investigation has determined that If your investigation has determined that

there is still a reasonable chance to there is still a reasonable chance to compare the treatment and control compare the treatment and control groups on important outcome measures, groups on important outcome measures, then proceed with Product and Impact then proceed with Product and Impact evaluation.evaluation.

As you do so, look not just at the As you do so, look not just at the quantitative results, but at organizational quantitative results, but at organizational change and climate in the two groups.change and climate in the two groups.

SummarySummary If the training has been effective in If the training has been effective in

producing results, you should see producing results, you should see quantitative indicators of gains in quantitative indicators of gains in productivity, etc.productivity, etc.

ANDAND

Increases in job satisfaction and Increases in job satisfaction and organizational health.organizational health.

Matching Purpose and Matching Purpose and MethodsMethods

Purpose

Formative Summative Comparative

Formative

Method Summative

Comparative

Matching Purpose and Matching Purpose and MethodsMethods

Purpose

Formative Summative Comparative

Feedback Inflated Ratings Inflated Ratings

Formative for Gaming the system Gaming the system

Improvement Systematic exclusion Systematic exclusion

Timeliness Information Lacks Controls

Method Summative Lacks Specificity About Weaker Causal

and Process Effectiveness Inference

Premature Premature Causal Inference

Comparative and and Relative Merits

Misleading Misleading Cost-Effectiveness

The Case of the NRSThe Case of the NRS

What is the purpose of the What is the purpose of the NRS?NRS?

How should the How should the consequential validity of the consequential validity of the NRS be determined?NRS be determined?

Purposes of NRSPurposes of NRS Provide feedback to programsProvide feedback to programs

– FormativeFormative Target training and technical Target training and technical

assistanceassistance– FormativeFormative

Help ensure effectiveness of Help ensure effectiveness of servicesservices– SummativeSummative

If formative…If formative…

What role will the HS monitoring What role will the HS monitoring system play in strengthening system play in strengthening programs current assessment programs current assessment practices?practices?

Timely feedback to programs, Timely feedback to programs, teachers, parents?teachers, parents?

Possible given commitment to Possible given commitment to aggregated data?aggregated data?

If formative…If formative…

Training for teachers and Training for teachers and administrators in how to administrators in how to translate assessment translate assessment information into meaningful information into meaningful program planning and program planning and individualized instruction.individualized instruction.

How will the results be tied How will the results be tied back to curriculum decisions?back to curriculum decisions?

If summative…If summative…

Have any of these measures Have any of these measures been developed or validated as been developed or validated as measures of program or measures of program or classroom quality?classroom quality?

What causal inferences will be What causal inferences will be made?made?

Any design features that allow Any design features that allow strong causal inferences?strong causal inferences?

If summative…If summative…

Can programs collect their Can programs collect their own summative data?own summative data?

What rewards and sanctions What rewards and sanctions will be place?will be place?