More on research design

CH-7, APP 7-1 Copyright 2008 Werner et al 1

More On Research Design

Chapter 7

Appendix 1


Research Design Validity

• Defines the extent to which one can be confident that conclusions are true

• Four aspects– Internal validity– External validity– Construct validity– Statistical conclusion


Internal Validity

• A Judgment – Are the results due to the experimental

variables or due to other variables

• Was the increase in performance due to HRD or due to something else, such as better lighting?


Factors Affecting Internal Validity – 1

Appendix Table 7-1

• History – Unrelated events occurring during the training period that can

influence training measurements.

• Maturation – Ongoing processes within the individual, such as aging or

gaining job experience, which are a function of the passage of time.

• Testing – The effect of a pretest on posttest performance.

• Instrumentation – The degree to which criterion instruments may measure different

attributes of an individual at two different points in time.SOURCE: From Cascio,W. F. (1991). Applied Psychology in Personnel Management (4th ed, p. 395). Adapted by permission ofUpper Saddle River, NJ.



Appendix Table 7-1

• Statistical Regression – Changes in criterion scores resulting from selecting extreme

groups on a pretest.• Differential Selection

– Using different procedures for selecting individuals for experimental and control groups.

• Experimental Mortality – Differential loss of respondents from various groups.

• Interaction of Differential Selection and Maturation – Assuming that experimental and control groups were different to

begin with, the compounding of the disparity between the groups by maturational changes occurring during the experimental period.

SOURCE: From Cascio,W. F. (1991). Applied Psychology in Personnel Management (4th ed, p. 395). Adapted by permission ofUpper Saddle River, NJ.


Factors Affecting Internal Validity – 3Appendix Table 7-1

• Interaction of Pretest with the Experimental Variable – During the course of training, something reacting with

the pretest in such a way that the pretest affects the trained group more than the untrained group.

• Interaction of Differential Selection with Training– When more than one group is trained, because of

differential selection, the groups are not equivalent on the criterion variable to begin with and may react differently to the training.

SOURCE: From Cascio,W. F. (1991). Applied Psychology in Personnel Management (4th ed, p. 395). Adapted by permission ofUpper Saddle River, NJ.



Appendix Table 7-1

• Reactive Effects of the Research Situation– When the research design itself so changes

the trainees' expectations and reactions that results cannot be generalized to future applications of the training.

• Multiple Treatment Interference– Differential residual effects of previous

training experiences.SOURCE: From Cascio,W. F. (1991). Applied Psychology in Personnel Management (4th ed, p. 395). Adapted by permission ofUpper Saddle River, NJ.


Non-Experimental Designs

• Do not support that training caused the improved performance

• Types include– Case study– Relational research– One-Group Pre-Test Post-Test Design


Case Study

• An intensive, descriptive study of a particular trainee, training group, or organization to determine what has occurred and reactions to it

• Diagrammed– Training Post-training measures and descriptions

• Many threats to internal validity – No pre-training information is collected,– No untrained comparison group


Relational Research

• Measuring two or more variables in an attempt to describe or explain their relationship to one another


One-Group Pretest-Posttest Design

• Trainees are assessed on the variables being observed before training and again after training

• DiagrammedPretest Training Posttest

• Have trainees changed as a result of the training program?

• Many factors could threaten its internal validity, such as history, maturation, and instrumentation


Appropriateness of Non-Experimental Design in HRD

• They do answer some HRD-related questions–How much change has occurred?

–Has a particular target level of KSA or performance been achieved by trainees?


Experimental Designs

• Two important factors:– A control group that does not receive training– Random assignment of participants to the

training and control groups• Allows researcher to assume groups are

equivalent


Pretest-Posttest with Control

• Allows control for outside influences by including untrained group as a comparison group

• Diagrammed– Group 1 : Pretest Training Posttest

– Group 2 : Pretest Posttest

• Allows three comparisons– Are the two groups in fact equivalent before training?– Did training improve the trainees? – Did the untrained group remain unchanged during the

course of the study?


Posttest Only with Control

• Eliminates factors resulting from pretest• Diagrammed

– Group 1 : Training Posttest– Group 2 : Posttest

• Relies on the assumption that the two groups are equivalent before the study


Solomon Four-Group• Uses one experimental group and three

different groups to control for the effects of the pretest

• Diagrammed:– Group 1 : Pretest Training Posttest– Group 2 : Pretest Posttest– Group 3 : Training Posttest– Group 4: Posttest


Quasi-Experimental Designs

• Researcher attempts to control as many threats to validity as possible, while matching evaluation research concerns to organizational constraint

• Nonequivalent Control Group

• Time Series


Nonequivalent Control Group

• One cannot assume control and experimental group are equivalent

• Threats to validity– selection-maturation interaction– testing-training interaction– regression effects

• Researcher must determine differing factors and control them


Time Series

• Multiple measurements before and after training• Diagrammed

– Simple Time Series Design– Group 1 : M1 M2 M3 M4 M5TrainingM6 M7 M8

M9 M10– Multiple Time Series Design– Group 1 : M1 M2 M3 M4 M5TrainingM6 M7 M8

M9 M10– Group 2 : M1 M2 M3 M4 M5M6 M7 M8 M9 M10


Statistical Power – 1

• Ensuring that a change will be detected if it exists

• Statistical Conclusion Validity – extent to which the research design and statistical

analyses performed can lead to false conclusions.– To the extent that the design and analyses are

appropriate, one can be confident in the results of the study

• keeping in mind the issues of internal, external, and construct validity


Statistical Power – 2

• Probability of concluding there is a difference between the training and control groups when a difference actually exists

• Higher the power of a design or analysis, the greater the chances of finding a difference


Selecting a Research Design

• Factors to consider– Validity of conclusions– Investigator may make incorrect conclusion based on

study– Certain designs use a lot of resources (e. g., Solomon

Four-Group)– Organizational support and commitment is needed

• Highest level of evidence has substantial cost in time, money, and personnel

• Companies are increasingly knowledgeable about statistics and understand them

• Use your company’s statisticians!

Education

More on research design