1

Chapter 2

Methods and Statistics

in I-O Psychology

Roy

alty

-Fre

e/C

OR

BIS

2

Module 1: Science

• What is science?

• Science has common methods• Science is a logical approach to investigation

– Based on a theory, hypothesis or basic interest

• Science depends on data– Gathered in a laboratory or the field

3

Common Methods (cont'd)

• Research must be communicable, open, & public

– Research published in journals, reports, or books

1) Methods of data collection described

2) Data reported

3) Analyses displayed for examination

4) Conclusions presented

4

Common Methods (cont'd)

• Scientists set out to disprove theories or hypotheses– Goal: Eliminate all plausible explanations

except one

• Scientists are objective– Expectation that researchers will be objective &

not influenced by biases or prejudices

5

Role of Science in Society

• Expert witnesses in a lawsuit– Permitted to voice opinions about practices

– Often a role assumed by I-O psychologists

6

Daubert Challenge

• Challenging testimony of an expert on the grounds it is not scientifically credible

• Daubert v. Merrill-Dow, 1993– Resulted in introduction of a method for

distinguishing between “legitimate science” & “junk science”

7

Scientific Testimony in Court

• Theories presented in court must:– Be recognized by particular scientific area as

worthy of attention– Be peer reviewed or subjected to scientific

scrutiny– Have a known “error rate”– Be replicable or testable by other scientists

8

Module 1 (cont'd)

• Why do I-O psychologists engage in research?– Better equip HR professionals in making

decisions in organizations– Provide an aspect of predictability to HR

decisions

9

Module 2: Research

• Research design– Experimental

• Random assignment of participants to conditions

• Conducted in a laboratory or the field

– Quasi-experimental• Non-random assignment of participants to conditions

10

Research Design (cont'd)

• Non-Experimental– Doesn’t include “treatment” or

assignment to different conditions

– 2 common designs:• Observational design• Survey design

11

Methods of Data Collection

• Quantitative methods– Rely on tests, rating

scales, questionnaires, & physiological measures

– Yield results in terms of numbers

C. Borland/PhotoLink/Getty Images

12

Methods of Data Collection

• Qualitative methods– Include procedures like observation, interview,

case study, & analysis of written documents– Generally produce flow diagrams & narrative

descriptions of events/processes

13

Quantitative & Qualitative Research

• Not mutually exclusive

• Triangulation, (Rogelberg & Brooks, 2002)

– Examining converging information from different sources (qualitative and quantitative research).

14

K Lewin

• B = f (p*e)– Behavior is a function of

• Person X environmental influences

15

Experimental v. Corr research

• “I” side: focus on Individual differences– Person attributes:

• E.g. Personality, behaviors, cognitive ability

• “O” side: focus on Environmental influences– Situation variables:

• E.g. work conditions, leadership style, pay for performance

16

I v. O

• Which is most likely to use – Experimental designs?– Correlational designs?

• Why?

17

Generalizability in Research

• Application of results from one study or sample to other participants or situations– Benefit of using theory

• Every time a compromise is made, the generalizability of results is reduced

18

Sampling Domains for I-O Research

Figure 2.1: Sampling Domainsfor I-O Research

19

Observational Unit

• Worker

• Team

• Department

• Organization

• Industry

• Others?

20

Measurement Unit(one of something)

• Identify a measurement unit for:– Worker’s performance score– Years of experience– Absenteeism– Motivation– Sales performance– Cognitive ability

21

Control in Research

• Experimental control– Influences that make results less reliable or

harder to interpret are eliminated

• Statistical control– Statistical techniques used to control for

influences of certain variables

22

Ethics

• Ethical standards of the APA

• Collection of 61 cases endorsed by SIOP– Illustrates ethical issues likely to arise in I-O

psychology (Lowman, 1985a,1998)

23

Module 3: Data Analysis

• Descriptive statistics– Summarize, organize, describe sample of data

Frequency Distribution:– Horizontal axis = Scores running low to high– Vertical axis = Indicates frequency of

occurrence

24

Describing a Score Distribution

• Measures of central tendency

• Mean• Mode• Median

Ryan McVay/Getty Images

25

Describing Score Distribution (cont'd)

• Variability– Standard deviation

• Lopsidedness or skew

Ryan McVay/Getty Images

26

Descriptive Statistics:Two Score Distributions (N = 30)

Figure 2.2 Two Score Distribution (N=30)

27

Two Score Distributions (N = 10)

Figure 2.3

28

Inferential Statistics

• Aid in testing hypotheses & making inferences from sample data to a larger sample/population

• Include t-test, F-test, chi-square test

29

Statistical Significance

• Defined in terms of a probability statement

• Threshold for significance is often set at .05 or lower– p < .05 (likelihood of this effect size would

occur less than 5 times in a hundred)

30

Statistical Power

• Likelihood of finding statistically significant difference when true difference exists

• Smaller the sample size, lower the power to detect a true or real difference

31

Concept of Correlation

Positive Linear Correlation

Figure 2.4Correlation betweenTest Scores andTraining Grades

32

Concept of Correlation (cont'd)

• Scatterplot– Displays correlational relationship between 2

variables

• Regression– Straight line that best fits the scatterplot

33

Correlation Coefficient

• Statistic or measure of association

• Reflects magnitude (numerical value) & direction (+ or –) of relationship between 2 variables

34

Correlation Coefficient

• Positive correlation → High values of one variable are associated with high values in the other variable (& vice versa)

• Negative correlation → High values of one variable are associated with low values in the other (& vice versa)

35

Figure 2.6: Scatterplots of Various Degrees of Correlation

36

Curvilinear Relationship

• Although correlation coefficient might be .00, it can’t be concluded that there is no association between variables

• A curvilinear relationship might better describe the association (eta η)– SPSS can provide η with F test

37

Curvilinear Correlation

Figure 2.7An Example ofa CurvilinearRelationship

38

Multiple Correlation

• Multiple correlation coefficient– Overall linear association between

several variables & a single outcome variable (R) • R2 = Proportion of variance in DV (outcome)

accounted for all preditors (several vars)

39

Meta-Analysis

• Statistical method for combining results from many studies to draw a general conclusion

• Statistical artifacts– Characteristics of a particular study that distort

the results– Sample size is most influential

40

Module 4: Interpretation

• Reliability– Consistency or stability of a measure

– Test-retest reliability• Calculated by correlating measurements

taken at Time 1 with measurements taken at Time 2

41

High and LowTest-Retest Reliability

Figure 2.8Examples of High and LowTest-Retest Reliability: Score Distributions of IndividualsTested on Two DifferentOccasions

42

Reliability (cont'd)

• Equivalent forms reliability– Calculated by correlating measurements

from a sample of individuals who complete 2 different forms of same test

• Internal consistency (Cronbach alpha α)

– Assesses how consistently items of a test measure a single construct

43

Reliability (cont'd)

• Inter-rater reliability– Can calculate various statistical indices to show

level of agreement among raters• Intraclass correlation ICC• Rwg

– Generalizability theory• Simultaneously considers all types of error in

reliability estimates

44

Validity

• Whether measurements taken accurately & completely represent what is to be measured

• Predictor– Test chosen or developed to assess identified abilities

• Criterion– Outcome variable describing important aspects or

demands of the job

45

Figure 2.9: Validation Process from Conceptual and Operational Levels

Figure 2.9

46

Criterion-Related Validity

• Correlate a test score with a performance measure (validity coefficient)

• Predictive validity design– Time lag between collection of test data &

criterion data– Test often administered to job applicants

47

Criterion-Related Validity (cont'd)

• Concurrent validity design– No time lag between collection of test data &

criterion data– Test administered to current employees,

performance measures collected at same time– Disadvantage: No data about those not

employed by the organization

48

Content-Related Validity

• Demonstrates that content of selection procedure represents adequate sample of important work behaviors & activities or worker KSAOs defined by job analysis

49

Construct-Related Validity

• Investigators gather evidence to support decisions or inferences about psychological constructs

• Construct - concept or characteristic that a predictor is intended to measure; examples include intelligence and extraversion

50

A Model for Construct Validity

Figure 2.10A Model forConstruct Validity

51

Construct Validity Model of Strength and Endurance Physical Factors

Figure 2.11