Embed Size (px)
Citation preview
Schmidt & Hunter Approach to r
Artifact Corrections
Statistical Artifacts
• Extraneous factors that influence observed effect– Sampling error*– Reliability*– Range restriction*– Computational error– Dichotomization of variables
*Addressed in the analysis
Psychometric Meta-Analysis
Disattenuation for reliability
yyxx
xyC
rr
rr
xx
xyC
r
rr
yy
xyC
r
rr
Correction for both
Correction for IV
Correction for DV
Suppose rxy = .30, rxx = ryy = .80. Then:
375.80.
30.2
yyxx
xyC
rr
rr 33541.
8.
3.
xx
xyC
r
rr
Range Restriction/Enhancement
These are examples of direct RR.
Direct Range Restriction/enhancement
1)1( 22
xyX
xyXC
rU
rUr
edSDrestrict
ctedSDunrestriU X
Suppose rxy = .33, SD1=12, SD2 = 20. Then:
67.112
20
edSDrestrict
ctedSDunrestriU X
50.1)33)(.167.1(
)33(.67.1
1)1( 2222
xyX
xyXC
rU
rUr
Can also invert by uX = 1/UX
Indirect RR
xxir
xxar
Reliability of IV in restricted sample (job incumbents in I/O validation study).
Reliability of IV in unrestricted sample (job applicants in I/O validation study).
T
TT S
su Ratio of SD of true scores; analogous to uX.
)1(1 2xxiXxxa rur
2/12 )1(
xxa
xxaXT r
ruu
You will need uT AND rxxi for INDIRECT range restriction correction.
You will need rxxa for DIRECT range restriction correction.
)1(1 2xxaXxxi rUr
Meta-Analysis of corrected r
• If information is available can correct r for each study
• Compute M-A on the corrected values
• Can also be done with assumed distributions, but I don’t recommend it.
Steps (1)Record data (N, r, artifact values rxx, etc.)
Compute the corrected correlation for each study:
a
rr o
C
Note ro is observed and rC is corrected.
If there is only 1 kind of artifact, disattenuation is simple:
Where a is the disattenuation factor.
If there is range restriction, things are tricky. If INDIRECT range restriction, then use Ut instead of Ux and disattenuate for reliability before adjusting for range restriction. Use reliabilities from the restricted group.
If DIRECT range restriction, adjust for ryy, then range restriction, then rxx, but rxxa, the reliability in the unrestricted group.
Steps (1b)
Compute sampling variance of uncorrected r:
)1/(]1[)( 22 ioio NreVar
Note this is sampling variance for one study.
For each study, compute compound attenuation factor:
C
o
r
rA
Steps (2)
Compute sampling variance of disattenuated r:
2/)( AeVarev o
If there is range restriction, then do the following 2 steps. Compute adjustment for range restriction:
]1)1/[(1 22 oXrr rUa
Adjust sampling variance of disattenuated r:evave rr 2
Compute weights:
2iii ANw Note A is the compound attenuation
factor.
Steps (3)Compute the weighted mean:
i
CiC w
rwr i
Compute the weighted variance:
i
CCiC w
rrwrVar i
2][)(
Compute average corrected r sampling error:
i
ii
w
vewveAve )(
Compute variance of rho: )()(ˆ)( 2 veAverVarVar C
Psychometric M-A data
Study Ni r rxxi ryy Ux
1 200 .20 .90 .80 1.5
2 100 .20 .80 .82 1.5
3 150 .40 .85 .88 1.0
4 80 .40 .85 .90 1.2
Mean 132.5 .30 .85 .85 1.3
We’ve already done the bare-bones analysis of these data. Now we’ll analyze 3 ways: (1) just criterion reliability, (2) all artifacts with INDIRECT RR, (3) all artifacts DIRECT rr.
Correct ryy only (1)
Study 1 r = .20, rxx = .90, ryy = .80, Ux = 1.5
Disattenuation ryy : rC = .2/sqrt(.8) = .223607. Compound attenuation factor A = .20/.223607 = .894.
Suppose we only wish to correct for criterion unreliability.
Correct ryy only (2)Study r A rC N V1 V2
1 .2 .894 .224 200 .0042 .0053
2 .2 .906 .221 100 .0085 .0104
3 .4 .938 .426 150 .0057 .0064
4 .4 .949 .422 80 .0107 .0118
2868.or )1/(]2868.1[)( 221 io NeVarV
22 /)( AeVarV o
Correct ryy only (3)
Study rC A Niwi wirC
1 .224 .894 200 160 35.78
2 .221 .906 100 82 18.11
3 .426 .938 150 132 56.28
4 .422 .949 80 72 30.36
Sum 446 140.53
2iii ANw 315.446/53.140ˆ
i
CiC w
rwr i
Correct ryy only (4)
Study rCwi
Wi[rC-rbarC]2 V2 wV2
1 .224 160 1.339 .0053 .8465
2 .221 82 .728 .0104 .8508
3 .426 132 1.635 .0064 .8479
4 .422 72 .817 .0118 .8529
Sum 446 4.52 3.3981
0101.446/52.4][
)(2
i
CCiC w
rrwrVar i315.Cr
0076.446/3981.3)( 2
i
ii
w
VwveAve
Correct ryy only (5)
0502.)(
0025.0076.0101.)()()(
VarSD
veAverVarVar C
Bare-Bones
ryy corrected
M .2868 .315
V(r) .0098 .0101
SDrho .0585 .0502
95CRlow .17 .22
95CRup .40 .41
All corrections, Indirect RR
Study Ni r rxxi ryy Ux
1 200 .20 .90 .80 1.5
2 100 .20 .80 .82 1.5
3 150 .40 .85 .88 1.0
4 80 .40 .85 .90 1.2
Mean 132.5 .30 .85 .85 1.3
2868.or Already know bare-bones mean.
Indirect RR (2)Study ro rxxi ryy Ux ux rxxa UT
1 .20 .90 .80 1.5 .67 .96 1.55
2 .20 .80 .82 1.5 .67 .91 1.60
3 .40 .85 .88 1.0 1 .85 1
4 .40 .85 .90 1.2 .83 .90 1.23
2/12 )1(
xxa
xxaXT r
ruu)1(1 2
xxiXxxa rur
Indirect RR (3)Study ro rxxi ryy UT rc1 rc A
1 .20 .90 .80 1.55 .236 .351 .570
2 .20 .80 .82 1.60 .247 .378 .530
3 .40 .85 .88 1 .462 .462 .865
4 .40 .85 .90 1.23 .457 .535 .747
yyxxioc rrrr /1 1)1( 2
12
1
cT
cTc
rU
rUr
co rrA /
Indirect RR (4)Study Ni rc A wi wrc
1 200 .351 .570 64.93 22.79 .480
2 100 .378 .530 28.04 10.59 .099
3 150 .462 .865 112.2 51.89 .073
4 80 .535 .747 44.69 23.92 .431
Sum 249.86 109.19 1.082
2iii ANw 437.
86.249
19.109Cr
2)( CCi rrw
0043.86.249
082.1)( CrVar
Indirect RR (5)Study Ni rc A wi V1 V2 arr V3 wV3
1 200 .351 .570 64.93 .0042 .013 .95 .012 .76
2 100 .378 .530 28.04 .0085 .030 .94 .027 .75
3 150 .462 .865 112.2 .0057 .008 1 .008 .85
4 80 .535 .747 44.69 .0107 .019 .92 .016 .73
Sum 249.86 3.09
)1/(]2868.1[)( 221 io NeVarV 2
2 /)( AeVarV o
]1)1/[(1 22 oTrr rUa 22
3 VaV rr 0124.86.249
09.3)( veAve
Indirect RR (6)
0)(
008.0124.0043.)()()(
VarSD
veAverVarVar C
Bare-Bones
ryy corrected
Full indirect correction
M .2868 .315 .437
V(r) .0098 .0101 .0043
SDrho .0585 .0502 0
95CRlow .17 .22 .44
95CRup .40 .41 .44
Direct Range Restriction (1)Study ro rxxi ryy Ux rxxa rc1 rc2 rc
1 .20 .90 .80 1.5 .96 .22 .33 .33
2 .20 .80 .82 1.5 .91 .22 .32 .34
3 .40 .85 .88 1.0 .85 .43 .43 .46
4 .40 .85 .90 1.2 .90 .42 .49 .51
)1(1 2xxiXxxa rur
yyoC rrr /1 1)1( 21
2
12
CX
CXC
rU
rUr
xxaCC rrr /2
Direct RR (2)Study ro rc A Ni w wrc
1 .20 .33 .60 200 72.20 24.03
2 .20 .34 .59 100 35.23 11.87
3 .40 .46 .86 150 112.2 51.89
4 .40 .51 .78 80 48.30 24.86
Sum 267.92 112.66
2iii ANw 42.92.267/66.112ˆ
i
CiC w
rwr i
Direct RR (3)Study rc w
1 .33 72.20 .55
2 .34 35.23 .25
3 .46 112.2 .20
4 .51 48.30 .43
Sum 267.92 1.43
42.Cr
2)( CCi rrw
0053.92.267
43.1)( CrVar
Direct RR (4)Study Ni rc A wi V1 V2 arr V3 wV3
1 200 .33 .60 72.20 .0042 .012 .95 .011 .77
2 100 .34 .59 35.23 .0085 .024 .95 .022 .77
3 150 .46 .86 112.2 .0057 .008 1 .008 .85
4 80 .51 .78 48.30 .0107 .018 .93 .015 .74
Sum 267.92 3.13
)1/(]2868.1[)( 221 io NeVarV 2
2 /)( AeVarV o
]1)1/[(1 22 oXrr rUa 22
3 VaV rr 012.92.267
13.3)( veAve
Direct RR (5)
0)(
006.0117.0053.)()()(
VarSD
veAverVarVar C
Bare-Bones
ryy corrected
Full indirect correction
Full direct correction
M .2868 .315 .437 .42
V(r) .0098 .0101 .0043 .005
SDrho .0585 .0502 0 0
95CRlow .17 .22 .44 .42
95CRup .40 .41 .44 .42
