Upload
jermaine-widmer
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Degree to which inferences made using data are justified or supported by evidence
Some types of validity◦ Criterion-related ◦ Content◦ Construct
All part of unitarian view of validity Constructs - theoretical abstractions aimed
at organizing and making sense of our environment; they are LATENT
Validity
Importance of Criteria A criterion is any variable you wish to
explain and/or predict They are the key to well-developed theory,
good measurement, and strong research design
Ultimate criterion Multidimensional nature of criteria Intermediate criteria
Criterion-Related Validity Process of establishing a relationship
between variables Predictive, concurrent, postdictive Usually based on correlation or
regression equation Low reliability will attenuate or mask
relationships
Selection Ratio – proportion of the individuals in the sample who are selected of the total number
Base rate – percent of successful individuals under random selection
Range Restriction Differential Prediction for different
subgroups
Usefulness of criterion-related validity estimate
DecisionsSR,BR=.50
FN
FP
VP
VN
Xc
YcFN+VP=BRVN+FP=1-BR
VP+FP=SRFN+VN=1-SR
Successful
Unsuccessful
Reject Accept
False Negatives
False Positives
DecisionsSR=.15,BR=.50
FN
FP
VP
VN
Xc
Yc
FN+VP=BRVN+FP=1-BR
VP+FP=SRFN+VN=1-SR
Successful
Unsuccessful
Reject Accept
False Negatives
False Positives
DecisionsSR=.85,BR=.50
FN
FP
VP
VN
Xc
Yc
FN+VP=BRVN+FP=1-BR
VP+FP=SRFN+VN=1-SR
Successful
Unsuccessful
Reject Accept
False Negatives
False Positives
Criterion-Related Validity Even low correlations can lead to large
increases in selection efficiency SR and BR have strong influences When SR is small (choose few), fewer FP and
more FN When SR is large, fewer FN and more FP When BR is large (many can be successful), SR
and validity have little effect on selection efficiency
Most gains in success ratio when BR = .50 and SR is small (e.g., .10)
The tradeoffs depend on purpose of selection
Content Validity (Logical Analysis)
Extent to which items or measures cover the content area the test purports to measure◦ Expert judges determine if a measure came from
a particular content domain◦ Scoring and content is based upon theory◦ If measures are from same content domain,
should demonstrate high reliability ◦ If low internal consistency reliability, low content
validity
Construct Validity
Validity of inferences about latent unobserved variables on the basis of observed variables
Does a measure assess what it is intended to assess? Do the variables relate in theoretically meaningful ways?
Low reliability will make it difficult to assess the nature of a particular construct and attenuate relationships with other constructs
Construct ValidityConstruct Validity
Can we generalize to the constructs from the measures?
Theory What you think
CauseConstruct
EffectConstruct
Measure orManipulation
ObservedOutcomes
ObservedRelationship
TrueRelationship
What you see
Construct Validity
Anxiety
Test Score(Y)
Measureof Anxiety
(X)
Abilityto Learn
1
2
3
4
SaladsEaten (Z)
Vegetarianism
5
Internal Structure Analysis Cross Structure Analysis Nomological network (Cronbach & Meehl)
Ways to Establish Construct Validity
Internal Structure Analysis
Factor Analysis◦ Used to identify factors or dimensions that underlie
relations among observed variables Exploratory - Useful When:
◦ No info on internal structure available◦ Factor structures may look different than original scale◦ You have reservations about previous factor analyses
Confirmatory - Useful When:◦ You have some idea of the internal structure◦ Confirming factor structures from previous studies
Necessary but not sufficient to establish construct validity
Cross-Structure Analysis Embedded in nomological network
(nomological validity) Test of hypotheses by examining
relationships between different indicators of underlying constructs ◦ e.g., leadership style based on reports from
subordinates and leadership self-report inventory
Relies on multiple methods of measurement
A representation of constructs of interest in a study, their observable manifestations (measures), and the interrelationships among and between them
Cronbach & Meehl said this is necessary to establish construct validity
Elements include:◦ Specify linkage between constructs (hypotheses)◦ Operationalize constructs (specify measurement)
Nomological Network
Convergent and Discriminant Validity Convergent validity - Convergence
among different methods designed to measure the same construct
Discriminant validity - Distinctiveness of constructs, demonstrated by divergence of methods designed to measure different constructs
Multi-Trait Multi-Method
Heterotrait-Monomethod◦ Different traits, same method
Heterotrait-Heteromethod◦ Different traits, different methods
Monotrait-Heteromethod◦ Same trait, different methods◦ Validity diagonals
Monotrait-Monomethod◦ Same trait, same method◦ Reliability diagonals
MTMM
Method1 Method2 Method3 A1 B1 C1 A2 B2 C2 A3 B3 C3
M1 A1 (.89) B1 .51 (.89) C1 .38 .37 (.76)
M2 A2 .57 .22 .09 (.93) B2 .22 .57 .10 .68 (.94) C2 .11 .11 .46 .59 .58 (.84)
M3 A3 .56 .22 .11 .67 .42 .33 (.94) B3 .23 .58 .12 .43 .66 .34 .67 (.92) C3 .11 .11 .45 .34 .32 .58 .58 .60 (.85)
MTMM
Steps to Establish Construct Validity Specify the nomological net (expected +
and - relationships) of expected relations Establish reliability Check convergence with other preexisting
measures of the construct (convergent validity)
Factor analysis Empirical studies of relatedness Empirical studies of discriminability
Take the hypotheses you developed in assignment 2 and the variables that were included in them. ◦ Draw a picture of what you believe the nomological
network of these variables would look like◦ What alternative measures of each variable might
you use (different than those specified in Assignment 3) to establish convergent validity?
◦ Draw what an MTMM construct validity chart would look like that includes each variable in your study and the original and alternative measures you identified for each construct. Specify whether each correlation would be expected to be Hi, Low or Moderate.
Assignment 4