Validity, Reliability, & Sampling

Psych 231: Research Methods in Psychology

Errors in measurement Reliability

If you measure the same thing twice do you get the same values?

Validity Does your measure really measure what it is supposed to measure??

reliablevalid

reliable

invalid

unreliable invalid

Reliability

True score + measurement error A reliable measure will have a small amount of error

Multiple “kinds” of reliability• Test-retest• Internal consistency• Inter-rater

Reliability

Test-restest reliability Test the same participants more than once• Measurement from the same person at two different times

• Should be consistent across different administrationsReliable Unreliable

Reliability

Internal consistency reliability Multiple items testing the same construct

Extent to which scores on the items of a measure correlate with each other•Cronbach’s alpha (α)•Split-half reliability

• Correlation of score on one half of the measure with the other half (randomly determined)

Reliability

At least 2 raters observe behavior

Inter-rater reliability

Extent to which raters agree in their observations• Are the raters consistent?

Requires some training in judgment

Validity

Does your measure really measure what it is supposed to measure? There are many “kinds” of validity

VALIDITY

CONSTRUCT

CRITERION-ORIENTED

DISCRIMINANT

CONVERGENTPREDICTIVE

CONCURRENT

FACE

INTERNAL EXTERNAL

Many kinds of Validity

VALIDITY

CONSTRUCT

CRITERION-ORIENTED

DISCRIMINANT

CONVERGENTPREDICTIVE

CONCURRENT

FACE

INTERNAL EXTERNAL

Many kinds of Validity

Construct Validity

Usually requires multiple studies, a large body of evidence that supports the claim that the measure really tests the construct

Face Validity At the surface level, does it look as if the measure is testing the construct?

“This guy seems smart to me, and

he got a high score on my IQ measure.”

Internal Validity

Did the change in the DV result from the changes in the IV or does it come from something else?

The precision of the results

Threats to internal validity

History – an event happens the experiment Maturation – participants get older (and

other changes) Selection – nonrandom selection may lead to

biases Mortality – participants drop out or can’t

continue Testing – being in the study actually

influences how the participants respond

The precision of the results

External Validity

Are experiments “real life” behavioral situations, or does the process of control put too much limitation on the “way things really work?”

External Validity Variable representativeness

Relevant variables for the behavior studied along which the sample may vary

Setting representativeness Are the properties of the research setting similar to those outside the lab (Ecological validity)

Subject representativeness Characteristics of sample and target population along these relevant variables

Sampling

Why do we do we use sampling methods? Typically don’t have the resources to test everybody, so we test a subset

Sampling

Population

Everybody that the research is targeted to be about

The subset of the population that actually participates in the research

Sample

Sampling

Sample

Inferential statistics used to generalize back

Sampling to make data collection manageable

Population

Sampling Why do we do we use sampling methods?

Goals of “good” sampling:– Maximize Representativeness:

– To what extent do the characteristics of those in the sample reflect those in the population

– Reduce Bias:– A systematic difference between those in

the sample and those in the population

Sampling Methods Probability sampling

Simple random sampling

Systematic sampling Stratified sampling

Non-probability sampling Convenience sampling Quota sampling

Have some element of random selection

Susceptible to biased selection

Simple random sampling Every individual has a equal and independent chance of being selected from the population

Systematic sampling Selecting every nth person

Stratified sampling Step 1: Identify groups (strata) Step 2: randomly select from each group

Convenience sampling Use the participants who are easy to get

Quota sampling Step 1: identify the specific subgroups Step 2: take from each group until desired number of individuals

Next time Read: Chpt 8