Challenges of Behavioral Science Research

Class 11

Face Validity

Construct

Hostility

Prejudice

Need for affiliation

Attraction

Empirical Realization

No. of shocks to harasser

RT: see Black or White target, ID pos/neg words

Choosing to be alone or with another person

Pupil dilation

Face Valid?

(High, Med., Low)

High

Med/low

High

Low

Criterion Validity

Match or fit between specific, empirical DV (i.e., the one used in study) and an independent (and presumably stable, encompassing) measure of DV (i.e., the criterion).

Concordant validity: Criterion exists in the present.

Predictive validity: Criterion represented by future behavior.

DV Criterion

"How fit are you today?" Treadmill Endurance

Conscientiousness # of missed classes Survey (at week 1) at end of semester

Type

Concordant

Predictive

Construct Validity

Face validity and criterion validity refer largely to the validity of the measure.

Construct validity refers to the validity of the underlying conceptual DV.

Typically requires multiple measures

Convergent Validity

Different measures that have only the underlying construct in common.

Neurotic: moderately related to stress, negative affect, self-preoccupation, fear of judgment

Divergent Validity

Measure is not tightly related to similar constructs

Neurotic introversion, perfectionism

CONSTRUCT: Neuroticism

Validity

Sexism scale items include: Women demand too many rights Wives should vote as their husbands do

Validity Type

Face Validity

“Acrophobia Survey” verified with heart rate, sweating, hyperventilation

Criterion

Aggressiveness Measure taken at prescreening predicts shocks delivered in experiment, 3 weeks later

Predictive

Method

Validity (continued)

“Nurturance” = Attn. to other’s emotions Listening to problems Willing to help AND ≠ gregariousness

Validity Type

Construct Validity

Self Esteem measure is moderately related to: Self-confidence; Self-Clarity; acceptance

Convergent Validity

Method

Self Esteem is not highly correlated with self-efficacy, positive mood

Divergent Validity

Boosting Validity

Avoid “response set”: Alternate (or mix) the positive and negative valence of questions, in survey DV.

Systematic Replication: Several experiments, each one accounts for alternative explanation (Batson Empathy and Helping research, e.g.)

Disguise measure: “Chicken game” in Culture of Honorstudies—non obvious measure of aggression.

DV outside of conscious control, e.g., physio reactions

Weighing the Alternatives

I had to shed 20 pounds or else I’d lose my job and my wife would leave me and I’d die an early death. I was desperate for a solution. Then I found Chubby Checkers ®. Within 2 months I lost 15

pounds! You can, too!

Implied causal story?

Alternative explanations?

Program led to weight loss

Sense of urgency, increased motivation, etc.

Saturday Academy Research Design

Pre-test SAT= 940

Class sessions

Post-testSAT = 991

SAT gain pre to post test = 51 points, p < .01

Implied causal story?

Alternative explanations?

What does design need to do to address alternative explanations?

Program produced results

Motivation, other influences, etc.

Control Condition

Control Groups

Purpose: To establish causality; that it is IV, and only IV, that accounts for DV.

Attributes of Control Group:

1. Random selection: * Each participant is equally likely to be

assigned to expt'l or control conditions. * Provides a check on systematic error

But, does not control for random error

2. Control condition should mimic experimental condition in all respects other than the IV.

3. Assign Ss to control or experimental conds. just before

introducing IV

Counterbalancing

Sub. No.

1

2

3

4

5

6

7

8

Confederate Race

Black

White

Black

White

Black

White

Black

White

Confederate Temperament

Friendly

Friendly

Unfriendly

Unfriendly

Friendly

Friendly

Unfriendly

Unfriendly

Essay Topic

TV

TV

TV

TV

Environ

Environ

Environ

Environ

MID TERM POSTPONE TO

THURSDAY OCTOBER 20

Single Factor Design

(1 X 2) (From Interracial Feedback Research)

  Black White

Writer Race I II

1 Factor: Race of Writer

2 Levels: Black writer or White writer

Shows main effect only (whether phenomenon exists).

Does not show interaction

Does not show moderation

Components of Factorial Design

Factors: The independent variables

Factor 1: Writer race Factor 2: Writer friendliness

Levels: The dimensions within factors

Race: Level 1 = Black, Level 2 = WhiteFriendliness: Level 1 = Friendly Level 2 = Unfriendly

Conditions: The combination of factors and levels

Condition I: Friendly, Black writerCondition IV: Unfriendly, White writer

Black White

Friendly Black/Friendly White/Friendly

Unfriendly Black/Unfriendly White/Unfriendly

2 X 2 Friendly Unfrnd

Black Ia 1b

White 2a 2b

2 X 3 Friendly Unfrnd Neutral

Black 1a 1b 1c

White 2a 2b 2c

3 X 3 Friendly Unfrnd Neutral

Black 1a 1b 1c

White 2a 2b 2c

Asian 3a 3b 3c

Friendly Unfrnd

Black Ia 1b

White 2a 2b

Friendly Unfrnd

Black Ia 1b

White 2a 2b

2 X 2 X 2Race-relevant Essay Race-Irrelevant Essay

Factorial Designs as Coherent Sentences

Number of Factors

Factorial “Sentence”

1 Ethical decisions (blind/don/t blind) are affected by discussion opportunity (discuss vs. don’t discuss). NOTE: Blinding is DV.

2 Ethical decisions are affected by group discussion and social contexts (Ghakistan vs. NY)

3 Ethical decisions are affected by group discussion and by social context as a function of gender.

4 Ethical decisions are affected by group discussion and by social context, as a function of gender—but only among the college educated.

5 Ethical decisions are affected by group discussion and by social context, as a function of gender—but only among the college educated, who specialized in humanities rather than engineering.

Factorial Design Not Requiring Neutral Control

Wegner “White Bear” Paradigm

Rate of Unbidden Thoughts as a Function of Mental Control and Nature of Topic

Topic

Sex Dean of Students

Mental Suppress High Low

Task Express Low Low

Note: What would be “neutral” condition in this study? Is it possible for there to be one?

Determining Number of Levels w/n Factors

How does arousal affect test performance?

0

2

4

6

8

10

Low Arousal High Arousal

02468

10

LowArousal

Mod.Arousal

HighArousal

Yerkes-Dotson Law

Perfo

rman

ce L

evel

Low Moderate High

Arousal Level

TIME CHATTING W' NEW ACQUAINTANCE

Perceived Similarity and Friendliness

Impact and Control

Control Reduce the effect of anything that can obscure thetrue effect of IV on DV.

Impact Create strong enough empirical realizations of IVand DV to demonstrate how conceptual IV effectsconceptual DV.

Goals of Control and Impact Often Conflict

Helping in a Hurry(A shameless Cribbing of Darley & Batson)

Hypothesis: The amount of time people spend helping strangers is determined by time pressure

Design: Ss arrive at study, told that "kindness expt." has been moved to another location, and they have either little time (hurry cond.) or much time (no hurry cond.) to get to lab.

On way to lab, Ss stopped by harried stranger who asks fordirections to UMDNJ. The stranger is confused, and repeatedly requests clarifications on directions.

Outcome measure: How much time does S spend giving directions to stranger?

Predictions: Ss in “hurry” cond. will spend less time giving directions than will Ss in “no hurry” cond.

Time (seconds) Talking to Stranger

Time Giving Help Due to Hurry/No Hurry

Hurry No Hurry

Time (seconds) Talking to Stranger

Time Giving Help Due to Hurry/No Hurry

Time (seconds) Talking to Stranger

Time Giving Help Due Hurry/No Hurry

Time (seconds) Talking to Stranger

Time Giving Help Due Hurry/No Hurry

Impact 

Control

Advantages Drawbacks   Advantages Drawbacks

Closer realization of conceptual IV

Isolation of IV is muddled.

  Cleaner realization of IV

Weaker realization of IV

Tells a more compelling story

Tells a more complicated story.

  Tells a concise, specific story

Tells a less compelling, more narrow story

Increases between group differences

Increases within group differences

  Reduces within subject differences

Reduces between group differences

More powerful Less precise   More precise Less Powerful

Stronger influence on behavior

Raises ethical problems

  Fewer ethical problems.

Weaker influence on behavior.

Trade-Offs Between Impact and Control

The "frustration experience" involved subjecting the subjects to some ten to twelve hours of food deprivation, inducing them to drink from a pint to a quart of water and preventing urination for approximately three hours, taking a blood sample with a sterilized spring lancet in such a way as to cause considerable pain, and finally forcing them to fail in a group situation. The latter aspect of the situation was the most crucial, the earlier, physiological assault being designed mainly to lower the subject's threshold of annoyance or frustration.

Super-High Impact

"An Experimental Examination of the Scapegoat Theory of Prejudice"

Lindzey, G (1950) Journal of Abnormal and Social Psychology, 45, 296-309

Content Vs. Mechanics If Done as Between-Group Factor

  Content MechanicsBlack Writer n = 20 n = 20White Writer n = 20 n = 20

0123456789

10

Black Writer Cond White Writer Cond

ContentMechanics

Within subjects SDBetween subjects SD

Non-Obvious Within Subjects ManipulationHarber & Hartley, 1983

Examples of Multiple Meanings

1. Terror Management:a. Conceptual IV: mortality fearsb. Empirical realization: Write about last 15 minutes

prior to own death.

Alternative Explanation? Bad Mood

2. Happinessa. Conceptual IV: Positive mood stateb. Empirical realization: Succeed on a task

Alternative Explanation?Self-Esteem

Self-efficacy

Definition Problem

1. Aggression = Harmful act?= Willingness act harmfully?= Intent to cause harm?

2. Empathy = Perspective taking?=

Compassion?= Simulating?

Systematic Replications of Independent Variable in Biased Feedback Research

Conceptual Variable: Performance on a task that can be evaluated in

terms of subjective and objective features.

Empirical Realizations of Independent Variable

  Task Subjective Objective

1.

Written Work

Arguments, Beliefs

Spelling, Grammar

2.

Landscape Painting Harmony, tone Proportions, Shading

3.

Photo Journalism Honesty, immediacy Focus, Depth of Field

4.

Architecture Originality, comfort Ventilation, Lighting

5.

Modern Dance Mood, Energy Timing, Control

Systematic Replications in Dependent Variable: Feedback Studies

1. Number/type of critical comments supplied in summary rating sheet and copy-edit comments. (Harber, 1998)

2. Number/type of critical comments supplied during face-to-face interaction. (Harber, 2004)

Cognitive Dissonance 

Attachment Theory

* People are more likely to help a fellow shopper pick up spilled groceries after they, themselves, break a stranger’s camera.  

* People who form ambivalent romantic relationships tend to be unsure about the existence of a supreme being.

* Paying children to do a task that they already like doing will make them like the task less. 

* Young children are more likely to freely explore if they have non-depressed mothers.

* People become more loyal believers in a cult if the cult’s own predictions don’t come true. 

* People who tend to avoid close relationships also tend to be more mentally rigid. 

Classic Systematic Replications

Successful and Unsuccessful Replications

Successful Replication: Increases confidence that effects are reliable and generalizable. Greater confidence that conceptual variable is valid.

Unsuccessful Replication

Efficacy of replication Implications for Implications forOriginal Findings Conceptual Variable

Replication correctly done

Replication incorrectly done

Orig. study did not account for all vars.

None

None

None

U.S. IMMIGRATION IQ TEST (circa 1900-1920)

Type of Question

Nationality U.S. Specific E. European Specific

E. Europeans Perform Worse Perform Better

Native Born U.S. Citizens Perform Better Perform Worse

E. European U.S. Residents

Perform Better Perform Better

MIXED (DIRECT AND SYSTEMATIC) REPLICATION

Test Questions:

1. Who was Benedict Arnold?

2. What is the name of Paul Bunyon's ox?3. Where was the Maine sunk?

1. Where was Mozart born?

2. What is the name of King

Arthur's sword?3. What city did Vesuvius destroy?

Experimental Realism and Mundane Realism

Experimental Realism: The experiment appears real to the subject.

Mundane realism: The experiment resembles real world situations.

Experimental realism and mundane realism are not polar opposites.

Experimental Realism

Mundane Realism High Low

High Feedback: Confederate studyCult. Honor: "Chicken" game

Adler: Police Vignettes

Low Asch: Conformity StudyMilgram: Obedience Study

Kahneman & Tversky: Framing study

Internal Validity and External Validity

Internal Validity: The degree that treatment, or IV, has a significant effect on the DV. And this effect is due to IV and not some extraneous variable.

* IV & DV should have impact on Ss -- Take them seriously; attend to them, engage w’ them * Appear to Ss as examples of conceptual IV & DV * Do not appear to represent other kinds of IV or DV * Internal validity boosted by expt’l realism

External Validity: The effects of study are generalizable. The empirical realization is a good representation of underlying conceptual variable.

* Systematic replication * Increase expt’l realism: study works w’ all subs, not just cooperators * Heterogeneous populations

Which is more important, Internal Validity or External Validity?