Upload
sister-edith-bogue
View
19.528
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Lecture from a research methods class.
Citation preview
Graziano and RaulinResearch Methods: Chapter 8
Research Methods:Validity and Threats to Validity
Generating Research Hypotheses(Review)
Ideas lead toobservationslibrary research
Statement of problemProblem statements
become research hypotheses when constructs are operationalized
Characteristics of a Good Study Question
“FINER”F Feasible
I Interesting
N Novel
E Ethical
R Relevant
Actually testing three sets of hypothesesThe null hypothesisThe confounding variable hypothesesThe causal hypothesis
Accept causal hypothesis only if youreject null hypothesis (statistical analysis)rule out each potential confounding variable
hypothesis (based on appropriate controls)
Testing Research Hypotheses
Correlation (also called covariation)Relationship found between variables
Time orderCause must occur before result
NonspuriousnessAlternative explanations must
be eliminated from possibilityExperiments are intended to
reduce or rule out alternative explanations and confoundingvariables
Criteria for Nomothetic Causality
Asking “the Question”The PICO format:
P PopulationI Intervention or Interest areaC Comparison intervention or statusO Outcome
Applying the “PICO” format“What is the usefulness or accuracy of the
current 1-10 pain scale assessment in treating a patient’s pain, and what are other options that may prove more useful?”
Does a 10 point pain Visual Analog Scale (____, ____) accurately assess pain in the first day postop abdominal total hysterectomy patient when compared with the Faces Pain Scale (Pasaro, 1997)?
Evaluating Hypotheses
Statistical Validity – carrying out the actual statistical analysis properly
Construct Validity refer most often to a characteristic of an instrument but also to the whole study
External Validity refers to the generalizability of study findings
Internal Validity refers to a characteristic of a study’s design
Types of Validity
Are the statistical tests accurate?
Threatened byUnreliable measuresViolations of statistical assumptions How do we detect these problems???
Strengthened byUsing well validated measuresHaving approximately equal sample sizes in
each group)
Statistical Validity
Is our theory the best explanation for the results?
Threatened byAny alternative explanation for the resultsHOW do we locate these alternative
explanations?Strengthened by
Using well-validated constructs to build the theoretical predictions for the study
Construct Validity
Do the results apply to the broader population?
Threatened byUnrepresentative samplesGeneralizing beyond the limits of the sampleHOW do we know when this problem is
present???Strengthened by
Gathering a representative sample (if possible)Clearly describing sample, so that other
researchers will know the limits of generalization
External Validity
Is the independent variable responsible for the observed changes in the dependent
variable?
Threatened by Confounding variablesHOW do we detect the presence of
confounding variables????Strengthened by
Adding adequate controls to reduce or eliminate confounding
Internal Validity
Confounding and internal validityMany sources for confounding (covered next)With proper controls, confounding can be
virtually eliminated (see Chapter 9)
Confounding and construct validityMake sure that you have considered alternative
theoretical explanations for the anticipated phenomenon
HOW????
Avoiding Confounding
Pretest-Posttest Research Design
Single-group, pretest-posttest design compares pre-treatment and post-treatment scores to determine improvement
Fails to control most sources of confounding
Historical events may occur during the course of the experiment.Remember Pygmalion effect & its story
Maturation of the subjects.Testing and retesting can
influence awareness of variables or behaviorLearn Hawthorne effect & its story.
Instrumentation – measurement methods or procedures may not be equivalent
Sources of Internal Invalidity
Statistical regression of subjects starting out in extreme positions.
Selection biases (we will see several types)
Experimental mortality(a.k.a. attrition) – subjects drop out of the study before it's finished.
Sequence effects – Performance on one measure is related to previous experience with other measures. Outcome depends on the sequence of measures.
Sources of Internal Invalidity
Demoralization subjects incontrol group find out, loseinterest in study, stop trying
Diffusion of treatment (those who get the experimental stimulus spread it to controls)
Rivalry (controls change behavior to try to beat the experimental group)
Equalization of treatment (researcher compensates controls for not getting treatment)
Social threats to validity
Social Psychology Experiment:Conformity to Norms (file)
Participants are not passiveThey try to understand the study to help them
to know what they “should do” (termed subject effects)
Respond to subtle cues about what is expected (termed demand characteristics)
Placebo effect: treatment effect due to expectations that the treatment will work
Subject Effects
Based on the expectations of the researcher
Can affect the outcome of studies if not controlled
May be due to the experimenter providing demand characteristics to the participant
Not the same as scientific fraud (which is deliberate)
Experimenter Effect
Three closely-tied concepts
Validity The accuracy of the study or procedureIncreased by using appropriate control
procedures
The more controls we employ, the higher the level of constraint
Controls may increase some types of validity while, by their unnatural aspect, decreasing other types of validity.
Validity, Control, and Constraint
Risk is balanced by rewardA poorly designed study will provide no useful
information; therefore, any risk would be unacceptable
Informed ConsentVirtually guarantees that you will have
confounding due to selection because some people will refuse to participate
A small price to pay to maintain ethical standards
Ethical Principles
Start by building a research hypothesisTesting the research hypothesis is actually
testing three hypotheses(1) null; (2) confounding-variable; (3) causal
Several types of validityMany potential confounding variablesSubject and experimenter effects can also
affect the outcome of the study
Summary