Upload
rozykalsi
View
1.556
Download
0
Embed Size (px)
Citation preview
Types of design - experimentsTypes of design - experiments 11
Types of research design – experimentsTypes of research design – experiments
Chapter 8 in Babbie & Mouton (2001)Chapter 8 in Babbie & Mouton (2001)
Introduction to all research designsIntroduction to all research designs All research designs have specific All research designs have specific
objectives they strive forobjectives they strive for Have different strengths and limitationsHave different strengths and limitations Have validity considerationsHave validity considerations
Types of design - experimentsTypes of design - experiments 22
Validity considerationsValidity considerations
When we say that a knowledge claim (or proposition) When we say that a knowledge claim (or proposition) is valid, we make a JUDGEMENT about the extent to is valid, we make a JUDGEMENT about the extent to which relevant evidence supports that claim to be which relevant evidence supports that claim to be truetrue
Is the interpretation of the evidence given the only Is the interpretation of the evidence given the only possible one, or are there other plausible ones?possible one, or are there other plausible ones?
"Plausible rival hypotheses" = potential alternative "Plausible rival hypotheses" = potential alternative explanations/claimsexplanations/claims e.g. New York City's "zero tolerance" crime fighting strategy e.g. New York City's "zero tolerance" crime fighting strategy
in the 1980s and 1990s - the reverse of the "broken in the 1980s and 1990s - the reverse of the "broken windows" effectwindows" effect
Types of design - experimentsTypes of design - experiments 33
The logic of causal social research in the The logic of causal social research in the controlled experimentcontrolled experiment
Explanatory rather than descriptiveExplanatory rather than descriptive Different from correlational research - one variable is Different from correlational research - one variable is
manipulated (IV) and the effect of that manipulation manipulated (IV) and the effect of that manipulation observed on a second variable (DV)observed on a second variable (DV)
If … then ….If … then ….
E.g.E.g. "Animals respond aggressively to crowding" (causal)"Animals respond aggressively to crowding" (causal) "People with premarital sexual experience have more stable "People with premarital sexual experience have more stable
marriages" (noncausal)marriages" (noncausal)
Types of design - experimentsTypes of design - experiments 44
Three pairs of components:Three pairs of components:
Independent and dependent variablesIndependent and dependent variables Pre-testing and post-testingPre-testing and post-testing Experimental and control groupsExperimental and control groups
Types of design - experimentsTypes of design - experiments 55
ComponentsComponents
VariablesVariables Dependent (DV)Dependent (DV) Independent (IV)Independent (IV)
Pre-testing and post-testingPre-testing and post-testing OO XX O O
Experimental and control groupsExperimental and control groups To off-set the effects of the experiment itself; To off-set the effects of the experiment itself;
to detect effects of the experiment itselfto detect effects of the experiment itself
Types of design - experimentsTypes of design - experiments 66
The generic experimental design:The generic experimental design:
RR OO11 XX OO22
RR OO33 OO44
The IV is an active variable; it is manipulatedThe IV is an active variable; it is manipulated The participants who receive one level of the IV The participants who receive one level of the IV
are equivalent in all ways to those who receive are equivalent in all ways to those who receive other levels of the IVother levels of the IV
Types of design - experimentsTypes of design - experiments 77
SamplingSampling
1. Selecting subjects to participate in the 1. Selecting subjects to participate in the researchresearch Careful sampling to ensure that results can be Careful sampling to ensure that results can be
generalized from sample to populationgeneralized from sample to population The relationship found might only exist in the The relationship found might only exist in the
sample; need to ensure that it exists in the sample; need to ensure that it exists in the populationpopulation
Probability sampling techniquesProbability sampling techniques
Types of design - experimentsTypes of design - experiments 88
SamplingSampling
2. How the sample is divided into two or 2. How the sample is divided into two or more groups is importantmore groups is important to make the groups similar when they start offto make the groups similar when they start off randomization - equal chancerandomization - equal chance matching - similar to quota sampling matching - similar to quota sampling
proceduresprocedures match the groups in terms of the most match the groups in terms of the most
relevant variables; e.g. age, sex, and racerelevant variables; e.g. age, sex, and race
Types of design - experimentsTypes of design - experiments 99
Variations on the standard experimental Variations on the standard experimental designdesign
One-shot case studyOne-shot case study
XX OO
No real comparisonNo real comparison
Types of design - experimentsTypes of design - experiments 1010
A famous one-group posttest-only designA famous one-group posttest-only design
Milgram's study on obedienceMilgram's study on obedience Obedience to authorityObedience to authority The willingness of subjects to follow E's orders to give The willingness of subjects to follow E's orders to give
painful electrical shocks to another subjectpainful electrical shocks to another subject A real, important issue here: how could "ordinary" A real, important issue here: how could "ordinary"
citizens, like many Germans during the Nazi period, citizens, like many Germans during the Nazi period, do these incredibly cruel and brutal things?do these incredibly cruel and brutal things?
If a person is under allegiance to a legitimate If a person is under allegiance to a legitimate authority, under what conditions will the person defy authority, under what conditions will the person defy the authority if s/he is asked to carry out actions the authority if s/he is asked to carry out actions clearly incompatible with basic moral standards?clearly incompatible with basic moral standards?
Types of design - experimentsTypes of design - experiments 1111
One-group pre-test post-test designOne-group pre-test post-test design
OO11 X X OO22
Types of design - experimentsTypes of design - experiments 1212
ExampleExample
We want to find out whether a family literacy We want to find out whether a family literacy programme enhances the cognitive development of programme enhances the cognitive development of preschool-age children.preschool-age children.
Find 20 families with a 4-year old child, enrol the family Find 20 families with a 4-year old child, enrol the family in a high-quality family literacy programmein a high-quality family literacy programme
Administer a pretest to the 20 children - they score a Administer a pretest to the 20 children - they score a mean of say 50 on the cognitive testmean of say 50 on the cognitive test
The family participates in the programme for twelve The family participates in the programme for twelve monthsmonths
Administer a post-test to the 20 children; now they Administer a post-test to the 20 children; now they score 75 on the test - a gain of 25score 75 on the test - a gain of 25
Types of design - experimentsTypes of design - experiments 1313
Two claims/conclusions:Two claims/conclusions:
1 The children gained 25 points on 1 The children gained 25 points on average in terms of their cognitive average in terms of their cognitive performanceperformance
2 the family literacy programme caused 2 the family literacy programme caused the gain in scoresthe gain in scores
VALIDITY - rival explanationsVALIDITY - rival explanations
Types of design - experimentsTypes of design - experiments 1414
Static-group comparisonStatic-group comparison
XX OO OO
Types of design - experimentsTypes of design - experiments 1515
Evaluating research (experiments)Evaluating research (experiments)
We know the structure of researchWe know the structure of research We understand designsWe understand designs We know the requirements of "good" We know the requirements of "good"
researchresearch Then we can evaluate a studyThen we can evaluate a study Is it good? Can we believe its conclusions?Is it good? Can we believe its conclusions? Back to plausible rival hypothesesBack to plausible rival hypotheses
Types of design - experimentsTypes of design - experiments 1616
Validity in designsValidity in designs
If the design is not valid, then the If the design is not valid, then the conclusions drawn are not supported; it is conclusions drawn are not supported; it is like not doing research at alllike not doing research at all
Validity of designs come in two parts:Validity of designs come in two parts: Internal validity Internal validity
can the design sustain the conclusions?can the design sustain the conclusions?
External validity External validity can the conclusions be generalized to the can the conclusions be generalized to the
population?population?
Types of design - experimentsTypes of design - experiments 1717
Internal validityInternal validity
Each design is only capable of supporting certain Each design is only capable of supporting certain types of conclusionstypes of conclusions e.g. only experiments can support conclusions about causalitye.g. only experiments can support conclusions about causality
Says nothing about if the results can be applied Says nothing about if the results can be applied to the real world (generalization)to the real world (generalization)
Generally, the more controlled the situation, the Generally, the more controlled the situation, the higher the internal validityhigher the internal validity
The conclusions drawn from experimental results The conclusions drawn from experimental results may not accurately reflect hat has gone on in may not accurately reflect hat has gone on in the experiment itselfthe experiment itself
Types of design - experimentsTypes of design - experiments 1818
Sources of internal invaliditySources of internal invalidity
These sources often discussed as part of These sources often discussed as part of experiments, but can be applied to all experiments, but can be applied to all designs (e.g. see reactivity)designs (e.g. see reactivity)
HistoryHistory Historical events may occur that will be Historical events may occur that will be
confounded with the IVconfounded with the IV Especially in field research (compare the Especially in field research (compare the
control in a laboratory, e.g. nonsense syllables control in a laboratory, e.g. nonsense syllables in memory studiesin memory studies
Types of design - experimentsTypes of design - experiments 1919
MaturationMaturation
Changes over time can be caused by a Changes over time can be caused by a natural learning processnatural learning process
People naturally grow older, tired, bored, People naturally grow older, tired, bored, over timeover time
Types of design - experimentsTypes of design - experiments 2020
Testing (reactivity)Testing (reactivity)
People realize they are being studied, and People realize they are being studied, and respond the way they think is appropriaterespond the way they think is appropriate
The very act of studying something may The very act of studying something may change itchange it
In qualitative research, the "on stage" In qualitative research, the "on stage" effectseffects
Types of design - experimentsTypes of design - experiments 2121
The Hawthorne studiesThe Hawthorne studies
Improved performance because of the Improved performance because of the researcher's presence - people became researcher's presence - people became aware that they were in an experiment, or aware that they were in an experiment, or that they were given special treatmentthat they were given special treatment
Especially for people who lack social Especially for people who lack social contacts, e.g. residents of nursing homes, contacts, e.g. residents of nursing homes, chronic mental patientschronic mental patients
Types of design - experimentsTypes of design - experiments 2222
Placebo effectPlacebo effect
When a person expects a treatment or When a person expects a treatment or experience to change her/him, the person experience to change her/him, the person changes, even when the "treatment" is changes, even when the "treatment" is know to be inert or ineffectiveknow to be inert or ineffective
Medical researchMedical research "The bedside manner", or the power of "The bedside manner", or the power of
suggestionsuggestion
Types of design - experimentsTypes of design - experiments 2323
Experimenter expectancyExperimenter expectancy
Pygmalion effect - self-fulfilling prophecies of Pygmalion effect - self-fulfilling prophecies of e.g. teachers' expectancies about student e.g. teachers' expectancies about student achievementachievement
Experimenters may prejudge their results - Experimenters may prejudge their results - experimenter biasexperimenter bias
Double blind experiments:Double blind experiments: Both the researcher and the research participant Both the researcher and the research participant
are "blind" to the purpose of the study.are "blind" to the purpose of the study. They don't know what treatment the participant They don't know what treatment the participant
is gettingis getting
Types of design - experimentsTypes of design - experiments 2424
InstrumentationInstrumentation
Instruments with low reliability lead to Instruments with low reliability lead to inaccurate findings/missing phenomenainaccurate findings/missing phenomena
e.g. human observers become more e.g. human observers become more skilled over time (from pretest to posttest) skilled over time (from pretest to posttest) and so report more accurate scores at and so report more accurate scores at later time pointslater time points
Types of design - experimentsTypes of design - experiments 2525
Statistical regression to the meanStatistical regression to the mean
Studying extreme scores can lead to Studying extreme scores can lead to inflated differences, which would not inflated differences, which would not occur in moderate scorersoccur in moderate scorers
Types of design - experimentsTypes of design - experiments 2626
Selection biasesSelection biases
Selection subjects for the study, and Selection subjects for the study, and assigning them to E-group and C-groupassigning them to E-group and C-group
Look out for studies using volunteersLook out for studies using volunteers
Types of design - experimentsTypes of design - experiments 2727
AttritionAttrition
Sometimes called experimental (or Sometimes called experimental (or subject) mortalitysubject) mortality
If subjects drop out, it creates a bias to If subjects drop out, it creates a bias to those who did notthose who did not e.g. comparing the effectiveness of family therapy with e.g. comparing the effectiveness of family therapy with
discussion groups for treatment of drug addictiondiscussion groups for treatment of drug addiction addicts with the worst prognosis more likely to drop out of the addicts with the worst prognosis more likely to drop out of the
discussion groupdiscussion group will make it look like family therapy does less well than will make it look like family therapy does less well than
discussion groups, because the "worst cases" were still in the discussion groups, because the "worst cases" were still in the family therapy groupfamily therapy group
Types of design - experimentsTypes of design - experiments 2828
Diffusion or imitation of treatmentsDiffusion or imitation of treatments
When subject can communicate to each When subject can communicate to each other, pass on some information about the other, pass on some information about the treatment (IV)treatment (IV)
Types of design - experimentsTypes of design - experiments 2929
CompensationCompensation
In real life, people may feel sorry for C-In real life, people may feel sorry for C-group who does not get "the treatment" - group who does not get "the treatment" - try to give them something extratry to give them something extra e.g. compare usual day care for street e.g. compare usual day care for street
children with an enhanced day treatment children with an enhanced day treatment conditioncondition
service providers may very well complain service providers may very well complain about inequity, and provide some enhanced about inequity, and provide some enhanced service to the children receiving usual careservice to the children receiving usual care
Types of design - experimentsTypes of design - experiments 3030
Compensatory rivalryCompensatory rivalry
C-group may "work harder" to compete C-group may "work harder" to compete better with the E-groupbetter with the E-group
Types of design - experimentsTypes of design - experiments 3131
DemoralizationDemoralization
Opposite to compensatory rivalryOpposite to compensatory rivalry May feel deprived, and give upMay feel deprived, and give up
e.g. giving unemployed high school dropouts e.g. giving unemployed high school dropouts a second chance at completing matric via a a second chance at completing matric via a special education programmespecial education programme
if we assign some of them to a control if we assign some of them to a control group, who receive "no treatment", they group, who receive "no treatment", they may very well become profoundly may very well become profoundly demoralizeddemoralized
Types of design - experimentsTypes of design - experiments 3232
External validityExternal validity
Can the findings of the study be Can the findings of the study be generalized?generalized?
Do they speak only of our sample, or of a Do they speak only of our sample, or of a wider group?wider group?
To what populations, settings, treatment To what populations, settings, treatment variables (IV's), and measurement variables (IV's), and measurement variables can the finding be generalized?variables can the finding be generalized?
Types of design - experimentsTypes of design - experiments 3333
External validityExternal validity
Mainly questions about three aspects:Mainly questions about three aspects: Research participantsResearch participants Independent variables, or manipulationsIndependent variables, or manipulations Dependent variables, or outcomesDependent variables, or outcomes
Says nothing about the truth of the result that Says nothing about the truth of the result that we are generalizingwe are generalizing
External validity only has meaning once the External validity only has meaning once the internal validity of a study has been establishedinternal validity of a study has been established
Internal validity is the basic minimum without Internal validity is the basic minimum without which an experiment is uninterpretablewhich an experiment is uninterpretable
Types of design - experimentsTypes of design - experiments 3434
External validityExternal validity
Our interest in answering research questions is rarely Our interest in answering research questions is rarely restricted to the specific situation studied - our interest is restricted to the specific situation studied - our interest is in the variables, not the specific details of a piece of in the variables, not the specific details of a piece of researchresearch
But studies differ in many ways, even if they study the But studies differ in many ways, even if they study the same variables:same variables: operational definitions of the variablesoperational definitions of the variables subject population studiedsubject population studied procedural detailsprocedural details observersobservers settingssettings
Generally bigger samples with valid measures lead to Generally bigger samples with valid measures lead to better external validitybetter external validity
Types of design - experimentsTypes of design - experiments 3535
Sources of external invaliditySources of external invalidity
Subject selection - Selecting a sample which Subject selection - Selecting a sample which does not represent the population well, will does not represent the population well, will prevent generalizationprevent generalization
Interaction between the testing situation and the Interaction between the testing situation and the experimental stimulusexperimental stimulus
When people have been sensitized to the issues When people have been sensitized to the issues by the pre-testby the pre-test
Respond differently to the questionnaires the Respond differently to the questionnaires the second time (post-test)second time (post-test)
Operationalization Operationalization
Types of design - experimentsTypes of design - experiments 3636
OperationalizationOperationalization
We take a variable with wide scope and We take a variable with wide scope and operationalize it in a narrow fashionoperationalize it in a narrow fashion
Will we find the same results with a Will we find the same results with a different operationalization of the same different operationalization of the same variable?variable?
Types of design - experimentsTypes of design - experiments 3737
Field experimentsField experiments
"natural" - e.g. disaster research"natural" - e.g. disaster research Static-group comparison type Static-group comparison type Non-equivalent experimental and control Non-equivalent experimental and control
groupsgroups
Types of design - experimentsTypes of design - experiments 3838
Strengths and weaknessesStrengths and weaknesses
StrengthsStrengths ControlControl Manipulating the IVManipulating the IV Sorting out extraneous variablesSorting out extraneous variables
Weaknesses Weaknesses Articifiality - a generalization problemArticifiality - a generalization problem ExpenseExpense Limited range of questionsLimited range of questions
Types of design - experimentsTypes of design - experiments 3939
IN CONCLUSIONIN CONCLUSION
Donald Campbell often cited Neurath's Donald Campbell often cited Neurath's metaphor:metaphor: "in science we are like sailors who must repair "in science we are like sailors who must repair
a rotting ship while it is afloat at sea. We a rotting ship while it is afloat at sea. We depend on the relative soundness of all other depend on the relative soundness of all other planks while we replace a particularly weak planks while we replace a particularly weak one. Each of the planks we now depend on one. Each of the planks we now depend on we will in turn have to replace. No one of we will in turn have to replace. No one of them is a foundation, nor point of certainty, them is a foundation, nor point of certainty, no one of them is incorrigible"no one of them is incorrigible"