Why Use Experimental Designs

8/10/2019 Why Use Experimental Designs

1/15

Why Use Experimental Designs?Experimental research is used to establish cause and effect relationship.


2/15

internal validity is at the center of all causal or cause-effect inferences. When you want to determine

whether some program or treatment causes some outcome or outcomes to occur, then you are

interested in having strong internal validity. Essentially, you want to assess the proposition:

If X, then Y

or, in more colloquial terms:

If the program is given, then the outcome occurs

Unfortunately, it's not enough just to show that when the program or treatment occurs the expected

outcome also happens. That's because there may be lots of reasons, other than the program, for why

you observed the outcome. To really show that there is a causal relationship, you have to

simultaneously address the two propositions:

If X, then Yand

If n o t X, then n o t Y

Or, once again more colloquially:

If the program is given, then the outcome occurs

and

If the program is n o t given, then the outcome does n o t occur


3/15

Experimental DesignsTwo categories

Lab experimentsAn artificial or contrived environment.

2.Field experimentsNatural / non-contrived environment where activities regularly take place.

When the causal effects can be tested. Where controls and manipulations are introduced toestablish cause and affect relationships in an artificial setting. It is possible in lab. When we postulatecause and affect relationships b/w two variables x (Ind.) & y (dep.). Another factor Z may influence Y.Here it is not possible to see effect of X or Z or Y to control Z effect is called control. Z is acontaminating/nuisance factor. e.g. Training Improve performance but some have previousexperience, they may perform well so excluding experience is control & providing training to fresh

only.

Lab experiments involves controlling the contaminating variables through the process of eithermatching or randomization and manipulation of the treatment.

ManipulationIn order to examine the causal effects of an independent Variable on a dependent variable, certainmanipulation are tried. It means create diff levels of the independent variable to assess the impacton the dependent variable. For example treatment of depression (CBT, SSRI, CBT+SSRI) isindependent variable then level of depression can be dependent variable , We measure level ofdepression before & after treatment.


4/15

Nuisance Variables/FactorsControlling the contaminating exogenous or nuisance variables is an important task. Two ways canbe used matching and randomization .

One way of controlling nuisance variables is to match the various groups by taking their

characteristics and deliberately spreading them across groups.

e.g. Students 20 each group (04) Members

Groups 05 (02 Boys 02 Girls)

Gender, Age, Experience Characteristics or Contaminating or nuisance variables. Motivation &Grades ? of 20 students.

Another way of controlling contaminating variables is randomization No predetermination to 5groups every member has a known & equal chance of being assigned any of there 5 groups. So the

characteristics or nuisance variables (gender, Age, Experience) are also randomly and equallydistributed. Here each group is comparable And all variables are controlled. Now we cannot saycause and effect relationship is affected by some nuisance variables. They have been controlled byrandomization. So here we have more internal validity or confidence in the cause are effectrelationship.

1st draw 1st group (more age, Gender, less exp.)

2nd draw 2nd group (more exp. less age, Gender)

3rd draw 3rd group (more exp. less Age, Gender.)

4th draw 4th group (and so on)

Internal ValidityIt refers to the confidence we place in the cause and effect relationship or To what extent does theresearch design permit us to say that the independent variable A causes a change in the dependentvariable B?

Low internal validity less causality or here cause & effect relationship not substantiated

Seven (7) factors or threats to internal validity.1. History,2. Maturation3. Testing4. Instrumentation5. Selection6. Statistical regression7. Mortality8. Design contamination


5/15

1. History: Did some unanticipated event occur while the experiment was in progress and did theseevents affect the dependent variable? History is a threat for the one group design but not for thetwo group design. In the one group pre -post test design, the effect of the treatment is the

difference in the pre-test and post-test scores. This difference may be due to the treatment or tohistory. Is not a threat for the two group (treatment/experimental and comparison/control) designbecause the comparison is between the treatment group and the comparison group. If the historythreat occurs for both groups, the difference between the two groups will not be due to the historyevent . E.g.

In a short experiment designed to investigate the effect of computer-based instruction, Some missedsome instruction because of a power failure at the school.

2. Maturation : Were changes in the dependent variable due to normal developmental processes

operating within the subject as a function of time? Is a threat to for the one group design. Is nota threat to the two group design, assuming that participants in both groups change (mature)at same rate.

e.g. The performance of first graders in a learning experiment begins decreasing after 45 minutesbecause of fatigue.

3. Statistical regression : An effect that is the result of a tendency for subjects selected on thebases of extreme scores to regress towards the mean on subsequent tests. When measurement ofthe dependent variable is not perfectly reliable, there is a tendency for extreme scores to regress ormove toward the mean. The amount of statistical regression is inversely related to the reliability ofthe test.

e.g. In an experiment involving reading instruction, subjects grouped because of poor pre-testreading scores show considerably greater gain than do the groups who scored average and high onthe pre-test.

4. Selection :Refers to selecting participants for the various groups in the study. Are the groupsequivalent at the beginning of the study? If subjects were selected by random sampling andrandom assignment, all had equal chance of being in treatment or comparison groups, and thegroups are equivalent. Were subjects self-selected into experimental and comparison groups? This

could affect the dependent variable. Selection is not a threat for the one group design but it is athreat for the two group design.

e.g. he experimental group in an instructional experiment consisted of a high-ability class, while

the comparison group was an average ability class.

5. Experimental Mortality: Differential loss of participants across groups. Did some participantsdrop out? Did this affect the results? Did about the same number of participants make it throughthe entire study in both experimental and comp arison groups? Is a threat for any design withmore than one group.


6/15

E.g. In a health experiment designed to determine the effect of various exercises, those subjects

who find the exercise most difficult stop participating.

6. Testing: Did the pre-test affect the scores on the posttest? A pre -test may sensitize participantin unanticipated ways and their performance on the post-test may be due to the pre-test, not to the

treatment, or, more likely, and interaction of the pre-test and treatment. Is a threat to the onegroup design. Not a threat to the two group design . Both groups are exposed to the pre-test andso the difference between groups is not due to testing.

e.g. In an experiment in which performance on a logical reasoning test is the dependent variable, a

pre-test cues the subjects about the post-test .

7. Instrumentation : Did any change occur during the study in the way the dependent variablewasmeasured? (Is a threat to the one group design; not to the two group design. )

e.g. Two examiners for an instructional experiment administered the post-test with different

instructions and procedures.

8. Design contamination : Did the comparison group know (or find out) about the experimentalgroup? Did either group have a reason to want to make the research succeed or fail? Often,investigators must interview subjects after the experiment concludes in order to find out if designcontamination occurred.

e.g. In an expectancy experiment, students in the experimental and comparison groups comparenotes about what they were told to expect.

Some other threats are:

Compensatory rivalry . When subjects in some treatments receive goods or services believed to bedesirable and this becomes known to subjects in other groups, social competition may motivate the

latter to attempt to reverse or reduce the anticipated effects of the desirable treatment levels.

Resentful demoralization . If subjects learn that their group receives less desirable goods or services,they may experience feelings of resentment and demoralization.


7/15

External ValidityExternal validity refers to the degree to which the results of an empirical investigation can begeneralized to and across individuals, settings, and times.

How representative is the sample of the population? The more representative, the more confidentwe can be in generalizing from the sample to the population. How widely does the finding apply?Generalizing across populations occurs when a particular research finding works across manydifferent kinds of people, even those not represented in the sample.

Ecological validity is present to the degree that a result generalizes across settings. Types include:

Interaction effect of testing


8/15

Interaction effects of selection biases and experimental treatment Reactive effects of experimental arrangements Multiple -treatment interference Experimenter effects

Quasi-Experimental Designsquasi(meaning having some but not all of the features) preceding the term experimentalindicates that we are dealing with a design that resembles an experiment but is not exactly anexperiment. How does a quasi-experimental design differ from an experimental design? 1)Sometimes the difference is the lack of a control group or a comparison group, that is, only onegroup is given a treatment and then assessed . 2) At other times the independent variable is not atrue manipulated independent variable; instead, it is a participant variable or a nonmanipulatedindependent variable . And 3) finally, some designs may be considered quasi-experimental becauseparticipants were not randomly assigned to conditions, that is, they were already part of a groupand the researcher attempted to manipulate a variable between preexisting groups.

TYPES OF QUASI-EXPERIMENTAL DESIGNS

Single-Group Post-test-Only Design simplest quasi-experimental design.


9/15

As the name implies, it involves the use of a single group of participants to whom sometreatment is given.

The participants are then assessed on the dependent variable. there is neither a comparison group nor a comparison of the results to any previous

measurements

For example, a new educational technique such as interactive learning, outcomes learning,or computer-assisted learning is proposed, and school systems begin to adopt it. Post-testmeasures are then taken to determine the amount learned by students.

This design is open to so many criticisms and potential flaws Chances of maturation effect history effect,

Group Pre-Test Treatment Post-testExperiment NO X1 YES

Single-Group Pretest/Posttest Design measures are taken twice: before the treatment and after the treatment. The two measures can then be compared, and differences in themeasures are assumed to

be the result of the treatment

For instance, if a single group of depressed individuals wanted to receive treatment(counseling) for their depression, we would measure their level of depression before thetreatment, we would then have them participate in the counseling, and finally, we wouldmeasure their level of depression after the treatment.

With no comparison group, we do not know whether any observed change in depression isdue to the treatment or to something else that may have happened during the time of thestudy ie chances of history, testing and instrumentation effect.

Maybe the pretest depression measure was taken right after the holidays when depressionis higher than during the rest of the year for many people. Consequently the participantsmight have scored lower on the posttest depression measure regardless of the counseling.

Group Pre-Test Treatment Post-testExperiment Yes X1 YESControl No No No

Single-Group Time-Series Design involves using a single group of participants, taking multiple measures over a period of time

before introducing the treatment, and then continuing to take several measures after thetreatment.

multiple measures allow us to see whether the behavior is stable before treatment and how,or if, it changes at the points in time at which measures are taken after treatment.


10/15

Cannot eliminate statistical regression, or regression to mean.

Group Pre-Test Treatment Post-test1 Post test 2 Post test n..Experiment Yes X1 YES Yes.. Yes

Control No No No No No

Non-equivalent Control Group Posttest-Only Design similar to the singlegroup posttest-only design, but a non-equivalent control group is added

as a comparison group.

Group Pre-Test Treatment Post-testExperiment Yes X1 YESControl yes No YesThis design is still not a true experimental one because as with the previous designs participants arenot randomly assigned to the two conditions.

e.g. The researchers found a small Canadian town that had no television reception until 1973; theydesignated this town the Notel group. Life in Notel was then compared to life in two othercommunities: Unitel, which received only one station at the beginning of the study, and Multitel,which received four channels at the beginning of the study. A single channel was introduced toNotel at the beginning of the study. During the 2 years of the study Unitel began receiving threeadditional stations. The researchers measured such factors as participation in community activitiesand aggressive behavior in children in all three groups, both before and after the introduction oftelevision in Notel. Results showed that after the introduction of television in Notel, there was asignificant decline in participation in community activities and a significant increase in aggressivebehavior in children.

Sources of Invalidity for Basic Research Designs


11/15

single-case designIn certain types of research researchers use methods that minimize the number of participants in astudy. This procedure may sound contrary to the basic principles of design discussed so far.However, in these methods often referred to as single case designs, only one person is measuredrepeatedly. Frequently the research is replicated on one or two other participants. Thus wesometimes refer to these studies as small-n designs. Such studies can also be thought of as avariation of the pretest/posttest quasi-experimental design.

Researchers may choose a single-case design for several reasons. They may want information ononly the single participant being studied. They may not be interested in trying to generalize theresults to a population, but rather they may only be interested in how the one participant reacts tothe manipulation. Single-case research is often used in clinical settings. In clinical studies manyresearchers believe that it is unethical to use traditional experimental methods in which one groupof participants receives the treatment and the other group serves as a control. They believe it isunethical to withhold treatment from one group, particularly when the participants may really needthe treatment. In such instances single-case or small-ndesigns are more ethically appealing becausethey involve providing treatment to all who participate in the study.

One problem with group designs, is that they do not allow for adequate replication of resultswhereas singlecase designs do. Consequently single-case designs are better at demonstrating a


12/15

reliable effect of an independent variable. A second problem is that group designs contribute toerror variance in a study. Error variance is the random differences in scores found within theconditions of an experiment. Using many people in a group design increases error variance resultingfrom individual differences. The increase in error variance may make it difficult to identify arelationship between the variables in the study.

A third problem that Sidman notes is that when using group designs, we typically look at the meanperformance in each group. However, a mean score for a given condition may not accuratelyrepresent the performance of all the participants in that condition. Once we have drawn conclusionsbased on the mean performance within a group, we then attempt to generalize the results toindividuals. Psychologists thus draw conclusions about individual behavior based on studying theaverage performance of a group of people.

Single-case and small-ndesigns address each of these problems. To determine the reliability of theeffect, we can either repeatedly manipulate the independent variable with the same participant or

perform replications with a few other participants. Further, error variance resulting from individualdifferences is eliminated because only one participant is used. Finally, rather than looking at groupmeans and conducting the appropriate statistical analyses, we look at only the performance of thesingle participant in the study to determine the relationship between the independent anddependent variables. Most commonly we graph the performance of the single participant andexamine the resulting graph. The effect of the independent variable is determined by how much theparticipants behaviour changes from one condition to another. Also because the findings are basedon an individuals performance, it makes sense to generalize the results to other individuals.

Summary

Single-subject design or single-case research design is a research design most often used inapplied fields of psychology, education, and human behavior in which the subject servesas his/her own control, rather than using another individual/group. Researchers usesingle-subject design because these designs are sensitive to individual organismdifferences vs group designs which are sensitive to averages of groups. Often there will belarge numbers of subjects in a research study using single-subject design, however because the subject serves as their own control, this is still a single-subject design.[1]These designs are used primarily to evaluate the effect of a variety of interventions in

applied research.[2]

The following are requirements of single-subject designs:[3]

Continuous assessment: The behaviour of the individual is observed repeatedly over thecourse of the intervention. This insures that any treatment effects are observed longenough to convince the scientist that the treatment produces a lasting effect.

Baseline assessment: Before the treatment is implemented, the researcher is to look forbehavioural trends. If a treatment reverses a baseline trend (e.g., things were getting worse

as time went on in the baseline but the treatment reversed this trend) then this is powerfulevidence suggesting (though not proving) a treatment effect.


13/15

Variability in data: Because behavior is assessed repeatedly, the single-subject designallows the researcher to see how consistently the treatment changes behavior from day-to-day. Large-group statistical designs do not typically provide this information becauserepeated assessments are not usually taken and the behavior of individuals in the groups

are not scrutinized; instead, group means are reported.

Phases within single-subject design

Baseline: this phase is one in which the researcher collects data on the dependent variablewithout any intervention in place.

Intervention: this phase is one in which the researcher introduces an independent variable(the intervention) and then collects data on the dependent variable .

Reversal: this phase is one in which the researcher removes the independent variable(reversal) and then collects data on the dependent variable.

It is important that the data are stable (steady trend and low variability) before theresearcher moves to the next phase. Single-subject designs produce or approximate threelevels of knowledge: (1) descriptive, (2) correlational, and (3) causal.

Between-participants design participants in each group are different, that is, different people serve in the control and

experimental groups

most basic ideas behind an experiment is that there are at least two groups to compare the control group serves as the baseline, or standard, condition. The experimental group

receives some level of the independent variable.

By randomly assigning participants, we are trying to make the two groups as equivalent aspossible.

posttest-only control group design


14/15

Group (random) Pre-Test Treatment Post-testExperiment No X1 YesControl No No Yes

pretest/posttest control group design,Group (random) Pre-Test Treatment Post-testExperiment Yes X1 YesControl Yes No Yes

WITHIN-PARTICIPANTS EXPERIMENTAL DESIGNSwithin-participants design the same participants are used in all conditions. Within-participants

designs are often referred to as repeated-measures designs because we are repeatedly measuringthe same individuals. A random sample of participants is selected, but random assignment is notrelevant or necessary because all participants serve in all conditions. Within-participants designs arepopular in psychological research for several reasons. within-participants design to minimize thenumber of participants needed is advantageous. Second, within-participants designs usually requireless time to conduct than between-participants designs. The study is conducted more quicklybecause participants can usually take part in all conditions in one session; the experimenter does notuse a participant in one condition and then wait around for the next person to participate in the nextcondition. Further, the instructions need to be given to each participant only once. If there are 10participants in a within-participants design and participants are run individually, the experimentneed only be explained 10 times. If there are 10 participants in each condition in a between-participants design in which participants are run individually, the experiment needs to be explained20 times.

Third, and most important, within-participants designs increase statistical power. When the sameindividuals participate in multiple conditions, individual differences between the conditions areminimized.


15/15

Documents

Why Use Experimental Designs