The Basics of Designing and Running an Economics

Experiment

Economics 328Spring 2005

The Purposes of Running an Experiment

Test a Theory Gather Empirical Regularities to Inform a

Theory Test Institutions

What Makes a Good Experiment?

Should an experiment replicate reality? No! We have field studies for that.

Should an experiment replicate a formal model?

No! We have theory for that. A good experiment tries to capture the most

relevant features of reality in a simple, carefully controlled environment. Good experiments are usually designed to test specific hypotheses, sometimes derived from the implications of some economic theory and other times based on previous observations in either experiments or field data.

What is a Controlled Experiment?

As much as possible, the experiment controls all elements of the environment in which the experiment takes place.

The experimenter determines the rules – what choices are available to subjects, when decisions are made, and what the consequences of these decisions will be.

The experimenter controls subjects’ payoffs as a function of the actions they take.

The experimenter controls the information that is available to subjects.

What is a Controlled Experiment?

Between treatments, the experimenter only changes variables which are directly relevant to the hypothesis being tested, otherwise holding the environment fixed.

Control vs. Treatment Sessions Avoid Confounds (e.g. Don’t change more than one thing at a time.) Variables that cannot be directly controlled are typically controlled via

randomization.

Example: Many experiments are designed to test subjects’ attitudes towards fairness are affected by some treatment variable. To the extent that subjects enter the lab with differing attitudes about fairness, a true controlled experiment can’t be run. However, by randomly assigning subjects to treatments, we can eliminate subjects differing attitudes as a cause of differences between treatments. This relies on the law of large numbers, implying that a large sample may be necessary.

Designing an Experiment Identify an interesting question or questions. These should be issues

that are better addressed through a controlled experiment than through gathering field data.

Determine hypotheses you would like to test.

Example: Increased payoffs lead to fewer mistakes in solving logic problems.

Example: Face-to-face interaction will lead to greater concern with fairness.

Design a simple environment that allows you to test the hypotheses you are interested in. Remember, the more complicated the environment is, the more likely you are to lose control. An experiment is not meant to replicate reality. The reason we use controlled experiments is because reality is too complicated.

Brandts and Cooper:The Weak Link Game

The employee payoff function is given by the following equation:

Having all five players choose the same effort level is a Nash equilibrium.

The critical variable here is B, the bonus rate. Higher values of B give greater benefits to successful coordination.

Trying to move to a higher effort level is a risky strategy.

i

e i jj 1,2,3,4

π 200 5E B min E

Worker i's Payoff Table, B = 6

Minimum Effort by Other Workers0 10 20 30 40

0 200 200 200 200 200Effort 10 150 210 210 210 210

by 20 100 160 220 220 220Worker i 30 50 110 170 230 230

40 0 60 120 180 240

Worker i's Payoff Table, B = 14

Minimum Effort by Other Workers0 10 20 30 40

0 200 200 200 200 200Effort 10 150 290 290 290 290

by 20 100 240 380 380 380Worker i 30 50 190 330 470 470

40 0 140 280 420 560

Brandts and Cooper:Research Questions

Can an increase in the bonus rate enable a firm to overcome coordination failure?

Does the magnitude of the bonus rate increase matter or is the simple fact of an increase effective as such?

If an increase in the bonus rate brings about improved coordination, can the bonus rate increase be revoked without affecting the improved outcome?

Does the length of time a firm has been underperforming affect the impact of the increase?

Designing an Experiment Test your hypotheses by varying a small number of

variables within the experiment. If at all possible, you should not vary more than one variable at a time.

Within vs. Between Subject Designs Some important factors to remember . . .

Subjects get bored. Subjects get confused.

Subjects learn as they gain experience within the experiment. You should anticipate changes in their behavior (order effects).

Design for Brandts and Cooper

List of Treatments

Cell 1 Cell 2 Cell 3 Cell 4 Cell 5 Cell 6 Bonus Rate

Rounds 1 – 10 6 6 6 6 6 6

Bonus Rate Rounds 11 – 20

14 10 8 14 14 6

Bonus Rate Rounds 21-30

14 10 8 10 6 14

Results for Brandts and Cooper

Comparison of Treatments, Second Block

0

10

20

30

40

10 11 12 13 14 15 16 17 18 19 20

Period

Ave

rag

e M

inim

um

Eff

ort

Treatment 6/6/14 Treatment 6/8/8 Treatment 6/10/10 Treatment 6/14/X

Results of Classroom Experiment

Effect of Bonus Rate Increase

0

10

20

30

40

5 6 7 8 9 10

Round

Ave

rag

e M

inim

um

Eff

ort

B = 8 or B = 10 B = 14

Results of Classroom Experiment

Effect of Bonus Rate Increase

0

10

20

30

40

5 6 7 8 9 10

Round

Ave

rag

e E

ffo

rt

B = 8 or B = 10 B = 14

Designing an Experiment Avoid the use of deceit. This is a very tempting trap, but one

that is highly frowned upon by the profession. In particular, what is going to happen the next time you try to run an experiment?

Anonymity: Most experiments guarantee subjects anonymity. In other words, subjects are guaranteed that no other subjects (or indeed, nobody other than the researchers) will be able to ever identify their actions or payoffs. For some experiments, abandoning anonymity is an important part of the design. If so, subjects should know what information about them is be revealed publicly. Subjects should have the option of withdrawing if they do not want information about them revealed publicly.

Subjects

What population should you use as subjects? Undergraduates are the easiest to get, but subjects with relevant

experience are often more interesting to study. How do you get subjects?

Advertisements can be placed on posters, electronic bulletin boards, or in newspapers. Spamming, while effective, is generally not a good way to make friends.

What should be in the advertisement? It should include a very brief description, a summary of payoffs (average

payoff is sufficient), the time necessary to complete the experiment, and contact information. (E-mail addresses work well. A website where people can sign up for the experiment works even better.)

The advertisement should stress the monetary payoffs. Remember, you want controlled experiments. This means that you want subjects who care about the monetary payoffs being offered.

Play Games!!! Earn Cash!!!

Participate in an Experiment in Economic Decision Making!!! Average Earnings = $20 - 25!!!

Takes 1½ Hours or Less!!!

Wednesday, September 4th, 4:30 Thursday, September 5th, 4:30 Monday, September 9th, 4:30

Tuesday, September 10th, 9:30 More Times Available

Sign up at :

www.weatherhead.cwru.edu/exper

Play Games!!!

Earn $$$

Subjects

Avoid unintentional selection of the subject population. For example, if you post all of you ads in the weight room in Veale, your subject population will be skewed towards men. You should also avoid unintentional clustering (e.g. don’t sign up ten people from the same fraternity for a single session).

Have times and locations for sessions selected before posting any ads. Be certain to schedule potential subjects as soon as they respond to your ad. Be certain to get contact information from your subjects. Confirm that all subjects will attend the day before the session. Even with confirmation, you should anticipate that only about 80% of subjects will show up (CWRU students tend to be pretty good about showing up).

Keep careful track of all contacts with potential subjects. You should never use a subject twice for the same set of experiments. Most experimental groups keep a black list – subjects who don’t show up or disrupt sessions are undesirable.

Instructions

Most experiments include some form of consent form. For those of you who plan on publishing your research, this brings up the issue of the dread IRB . . .

The overarching goal of instructions is to make certain that the subjects understand the rules of the experiment. If subjects do not understand the rules, you have immediately lost control.

Instructions should be as clear and complete as possible. Any critical points should be repeated at least once. If in doubt, make the instructions too detailed. Nobody likes sitting through lengthy instructions, but remember your goal – the subjects need to understand the rules of the experiment.

Context: Many experiments like to frame their instructions in a generic context that does not have any real world connotations. In my opinion, this often generates confusion for experimental subjects, but that is a controversial point. All experimenters agree that you should avoid any loaded terms.

Example: In a Prisoner’s Dilemma Game, you wouldn’t want to label the two strategies as “Cooperate” and “Cheat” even though this is how game theorists often refer to them.

Instructions

Demand Effects: A common flaw of experimental instructions is that they suggest how the subjects ought to behave in a way that compromises the experimental design. This should be avoided scrupulously. Sometimes demand induced effects can be quite subtle.

Example: “The employees of a firm are always better off if all of them spend more hours on Activity A.”

Example: All of the examples in your instructions have exactly the same choices being made.

Payoff Quizzes: Payoff quizzes should be used to ensure that subjects understand the relationship between their choices, the choices of others, and their payoffs. Payoff quizzes also serve as a device for reinforcing important points from the instructions. Take care to avoid demand effects in writing your payoff quiz.

Instructions

Answering Questions: You should have subjects raise their hands and tell an experimenter any questions privately. (Use someone other than the person reading instructions.) This limits the chance of losing control by having one subject sharing his/her strategy with everyone in the session. Anything that you think is substantive, you should repeat aloud to the entire class. A common question is "Should I do blah, blah, blah . . ." or "What should I do . . ." Do not answer these questions directly, as this leads to demand effects. I always answer something like, "We don't know what you should do. The whole purpose of running the experiments is to see what you will do. You should do whatever seems best to you." Nobody has slugged me yet, although I imagine they have been tempted.

Actually Running the Experiment Computers vs. Paper & Pencil Experiments Run pilots in advance! Getting experimental subjects is hard work.

You don't want to waste a room full of subjects because there is a problem with your procedures. Think everything out in advance and do some practice.

Setting Up the Room Always set the room up in advance. Time is precious, so you don't want to

waste time moving desks or handing out papers. Separate people who arrive at the same time. You generally don't want

subjects talking during the experiment, and people who arrive together are often friends.

Be certain that subjects can easily see you and any materials you might post in giving out instructions. Make it as hard as possible for them to see the materials being filled out by other subjects. Many established labs will have either dividers or sunken terminals. For our purposes, get a big room and spread people out.

Actually Running the Experiment

Keep careful records of subjects' arrivals to the lab. I usually keep a checklist. I usually will not allow any subject who is more than five minutes late participate in the session (10 if I'm desperate). I sometimes have done "bully recruiting," but this is generally best avoided.

You should generally have one person who runs the experiment (a "conductor) and a group of people who help out ("assistants"). The conductor is the front man. He/she reads the instructions, gives any general instructions, and oversees the experiment. The conductor need not have any particular authority in reality – the goal is to have someone with a loud voice who will seem credible to the subjects. The assistants do all the other myriad tasks involved in running an experiment.

Both the conductor and the assistant should be careful to avoid any demand effects. Do not lurk over the subjects while they are making decisions, do not rush them if they are thinking about a choice, and don't talk among yourselves unnecessarily during the session.

Actually Running the Experiment

You will want to have all forms made up in advance. In general, you will need the following sorts of forms. Make up a checklist before the experiment to make certain you don't forget anything.

Consent Forms (for publishable research) Instructions Payoff Tables: Make certain these are easy to read. Have these

separate from the instructions so subjects can easily refer to them. Record Sheets: You generally want a sheet that allows subjects to

record all the information they have received, all the choices they have made, and all the payoffs they have received. This both allows them to know the history of their past choices and to confirm that you are paying them what they are owed.

Play Sheets: Sheets where subjects record their actions. Receipts

Actually Running the Experiment

Paying Subjects: You should pay subjects in cash as privately as possible immediately after the end of the session. As much as possible, you should calculate the payoffs yourself rather than having the subjects do it. Be certain to get receipts!!! These need the subject's name, SS#, the date, the amount paid, and a signature.

Collecting Data: In designing your experiment, you should try to make it as easy as possible to figure out afterwards what happened. In particular, design the action sheets carefully and store them neatly as the experiment progresses. Buying lots of folders and labeling them in advance is a lot cheaper than spending days deciphering a mess. Do not rely on subjects' record sheets – these are often inaccurate.

Some Issues of Experimental Techniques As much as possible, random number generation should be public and

credible. For pencil and paper experiments, I try to use devices people will understand – rolling dice and flipping coins. Occasionally pre-generating the random numbers is desirable, although I try to avoid this.

Many experiments involve rematching subjects. If they are not supposed to be matched with the same person, you need to think out the matchings in advance to make certain that they won't be rematched. This is easy for 1 x 1 matchings, harder for group matchings.

Anonymity: For many experiments, we don't want subjects to know whom they are playing. For most experiments, using ID numbers rather than names will be good enough. For experiments with rematching, you may want to conceal the ID numbers. Some experimenters use more extreme techniques. For most cases, I think these are overkill.

Some Issues of Experimental Techniques

Risk Aversion: To limit the effects of risk aversion for experiments where it is a nuisance variable, many experiments use a "lottery technique." The value of this technique is dubious and it confuses the living daylights out of subjects. You should be sensitive to the possible effects of risk aversion on your results.

Feedback: Experimental results are quite sensitive to what feedback subjects are given. Any sort of feedback is acceptable, just think about its effects in advance and be certain that you aren't accidentally leaking additional information. Often times the feedback is an important component of the experimental design.

Asymmetric Information: Often times you want some information to only go to some subjects. Be certain that this doesn't slip over into deception. The other common problem is accidentally revealing information to people who aren't supposed to have it.