Embed Size (px)
Citation preview
O
Pa
b
a
ARRA
KDES
1
insse
dmotd
ecpthuati
SfM
e
1h
The Journal of Socio-Economics 41 (2012) 714– 720
Contents lists available at SciVerse ScienceDirect
The Journal of Socio-Economics
j o ur nal homep ag e: www.elsev ier .com/ locate /soceco
n “lab rats”�
ablo Guilléna,∗, Róbert F. Vesztegb
Faculty of Economics and Business, Discipline of Economics, Room 340, Merewether Building (H04), The University of Sydney, NSW 2006, Sydney, AustraliaSchool of Political Science and Economics, Waseda University, 1-6-1 Nishiwaseda, Shinjuku-ku, 169-8050 Tokyo, Japan
r t i c l e i n f o
rticle history:eceived 10 June 2011
a b s t r a c t
Experimental subjects usually self-select to the laboratory and this may introduce a bias to the con-clusions derived from observing their behavior. We analyze data stored by a subject-pool management
eceived in revised form 29 June 2012ccepted 7 July 2012
eywords:emographic characteristicsxperiments
program at an experimental laboratory and speculate about the effect of individual decisions on returning.Specifically, we test whether experience and earnings in previous sessions together with demographicvariables explain the decision to return to the laboratory. We find that males and (in monetary terms)well-performing subjects are more likely to participate in experiments again.
© 2012 Elsevier Inc. All rights reserved.
Harrison et al. (2009) conducted such experiments and analyze theproblem of self-selection using both laboratory and field exper-iments. They observe that by changing the reward scheme, the
ubject pool
. Introduction
The wide acceptance that experimental economics has achievedn the profession over the past three decades offers an opportu-ity for self-inspection. It is time now to put our method undercrutiny – if for no other reason than to dissipate the remainingkepticism – by taking a critical look at the basic rules of runningxperiments.
Generally speaking, the purpose of laboratory experiments (onecision-making) is to test hypotheses in a controlled environ-ent. However, the composition of the subject pool is often out
f the experimenter’s control and subjects’ individual characteris-ics are rarely included in the statistical analysis of experimentalata.
In this paper, we study the evolution of the subject pool at anxperimental laboratory and speculate about its consequences ononclusions derived from the collected observations. We do notose more specific research questions, as our approach reverseshe usual experimental steps. As we make use of a database thatas been created for administrative reasons, we do not have all thesual experimental controls. Our aim is to check whether there are
ny general regularities in subjects’ return decisions. We believehat either a positive or a negative answer would represent anmportant finding for the experimental method.� We thank Jordi Brandts, Mathieu Durand, Glenn Harrison, Al Roth, Carmitegal and Robert Slonim for their comments and precious help. Veszteg grate-ully acknowledges the financial support from project SEJ2006-10087 of the Spanish
inistry of Education and Sciences, and from PIUNA of the Universidad de Navarra.∗ Corresponding author.
E-mail addresses: [email protected], [email protected] (P. Guillén), [email protected] (R.F. Veszteg).
053-5357/$ – see front matter © 2012 Elsevier Inc. All rights reserved.ttp://dx.doi.org/10.1016/j.socec.2012.07.002
Subject pools used in economic experiments typically consistof college students recruited online or on campus on a voluntarybasis. In other words, they self-select to the laboratory. This mayintroduce a bias to the conclusions. This bias, unless the possibleadverse effects of self-selection are explicitly controlled for, cannotbe overcome by randomization (assigning subject to treatments in arandom way) and by founding the inference upon across-treatmentcomparisons of observed behavior.
Selection bias is tightly related to the broader problematics ofexternal validity.1 Firstly, students obviously differ significantlyand in important ways from the general population in terms oftheir age, educational level and experience.2 Secondly, experimen-tal samples of student subjects may fail to be representative evenfor the general population of students (at their school, college, city,etc.) due to selection bias.
Andersen et al. (2010), for example, argue that the subject poolin the laboratory may not constitute a representative sample of thebroader population and call for complementary field experiments.3
1 For an excellent discussion of external validity, refer to Bardsley et al. (2010),who dedicate an entire chapter to it.
2 Instead of reviewing the literature related to this point, we refer to the second-order meta-analysis conducted by Peterson (2001). The author finds that studentsamples (with observations on behavioral or psychological relationships) tend to bemore homogeneous. More importantly, effect sizes inferred from student samplesfrequently differ (both in direction and in magnitude) from those estimated fromnon-student samples.
3 They consider experiments to elicit preference heterogeneity and claim that“the lab might not be the best place to search for demographic effects”.
l of So
csalt
eepbmdenesinBtfacsaip
eresbooeeb
vlersta
amrmbtdtaf
waraa
rience, EXP, we have counted the total number of occasions thesubject appears in the database prior to the experimental sessionin question (we have only taken into account those sessions that
5 While the university has given us its written permission to analyze the collecteddata, we are not allowed to refer to it by name.
6 The administrative system that gathered the data was an in-house design thatpredates the popular ORSEE software. All participants were attracted to the recruit-ment website by on-campus posters and required to create an account (with apersonal username and password). Each time they logged in they could see theexperiments which they were eligible for. In principle, all eligible (as determined bythe experimenter) volunteers could sign up for a given session. Note that a systemlike ORSEE, that sends invitations to a random sample of eligible potential partic-ipants, would only help in solving the problems caused by selection bias if therewere enough eligible and active participants in the registered subject pool.
7 This selection is in line with the philosophy of the regression analysis we per-form on the data, since ignoring fixed and quasi-fixed payments guarantees largervariance in the dependent variable. As a result, we effectively exclude most tour-nament experiments, and also those sessions with fixed payments in which a few
P. Guillén, R.F. Veszteg / The Journa
onstitution of the subject pool (e.g., the proportion of risk-averseubjects) also changes. Ruström (1998), Eckel and Grossman (2000),nd Casari et al. (2007) derive similar conclusions using data fromaboratory experiments on the Vickrey and the English auctions,he dictator game, and common-value auctions, respectively.
Others, using different approaches and different games in theirxperiments, fail to identify any bias related to self-selection how-ver. For example, Cleave et al. (2010) look at social and riskreferences and do not find significant differences between theroader student population and the group of self-selected experi-ental subjects. Anderson et al. (2010) report similarly negligible
ifferences when comparing measures for other-regarding pref-rences across three samples (self-selected students, self-selectedon-students, and non-self-selected non-students). Despite differ-nces in their scope, the details in their protocol and the statisticalignificance of the reported differences, these studies open anmportant line of research, concerned with the basic rules of run-ing laboratory experiments, on which the collection of papers byardsley et al. (2010) constitutes a milestone. In this paper, we aimo contribute to the discussion by considering administrative datarom the recruitment system of an experimental laboratory locatedt a university in the Northeastern United States. The databaseontains individual participation and earnings histories along withome demographic information. We use simple statistical tools tonalyze the subjects’ decision to return to the laboratory by treat-ng it as a function of variables like gender, age, experience andrevious earnings.
We find previous earnings and age to have a significant positiveffect on the probability of returning to the laboratory.4 Expe-ience, on the other hand, seems to have a significant negativeffect. Although statistically significant, these effects tend to bemall in size. As for the effects of the other personal characteristics,eing a male student, especially one attending Boston Universityr Harvard, and majoring in social sciences increase one’s oddsf returning to the laboratory where our data were recorded. Theffect of these characteristics cannot be considered negligible asach increases the odds of returning to the laboratory by a factoretween 12 and 34% points.
The positive effect of studying at Boston University or at Har-ard can be traced back to their proximity to the experimentalaboratory. The positive effect of studying social sciences might bexplained by the fact that the word “economics” is often used in theecruitment material and that this may bias the sample towardstudents with an interest in the field. Interestingly, even if we con-rol for the other personal characteristics, it turns out that malesre more prone to return.
In experimental research, demographic variables (typically agend gender) are mostly taken into account only if the author’s pri-ary objective is to analyze the effect of such variables. For this
eason, the abundance of male experimental subjects (for they areore prone to return to the laboratory) might introduce a serious
ias. Croson and Gneezy (2008) present some empirical evidencehat supports this conjecture. Their study surveys research on gen-er differences in risk aversion, social preferences and preferenceowards competition and finds that women tend to be more risk
verse, to act according to social cues rather than principles whenacing social dilemmas, and to dislike competition.4 This earning effect is in line with the findings by Eckel and Grossman (2000)ho compare results from the dictator game in the classroom (pseudo-volunteers)
nd the experimental laboratory (volunteers). They observe that “when subjects areecruited to an independent location and paid for their appearance, they behave in
less extreme manner”, and find “some indirect evidence that volunteer subjectsre more motivated by [monetary] incentives”.
cio-Economics 41 (2012) 714– 720 715
2. Descriptives
Our data set consists of 8755 observations. Each observationrepresents a subject’s participation in an experimental session. Ouranalysis covers a total of 2408 subjects who participated in 597experimental sessions corresponding to 74 different studies. Allthe data come from the same laboratory located at a universityin the Northeastern United States.5 We use entries recorded afterApril 2003, as this is the month when the laboratory first startedgathering participants’ personal data on a regular basis.6
The latest observations included in this analysis were gatheredin January 2006. The available data include subjects’ self-reportedpersonal characteristics. This includes their gender, age, and theuniversity (if any) they are affiliated with, along with their earningsin the experiment.
In some experiments, subjects’ payment does not depend, ordepends very little, on the subjects’ behavior. Since our objectiveis to study economic experiments where monetary incentives arethe norm, we have omitted observations from sessions with fixedpayments or in which payments do not vary much. We excluded allthe sessions in which 80% or more of the participants received thesame amount of money.7 We have not discovered important qual-itative changes in the results when performing the same analysisusing cutoffs of 50%, 90% and 100%, instead of 80%.8 We have alsoomitted all records with zero dollar payments, and repeated entries,keeping the one with the highest payment.9 These two categoriesare a result of faulty data entries: no subject actually received zeropayment, and no subject was paid more than once for the sameexperiment. We found a total of 176 zero entries and 277 repeatedentries. This leaves 8755 observations in the data set.10
Apart from the recorded personal data such as age, gender, racialgroup, educational level (with intended major and the name of thecollege), and the basic characteristics of the experimental session(final payment, experiment id), we also created some additionalvariables for the empirical analysis. Of the 2408 subjects in thedatabase, 70% came to the laboratory more than once over thecourse of the period we investigate. In order to control for expe-
of the subjects earned more money, due to the “early show-up fee” that rewardspeople who arrive at experiments early with an extra payment.
8 The excluded observations belong to participants who on average earned almost$4 less than the included ones. They also tend to be older (by 0.78 years) and aremore likely to be college students (by 2.3% points).
9 We enquired and found that the lower payments usually correspond to show-upfees, while the higher payments include all monetary payoffs.
10 The laboratory has recently started collecting information on subjects’ ethnicgroup. As there are only 4559 (52.97%) entries that contain a value for this variable,we decided not to include this variable in the final data set in order not to reducethe number of observations in the analysis. If we compare the mean payoffs acrossthe nine ethnic groups using analysis of variance (ANOVA) tables we cannot rejectthe null hypothesis of them being equal. The P-value in this case is of 0.14.
7 l of Socio-Economics 41 (2012) 714– 720
mhhanit
RtdhiwiycTeu2otfdtsi
isetETmsttirob$sti
3
atu1iR
aaAetr
Table 1Subject descriptives. Newcomers: subjects with no experience in incentive-basedexperiments. Experts: subjects with experience of more than 27 incentive-basedexperiments. Measurement units in second column between parentheses.
Characteristics Variable Newcomers All Experts
Age (years) AGE 23.03 23.56 35.86Gender (%) MALE
Female 48.59 44.81 19.05Male 41.69 46.61 64.29n/a 9.72 8.58 16.67
Education (%)Some high school E1 0.26 0.27 0.00High school diploma E2 8.93 7.86 0.00Some college E3 53.13 52.27 9.52Associate level degree E4 0.73 0.51 0.00Bachelor level degree E5 16.51 18.52 17.86Other masters E6 6.37 6.72 20.24MBA E7 0.31 1.11 0.00Doctoral level degree E8 0.94 1.28 19.05n/a 12.80 11.47 33.33
College (%)Other U1 4.60 3.86 0.00Univ. of Massachusetts U2 0.47 0.95 27.38Tufts University U3 2.04 2.44 8.33Northeastern University U4 1.52 2.22 0.00Boston College U5 0.73 0.62 0.00Boston University U6 11.13 12.26 20.24Harvard University U7 46.92 47.64 0.00MIT U8 4.70 4.05 0.00n/a 27.90 25.96 55.95
Major (%)Natural sciences M1 13.27 12.51 0.00Social sciences M2 30.25 32.24 0.00Engineering M3 4.65 5.15 20.24Humanities M4 12.33 11.98 9.52Other M5 3.97 4.20 17.86n/a 31.71 30.47 52.38
Earnings (USD/session) EARN 23.21 23.80 22.79Standardized per session ST EARN −0.02 0.00 −0.01
Experience (session) EXP 0.00 4.80 32.99Return (%)
In 30 days RET 30 0.47 0.57 0.56In 60 days RET 60 0.53 0.64 0.80In 90 days RET 90 0.55 0.67 0.84
control for differences in the average payment across experimentalsessions. Also, this is the only way we can control for possible differ-
16 P. Guillén, R.F. Veszteg / The Journa
ake the 80% cutoff mentioned earlier). It turns out that subjectsave a long experience record. As of the time of our analysis subjectsave participated in almost 5 incentive-based sessions on aver-ge (in more than 6 if we consider all recorded experiments). Thisumber is in line with the usual concern on the validity of exper-
mental results as explained by List and Levitt (2006) who discusshe problem of subjects’ self-selection into experiments.
As for the return decisions themselves, the variablesET 30/60/90/365 take value 1 if the subject returns to par-icipate in an experiment in the subsequent 30, 60, 90 and 365ays, respectively. Although our database is very large, it obviouslyas its (time) limits. While we consider all data recorded since the
mplementation of systematic data collection at the institution,e do not have any information about subjects’ prior experience
n experiments inside or outside the laboratory. This constituteset another unobservable personal characteristic that we can onlyontrol for with the help of an appropriate statistical technique.he laboratory is currently active and continues to host numerousxperiments, but as the new directors were unwilling to provides the more recent records, our last observation is from January006. In any case, fixing some arbitrary day as the upper limit forbservations used in the study is unavoidable. In order to eliminatehe survivorship bias from our analysis, we excluded observationsrom the very last 30, 60, 90 and 365 days from the database,epending on which return decision was being considered, givenhat we cannot know whether those observation belong to aubject who wishes to (and actually does) return to the laboratoryn the future or not.
Table 1 displays descriptive statistics on all the variablesncluded in our analysis for the whole sample and also for two sub-amples. We call “newcomers” all those participants who have noxperience with incentive-based experiments, while we refer tohose who have participated in more than 27 sessions as “experts”.xperts make up 1% of the subjects with the most experience.11
he picture that this table reflects is typical for laboratory experi-ents in economics. The vast majority of the subjects are college
tudents (74%), and almost 82% have experience in experiments byhe time of participation. The largest share of subjects comes fromhe area of social sciences, though other specialities such as human-ties or natural sciences are also well represented. Participants earnoughly $24 per experimental session on average. The distributionf payments is positively skewed as income distributions tend toe. Its variance is large, with a standard deviation of approximately9. The subject pool of the experimental laboratory that we studyeems to be well-trained, as subjects have experience from morehan six experiments on average (almost five if we consider onlyncentive-based sessions).
. Return decisions
One would expect a high correlation between the fact of being student and the decision to return to the laboratory on a volun-ary basis. Pearson’s �2 confirms such a positive relationship at anysual significance level. The values of Cramer’s V statistics is around
0% and indicates that the association is very low, however.12 Its equal to 10.63%, 10.52%, 10.02%, and 12.94% for the variablesET 30/60/90 and 365 respectively.
11 The experts are predominantly men above the usual college age, and actuallyround half of them are non-students. For these are rather unusual characteristics forn experimental subject, we have performed our analysis also without non-students.ll the reported effects remain (essentially with the same statistical significance andffect size) except that age loses its explanatory power in determining experimen-al earnings. This is not a surprising result, since age does not vary much in theemaining student population.12 With usual, we refer to significance levels between 1% and 10%.
In 365 days RET 365 0.68 0.82 1.00Number of observations N 1914 8755 84
We ran logistic regressions with subject-specific fixed effectsin order to explain the decision of returning to the experimen-tal laboratory in the next 30, 60, 90 and 365 days with personalcharacteristics.13 Due to the lack of prior hypotheses on how exactlypersonal characteristics influence return decisions, we transformedthe continuous variables into categorical ones by substituting theiroriginal values with their quartile rank.14 All our regression modelsuse the 1st quartile as a reference for all affected variables. Com-ments on the monotonicity of the estimated effects are based onstatistical tests comparing the coefficients across quartiles.
The list of regressors also includes standardized earnings (com-puted for each session separately), ST EARN, with which we try to
ences in the length of experimental sessions, as these differences
13 We estimate logit models, because we wish to study subject-related fixedeffects both in the return decisions and in individual earnings from experiments.As Wooldridge (2002) points out (in Chapter 15.8.3), the unobserved-effect logitmodel has an important advantage over the probit, because a consistent estima-tor can be obtained without any assumption about how the unobserved effects arerelated to the observed ones. We do not report marginal effects, but odds ratiosbased on coefficient estimates, as the fixed-effect specification makes the computa-tion of the former impossible. The estimated odds ratios are the standard measurefor the effect size for binary data (Ellis, 2010).
14 We are thankful to an anonymous referee for suggesting this approach.
P. Guillén, R.F. Veszteg / The Journal of Socio-Economics 41 (2012) 714– 720 717
Table 2Logit analysis of return decisions (part 1).
RET 30 RET 60 RET 90 RET 365
EARN2nd quartile 1.63*** 1.64*** 1.82*** 2.23***
3rd quartile 1.57*** 1.53*** 1.59*** 1.46*
4th quartile 1.47*** 1.40*** 1.29** 1.74***
ST EARN2nd quartile 0.83** 0.87 0.95 0.873rd quartile 0.70*** 0.80** 0.86 0.864th quartile 0.85** 0.87 1.01 0.89
AGE2nd quartile 2.99*** 3.68*** 3.48*** 2.123rd quartile 8.10*** 11.58*** 9.86*** 0.234th quartile 11.75*** 25.62*** 25.38*** 0.00
EXP2nd quartile 0.45*** 0.39*** 0.29*** 0.04***
3rd quartile 0.27*** 0.22*** 0.13*** 0.00***
4th quartile 0.13*** 0.09*** 0.04*** 0.00***
Control Subject Subject Subject SubjectPseudo-R2 0.06 0.07 0.11 0.38Obs. 6683 6104 5416 2043
Note. The reported numbers are odds ratios.* Estimates significantly different from one at 10%.
** Estimates significantly different from one at 5%.***
avdtpoown
rsrs(s
ttiriIolorttsmpdi
tOd
Estimates significantly different from one at 1%.
re unfortunately not recorded in the database. Given a nominalalue of earnings (EARN), a high value for the corresponding stan-ardized earnings (ST EARN) suggests that earnings in general (inhe experimental session) were rather low, as with that particularayoff it was possible to end up close to the higher end of the pay-ff range (as measured by the standardized earnings variable). Inther words, if they turn out to be significant in the same regression,e expect earnings to have a positive and standardized earnings aegative effect on the odds of returnings.
Table 2 shows the estimation and the related test results. Expe-ience and earnings from previous experiments along with theubject’s age have a significant effect on the return decisions. Expe-ience affects the odds of returning to the laboratory negatively (in atrictly monotonic way), while age and earnings affect it positivelyin a weakly monotonic way, because the two highest quartilestatistically tend to have the same effect).
Previous earnings unsurprisingly incentivize subjects to returno the laboratory (the odds of returning are roughly 60% higher ifhe earnings are above the first quartile).15 Relative, i.e. standard-zed, earnings seem to matter only in the short run. They have aather small, but significant, negative effect on the decision regard-ng whether to return or not in 30 days following the experiment.t seems that participating in experiments is not addictive, as thedds of repeating quickly decrease with experience. The relativelyarge positive effect of age on the odds of returning may be foundedn two pillars. On one hand, while college students, encouraged byandomly posted adds on campus, often impulsively decide to enterhe laboratory, older participants tend to look for experiments andherefore be “more frequent guests”. As List and Levitt (2006) argue,ubjects self-select into experiments, and therefore people who areore interested in the announced research topic are more likely to
articipate. On the other hand, we cannot be sure that causalityoes not run in the opposite direction, given that those who return
n the future are necessarily older. It is interesting that, as is the case
15 In the interpretation of the logit estimates, we use the fact that a small change inhe logarithm of a variable (now the odds) is approximately its percentage change.ur tables report odds ratios that should be compared to 1 for reference, as theyescribe by how much odds multiply when a given category is considered.
with standardized earnings, the effects of age tend to fade away thelonger we look into the future for return decisions.
The increasing goodness-of-fit of the model suggests that short-term decisions are more random than long-term ones. The rathersmall values also indicate that decisions are largely influenced byfactors that lie outside the scope of our analysis.
The regression models with subject-specific fixed effects havea large number of dummy variables and do not allow for studyingwhether gender, college, education level or intended major haveany significant effect on the return decisions. This is why Table 3reports estimation results for “simple” logit models that includethese as regressors to explain RET 30.16 Although our databaseis relatively large, it does not allow for the inclusion of all thesedummy variables in the same model.
Males tend to return to the laboratory more frequently thanfemales (the odds of returning are 12–14% higher for males thanfemales). The location of the experimental laboratory relative to BUand Harvard seem to have a significant effect too. Education leveldoes not seem to be an important determinant. Students of socialsciences, however, are more likely to return than others (their oddsof returning are 23% higher). In the logit estimates in Table 3 thevariables related to age and previous experience have the oppositesign when compared to the results in Table 2. While the latter canbe interpreted on an individual level (“as subjects grow older andgain more experience. . .”), the former capture population effects(“older subjects and subjects with experience. . .”). That is, oldersubjects are less likely to return than younger subjects, and thosewho have participated in more experiments are likelier to returnthan those who have participates in less.
4. Earnings
Personal earnings in experiments have a significant posi-tive effect on the decision to return to the lab. This sectionexamines whether personal characteristics have any explanatory
16 The regressors have similar power in explaining the other return decisions,therefore those estimation results are omitted.
718 P. Guillén, R.F. Veszteg / The Journal of Socio-Economics 41 (2012) 714– 720
Table 3Logit analysis of return decisions (part 2).
RET 30
EARN EARN EARN2nd quartile 1.57*** 2nd quartile 1.58*** 2nd quartile 1.57***
3rd quartile 1.60*** 3rd quartile 1.59*** 3rd quartile 1.59***
4th quartile 1.57*** 4th quartile 1.58*** 4th quartile 1.60***
ST EARN ST EARN ST EARN2nd quartile 0.87* 2nd quartile 0.88* 2nd quartile 0.85**
3rd quartile 0.75*** 3rd quartile 0.75*** 3rd quartile 0.75***
4th quartile 0.88* 4th quartile 0.89* 4th quartile 0.88*
AGE AGE AGE2nd quartile 0.90 2nd quartile 0.91 2nd quartile 0.923rd quartile 0.90 3rd quartile 0.85** 3rd quartile 0.914th quartile 0.82* 4th quartile 0.70*** 4th quartile 0.79*
EXP EXP EXP2nd quartile 1.23*** 2nd quartile 1.25*** 2nd quartile 1.25***
3rd quartile 1.42*** 3rd quartile 1.42*** 3rd quartile 1.44***
4th quartile 1.47*** 4th quartile 1.49*** 4th quartile 1.44***
MALE 1.14*** MALE 1.13** MALE 1.12**
U1 1.15 – – M1 1.12U2 0.67* E2 0.50 M2 1.23***
U3 1.05 E3 0.48 M3 1.03U4 1.08 E4 0.53 M4 1.12U5 1.24 E5 0.49 M5 1.05U6 1.34*** E6 0.53 – –U7 1.21** E7 0.64 – –U8 1.00 E8 0.53 – –
Control – – –Pseudo-R2 0.02 0.02 0.02Obs. 7700 7419 7049
Note. The reported numbers are odds ratios.* Estimates significantly different from one at 10%.
** Estimates significantly different from one at 5%.
psh
msmn
mvl
TO
*** Estimates significantly different from one at 1%.
ower in the determination of these earnings. We use regres-ion analysis to separate the effects that these variables mightave.
On top of these demographic variables we include the mini-um (EARN MIN) and the maximum (EARN MAX) payment in the
ession. These two variables control for subjects’ payment-inducedotivation. For example, when payments are too low, subjects may
ot take as much care in their decision making.
Table 4 shows the estimation results for linear regressionodels with robust variance estimates that explain the observedariation in real money earnings. We use the standard ordinaryeast squares (OLS) method to estimate the coefficients. In order
able 4LS analysis of earnings (part 1).
EARN
AGE2nd quartile 0.41* 0.92*
3rd quartile 0.09 0.09
4th quartile −0.57*** 0.91
EXP2nd quartile 0.22 −0.16
3rd quartile −0.10 −0.87***
4th quartile 0.41* −0.20
EARN MIN 0.59*** 0.59***
EARN MAX 0.29*** 0.30***
MALE 0.29* –
Const. 4.02*** 3.95***
Control – Subject
Adj-R2 0.34 0.37
Obs. 8004 8004
* Estimates significantly different from zero at 10%.** Estimates significantly different from zero at 5%.
*** Estimates significantly different from zero at 1%.
to account for unobservable subject, session or experiment-relatedeffects, we studied several specifications of our regression modelincluding fixed effects.
It seems that personal earnings are mainly determined by therules set by the experimenter, i.e. the minimum and the maximumpayment in the session, and by variables other than the demo-graphic ones included here. This is in line with the philosophyof incentives-based economic experimental studies which usually
assume that earnings do not differ across demographic groups, butdo vary with individual behavior and decisions that are beyond thescope of our analysis. This probably explains why the goodness-of-fit of these models is around 40%.0.42* 0.40*
0.21 0.17−0.42* −0.41*
0.05 0.10−0.21 −0.200.14 0.24– 0.32***
– 0.23***
0.29* 0.27*
23.75*** 10.42***
Session Experiment0.41 0.408004 8004
P. Guillén, R.F. Veszteg / The Journal of Socio-Economics 41 (2012) 714– 720 719
Table 5OLS analysis earnings (part 2).
EARN
AGE AGE AGE2nd quartile 0.45** 2nd quartile 0.46* 2nd quartile 0.53**
3rd quartile 0.31 3rd quartile 0.26 3rd quartile 0.344th quartile −0.16 4th quartile −0.34 4th quartile −0.29
EXP EXP EXP2nd quartile 0.04 2nd quartile 0.09 2nd quartile −0.063rd quartile −0.22 3rd quartile −0.20 3rd quartile −0.294th quartile 0.16 4th quartile 0.12 4th quartile 0.13
MALE 0.20 MALE 0.26 MALE 0.34**
U1 0.36 – – M1 0.46U2 0.09 E2 0.17 M2 0.46*
U3 0.54 E3 0.32 M3 0.37U4 0.33 E4 0.04 M4 −0.06U5 0.68 E5 0.14 M5 0.56U6 0.14 E6 0.44 – –U7 0.51* E7 0.21 – –U8 1.15*** E8 −0.33 – –Const. 23.34*** Const. 23.49*** Const. 23.38***
Control Session Session SessionAdj-R2 0.41 0.41 0.42Obs. 8004 7717 7319
* Estimates significantly different from zero at 10%.** Estimates significantly different from zero at 5%.
*** Estimates significantly different from zero at 1%.
tiwies(baftg
5
topf
d(riapppWa
ebr
17 Our database does include some information on the type of the experimentreported by the experimenters. We, nevertheless, have decided to exclude thesetypes from the discussion because the reports are often ambiguous, that is the cat-egories are not mutually exclusive. Even if we included them, they would not alterthe described conclusions, but would add a list of statistically significant effects forwhich we have no plausible explanation.
18 For example, Casari et al. (2007) look at common-value-auction experiments andargue that some subjects are just better, i.e. smarter and more able, and therefore
When significant, age affects earnings positively when the firstwo quartiles are compares, but when the fourth quartile is reachedt has a considerable negative effect. Experience works the other
ay around; its effects are negative (if they are significant), exceptn the fourth quartile. As for the qualitative variables, males tend toarn more than females, and MIT students tend to earn more thantudents attending other universities. The model specificationsthat include information on college, educational level or major)ehind the results in Table 5 assume session-specific fixed effect,nd the session minimum and maximum are therefore excludedrom the regressor list. Remarkably, only few of the analyzed fac-ors appear to have a significant (positive) effect on earnings: age,ender and studying at MIT.
. Conclusion
Our analysis of return decisions to the experimental labora-ory reveals that although demographic variables have little effectn individual decisions, males and (in monetary terms) well-erforming subjects are more likely to return. That is, we haveound some evidence of “lab rats” being raised.
Research on gender differences has already demonstrated men’sistinct attitudes towards risk, social preferences and competitione.g. Croson and Gneezy, 2008). Thus the fact that male subjectseturn more often to the lab may introduce some behavioral biasn experimental results. Likewise, it is possible that subjects whore more familiar with the laboratory environment and have beenerforming well in the past may behave differently from inex-erienced subjects. Unfortunately our database does not containrecise information about behavior in a particular experiments.e only observe earnings, and therefore we cannot be more
ssertive.What our analysis has done is identify a robust positive effect of
arnings on return decisions. Whether this introduces a systematicias in the experimental findings depends on how earnings are cor-elated with personal and institutional characteristics that explain
behavior and the corresponding monetary outcomes.17 In experi-ments with a more sophisticated game, like an auction or matchingmechanism, one might suspect that high earners are the smartsubjects.18 In public-good games, perhaps they are the more selfishparticipants. In market games, they may be the more competitiveor the least risk-averse ones.19 In this paper we have analyzedthe upper layer of the problem but with the available informationcannot go deeper.
In order to overcome this caveat, demographic informationshould be routinely and systematically collected in experimentallaboratories across the world. Experimentalists might also wantto balance their subject pool in terms of gender and subjects’prior experience. Even more, precise information about the typeof experiments should be stored and made widely available.20
By doing this, the effect of subject pools on experimental resultscould be studied more precisely than it is done in our exercise,and future analysis could compare different subjects pools: maleswith females, students with non-students in numerous differentexperimental settings.
earn more and are more likely to return.19 We are thankful to our anonymous referees for calling our attention to this
argument.20 At least gender, age, student status, experience and college status should be
routinely recorded. The experimental community should reach an agreement inorder to give access of the collected data, as replicas of well known experimentaldesigns can be very useful for comparative studies similar to ours.
7 l of So
R
A
A
B
C
C
C
450–461.
20 P. Guillén, R.F. Veszteg / The Journa
eferences
ndersen, S., Harrison, G.W., Lau, M.I., Ruström, E.E., 2010. Preference heterogene-ity in experiments: comparing the field and laboratory. Games and EconomicBehavior 73, 209–224.
nderson, J., Burks, S.V., Carpenter, J., Götte, L., Maurer, K., Nosenzo, D., Potter, R.,Rocha, K., Rustichini, A., 2010. Self selection does not increase other-regardingpreferences among adult laboratory subjects, but student subjects may be moreself-regarding than adults. IZA Discussion Paper No. 5389.
ardsley, N., Cubitt, R., Loomes, G., Moffatt, P., Starmer, C., Sugden, R., 2010. Experi-mental Economics, Rethinking the Rules. Princeton University Press.
asari, M., Ham, J.C., Kagel, J.H., 2007. Selection bias, demographic effects, and abilityeffects in common value auction experiments. American Economic Review 97,
1278–1304.leave, B.L., Nikiforakis, N., Slonim, R., 2010. Is There Selection Bias in LaboratoryExperiments? Economics Working Paper Series. The University of Sydney.
roson, R., Gneezy, U., 2008. Gender differences in preferences. Journal of EconomicLiterature 47, 1–27.
cio-Economics 41 (2012) 714– 720
Eckel, C.C., Grossman, P.J., 2000. Volunteers and pseudo-volunteers: the effectof recruitment method in dictator experiments. Experimental Economics 3,107–120.
Ellis, P.D., 2010. The Essential Guide to Effect Sizes: Statistical Power, Meta-analysis,and the Interpretation of Research Results. Cambridge University Press.
Harrison, G.W., Lau, M.I., Ruström, E.E., 2009. Risk attitudes, randomization to treat-ment, and self-selection into experiments. Journal of Economic Behavior andOrganization 70, 498–507.
List, J.A., Levitt, S.T., 2006. What Do Laboratory Experiments Tell Us About the RealWorld? NBER Working Paper.
Peterson, R.A., 2001. On the use of college students in social science research:Insights from a second-order meta-analysis. Journal of Consumer Research 28,
Ruström, E.E., 1998. Home-grown values and incentive compatible auction design.Game Theory 27, 427–441.
Wooldridge, J.M., 2002. Econometric Analysis of Cross Section and Panel Data. MITPress.
