The Influence of Design Dimensions on Stated Choices ... · The Influence of Design Dimensions on Stated Choices – An Example from Environmental Valuation Using a Design of Designs

The Influence of Design Dimensions on Stated Choices – An Example from Environmental Valuation Using a Design of Designs Approach

By

Jürgen Meyerhoff, Malte Oehlmann, Priska Weller

033 /2013

WORKING PAPER ON MANAGEMENT

IN ENVIRONMENTAL PLANNING

Working Paper on Management in Environmental Planning 33/2013 Arbeitspapiere zum Management in der Umweltplanung 33/2013

Authors Jürgen Meyerhoff * Technische Universität Berlin Institute for Landscape and Environmental Planning Straße des 17. Juni 145 D-10623 Berlin [email protected] Malte Oehlmann Technische Universität Berlin Institute for Landscape and Environmental Planning Straße des 17. Juni 145 D-10623 Berlin [email protected] Priska Weller Johann Heinrich von Thünen Institute Institute of Forest Economics Leuschnerstraße 91 D-21031 Hamburg [email protected] * corresponding author

ABSTRACT This paper investigates the influence of task complexity on dropout rates and model results in stated choice experiments in the context of environmental valuation. We systematically vary the number of choice sets, the number of available alternatives, the number of attributes and their levels as well as the level range presented to each respondent. Largely, we follow a design of designs approach originally introduced in the context of transportation using 16 different split samples. First, we relate choice task complexity to participants dropout behav-ior. We find that the probability to drop out of the survey is influenced by socio-demographic characteristics and increases with the number of choice sets as well as by the number of alternatives. Second, we investigate the impact of the design dimensions on stated choices by estimating a multinomial logit model and a heteroskedastic logit model. Results show that with the exception of the number of choice sets all design dimensions influence the error term variance. Finally, we compare willingness to pay measures from both models finding that the absolute willingness to pay estimates differ between both models. However, the point estimates are each time included in the confidence interval of the other model for the same attribute. Keywords: choice complexity, design of designs, stated choice experiment, error variance

- 1 -

Working Paper on Management in Environmental Planning 33/2013

1. Introduction

Discrete Choice Experiments (CE) are widely used to elicit consumer preferences in different

fields of application such as marketing, health economics, transportation and environmental

valuation. CEs usually consist of a series of hypothetical scenarios or choice sets which are

composed of two or more alternatives. The characteristics or attributes of the alternatives are

varied in a systematic way by means of their levels. For each choice set, respondents are

asked to choose their preferred alternative (Louviere et al. 2000). The number of choice sets,

alternatives, and attributes as well as the number of levels and their range can be seen as

the design dimensionality of the CE to be specified by the researcher. Across different stud-

ies, the dimensionality of the choice experiment can vary significantly. For instance, in the

majority of CE studies respondents are offered 4 to 10 choice sets and 2 to 4 alternatives.

However, there are occasions in which participants are asked to assess up to 26 or even

more choice questions (Czajkowski et al. 2012) and choose among 12 or more alternatives

(Chung et al. 2011). Similar observations can be made for the other design dimensions.

Assuming neoclassic economic theory, which suggests that individuals are omnipotent,

fully rational decision makers who have stable preferences and utility functions, the design

dimensionality should not influence choice outcomes. As a consequence, the complexity of

the choice experiment, the ability of the individual to make complex decisions and the effect

of the choice context on decision strategies are often not considered in statistical models

(Swait and Adamowicz 2001). However, many authors have highlighted that the limited abil-

ity of individuals to process complex information needs to be taken into account. For in-

stance, Heiner (1983) argued that choice complexity can influence choice consistency and

that the more complex the choice task is, the higher is the gap between individual’s cognitive

ability and cognitive demand. This result leads to a trade-off to be made by the researcher.

On the one hand, one might be tempted to increase task complexity to, for instance, increase

the number of observations or to gather as much information as possible by increasing the

number of attributes. However, this comes at the cost of higher cognitive burden and the

possibility of significantly distorted model estimates (DeShazo and Fermo 2002). As a result,

there is so far no agreement in the literature on the optimal task complexity in CEs.

Task complexity, in general, can be seen as a part of the unobserved factors influencing

choice outcomes. The suit of unobserved candidate influences can be classified as follows:

omitted variables, measurement error in the observed attributes and alternatives, true task

complexity that imposes variation in cognitive difficulty, and uncertainty attributable to many

sources such as stimulus ambiguity, beliefs about future states and peer impacts (Hensher

2006). In our study, we focus on the issue of true task complexity expressed through the var-

iation of five design dimensions. Our research is largely motivated by a series of studies in-

vestigating the influence of the number of choice sets, the number of alternatives in each

choice set, the number of attributes, the number of attribute levels, and level range on choice

experiment outcomes in the context of transportation (Hensher 2004, Caussade et al. 2005,

Hensher 2006, Rose et al. 2009). In these studies, 16 different treatments were used which

were generated by a design of designs (DoD) approach originally introduced by Hensher

(2004). The attributes used considered different travel times and cost. Our study is based on

- 2 -


a nation-wide online survey in Germany carried out in the context of land use changes in

December 2012. We incorporated aspects of land consumption, the share of forest and dif-

ferent biodiversity attributes.

Similar to Hensher (2004) and Caussade et al. (2005), we used 16 different split samples

following a design master plan. To our knowledge, this is the first study to apply a DoD ap-

proach in environmental valuation systematically varying five design dimensions. We analyze

the influence of the five design dimensions in terms of two aspects: First, we investigate the

relationship between task complexity and participants dropout behavior by using descriptive

statistics and by specifying a binary logit model. Second, we estimate a joint multinomial logit

model (MNL) for all samples and a heteroskedastic logit model (HL) with the scale parameter

specified as a function of the design dimensions. This allows us to study the impact of the

five design dimensions on the error term variance which is inversely related to the scale pa-

rameter. Willingness to pay (WTP) estimates are subsequently obtained from both models

and compared to each other. The remainder of this paper is structured as follows: Section 2

outlines previous literature and the hypotheses to be tested in our study. Section 3 presents

the modeling approach before giving details on the study design and implementation in Sec-

tion 4. The main results are presented in Section 5 (dropout analysis) and Section 6 (estima-

tion results). Finally, main conclusions and further research will be discussed.

2. Literature Review and Hypotheses to be Tested

The influence of task complexity of stated choice experiments has been investigated in sev-

eral studies. Mostly, complexity issues have been analyzed in the context of health econom-

ics (Ryan and Wordsworth 2000, Ratcliffe and Longworth 2002, Bech et al. 2011), marketing

research (Dellaert et al. 1999, Dellaert et al. 2012) and transportation (Hensher et al. 2001,

Hensher 2004, Caussade et al. 2005, Hensher 2006, Rose et al. 2009). The research has

largely focused on dimensionality influences on the error variance or scale parameter

through effects of fatigue and learning (Dellaert et al. 1999, DeShazo and Fermo 2002,

Arentze et al. 2003, Caussade et al. 2005, Rolfe and Bennett 2009, Chung et al. 2011, Czaj-

kowski et al. 2012) and on WTP estimates (Ryan and Wordsworth 2000, Hensher 2004,

Hensher 2006, Bech et al. 2011, McNair et al. 2011). Additionally, effects on attribute weights

(Arentze et al. 2003), response rate (Hensher et al. 2001, Bech et al. 2011), decision time

(Dellaert et al. 2012) or perceived choice certainty (Rose et al. 2009, Brouwer et al. 2010,

Bech et al. 2011) have been analyzed.

Subsequently, we review the literature in order to develop hypotheses of possible influ-

ences on the relationship between dropout rates and error variance. With respect to the im-

pacts on the error variance and in the light of studies published in recent years, we largely

follow the hypotheses which were put forward by Caussade et al. (2005) in order to compare

their results with our findings in the context of environmental valuation.

- 3 -


2.1. Hypothesis on the Relation between Task Complexity and Dropout

Rates

To our knowledge, this is the first study to investigate the impact of task complexity on partic-

ipants dropout behavior in discrete choice experiments. What we have found were two stud-

ies looking at the relationship between the number of choice sets and the response rate.

Hensher et al. (2001) found little differences in response rates for mail surveys with varying

numbers of treatments, while Bech et al. (2011) found no significant influence in an online

survey. However, it has to be noted that response and dropout rates need to be distin-

guished from each other. Whereas the former gives the ratio between those respondents

who completed the questionnaire and those contacted, the later shows the percentage of

those participants who prematurely abandoned the questionnaire (Vicente and Reis 2010).

Supported by studies carried out in other disciplines which concluded that the survey length

has significant influence on dropout rates (see for example Vicente and Reis 2010), we hy-

pothesize that the number of choice sets is positively related to dropout rates since the sur-

vey length increases with the number of choice questions. For the other design dimensions,

we also rely on insights from studies in other areas of research. Among others, Galesic

(2006) showed that the lower the experienced burden is, the lower is the risk of dropping out.

Based on this, we expect the dropout rate to grow with the number of alternatives and the

number of attributes. For the number of levels and their range, it may be argued that the

number of comparisons to be made increases with the number of levels and that compari-

sons might be easier to assess for attribute levels which have a narrow range (Caussade et

al. 2005). As a result, we also hypothesize a positive relationship between these design di-

mensions and dropout rates.

2.2. Hypothesis on the Relationship between Task Complexity

and Error Variance

2.2.1. Number of Choice Sets

The design dimension which has probably been analyzed most is the number of choice sets.

However, there is no consensus in the literature with respect to impacts on the error vari-

ance. On the one hand, Hensher et al. (2001) found little evidence for fatigue effects for even

32 choice sets. Brouwer et al. (2010) and Czajkowski et al. (2012) have also supported this

inference. On the other hand, Bradley and Daly (1994), who were probably the first to inves-

tigate fatigue effects in CEs, found the error variance to grow with the number of choice sets.

Caussade et al. (2005) and Chung et al. (2011) observed a U-shaped relation with the error

variance decreasing up to a threshold (9/10 sets in Caussade et al. 2005, 6 sets in Chung et

al. 2011) and increasing after this. A similar pattern was found by Bech et al. (2011). These

authors argue that the error variance initially decreases due to learning effects while subse-

quently fatigue effects cause the error variance to increase. We follow this argumentation

and expect a U-shaped pattern.

- 4 -


2.2.2. Number of Alternatives

The evidence for the impact of the number of alternatives tends to be mixed. Arentze et al.

(2003) found no effects on the error variance distinguishing between a design of 2 and 3 al-

ternatives. As opposed to this, Caussade et al. (2005) as well as Chung et al. (2011) found a

U-shaped pattern with the lowest error variance for 5 and 4 alternatives, respectively. A simi-

lar pattern emerged earlier in DeShazo and Fermo (2002), who argued that the initial de-

crease of the error variance results from a better match of preferences while the increase at

the later stage is caused by a more complex choice.

2.2.3. Number of Attributes

With respect to the number of attributes, there is clear evidence that an increase in attributes

results in an increase in the error variance. Caussade el al. (2005) found the number of at-

tributes to have a strong detrimental effect on the ability to choose contributing to higher error

variance. Similar inferences were drawn by DeShazo and Fermo (2002) and Arentze et al.

(2003). As the information load to be processed by the respondent grows with the number of

attributes, we also expect a positive relationship between the error variance and the number

of alternatives.

2.2.4. Number of Attribute Levels and Level Range

Based on the findings from Dellaert et al. (1999) and Caussade et al. (2005) and following

the same argumentation presented above regarding the influences on dropout rates, we ex-

pect a positive relationship between the number of attribute levels and the error variance and

an increase in the error variance with a wider level range.

Hypothesis 1:

There exists a U‐shaped relationship between the number of choice sets and the error variance.

Hypothesis 2:

There exists a U‐shaped relationship between the number of alternatives and the error variance.

Hypothesis 3:

There exists a positive relationship between the number of attributes and the error variance.

- 5 -


3. Econometric approach

Random utility theory assumes that the modeler does not possess complete information con-

cerning the individual decision maker (subscript n) and thus considers individual preferences

to be the sum of a systematic (Vin) and random (εin) components as

in in in inU V (x ) (1)

where Uin is the true but unobservable utility associated with alternative i out of a set of avail-

able alternatives, Vin is the measurable or deterministic part which is itself a function of the

attributes (xinß), ß is a vector of coefficients reflecting the desirability of the attributes, and εin

is a random term with zero mean. The error term εin represents attributes and characteristics

unknown to the researcher, measurement error and/or taste heterogeneity among respond-

ents. Selection of one alternative over another implies that the utility (Uin) of that alternative is

greater than the utility of the other alternative:

i i j jP(i) Pr ob(V V ) j C, j i (2)

Assuming that the error components are distributed independently and identically (IID) fol-

lowing a type 1 extreme value distribution, we get the multinomial logit (MNL) model where

the probability of choosing alternative i chosen by individual n takes the form:

exp( )

exp( )in

injn

j C

VP

V

(3)

where μ is a scale parameter which is commonly normalized to 1 in practical applications for

any one data set as it can not be identified separately from the vector of parameters. The

scale parameter is inversely proportional to the error variance 2 :

26

(4)

The assumption of a constant error variance across individuals has been questioned and

alternatively a heteroskedastic logit model (HCL; e.g., DeShazo and Fermo 2002, Hole

2006). Here, the scale parameter is no longer a constant term as it allows for unequal vari-

Hypothesis 4:

There exists a positive relationship between the number of attribute levels and the error variance.

Hypothesis 5:

There exists a positive relationship between width of the level range and the error variance.

- 6 -


ances across survey characteristics such as different treatments, e.g., the design dimen-

sions, or individual characteristics. It can be described by

exp( )

exp( )n in

inn jn

j C

VP

V

(5)

where n is a function of survey characteristics, i.e., individuals assigned to specific treat-

ments, or individual characteristics that influence the scale parameter and accordingly the

error variance. The parameterization of n can be done as exp( )nZ with Zn a vector of the

individual characteristics and as a vector of parameters indicating the influence of those

characteristics on the error variance. If turns out to be zero, then the heteroskedastic logit

collapses to a conditional logit. For estimating the heteroskedastic logit model we use the

STATA program clogithet provided by Hole (2006).

4. Study Design and Implementation

4.1. Study Design

Our study design largely follows the design master plan introduced by Hensher (2004). Tak-

ing into account that several studies published in recent years have shown that respondents

can cope with a fairly large number of choice situations (Czajkowski et al. 2012), we slightly

adapted the design master plan of Hensher (2004) by using 6, 12, 18 and 24 choice situa-

tions rather than 6, 9, 12, and 15 choice sets. As distinct from Hensher (2004), we also in-

creased the number of attributes from using 4 to 7 insteads of 3 to 6. Other dimensions were

kept equal to those of Hensher (2004). In order to obtain 16 treatments, we generated 16

different generic designs using Ngene software. Unlike Hensher (2004) and Caussade et al.

(2005), who employed a D-efficient experimental design, we made use of the C-error as a

design criterion in each treatment since it allows to minimize the variance of the WTP esti-

mates (Scarpa and Rose 2008). Our adaptation of the design master plan can be seen in

Table 1.

- 7 -


Table 1: Our adaptation of the Design master plan

Design Sets Alternatives Attributes Levels Range

1 24 4 5 3 Base

2 18 4 5 4 +20%

3 24 3 6 2 +20%

4 12 3 6 4 Base

5 6 3 4 3 +20%

6 24 3 4 4 -20%

7 6 4 7 2 -20%

8 12 5 4 4 +20%

9 24 5 4 4 Base

10 6 5 7 3 +20%

11 6 4 6 4 -20%

12 12 5 5 2 -20%

13 18 4 7 2 Base

14 18 3 4 3 -20%

15 12 3 5 2 Base

16 18 5 6 3 -20%

The attributes in our study deal with land use changes. Three different groups of attributes

are distinguished: First, the attributes “share of forest” and “land consumption” were included

in all treatments. The number of levels as well as the level range were varied due to the de-

sign master plan (see Table 1). The different level values were expressed in percentage

changes compared to the current state. Second, different biodiversity attributes were used

which are based on an indicator using stocks of bird populations, which was developed as

part of an indicator system for sustainable development in Germany (BMU 2010). As the

indicator can be split up to bird populations in different parts of the landscape, e.g. “birds in

the whole landscape” equals “birds in agricultural landscapes” plus “birds in other land-

scapes”, we could vary the number of attributes across treatments following Hensher (2004),

who used different types of travel time and cost. This allowed us to aggregate and disaggre-

gate the biodiversity attribute as a combination of already existing attributes. Doing so, one

can systematically account for the influence of the number of attributes on model outcomes

(Hensher 2004). Figure 1 illustrates the split-up of the biodiversity attribute across different

designs. The level values were expressed as indicator values. Respondents were informed

- 8 -


that an indicator of 100 or more means that the landscape type is a good habitat for a variety

of species.

Third, “contribution to a landscape fund” was utilized as the payment vehicle. This attrib-

ute was presented in all designs with its number of levels and level range being constant

over all treatments. We were well aware that the payment vehicle used in our study might

cause problems concerning the issue of incentive compatibility. However, we decided not to

use taxes due to several experiences with respect to protest respondents in Germany. Table

2 summarizes the attributes used in our study as well as their levels for a base design with 2

levels and a base level range. The attribute levels for designs with 3 and 4 levels as well as a

narrow and a wide level range are available on request.

Figure 1: Split-up of the biodiversity attribute

- 9 -


Table 2: Attributes and their levels for the base design

Attribute Description Level

Share of forest Percentage changes in the share of forest -25; +25

Land consumption Percentage changes in the land consumption -50; 50

Bio_whole Biodiversity in the whole landscape including all land-

scape types

70; 100

Bio_agrar Agricultural landscape biodiversity 65; 100

Bio_forest Forest landscape biodiversity 80; 100

Bio_urban Urban area biodiversity 60; 100

Bio_other1 Biodiversity in other landscape types: Forests, urban

areas, mountains, waters

75; 100

Bio_other2 Biodiversity in other landscape types: Urban areas,

mountains, waters

60; 100

Bio_other3 Biodiversity in other landscape types: Mountains, wa-

ters

75; 100

Cost Contribution to a landscape fund in € per year 10; 25; 50;

80; 110; 150

As one can see from Table 1, the number of alternatives varies from 3 to 5 including the sta-

tus quo alternative, which was defined as the current situation with the cost attribute being 0.

As we asked respondents to assess land use changes for around 15 km of her or his resi-

dence without knowing the shape of the landscape for each respondent, the attribute levels

for the status quo alternative were chosen to be “like today”. This is another aspect which

distinguishes our study from Caussade et al. 2005. Further differences are summarized in

Table 3.

- 10 -


Table 3: Differences across complexity studies

Characteristic Hensher (2004), Caussade et al. (2005) Present Study

Application Transportation Environmental valuation

Number of choice sets 6, 9, 12, 15 6, 12, 18, 24

Number of attributes 3 – 6 4 – 7

Survey mode Computer aided personal interview (CAPI)

Online survey

Experimental design D-efficient C-efficient

Status quo Current route Current situation

Region Sydney / Santiago German-wide

Aim of assessment Route changes Land use changes around 15 km of resi-dence

Variation of attribute numbers

Based on different types travel time and cost

Based on Biodiversity in different landscapes types

Payment vehicle Travel cost Contribution to a land-scape fund

4.2. Survey Implementation

A nation-wide online survey was conducted between December 7th and December 21st

2012 using an online panel from a survey company. When participants entered the survey,

they were randomly assigned to 1 of the 16 designs. The questionnaire started by asking

respondents socio-demographic questions concerning date of birth, gender and education.

We did this at that stage of the questionnaire in order to be able to control for socio-

demographic characteristics within the dropout analysis. Then, participants were asked sev-

eral warm-up questions in order for respondents to get introduced to the topic including the

assessment of the current state of the landscape characteristics around 15 km of the re-

spondent’s residence. Before presenting the first choice set, participants were given an in-

struction page with information on the choice experiment as well as the attribute descriptions,

which varied across treatments with 4, 5, 6 and 7 attributes. The choice sets were subse-

quently presented in a randomized order with the number of choice sets depending on the

design respondents were assigned to. For each choice set, respondents were asked to as-

sess land use changes for an area around 15 km of their residence by choosing their pre-

ferred alternative. After finishing the CE, several follow-up questions were asked including

perceived choice certainty and the attendance to attributes. At each stage of the question-

naire, respondents could only abandon the survey by closing their internet browser. If so, the

exact position of dropout was recorded. In total, 2133 interviews were collected with 1673

(78.43%) respondents having completed the whole questionnaire and 460 partial completed

- 11 -


surveys (dropouts). The average interview length was measured to be 23 minutes and the

response rate was 29.49%.

5. Dropout Analysis

5.1. The Influence of the Design Dimensionality on Dropout Rates

As a first empirical illustration, Table 4 depicts the dropout rates while progressing through

the questionnaire. Around 8.20% of the participants dropped out before starting to answer

the CE. The highest dropout rate was observed to be within the choice experiment (11.02%),

while only 2.35% quit the survey after finishing the CE. The relatively high dropout rate in the

first part of the questionnaire has also been observed in studies carried out in other areas of

research. For instance, Hoerger (2010) found 10% of the participants to drop out of the sur-

vey instantaneously. This behavior may be explained by a low interest in the topic. Among

others, Galesic (2006) found that the lower respondent’s overall interest, the higher the drop-

out rate.

Table 4: Position of dropout

Dropout position Frequency %

Before CE 175 8.20

Within CE 235 11.02

After CE 50 2.34

Completed 1673 78.43

Total 2133 100.00

Next, Table 5 presents the 16 treatments with their corresponding design dimensions as

well as the number of interviews and the number of dropouts per design. We excluded those

respondents who quit the survey before starting to answer the choice experiment since all

questions were identical across designs up to this stage. The split samples which we ob-

served to have the highest dropout rates are designs 1, 3 and 9 each having 24 choice sets

with all other design dimensions varying across designs (3 to 5 alternatives, 4 to 6 attributes,

etc). With respect to the split samples with the lowest dropout rates, we found design 5, 11,

15 all to have 6 choice sets. Again, all other dimensions varied between, for instance, 4 to 6

attributes and 2 to 4 attribute levels. Based on this, there is already some indication that

dropout rates might especially be influenced by the number of choice sets.

- 12 -


Table 5: Design-dependent Dropouts while answering choice sets

Design Sets Alternatives Attributes Levels Range Interviews Completed

Droppers In %

1 24 4 5 3 Base 83 21 20.19

2 18 4 5 4 +20% 81 10 10.99

3 24 3 6 2 +20% 123 33 21.15

4 12 3 6 4 Base 80 12 13.04

5 6 3 4 3 +20% 102 8 7.27

6 24 3 4 4 -20% 81 16 16.49

7 6 4 7 2 -20% 219 29 11.69

8 12 5 4 4 +20% 80 13 13.98

9 24 5 4 4 Base 81 21 20.59

10 6 5 7 3 +20% 150 29 16.20

11 6 4 6 4 -20% 89 10 10.10

12 12 5 5 2 -20% 79 12 13.19

13 18 4 7 2 Base 112 24 17.65

14 18 3 4 3 -20% 84 18 17.65

15 12 3 5 2 Base 146 9 5.81

16 18 5 6 3 -20% 83 20 19.42

Total 1673 285 14.56

In order to analyse the relationship between the five design dimensions and dropout rates

in detail, we specified a binary logit model with the dependent variable being 0 if a participant

completed the survey and 1 if the respondent dropped out after starting to answer the choice

experiment. The results are presented in Table 6. As expected, the number of choice sets

has a highly significant positive impact on the probability to drop out. The same pattern was

observed for the number of alternatives and the number of attributes, although for the later

the coefficient became only significant at the 10% level. For the attribute levels and the level

range we found no significant influence. So we could not reject our null hypothesis of no ef-

fect. Since we also expected socio-demographic characteristics to be possible sources of

explanation of the probability to drop out, we included age, gender and education as further

independent variables. As shown in Table 6, age has a highly significant effect on the proba-

bility to drop out. The same applies to the dummy variable for being female.

- 13 -


Table 6: Model estimates for the influence between design dimensions and dropout

rates

Dimension Coefficient Standard Error Z-score

Age 0.019 0.005 3.71

Dummy gender female 0.617 0.135 4.56

Dummy degree after 10 years of school-ing

0.011 0.256 0.04

Dummy degree high-school degree (13 years of schooling)

-0.413 0.279 -1.48

Dummy university degree -0.145 0.256 -0.57

Dummy no degree 0.728 1.253 0.58

Number of choice sets 0.055 0.011 4.92

Number of alternatives 0.191 0.089 2.15

Number of attributes 0.141 0.080 1.77

Dummy narrow range 0.190 0.171 1.11

Dummy wide range 0.180 0.170 1.06

Number of levels 0.012 0.103 0.12

Constant -5.133 0.818 -6.28

Log-likelihood null -810.178

Log-likelihood model -772.701

Pseudo R2 0.046

Based on 1,955 observations.

5.2. Dropouts within the Choice Experiment

To finish our analysis on dropout rates, Table 7 shows the position of dropout within the

choice experiment. We observe the highest number of participants (126) to drop out within

the first six choice sets. This figure corresponds to 53.6% of the dropouts within the choice

experiment and a dropout rate of 6.54%. At the other extreme, only 14 respondents or 6% of

the dropouts abandoned the survey between choice set 19 and set 24. The total dropout rate

at this stage of the CE is therefore only 3.24%.

- 14 -


Table 7: Dropout rates within the choice experiment

Position dropout

Number of presented choice sets Total Number of respondents

Dropout rate

6 Sets 12 Sets 18 Sets 24 Sets

Set 1 to 6 59 25.1

20 8.5

21 8.9

26 11.1

126 53.6

1952 6.54%

Set 7 to 12 0 16 6.8

20 8.5

16 6.8

52 22.1

1272 4.09%

Set 13 to 18 0 0 18 7.66

25 10.6

43 18.3

835 5.15%

Set 19 to 24 0 0 0 14 5.96

14 6.0

432 3.24%

59 25.1

36 15.32

59 25.1

81 34.5

235 100.0

6. Estimation Results

We started our modelling by estimating a MNL model in which the data from all split samples

were pooled. Furthermore, we specified a HL model with the scale parameter as a function of

the design dimension. The results of the MNL and HL model are shown in Table 8. In both

models, all parameters are significant at a 1% level of significance and have the expected

sign. On average, respondents want a larger share of forests while at the same time prefer-

ring reduced land consumption in their surroundings. The biodiversity attributes all have posi-

tive signs indicating that respondents want to increase the levels of biodiversity as measured

on the underlying scale. The model also contains an alternative specific constant (ASCsq) for

the current situation. It is positive and statistically significant suggesting that on average re-

spondents have a propensity to choose the current situation instead of one of the hypothet-

ical alternatives describing future land use changes. Table 8 also contains the log-likelihoods

for the MNL and HL. To compare both models to each other, we conducted a log-likelihood

ratio test with the test statistic being 108.948. Since the critical Chi-squared value is 21.666

(1% level with 9 degrees of freedom), we can reject the null hypothesis that the HL model is

no better than the MNL model.

- 15 -


Table 8: Estimation results for the MNL and the HL model

Coefficients MNL model t-ratio HL model t-ratio

ASC_sq 1.374 17.18 2.514 5.51 Share of forest 0.017 35.59 0.030 5.59 Land consumption -0.009 38.45 -0.016 5.62 Bio_whole 0.011 12.85 0.022 5.37 Bio_agrar 0.005 8.40 0.009 4.78 Bio_forest 0.005 6.70 0.008 4.37 Bio_urban 0.003 3.42 0.004 2.09 Bio_other1 0.008 10.67 0.015 5.03 Bio_other2 0.003 4.76 0.006 3.62 Bio_other3 0.003 3.51 0.008 3.25 Cost -0.006 26.54 -0.007 6.13 Log-likelihood null -31278.157 -31278.157 Log-likelihood model -27616.061 -27507.133 Pseudo R2 0.117 0.121

Based on 90,831 observations.

Table 9 reports the estimates of the design dimension variables. For the number of choice

sets, we attempted to show a U-shaped relationship by estimating a linear and a quadratic

effect. However, neither the coefficient for the number of choice sets, nor the coefficient for

the squared number of choice sets became statistically significant. As a result, we cannot

reject the null hypothesis of no relationship between the number of choice sets and the error

variance. This finding contradicts the results from Caussade et al. (2005), but is in agreement

with other studies such as Czajkowski et al. (2012) or Hess et al. (2012). With respect to the

number of alternatives, we can reject the null hypotheses by observing a U-shaped pattern

with dummy variables specified for 4 and 5 alternatives. With respect to a base design with 3

alternatives, designs with 5 and 5 alternatives have a higher scale parameter (lower error

variance). However, the drop in the error variance is less pronounced for designs with 5 al-

ternatives. This suggests exactly the same pattern as observed by Caussade et al. (2005).

The same is true for the number of attributes, which we found to have a highly significant

negative impact on the scale parameter (positive on the error variance) allowing us to reject

our null hypothesis. For the number of levels, we could not find any statistically significant

linear effect. Nevertheless, when we specified dummy variables for 3 and 4 levels, coeffi-

cients became significant suggesting a non-linear relationship between the number of attrib-

ute levels and the error variance with respect to a design with 2 levels. Positive impacts on

the error variance were higher for 3 levels than for designs containing 4 levels with the coef-

ficient only significant at a 10% level. This contradicts the observation of a linear relationship

found by Caussade et al. (2005), but allows us to reject our null hypothesis. Similar inference

was drawn for the level range as we observe the error variance to decrease for designs with

a narrow range and to increase for designs with a wide range compared to the base. This

result is in line with Caussade et al. (2005).

- 16 -


Table 9: Coefficient estimates for the parameterization

Dimension Coefficient t-value

Number of choice situations -0.007 0.52 Squared number of choice situations 0.001 0.59 Dummy 4 alternatives 0.154 3.88 Dummy 5 alternatives 0.072 2.07 Number of attributes -0.956 4.59 Dummy 3 levels -0.314 7.35 Dummy 4 levels -0.079 1.72 Dummy narrow range 0.202 5.85 Dummy wide range -0.128 3.68

Based on 90,831 observations

Turning now to the willingness to pay (WTP) estimates, we calculate the marginal WTP as

the ratio between the attribute coefficients and the cost at constant utility levels. Table 10

reports the WTP values for the MNL as well as the HL model; the confidence intervals were

calculated using the Delta method. Note that the marginal WTP estimates refer to environ-

mental changes in the 15 km surrounding of respondent’s place of residence. Starting with

the WTP estimates based on the MNL model, respondents are, on average, willing to pay

2.73 € per year for a one percent increase in the share of forests, but would experience a

disutility of 1.39 € per year for a one percent growth in land consumption. The biodiversity

attributes, which were aggregated and disaggregated across designs, show values as ex-

pected. Respondents are willing to pay more for the aggregated attribute “whole landscape

biodiversity” (Bio_whole), which is 1.84 € per year for an improvement by one point of the

indicator (see Section 4.1), than for attributes at a lower aggregation level such a forest land-

scape biodiversity (0.78 € per year for one point improvement) or agricultural landscape bio-

diversity (0.77 € per year for an improvement by one point). Moreover, the WTP for

bio_other1, which includes biodiversity in forests, urban areas, mountains and waters, is

higher than bio_other2 considering biodiversity in mountains, urban areas and waters.

The absolute WTP estimates differ between the MNL and HL model but the point esti-

mates are each time included in the interval of the other model for the same attribute. Thus,

taking into account the impact of the design dimension on the error variance does not result

in statistically significant different WTP estimates.

- 17 -


Table 10: Marginal willingness to pay estimates for attributes in Euro per year

Attribute MNL model 95% - CI HL model 95% - CI

Share of forest 2.73 2.44 / 3.01 2.96 2.65 / 3.27 Land consumption -1.39 -1.51 / -1.28 -1.50 -1.63 / -1.37 Bio_whole 1.84 1.51 / 2.18 2.09 1.71 / 2.47 Bio_agrar 0.77 0.57 / 0.96 0.78 0.55 / 1.00 Bio_forest 0.78 0.54 / 1.03 0.79 0.50 / 1.08 Bio_urban 0.44 0.19 / 0.70 0.38 0.08 / 0.67 Bio_other1 1.29 1.02 / 1.56 1.34 1.04 / 1.64 Bio_other2 0.52 0.30 / 0.74 0.63 0.36 / 0.90 Bio_other3 0.49 0.21 / 0.77 0.63 0.30 0.96

7. Discussion and Conclusion

In this study, we analyzed the impact of the number of choice sets, the number of alternative

in each choice set, number of attributes as well as the number of levels and their range on

the error variance and on dropout rates. To our knowledge, it is not only the first study em-

ploying the design of design approach introduced by Hensher (2004) in environmental valua-

tion but also, this is the first attempt to investigate the relationship between task complexity in

discrete choice experiments and participant’s dropouts.

With respect to the dropout rates, we found that the probability to abandon the survey sig-

nificantly increases with the number of choice sets and the number of alternatives presented

on a choice set. All other design dimensions did not significantly influence the probability to

quit the survey. Additionally, older respondents as well as women are more likely to drop out.

Among the dropouts that happened while answering the choice tasks, it is noteworthy that

the majority of the respondents abandoned the choice experiment within the first 6 choice

questions. One reason for this could be that people did not like the choice format and thus

decided not to proceed. During focus groups, we have repeatedly discovered that some re-

spondents do not like to make comparisons among bundles of attributes, but would prefer to

rate each attribute separately. Another reason might be that people realized that choosing

among the alternatives on a choice sets also includes payments to the landscape fund and

that quitting the survey at this stage was motivated by protest votes. However, we do not

conclusively know the reasons why respondents abandon the questionnaire. We also re-

frained from sending those people another set of questions in order to find out why they act-

ed as they did. The survey company expected very low response rates for such a kind of

debriefing interview. Although this study is not sufficient to draw general conclusions, the

results suggests that if dropout rates in an online survey are of crucial importance, less

choice sets and fewer alternatives should be presented. These findings might depend very

much on the survey mode. In a CAPI interview, as used by Czajkowski et al. (2012), the re-

spondents might be more motivated or might feel more obliged to go through a series of 26

- 18 -


choice tasks as presented in their survey. Due to missing information on the dropout rates in

other studies, we are not able to compare our results to other survey formats.

Turning to the results with respect to influences of the design dimensions on the error var-

iance, we mainly find the same results as presented by Caussade et al. (2005). The main

difference is that we cannot find any impact of the number of choice tasks on the error vari-

ance. In all model specifications that we tried, there was no significant association. There-

fore, we cannot reject the null hypothesis that the error variance is not associated with the

number of choice sets. This is in line with the findings of Czajkowski et al. (2012) and Hess et

al. (2012), who also could not find any systematic influence of the number of choice tasks on

the scale parameter. For the other dimensions, we can each time reject our null hypothesis

of no relationship between the design dimensionality and the error variance.

However, there are some reasons to interpret these findings with a degree of caution.

First, our payment vehicle, a contribution to a fund, might not be as incentive compatible as

the costs respondents faced in the survey done by Caussade et al. (2005). Second, we did

not adress taste heterogeneity. Taste heterogeneity is likely to be present among the re-

spondents of our study when it comes to land use options. As the survey was conducted

nation-wide and respondents live in very different landscapes, it is likely that participants pre-

fer different changes. Similarly, the present situation in the 15 km surrounding of residence of

the respondents is very likely to be different and thus might strongly affect the propensity to

choose the status quo option. Within the survey, we asked participants to report on the cur-

rent situation with respect to the choice attributes but have not included these responses into

the models.

Finally, we estimated the willingness to pay for all attributes considered in our study using

the MNL as well as the HL model. We found that the amount participants are willing to pay is

always higher for the biodiversity aggregated attribute than for biodiversity attributes at a

lower level of aggregation. The willingness to pay estimates from both models are not signifi-

cantly different from each other since the point estimate of one model is included in the con-

fidence interval of the other.

Further research could start by applying a more flexible model in terms of taste heteroge-

neity. Moreover, we will include an analysis to which extent the design dimension influence

the number of times the status quo alternative is chosen. The results presented so far indi-

cate that the status quo option is more likely to be chosen when the choice design is more

complex (Boxall et al. 2009, Zhang and Adamowicz 2011). Additionally, effect of the choice

task on choices and WTP estimates will be analysed using an entropy based measure of

complexity This could provide further insights on whether the design dimensions have a

stronger impact on the results of stated choice experiments or whether researchers do not

have to care too much about this issue, at least within the ranges investigated here.

- 19 -


References

Arentze, T., A., Borgers, H., Timmermans and R., DelMistro, 2003. Transport Stated Choice Responses: Effects of Task Complexity, Presentation Format and Literacy. Transpor-tation Research Part E, 39, 229–244.

Bech, M., T., Kjaer and J. Lauridsen, 2011. Does the number of choice sets matter? Re-sults from a web survey applying a discrete choice experiment. Health Economics, 28(3), 273–286.

Boxall, P., W.L., Adamowicz A., Moon, 2009. Complexity in choice experiments: choice of the status quo alternative and implications for welfare measurement. Aust J Agr Resour Eco-nomics 53, 503-519.

Bradley, M. and A., Daly, 1994. Use of the logit scaling approach to test for rank-order and fatigue effects in stated preference data. Transportation 21, 167–184.

Brouwer, R., T. Dekker, J. Rolfe and J., Windle, 2010. Choice Certainty and Consistency in Repeated Choice Experiments. Journal of Environmental and Resource Economics, 46(1), 93–109.

Bundesministerium für Umwelt, Naturschutz und Reaktorsicherheit (BMU), 2010. Der In-dikatorenbericht 2010 zur Nationalen Strategie zur biologischen Vielfalt. Berlin.

Caussade, S., J., Ortúzar, L.I., Rizzi and D.A., Hensher, 2005. Assessing the influence of design dimensions on stated choice experiment estimates. Transportation Research Part B, 39, 621–640.

Czajkowski, M., M., Giergiczny and W.H., Greene, 2012. Learning and Fatigue Effects Revisited. The Impact of Accounting for Unobservable Preference and Scale Heterogeneity on Perceived Ordering Effects in Multiple Choice Task Discrete Choice Experiments. Work-ing Paper 8, Faculty of Economics, University of Warsaw.

Chung, C., T., Boyer and S. Han, 2011. How many choice sets and alternatives are opti-mal? Consistency in choice experiments. Agribusiness, 27(1), 114-125.

Dellaert, B., J., Brazell and J.J., Louviere, 1999. The effect of attribute variation on con-sumer choice consistency. Marketing Letters, 10, 139-147.

Dellaert, B.G.C., B., Donkers and A., van Soest, 2012. Complexity Effects in Choice Ex-periment–Based Models. Journal of Marketing Research 49(3), 424-434.

DeShazo, J.R. and G., Fermo, 2002. Designing choice sets for stated preference meth-ods: the effects of complexity on choice consistency. Journal of Environmental Economics and Management 44, 123-143.

Galesic, M., 2006. Dropouts on the Web: Effects of Interest and Burden Experienced Dur-ing an Online Survey. Journal of Official Statistics, 22(2) 313–328.

Heiner, R., 1983. The origin of predictable behaviour. The American Economic Review, 73, 560–595.

Hensher, D.A., P.A., Stopher and J.J., Louviere, 2001. An exploratory analysis of the ef-fect of numbers of choice sets in designed choice experiments: an airline choice application. Journal of Air Transport Management, 7(6), 373–379.

Hensher, D.A., 2004. Identifying the Influence of Stated Choice Design Dimensionlity on Willingness to Pay for Travel Time Savings. Journal of Transport Economics and Policy 38(3), 425–446.

- 20 -


Hensher, D.A., 2006. Revealing Differences in Willingness to Pay due to the Dimension-ality of Stated Choice Designs: An Initial Assessment. Journal of Environmental and Re-source Economics 34(1), 7-44.

Hess, S., D.A., Hensher and A., Daly, 2012. Not bored yet – Revisiting respondent fatigue in stated choice experiments. Transportation Research Part A, 626–644.

Hoerger, M., 2010. Participant dropout as a function of survey length in internet-mediated university studies: implications for study design and voluntary participation in psychological research. Cyberpsychology, Behavior and Social Networking, 13(6), 697-700.

Hole, A.R., 2006. Small-sample properties of tests for heteroscedasticity in the conditional logit model. Economics Bulletin 3, 1-14.

Louviere, J.J., D.A., Hensher and J.D., Swait, 2000. Stated Choice Methods. Analysis and Application. Cambgridge.

McNair, B.J., J. Bennett and D.A., Henser, 2011) A comparison of responses to single and repeated discrete choice questions. Resource and Energy Economics 33(3), 554-571.

Ratcliffe, J. L., Longworth, 2002. Investigating the structural reliability of a discrete choice experiment within health technology assessment. International Journal of Technology As-sessment in Health Care, 18(1), 139–144.

Rolfe, J., and J. Bennett, 2009. The impact of offering two versus three alternatives in choice modelling experiments. Ecological Economics 68, 1140-1148.

Rose, J.M., D.A., Hensher, S., Caussade, J., Ortúzar and J.-C., Jou, 2009. Identifying dif-ferences in willingness to pay due to dimensionality in stated choice experiments: a cross country analysis. Journal of Transport Geography 17, 21-29.

Ryan, M. and S., Wordsworth, 2000. Sensitivity of Willingness to Pay Estimates to the Level of Attributes in Discrete Choice Experiments. Scottish Journal of Political Economy 47(5), 504–524.

Scarpa, R. and J.M., Rose, 2008. Design efficiency for non-market valuation with choice modelling: how to measure it, what to report and why. The Australian Journal of Agricultural and Resource Economics 52, 253-282.

Swait, J.D. and W.L Adamowicz, 2001. Choice environment, market complexity and con-sumer behaviour: a theoretical and empirical approach for incorporating decision complexity into models of consumer choice. Organizational Behaviour and Human Decision Processes, 86, 141–167.

Vicente, P. and E. Reis, 2010. Using Questionnaire Design to Fight Nonresponse Bias in Web Surveys. Social Science Computer Review, 28(2), 251-267.

Zhang, J. and W.L., Adamowicz, 2011. Unraveling the Choice Format Effect: A Context-Dependent Random Utility Model, Land Economics 87(4), 730-743.

Documents

The Influence of Design Dimensions on Stated Choices ... · The Influence of Design Dimensions on Stated Choices – An Example from Environmental Valuation Using a Design of Designs