Upload
rachel-hart
View
215
Download
0
Embed Size (px)
Citation preview
Measuring Subjective Expectations in Surveys
Adeline Delavande
Institute for Social and Economic Research,University of Essex
2012 ESRC Research Methods FestivalSurvey Data Quality
Overview
1. Motivation
2. Measurement: how to elicit subjective expectations?
1. In developed countries
2. In developing countries
3. Data quality: Validation of answers1. Basic patterns of answers
2. Violation of properties of probabilities
3. Correlation of expectations with characteristics
4. Design issues
Motivation Social scientists and policy makers want to predict
choice behavior
Most decisions are made under uncertainty Deciding to retire Choosing a medical treatment
Typical assumption: individuals maximize subjective expected utility Basic identification problem with choice data alone: many
combinations of expectations and preferences generate same choices
Motivation
One possibility to mitigate the identification problem Elicit directly subjective expectations from survey
respondents Combine data on observed choice and subjective
expectations to make inference on preferences
Motivation
Other reasons to elicit subjective expectations from survey respondents
Find out whether there is systematic misperception in the population Do women at risk of unintended pregnancy have misperception
about effectiveness of birth control methods?
Ask expected choice behavior
Replace data on realizations Subjective expectations about survival instead of long panel data
on mortality to estimate differential mortality (Delavande and Rohwedder, forthcoming)
2. Measurement: how to elicit subjective expectations?
Eliciting subjective expectations
Attitudinal research and verbal expectations data
General Social Survey: Thinking about the next twelve months, how likely do you think it is that you will lose your job or be laid off—very likely, fairly likely, not too likely, or not at all likely?
Survey of Consumers: Now turning to business conditions in the country as a whole—do you think that during the next 12 months we’ll have good times financially, or bad times, or what?
Eliciting subjective expectations
Attitudinal research and verbal expectations data: various problems
Interpretation of answers Interpersonal comparability: Are the response
comparable across people? Very likely may mean different thing to different people
Intrapersonal comparability : Are the response comparable across event for the same individual?
Coarseness of response options Researchers gain only limited information
Eliciting subjective expectations
Eliciting probabilistic expectations Advantages:
well-defined numerical scale (comparability of answers)
Assessment of internal consistency of answers is possible Do answers respect basic properties of probabilities?
Answers can be compared with known events frequencies
Eliciting subjective probabilities in developed countries
Binary eventsTypical wording: percent chance format
Survey of Economic Expectations: What do you think is the percent chance that you will lose your job during the next 12 months?
Health and Retirement Study: What do you think is the percent chance that you will live to be 75?
My own survey: What do you think is the percentage chance that you would get pregnant during the next 12 months if you were using the birth control pill during that period?
Eliciting subjective probabilities in developed countries
Distribution of beliefs about a continuous variableTwo methods used so far:
Percent chance: elicit several points in the cumulative distribution
On a scale from 0 to 100, where 0 means no chance and 100 means you are absolutely certain, what do you think is the percent chance that your Social Security benefits will be more than $1,000 per month?
Visual format in Internet surveys (Delavande and Rohwedder, 2008)
Eliciting distribution of beliefs about a continuous variable
Eliciting subjective probabilities in developing countries
Binary events
Percent chance format (like in developed countries in a context of high literacy)“Frequentist” (consider 20 people like you)Visual format with physical objects (beans/stones)
Eliciting subjective probabilities in developing countries
Visual format (Delavande and Kohler, 2009)
“I will ask you several questions about the chance or likelihood that certain events are going to happen. There are 10 beans in the cup. I would like you to
choose some beans out of these 10 beans and put them in the plate to express what you think the
likelihood or chance is of a specific event happening.
One bean represents one chance out of 10.”
Eliciting subjective probabilities in developing countries
Distribution of beliefs about a continuous variable
physical objects used to allocate probability mass into intervals (e.g., Delavande, Gine and McKenzie, 2011)
3. Data quality: validations of expectations
Data quality: validations of expectations
Do people report their actual expectations?
Basic patterns of answers
Item non-response Tend to be low: People willing and able to answer
Respondents use the whole scale from 0 to 100 Some rounding to nearest 5 (see Manski and Molinari
2010)
Heterogeneity in answers
Often: focal answers at 0, 50, 100
Basic patterns of answers
0.0
5.1
.15
.2.2
5D
ens
ity
0 20 40 60 80 100r7liv75:w7 r probability to live 75+
Probability of being alive at age 75 HRS 2006
Basic patterns of answers
Heaping at 0, 50 and 100 may proxy for respondents’ burden
50-50: may reflect uncertainty rather than a true “1/2” probability In HRS 2006, respondents who answer 50% to the subjective
probability of survival are given a follow-up question asking whether they just do not know their survival probabilities or whether their belief is really that the chances are about 50 percent. It turns out that the fraction of 50s being simply uncertain is high – a little over 60 percent.
Rate of 50s tend to decrease with education and numeracy (Bruine de Bruin et al., 2000)
Rate of 50s varies with types of events
Violation of properties of probabilities
Do respondents understand concept of probabilities?
Delavande and Kohler (2008) ask about the probability of nested events in training questions in Malawi
Likelihood of going to the market within 2 days 2 weeks
Violation of properties of probabilities
99% of respondents provide answers coherent with probability theory:
P(market within 2 days)<=P(market within 2 weeks)
Elicited probabilities of going to the market
0
5
10
15
20
25
0 1 2 3 4 5 6 7 8 9 10
# beans
% r
es
po
nd
en
ts
Two days
Two weeks
Violation of properties of probabilities
Delavande, Gine and McKenzie (2011) ask a series of checks questions in a survey in India
Imagine I have 5 fishes, one of which is red and four of which are blue. If you pick one of these fishes without looking, how likely it is that you will pick the red fish?
Nested events Probability zero event: How likely do you think it is that you will not
catch any fish in the month of August if you go fishing 6 days a week?
Probability one event: How likely it is that you will eat fish at least once during the month of August?
Expectations and characteristics
Expectations vary with characteristics in the same way as actual outcomes
Likelihood of having to rely on family members in Malawi – by SES
Source: Delavande and Kohler (2009)
Design issues
Which design for better quality?
Eliciting distribution of beliefs about a continuous variable
Comparing percent chance versus visual format (Delavande and Rohwedder, 2008)
Randomized design to ask about Social Security benefits expectations HRS Internet survey in 2007
Half sample was randomized into percent chance format (4 thresholds)
Half sample was randomized into visual format
Eliciting distribution of beliefs about a continuous variable
Eliciting distribution of beliefs about a continuous variable
How do the 2 methods compare? Non-response rate: less than 2% for both formats Similar survey time Respondents take only limited advantage of the
additional precision of their format Bins and balls: only 1 percent chose to use more than five
bins Percent chance: less than 3.5 percent of the answers per
bin are not multiple of 5
Eliciting distribution of beliefs about a continuous variable
Unusable answers Percent chance format: respondents do not
respect the monotonicity across thresholds 20% of respondents who answer the 4 thresholds Those are less educated, less wealthy and in poorer
health Note: we did not inform respondents about the
violation (different from Dominitz and Manski, 2006) Visual format: respondents do not allocate all the
balls 3% of respondents
Eliciting distribution of beliefs about a continuous variable
Central tendency and spread of the distribution The visual and percent chance format generate similar
central tendency. Respondents tend to allocate most probability mass in the
middle bin
Yet, the percent chance format generates more spread-out distributions
Average probability mass:
First bin Second bin Middle bin Fourth bin Fifth bin
Bins and balls 2.74 16.98 59.93 16.96 3.38
Percent chance 9.57 16.13 43.02 18.17 13.12
Eliciting distribution of beliefs about a continuous variable
Does the visual format create anchoring towards the middle of the visual aid?
Experiment in the RAND American Life Panel Randomly allocate position of the bin containing
the point estimate (middle bin or third bin) Probability mass allocated in the bin containing
the point estimate is not sensitive to the position of that bin
Eliciting distribution of beliefs about a continuous variable
How to evaluate which format yields higher quality of answers?
1. Check how uncertainty in Social Security benefits correlates with uncertainty about related outcomes (subjective probability of reform, distance to claiming age, etc…) Elicited uncertainty correlates with other sources of
uncertainty in the expected direction for both formats Both yield high-quality data from a substantive point of view
Eliciting distribution of beliefs about a continuous variable
How to evaluate which format yield higher quality of answers?
2. Check whether uncertainty about future SS benefits predicts actual behavior related to retirement planning
Portfolio choice (Delavande and Rohwedder 2011) Find that higher levels of uncertainty about future benefits
hold a smaller share of their wealth in stocks in visual format only
Could also be due to smaller sample size or functional form assumptions that suit better visual format
Eliciting distribution of beliefs in developing countries Delavande, Gine and McKenzie (2011)
Experiment in India to test the sensitivity of elicited expectations to variations in three facets of the elicitation methodology: Number of beans (10 versus 20)
20 beans: more precise answers but may increase respondent’s burden
Design of the support (pre-determined with 20 intervals or self-anchored with 4 intervals based on elicited min/max) Self-anchor: more relevant to respondent but may increase
interviewer’s burden
Ordering of questions
Eliciting distribution of beliefs in developing countries
Subjective distribution elicited is robust to modifications in the elicitation design
When using a pre-determined support with many intervals, individuals with 20 beans use more intervals and have a distribution which is less uniform 20 beans does not seem to unduly increase respondent burden
and allows more precision
Compare individual-level realized distribution of daily catches during August to elicited distribution Greatest accuracy occurs when using 20 beans and the pre-determined support,
which results in a 38 percent improvement in accuracy relative to using 10 beans and a self-anchored support
Conclusions
Various ways can be used to assess expectations data quality Introduce training questions to check whether
respondents understand the concept of probabilities Correlate with characteristics Evaluate whether expectations predict actual
behavior
We may be able to improve data quality with more innovative elicitation formats
References Bruine de Bruin, W¨andi, Baruch Fischhoff, Susan G. Millstein, and Bonnie L. Halpern-Felsher (2000). “Verbal and
Numerical Expressions of Probability: ‘It’s a Fifty-Fifty Chance’.” Organizational Behavior and Human Decision Processes 81:115–31.
Delavande, Adeline, Xavier Giné and David McKenzie (2011) “Eliciting Probabilistic Expectations with Visual Aids in Developing Countries: How sensitive are answers to variations in elicitation design?”, Journal of Applied Econometrics.
Delavande, Adeline and Hans-Peter Kohler (2009) “Subjective Expectations in the Context of HIV/AIDS in Malawi”, Demographic Research.
Delavande, Adeline and Susann Rohwedder (2008). “Eliciting Subjective Expectations in Internet Surveys,” Public Opinion Quarterly.
Delavande, Adeline and Susann Rohwedder (2011). “Individuals’ Uncertainty about Future Social Security Benefits and Portfolio Choice ,” Journal of Applied Econometrics.
Delavande, Adeline and Susann Rohwedder (forthcoming). “Differential survival in Europe and the US: Estimates based on subjective probabilities of survival ,” Demography.
Manski, Charles F. and Francesca Molinari (2010). “Rounding Probabilistic Expectations in Surveys,” Journal of Business & Economic Statistics.