Upload
vantuyen
View
239
Download
0
Embed Size (px)
Citation preview
1
Advantages and Disadvantages of Internet Survey Methods for Official
Statistics
Mick P. Couper, PhDSurvey Research Center, University of Michigan
andJoint Program in Survey Methodology
4th International Workshop on Internet Survey Methods
September 2012
2
Outline
Background: Innovation in survey research and the rise of Internet surveys
The use of Internet data collection in national statistical offices• Internet for surveys of households or people• Internet for surveys of establishments
Possible futures for the role of the Internet
Challenges and research opportunities
2
3
Background: Research Sectors
There are four main research sectors with interest in online survey methods:• Market/consumer research – includes advertising, product
research, customer satisfaction, etc.• Public opinion and political polling research• Academic research• Government research
Each sector has a different focus and use of online surveys• Adopting Internet surveys for different reasons • But there is a lot of overlap between sectors• Some types of surveys (e.g., customer satisfaction) are done
by all four sectorsMuch of the innovation and push for online surveys has come from the market research world; the other sectors have been more cautious
4
Research Sectors and Key Concerns
Market and consumer research
Public opinion and political
polling
Government
Speed, cost Speed, cost, accuracy
Accuracy, broad representation
Academic
Cost, complexity
3
5
Internet Survey Adoption in Different Sectors
Market researchers have fully embraced online survey methods, and were early adopters• See next slide
Public opinion research is using Internet methods alongside RDD telephone surveys
Academic researchers doing theory testing are using Internet surveys for cost reasons
Governments have been relatively slow to adopt• Greater focus on mixed-mode surveys than stand-
alone Internet surveys
6
Growth in Online Market Research in U.S. and Europe
0200400600800
10001200140016001800200022002400
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Spen
ding
U.S. (millions $)Europe (millions €)
Source: Inside Research, February 2011
4
7
Competing Pressures
National Statistical Offices (NSOs) face a number of pressures:• Declining response rates and increasing costs of
traditional methods• General budgetary pressures – do more with less• Pressures to be modern and adapt to user
preferences (“digital government”)• Responsibility to collect data on all segments of
society (both the “haves” and “have not’s”)• Need for accuracy, reliability, and trustworthiness –
data serve as benchmarks and have vital policy implications
8
Competing Pressures (continued)
Tension between the need to adapt and innovate and the need to preserve comparability of data across time
Innovation tends to be slow and cautious, relative to other sectors• Innovation is rewarded in some sectors (competitive
edge, publications)• Risk of failure much higher in the government
sector, given relative scale of operations (e.g., censuses)
• Cost pressures not as great as other sectors
5
9
Reasons for Slow Adoption of Internet Surveys
Coverage concerns• Not everyone has Internet access• Those with access are different from those without (the digital
divide), and the differences are not disappearing• NSOs are particularly concerned about broad representation
Sampling concerns• There is no sampling frame of Internet users• There is no RDD-like method to sample e-mail addresses
Nonresponse concerns• Concern that Internet surveys may decrease response rates
and increase nonresponse bias
10
World Internet Penetration Over Time
6
11
Internet Penetration in Korea
44.7
56.659.4
65.5
72.7 73.578.1 78.8 81 81.6 83.7 83.8
0
10
20
30
40
50
60
70
80
90
100
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
Pece
nt
Source: ITU World Telecommunication / ICT Indicators
12
Internet Survey Response Rates
Evidence from two meta-analysis that Internet response rates generally lower than comparable modes
Lozar Manfreda et al. (2008)• Meta-analysis of 45 experimental comparisons of Web to
other modes (mail, phone, IVR, etc.) in 24 papers• Web has an average 11% lower response rate than the
alternativeShih and Fan (2008)• Meta-analysis of 39 experimental comparisons of Web to mail• Web has an average 11% lower response rate than mail• No significant effect of publication year (i.e., the gap is not
closing over time)
Response rate charts
7
13
So What Are NSOs Doing?
Because of these concerns, greater focus on mixed-mode surveys involving Internet than on stand-alone Internet surveys
Different approaches for household surveys and establishment surveys• I focus primarily on household surveys, then offer
some observations on establishment surveys
14
Household Surveys
Different types of surveys • Censuses• One-time or repeated cross-sectional surveys• Panel surveys
Internet as replacement/supplement for different modes• Mail• Interviewer-administered (face-to-face or telephone)
8
15
Internet Use in Household/Population Censuses
Typically used as an option in a sequential mixed-mode design• Mail-out census form includes option for Internet
completion• Non-experimental design• Varying rates of success across countries
Zewoldi (2011) noted that more than 30 countries are providing or experimenting with an Internet option in the 2010-2011 round of censusesSelected examples follow
16
Census Experiences with Mail+Web
Switzerland• In 2000, about 4.2% of forms returned via Internet• Register-based census in 2010
Norway• In 2001, about 9.9% of responses via Internet• Register-based census in 2011
United Kingdom• In 2011, about 27% of households responded online
9
17
Czech Republic and Hungary 2011
25.1 19
62.2
15
12.7
66
0102030405060708090
100
Czech Republic Hungary
EnumeratorMailInternet
18
Census Experiences with Mail+Web (cont.)
Canada• In 2006, 18.3% of households completed the census
online• In 2011, 54.4% of households did so
United States 2000 and 2010• In 2000, about 67,000 households (out of 105.5
million) completed the census form online, despite no advertising of the availability of a Web option
• Tested Internet in 2005 National Census Test• The Census Bureau did not have a Web option in
2010 – I return to this later
10
19
Census Experiences with Mail+Web (cont.)
New Zealand • In 2006, about 7.0% of responses via Web
Australia• In 2006, 10% of occupied households completed the
census online• Aimed for 30% to 40% uptake in 2010• Preliminary estimate of 27% uptake of Internet
option for August 2011 census
20
Census Experiences with Mail+Web (cont.)
Thailand• In 2010, planned for 10% online response• Only 1.4% responded via Internet or telephone call-
in South Korea• In 2005, 0.9% of responses via Internet• In 2010, planned for 30% of forms online, achieved
47.9% (47.7% for short form and 49.3% for long form)
• Estimated reduced expenditure of ±$13.6m USD in personnel costs and ±$4.1m USD in data processing (on base of $157.2m USD)
11
21
Singapore Census 2000 and 2010
Both 2000 and 2010 censuses were register-based, supplemented with sample of 200,000 households
Telephone involved both inbound and outbound calling
15
38
62
46
22 16
0102030405060708090
100
2000 2010
EnumeratorTelephoneInternet
22
Japan Census 2010
Tested Internet response option in Tokyo area only
Population of about 13 million, or 10% of total population of Japan
Expected 5% Internet return rate
Response mode (in percent) of households in Tokyo area
8.4
59.2
32.4
0
10
20
30
40
50
60
70
80
90
100
EnumeratorMailInternet
12
23
Comments on Census Internet Options
Success rates vary across countries and regions
Internet penetration rates do not explain all of this variation
Other factors may play a role:• Level of security requirements and ease of access
to online instrument• Length and complexity of instruments• Promotion of online version and reasons given• Trust in government, concerns about online privacy,
etc.
24
Internet Use in Personal/Households Surveys
Internet-only surveys are rare• Internet status (access or use) is not universal or
known• Except for some specialized populations (e.g.,
college graduates)Most NSOs are exploring mixed-mode designsTwo types of mixed-mode designs• Concurrent mixed-mode surveys, e.g., mail survey
with an Internet option• Sequential mixed-mode surveys, e.g., start with
Internet, then switch to mail, telephone, and/or face-to-face
13
25
Response Rates to Mail Survey With Internet Option
Recent meta-analysis (Medway and Fulton, 2012) of 19 experimental comparisons found that providing a Web option in a mail survey significantly decreases response rates OR=.87)• Meta-analysis results• Detailed examples• Response rate charts
The first study (Griffin et al., 2001) on the ACS led to concerns about implementing an Internet option for the U.S. census and delayed implementation of the Internet on ACS for many years
26
What Do We Make of These Findings?
Several hypothesized reasons have been offered for surprising effects
Cautions• Not clear if this holds for censuses too – two
exceptions were census examples• Censuses are generally short and mandatory• Length and complexity of surveys may be a factor• Cost and/or speed of return is also important
These findings have led many to argue that sequential mixed-mode designs are better
14
27
Sequential Mixed-Mode Designs
In general, the design is to start with a cheaper mode (e.g., Internet) and then follow up with increasingly more costly modes (e.g., telephone and face-to-face)In practice, because e-mail addresses are usually not available, this starts with a mailed invitation to complete an Internet survey• When accompanied by a paper questionnaire, this
is a concurrent or choice mixed-mode design• When followed by a mail survey, this is a sequential
designMany NSOs are exploring sequential designs
28
Research on Sequential Mixed-Mode Designs
There are various mode combinations involving the Internet• E.g., Web followed by mail or Web followed by telephone
and/or face-to-faceThere are different sequences• E.g., Web as the initial mode or as a follow-up mode
The sequential designs are compared to different designs• E.g., single mode designs (e.g., mail only) or choice designs
All of this makes drawing conclusions from a limited number of studies difficulty
Selected examples follow
15
29
Sequential Mail + Web Designs
Studies comparing a sequential design to a mail-only design:• Cantor et al. (2010)• Israel (2009)• ACS Internet tests (see later)
Studies varying the order of the modes in a sequential design:• Holmberg et al. (2010)• Börkan (2010)• Stevenson et al. (2011)• Converse et al. (2008)• Smyth et al. (2010)• Lebrasseur et al. (2010)
Studies comparing choice versus sequential approaches:• Bensky, Link, and Shuttles (2010)• Lesser et al. (2010)• Friese et al. (2010) • Lagerstrøm (2011)
30
Summary of Overall Response Rates for Sequential Mail + Web Designs
Mail only versus Web+mail sequential designs:• 3 studies (Cantor et al; Israel; Elliott et al.) find mail only >
sequential• 1 study (ACS Internet) finds sequential > mail only
Order of mail+Web sequential design:• 4 studies (Stevenson et al.; Converse; Smyth et al.; Friese et
al.) find mail→Web > Web→mail• 3 studies (Holmberg et al.; Börkan; Lebrasseur et al.) find
mail→Web ≈ Web→mailChoice versus sequential design:• 3 studies (Bensky et al.; Lesser et al.; Friese et al.) find choice
> sequential• 1 study (Lagerstrøm) finds choice ≈ sequential
16
31
Comments on Sequential Mail + Web Designs
The results are still quite mixed• The belief that a sequential design starting with the
Internet is best has mixed supportThe benefit of sequential mixed-mode design may lie in cost, speed of response, and data quality – rather than response rates• Even if overall response rates aren’t increased,
substantial cost savings and data quality improvements are possible at large volumes
• Thieme (2011) estimated that each paper form in the U.S. 2010 census cost $12-$14 to print, mail, and process
32
The American Community Survey (ACS)
The ACS is a rolling cross-sectional survey to replace the U.S. census long form
Early development and testing around the 2000 census; now fully implemented
Employs a sequential mixed-mode design with mail and Internet choice• Mail/Internet → telephone (if available) → face-to-
face (for subsample)Recent research on sequential mixed-mode designs has shown more promise
17
33
ACS Internet Tests
Several developments since early tests• Improved interface design• Made login process easier• Made Web choice more prominent• Tested sequential (push) designs
Two recent field tests show more promising results• April 2011 Test• November 2011 Test
Based on this, the Census Bureau is now implementing a sequential mixed-mode design
34
Sequential Designs Involving Interviewers
Many of the sequential designs employed by NGOs include follow-up by telephone and/or face-to-face• This includes the ACS and censuses discussed
earlierStatistics Netherlands is one NSO aggressively exploring use of the Internet in mixed-mode designs
18
35
Web and Interviewer-Administered Sequential Designs: Statistics Netherlands
Dutch Health Interview Survey (Van Nunspeet, Cuppen, and van der Laan, 2011)• Cross-sectional survey, sample drawn from
population register• Response rates slightly higher (64%) for the
sequential mixed-mode design than for old CAPI design (61%)
• About half of the respondents in the mixed-mode design reported via the Web
Dutch Labor Force Survey
Dutch Security Monitor
36
Web and Interviewer-Administered Sequential Designs: Other Examples
Netherlands Kinship Panel Study
Netherlands ESS Round 4 Experiment
UKHLS or Understanding Society innovation panel experiment
UK LFS and OPN tests
Danish LFS test
19
37
What About Web as Follow-up Mode?
Several organizations have explored following up nonrespondentswith a Web optionVoorpostel and Ryser (2011): Swiss Household Panel• Telephone refusals contacted and offered a Web survey• Only 87 of the 1,962 panel members opted for Web and were sent
login information; of these, only 43 completed the Web surveyBanning and Schouten (2009): Dutch Labor Force Survey • One person per household recontacted• One group offered a short instrument by phone, mail, or Web; 45%
did so (mode not reported)• Other group followed up using CATI and CAPI; 77% answered basic
questionsDykstra et al. (2007): W2 of NKPS• Web survey offered to those who refused participation in main data
collection• Only 1.6% of all completes via Web
Findings suggest this strategy is not particularly effective
38
Challenges of Mixed-Mode Surveys Using the Internet
Within-household selection• Some surveys require the selection of one person at random
within the household, but the frame is one of addresses• Several attempts at self-administered within-household
selection have yielded mixed resultsHave everyone complete questionnaire
Use a method such as last birthday to select
Accept a household respondent
Measurement error• Assumption of no measurement differences between modes is
being questioned• Development of statistical methods to adjust for measurement
differences is underway
20
39
Comments on Measurement Error Differences
Different reasons for measurement error differences, e.g.,• Self- versus interviewer-administration: more (i.e., better)
reporting of socially undesirable information in self-administration
• Visual versus aural presentation: question order and response order effects
• Cognitive demands of modes, including distributed cognition• Motivational differences
In general differences between visual modes (mail and Web) and between aural modes (face-to-face and telephone) are smaller than those crossing mode types
Growing evidence that the design of the instruments can have an impact on mode differences
40
Internet Use for Establishment Surveys
Much of the research on mode alternatives is conducted in the context of household surveys
While many NSOs use the Internet as part of a mixed-mode strategy, much less research evidence is available
For this reason, I offer a set of observations, rather than present detailed empirical evidence
21
41
Internet Surveys of Establishments Versus Households/Persons
Establishments include businesses, educational facilities, government agencies, farms, institutions, etc.
Varying levels of Internet use across sectors, but probably greater penetration than for households
42
Features of Establishment Surveys That Promote Internet Survey Use
Surveys are often mandatory
Surveys are usually repeated• Often at regular intervals (e.g., monthly, annually)
Surveys are often short• Small number of items asked repeatedly
Sampling frames usually more complete• No need for listing, screening, selection, etc.
Questions are mostly factual• May be less susceptible to measurement error
22
43
Features of Establishment Surveys That Promote Internet Survey Use (cont.)
Administrative data often available• Permits verification/validation of responses
Extensive editing required• Could be reduced through built-in edits
Speed of reporting is important• Increase reliability of preliminary estimates
44
Characteristics of Establishment Surveys That Impede Internet Survey Use
Information is often distributed or available in records• Requires questionnaire to be answered by several
different people• Requires looking up information in administrative
recordsNeed for documentation• Because of mandatory nature and repeated
measures, businesses often need copy of survey for records
Need access to prior survey’s data• Additional security requirements
23
45
How Widespread is Use of Internet in Establishment Surveys?
Data on this issue is much less easy to obtain than for household surveys• Two exceptions: Bremner (2011); Kovac et al. (2009)
Some estimates from Haraldsen et al. (2011):• Statistics Norway: more than 150 business surveys available
in Internet portal; in 2009 an average of 63% of businesses responded via Internet or electronic data interchange (EDI)
• Statistics Netherlands: about 50% of all business responses in electronic form (Web or EDI)
• Statistics Denmark: more than 45 Web questionnaires on portal; percent of respondents choosing Internet ranged from 2% to 80%
46
Research Opportunities in Establishment Surveys
Unlike household surveys, there is not a tradition of methodological research in establishment surveys• Other than in editing and estimation process• Or in qualitative design work
Yet, the potential benefits of improved design and implementation using the Internet may be greater in establishment surveys
The ongoing nature of establishment surveys lend themselves to incremental improvements
24
47
Research Opportunities in Establishment Surveys (cont.)
Data quality improvements could be achieved with improved design of online business forms• Very little empirical work in this area
Postsurvey processing could identify key areas of concernParadata (e.g., keystroke files, time stamps) could identify inefficiencies• E.g., items that trigger edit checks• Time taken to complete items
These could lead to design improvements that could be tested in the next round of the survey
48
What About “Organic Data”?
Groves (2011) introduced the notion of organic data• Designed data: data created and collected for a specific
purpose (as in surveys)• Organic data: data generated by digital transactions
Examples of organic data• Digital transactions: scanners recording purchases, RFID
tracking goods, vehicle tracking• Social media: blogs, tweets, Internet searches, FaceBook
updates, etc.Examples:• Google trends• Nielsen BlogPulse• Billion prices project• Mappiness
Potential for supplementing rather than replacing traditional survey methods
25
49
Organic Data
Advantages• Cheap, widespread, timely
Disadvantages• Usually only one variable at a time; no covariates
E.g., we know gas prices are rising, but we don’t know how households are changing other spending to accommodate the increase
• Miss the very poor, those offline, etc.E.g., can’t look at how those in poverty, or the old, are behaving relative to others
• Stability of estimates: availability and meaning may change over time
Organic data tend to be wide but thin, while survey data tend to be narrow but deep
50
Conclusions: Opportunities
Internet-only surveys are unlikely in the foreseeable future for NSOs• Except for customer satisfaction surveys, surveys of
special populations, or supplemental tracking studies
The Internet is likely be used in combination with other modes• Sequential mixed mode designs show the most
promiseIncreasing the proportion of surveys completed online is likely to reduce costs and improve data quality
26
51
Conclusions: Challenges or Research Opportunities
We know that increasing Internet data collection benefits NSOs, but we need to identify benefits for respondents (households or establishments) to increase take-up ratesWe need to deal with measurement error differences between modes• Identify under what circumstances they occur• Test instrument designs to minimize such differences• Develop statistical models to adjust for differences
We need more research on Internet surveys for establishment surveys• How to get businesses responding online and keep them there• How to improve the design of survey instruments to leverage
the advantages of the Internet, minimize respondent burden and improve data quality
Thank You