Technological Advances in Survey Data Collection Methods

Survey Research Laboratory

University of Illinois at Chicago

September 2011

Traditional Survey Methods

• Mail/self-administered questionnaires

� Census questionnaires go back centuries

• Face-to-face interviews

� Employed in the U.S. since 1930

• Telephone interviews

� In common use since the 1970s

Some New Survey Technologies

For face-to-face surveys:

•ACASI (audio computer-assisted self-interviews)

•Hand-held devices (PDAs, Smartphones, Tablets)

•MiFi to create mobile hot-spots in rural areas

For telephone surveys:

•IVR (interactive voice response),

aka: T-ACASI

•VOIP

For self-administered surveys:

•Web surveys

•Also hand-held devices

New computer technologies:

• Combine power of computer administration� Automated question routing

� Checking for bad or inconsistent answers

� Data encryption

� Wireless transmission

• With the advantages of self-administration� Elimination of interviewer variance (but replaced by device/browser variance)

� Improved reporting of sensitive information

ACASI Development

• Developed at the Research Triangle Institute in early-to-mid-1990’s

• Adapted for the National Household Survey on Drug Abuse in 1997-99

• Now understood that most respondents do not take advantage of audio features (Couper et al., POQ, 2009)

IVR Development

• First reported in late 1990s

• Two approaches:� Initial contact by recording & IVR survey

� Human contact, then IVR survey

• Developmental work centered on market research applications

• So far, less commonly applied in academic research

Web Surveys

• First reported use in early 1990’s

• Dramatic increase in use over the past decade

• Hundreds of web survey software packages now available

• Knowledge Networks panel

Typology of Web Surveys

With thanks to Mick Couper, University of Michigan Survey Research Center(see: Couper, Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly 2000;

64: 464-494)

Nonprobability Methods

1. Polls as entertainment

2. Unrestricted self-selected surveys

3. Volunteer opt-in panels

Probability Methods

4. Intercept surveys

5. List-based samples

6. Web option in mixed-mode surveys

7. Pre-recruited panels of internet users

8. Pre-recruited panels of full population

Sources of Survey Error

1. Sampling error

2. Coverage error

3. Nonresponse error

4. Measurement error

5. Processing error

Coverage Error

• No real effect of ACASI and IVR on coverage error

• Coverage error is a big problem for web surveys� Poor frame availability for general population

� Around 75% of U.S. population are ever internet users

� Digital divide: differences in accessibility by race/ethnic minorities

� Solutions: (1) weighting (2) web-enabled panels

European HH Internet Access: 2008Data Source: http://epp.eurostat.ec.europa.eu/

Darker colors = higher internet access

Nonresponse Error

• Unit nonresponse

� Little effect of ACASI

� IVR – breakoffs a big problem

� Low response rates in web surveys

• Item nonresponse

� In general, less missing data in ACASI, IVR and web surveys (when programmed correctly)

Measurement Error

• General belief that self-administered survey data of higher quality� More privacy = more reporting of sensitive behaviors

� ACASI, IVR and Web surveys are all enhancements to self-administered survey research

� More non-differentiation in web surveys

� Computer methods employ different sensory channels: visual vs. aural

Processing Error

• ACASI, IVR and Web surveys can all be used to program data consistency checks

• Skip patterns can be automated

• Out-of-range values prohibited

• Some data can be generated by the device directly (date, time, length)

• Programming errors can, however, be fatal

Other Advantages/Disadvantages

• Cost

� ACASI/IVR more expensive

� To purchase software & recording systems

� To develop programming expertise

� ACASI requires expensive hardware for each interviewer

� Requires availability of timely technical support

� Web surveys – relatively inexpensive

� Software systems are affordable for most organizations

� Additional hardware required for hosting web surveys� And hosting is more expensive

� Surrender control if hosting survey with outside vendor� IRBs may impose restrictions on use of outside vendors

Other Advantages/Disadvantages

• Cost Effectiveness

� ACASI most cost-effective for complex, large-sample, sensitive surveys of the general population

� Web surveys most cost-effective for relatively brief and less complex surveys of special populations

� But this is changing

Other Advantages/Disadvantages

• Time efficiency

� ACASI/IVR can require considerable time to program and test

� It can take months/years to develop skills with these systems

� Web surveys – can be done very quickly in comparison to most other methods

� Are now relatively simple to program

� And complex programming functions are becoming more available

Other Advantages/Disadvantages• Computer familiarity

� ACASI, IVR and Web surveys are very applicable with young, educated and professional samples

� These technologies can be a barrier to survey participation and data quality for some populations who are less familiar and/or comfortable with these technologies, or who have limited access

• Still some gap in technical competence and information literacy among several groups:

• Older citizens

• Minorities

• Low SES groups

Other Advantages/Disadvantages

• Data Complexity� ACASI can be used to collect lengthy and highly complex data sets

� IVR can also be used to collect complex data but data complexity may contribute to break-off problem

� Web surveys have historically been thought to be useful only for collecting relatively simple data and brief sets � this is not SRL’s experience

� ACASI/IVR/Web remove the interviewer. It is difficult to replace an interviewer’s ability to identify respondent fatigue and respond in ways that maintain respondent engagement.

Other Advantages/Disadvantages

• Questionnaire Design

� These methods facilitate question tailoring

� Complex skip patterns, contingent on responses to multiple questions

� Personalize/customize question wording based on earlier or pre-coded or pre-loaded data (‘fills’)

� Examples: use terminology most comfortable with, specific dates, avoid awkward-looking text (eg., his/her/yours)

Other Advantages/Disadvantages

• Data Quality

� These methods reduce interviewer error

� These methods can also improve the quality of open-ended responses, although:

� The typing skills of respondents are another consideration when asking open-ended questions

� Can be used to detect interviewer falsification

� Random respondent selection can be automated to avoid errors

Other Advantages/Disadvantages

• Administration

� ACASI, IVR will take longer, on average, to administer to each respondent

� In the field, use of computers can sometimes be ‘clunky’ or awkward

� but things are improving as computers become lighter and more mobile

� Web surveys can be made very convenient and user-friendly

� and respondents are becoming more comfortable with web surveys

Other Recent Technologies

• On-line focus groups• Sample block listings via:

� GPS systems� Other hand held devices

• Use of video-clips in ACASI & web surveys• ACASI digital recording of open-ended responses• Touch screen systems• Digital pens• Improvements in voice recognition systems• Daily uploading of encrypted, completed field

interviews via remote wireless connections• Data encryption

Continuing technical challenges:

•Wireless-only households

•Data security & IRB issues

State-level Estimates of the Percentage of Adults Living in Wireless-only Households: United States, July 2009-June 2010

Source: Wireless Substitution: Early Release of Estimates From the National Health Interview Survey, July - December 2008. by Stephen J. Blumberg, Ph.D., and Julian V. Luke, Division of Health Interview Statistics, National Center for Health Statistics

United States Flu Activity

• Google Flu Trends estimate• United States Data

Smart Phones

Relatively untapped in Survey Research

Technology being adopted at fast rate

Potentially very flexible tool, multiple modes on a single device

Types of Smartphone Surveys

Phone

SMS (Short Message Service) – Text messaging

Browser

APP

Hybrid (ie. Download APP via browser)