Facing the Nonresponse Challenge

http://ann.sagepub.com/of Political and Social Science

The ANNALS of the American Academy

http://ann.sagepub.com/content/645/1/23The online version of this article can be found at:

DOI: 10.1177/0002716212456815

2013 645: 23The ANNALS of the American Academy of Political and Social ScienceFrauke Kreuter

Facing the Nonresponse Challenge

Published by:

http://www.sagepublications.com

On behalf of:

American Academy of Political and Social Science

can be found at:ScienceThe ANNALS of the American Academy of Political and SocialAdditional services and information for

http://ann.sagepub.com/cgi/alertsEmail Alerts:

http://ann.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

What is This?

- Nov 26, 2012Version of Record >>

at UNIVERSITY OF MARYLAND on January 11, 2013ann.sagepub.comDownloaded from

http://ann.sagepub.com/

http://ann.sagepub.com/content/645/1/23

http://www.sagepublications.com

http://www.aapss.org/

http://ann.sagepub.com/cgi/alerts

http://ann.sagepub.com/subscriptions

http://www.sagepub.com/journalsReprints.nav

http://www.sagepub.com/journalsPermissions.nav

http://ann.sagepub.com/content/645/1/23.full.pdf

http://online.sagepub.com/site/sphelp/vorhelp.xhtml


ANNALS, AAPSS, 645, January 2013 23

This article provides a brief overview of key trends in the survey research to address the nonresponse chal-lenge. Noteworthy are efforts to develop new quality measures and to combine several data sources to enhance either the data collection process or the qual-ity of resulting survey estimates. Mixtures of survey data collection modes and less burdensome survey designs are additional steps taken by survey researchers to address nonresponse.

Keywords: nonresponse; survey quality; combining data sources; paradata; administrative data

Modern surveys are challenged by high non-response rates, a fact documented in sev-

eral contributions in this volume and a large number of previous articles. The challenge to initially recruit respondents to participate has been observed in numerous surveys regardless of their mode of data collection or the country in which they are conducted. Although the U.S. government’s flagship survey, the Current Population Survey, still achieves household response rates over 90 percent, most other face-to-face surveys in the United States and else-where fall significantly below this mark. The situation is worse in large-scale telephone surveys, such as the University of Michigan’s Survey of Consumer Attitudes or the federal government’s Behavioral Risk Factor Surveillance Survey, in

The Annals of the American AcademyFacing the Nonresponse Challenge

Frauke Kreuter is an associate professor at the University of Maryland in the Joint Program in Survey Methodology and, currently, head of the statistical methods research department at the Institute for Employment Research in Nuremberg, Germany. She has published extensively on the use of paradata and nonresponse adjustment.

NOTE: This article integrates thoughts from Mick Couper’s presentation at the CNSTAT planning meet-ing in 2009; a working paper written by the author in response to a request from the German Rat für Sozial- und Wirtschaftsdaten; and elements from a recent presentation at the 2010 Joint Statistical Meeting by Mick Couper, Lars Lyberg, and the author.

DOI: 10.1177/0002716212456815

Facing the Nonresponse

Challenge

ByFRAUkE kREUTER



24 THE ANNALS OF THE AMERICAN ACADEMY

which response rates have recently declined at a rate of one-half to one percentage point per year and are now likely to be found in the 30 to 40 percent range, response rates common to many household surveys in Europe. Years of research on survey nonresponse have helped neither to reverse nor to stop this trend.

The nonresponse challenge goes hand in hand with several other threats to the integrity of modern surveys. To some extent, declining response rates can be offset by spending more money on recruitment and incentives, but increased costs without corresponding increases in funding have left government programs struggling to maintain data quality. Rapid societal changes introduce new uncer-tainties into surveys at the design stage, making it more difficult to predict the costs and outcomes of design choices. Exacerbating these basic difficulties is the increasing complexity of surveys themselves, which often can no longer be con-ducted using a single sampling frame, a single mode of data collection, or even a single language. As hard as it was in the past to predict the cost per case for a survey that used a single collection mode, a single language, and a single sam-pling frame, it is now even more challenging to know what the nonresponse rate will be and whether cost targets can be met.

Good challenges can create new opportunities, of course, and the nonresponse challenge is no exception. The decline in response rates forces survey researchers to innovate and improve: to find better measures of evaluating surveys and to seek improved ways of using all data available, even if this means combining data from other sources and different cases. It may even require adapting and modify-ing survey operations on a continual basis. Indeed, innovation and improvement is happening on surveys. In recent years, for example, there has been a change in perspective, moving away from a focus on response rates per se and instead seek-ing to assess the degree and effects of nonresponse bias, as reflected in efforts to develop alternative measures for describing survey results.

Statisticians have been working fervently on new techniques to combine data from different surveys and to integrate administrative data with survey data. Fieldwork agencies, meanwhile, are moving toward greater flexibility, shifting away from one-size-fits-all strategies and implementing new, targeted interventions designed to increase response rates for specific groups. Designing surveys with an eye to nonresponse requires a new way of managing data collection, and the age of electronic data collection opens multiple possibilities in this arena. Of course, con-siderable development work lies ahead for those wanting to leave behind the charm and simplicity of evaluating surveys on the basis of response rates alone and instead move toward ensuring survey quality through strong process monitoring and con-trol. The following sections lay out in more detail the major actions survey research-ers have been taking to address the nonresponse challenge.

Shedding Bad Habits: New Measures of Survey Quality

The habit of judging surveys by their response rates alone is difficult to overcome, in part because national and international survey guidelines have long specified



FACING THE NONRESPONSE CHALLENGE 25

response rate targets. Prominent examples are the U.S. Office of Management and Budget’s guidelines and those provided by the Organisation for Economic Co-operation and Development (OECD) for international surveys such as the Program for International Student Assessment (PISA) and the Program for the International Assessment of Adult Competencies (PIAAC). In addition, journals such as Public Opinion Quarterly have, for very good reasons, routinely asked authors to report basic information about their surveys, including their response rates. Nonetheless, as discussed elsewhere in this volume, response rates do not necessarily inform data users about potential biases or provide a good indicator of survey quality, at least for rates in the range with which we are now familiar.

In response to research demonstrating the limited amount of information on survey quality that is actually captured by response rates, attempts have recently been made to develop alternative measures. Two examples of such alternative measures are the R-indicator (Schouten, Cobben, and Bethlehem 2009) and the fraction of missing information (FMI) indicator (Wagner 2010). These and simi-lar indicators are currently used in responsive designs (Groves and Heeringa 2006), where these indicators are monitored during implementation and continu-ously inform fieldwork decisions (Lepkowski et al. 2010; Lundquist and Särndal 2012). Each of these alternative measures depends on the availability and quality of auxiliary information about respondents and nonrespondents, however.

Using data available from the sampling frame, the R-indicator is designed to capture imbalances in response propensities between subgroups of sampled units and to measure the similarity between the original sample and the resulting com-position of respondents (in a given set of auxiliary variables). In its simplest form, the estimated R-indicator for a survey with sample size n is proportional to the standard deviation of the response propensities for each individual i estimated using a set of covariates. Assuming equal sampling probabilities, it is expressed as

Rn ii

n( ) ( )ρ ρ ρ= −

−−

=∑1 21

12

1 , (1)

where ρi is the individual response propensities and ρ- is the average response pro-

pensity over all sample cases (Schouten, Cobben, and Bethlehem 2009). The R-indicator uses available information on both respondents and nonrespondents to estimate response propensities, through either logistic regression models or classifi-cation trees, and is one of several measures developed as part of the European Union (EU) project to derive Representativity Indicators of Survey Quality, popularly known by the acronym RISQ.1 Funded by the EU, the project is administrated and conducted by researchers from the statistical institutes of the Netherlands, Norway, and Slovenia, as well as the Universities of Southampton and Leuven. Similar in spirit to the R-indicator is the q2 indicator, developed by Särndal and Lundström (2008), which is defined as the variance of the predicted inverse response probabili-ties. Larger values of q2 reflect a lower potential for nonresponse bias.

The R-indicators and other “balancing indicators” yield statistics at the survey level. In contrast, the FMI indicator includes information from individual survey

ˆ ˆˆ

ˆ ˆ




items, thus acknowledging that nonresponse bias likely varies across different survey outcomes (Groves 2006). The FMI (γD) seeks to measure uncertainty about values imputed for missing elements (Rubin 1987; Little and Rubin 2002), or in the present case the data imputed for nonresponding cases (Wagner 2010). The FMI represents a ratio of between-imputation variance in the estimate relative to total variance in the estimate, with

γD = (1 + 1 /D)B

D/T

D, (2)

where D is the number of multiple imputations, BD is the between-imputation variance, and T

D is the total variance associated with the average of a given vari-

able for D imputations. Less uncertainty in the imputations (and thus the esti-mates) indicates that more information is available in the (sometimes enriched) sampling frame dataset for that estimate.

The FMI indicator is equivalent to the nonresponse rate when the values used in the imputation model are not correlated with the survey variables. If the available auxiliary or sampling frame variables are correlated with the survey variables, the fraction of missing information will be reduced relative to the nonresponse rate. A limitation of the FMI is that it assumes the data to be miss-ing at random, thus assuming that after conditioning on the observed auxiliary or frame data the likelihood of being missing is independent of the survey out-come. Furthermore, the FMI will create a large set of item-specific indicators that can be difficult to communicate when survey questionnaires are long.

Estimators that do not rely on the missing-at-random assumption, conditional on covariates, have also been proposed in the literature (Andridge and Little 2011; West 2011). The authors proposing these methods generally advocate sen-sitivity analyses in which investigators examine the sensitivity of inferences to assumptions about the social mechanisms that produce the missing data. Severe fluctuations in inferences based on assumptions about the missing data mecha-nism could themselves be an indicator of poor quality and potential bias in the data available for the survey respondents.

A key feature of all these approaches is the need for additional information on cases for respondents and nonrespondents alike, such as that obtained from sam-pling frames; from auxiliary files purchased from commercial vendors; or, in the case of panel surveys, from prior waves. Fortunately (at least from a methodologi-cal viewpoint), the amount of information available for individual cases has been increasing and is often compiled systematically by data management companies, such as Acxiom in the United States or Microm in Germany (Smith 2011). At the same time, survey researchers increasingly have access to survey process data on all sampling units, which likewise can be useful for the approaches described here. Survey process data for nonresponse adjustment will play a large role in surveys if no other sources of auxiliary variables are available or survey research-ers are not allowed to link available auxiliary data to survey data (Iannacchione 2011).




Enriching Auxiliary Information

Survey designers have always relied on auxiliary data to make sampling design deci-sions. Ideally, knowing about the distribution of proxy variables for survey out-comes should guide decisions about stratification and selection procedures when designing area probability samples (Lohr 1999). Auxiliary data describing the sur-vey process, which Couper (1998) has labeled “paradata,” can also help survey researchers to address the nonresponse challenge by guiding fieldwork decisions and enabling adjustments for nonresponse once the survey is complete.

Paradata of potential use for nonresponse adjustment are now routinely col-lected in conjunction with the listing of housing units and the contact attempts of sample units. For example, the date and time of an attempted contact, the method of data collection (e.g., in person or by phone), and other information about the sampling unit are often recorded in the contact attempt record. The U.S. Census Bureau has, for several years, used an automated system to collect contact histo-ries for its computer assisted personal interviews (Bates, Dahlhamer, and Singer 2008), and other governmental organizations increasingly do the same.

In response to the work of Campanelli, Sturgis, and Purdon (1997), the Research Center of the Flemish Government has begun to use standard con-tact forms in its surveys. A standard contact form has also been implemented since the very first round of the European Social Survey (in 2002), and contact data were recently released by the U.S. National Center for Health Statistics for its 2006 National Health Interview Survey. Contact data such as these are very attractive to those concerned with biases stemming from nonresponse. The fact that contact data are available for each sample unit makes them suit-able for use in computing all three bias indicators discussed above. However, the best variables for bias estimation models are those that both predict nonre-sponse and correlate highly with other survey outcomes, and contact data might not fit this bill. In the case of the European Social Survey, at least, the relation-ship between contact history and other response indicators is much stronger than the relationship between contact history and key survey outcome variables (kreuter and kohler 2009).

Researchers continue to search for process measures—information on pro-cesses of respondent location and recruitment—that have strong associations with survey variables. Interviewer observations about the nature of sampled housing units as well as neighborhood circumstances are particularly strong can-didates. Observations such as these are embedded in several current surveys. In the United States, examples include the Health and Retirement Study, the Study of Early Child Care, the Survey of Consumer Finances, and the National Survey on Drug Use and Health. In Europe, they include the British Crime Survey and the Survey of Health, Aging and Retirement in Europe.

Some interviewer observations are explicitly designed to serve as proxy varia-bles for survey outcomes (kreuter et al. 2010). For example, interviewer assess-ments of housing tenure are gathered in the Consumer Expenditure Survey




(king et al. 2009), where respondents answer questions about their housing expenses. In the National Survey of Family Growth, where reproductive behav-ior is one of the key outcome variables, interviewers are asked to guess the sexual activity status of selected persons (Lepkowski et al. 2010). More recently, the educational status of sample units was collected in the German pretest for the Program for the International Assessment of Adult Competencies; and Maitland, Casas-Cordero, and kreuter (2009), for data collection in the National Health Interview Survey, suggest asking interviewers to guess sample members’ health status by observing signs of smoking, obesity, and other health conditions in the household. The U.S. Census Bureau is currently exploring indicators along these lines as well.

Besides paradata, two additional data sources are increasingly of interest to survey designers: information offered by commercial mailing vendors and data from administrative sources. The latter often provide limited potential for assess-ing the degree and nature of bias. However, because consent is generally required for linkage to respondent records, which can be obtained only from the respondents themselves, the utility of administrative information is limited. Exceptions are cases in which samples have been drawn directly from administra-tive lists and legislation has been passed to allow investigators to link back to the original administrative source (kreuter, Müller, and Trappmann 2010).

Reducing Burden and Combining Sources

Asking fewer questions or administering surveys to fewer people is another potential answer to the nonresponse challenge and the resultant increasing costs of data collection. To the extent that reluctance to respond is a function of the perceived burden of completing a survey, shorter questionnaires can help. A reduction in the number of people exposed to a survey request naturally tends to lower the overall survey burden for society. Both a reduction in length of the questionnaire and a lowering of the number of people asked to participate in any given survey can also save greatly on costs.

Under the rubric of “borrowing strength” or “leveraging existing data,” a dis-cussion has emerged among statisticians about the efficacy of combining infor-mation from several different surveys and the possibility of linking surveys to administrative data using statistical modeling techniques (Raghunathan, Reiter, and Rubin 2003; Abowd and Lane 2004; Raghunathan et al. 2007). Related in spirit is what has been called “multiple-matrix sampling,” a technique currently under consideration by researchers at the Bureau of Labor Statistics. This approach is designed to reduce the number of questions administered to each sample unit in the Consumer Expenditure Survey and, thus, lower respondent burden (Gonzalez and Eltinge 2008). Under a matrix sampling approach, a ques-tionnaire is divided into subsets and random subsamples of cases are selected to respond to different items on the questionnaire. Answers to the nonadministered




subsets are then imputed. To the extent that nonresponse is related to actual or perceived burden, such adaptations can be useful. However, this approach relies rather heavily on statistical models that are later used to re-create full datasets.

Linking sampled cases to administrative data can also reduce the number of survey items and lower respondent burden. Health surveys have for a long time asked respondents for permission to link to their medical records. In the past decade, many other social science surveys have followed suit. The usefulness of such links depends on how many cases are willing to consent, of course, but so far linkage consent rates in most surveys are reasonably high. Sakshaug, Couper, and Ofstedal (2010) report, for example, that 67 percent of respondents to the Health and Retirement Survey granted consent; Schmucker and Huber (2009) report a figure of 90 percent for the German Lifelong Learning Survey. Despite these encouraging numbers, we probably cannot rely on consent rates to remain high over the years or assume that they will remain high regardless of the nature of the administrative data in question. In many ways, requesting respondent con-sent for administrative linkage simply shifts the nonresponse problem to a differ-ent venue, and just as in the case of unit nonresponse, careful evaluation of possible consent bias is warranted.

Survey Operations: Mixed Modes and Adaptive Changes

Multimode surveys constitute another attempt to maintain costs while keeping response rates at desired levels. For example, as a cost-saving approach, the Behavioral Risk Factor Surveillance Survey explored inviting respondents by mail to participate in a Web survey that was followed by a shorter computer-assisted telephone interview (Link and Mokdad 2005). At present there are three principal approaches to the design and administration of multiple-mode surveys: (1) modes are administered in sequence, (2) modes are implemented simultane-ously, or (3) a primary mode is supplemented with a secondary mode (De Leeuw 2005).

The American Community Survey, for example, uses a sequential application of modes in which respondents are first contacted by mail and then nonrespond-ents to the mail survey are contacted by phone (if telephone numbers can be obtained). If phone interviews cannot be achieved, in-person follow-ups are made to the sample of respondents that remained uninterviewed (Griffin and Obenski 2002). In contrast, surveys associated with the Current Employment Statistics Program deploy several modes in parallel, as respondents (in this case, business establishments) first choose between mail and fax as a mode of response, and then nonrespondents are separately contacted for a computer-assisted telephone interview. Likewise, the European Social Survey program recently launched a special mixed-mode design in four countries to examine appropriate ways to tailor data collection strategies to respondents.




The core idea of mixed-mode surveys is to tailor response options to the con-venience of individual survey respondents in specific subgroups. Given that how people are contacted differs across modes, it is likely that ease of contact does, too, and that a broader set of people could be reached using multiple modes. With respect to the participation decision, for example, certain subgroups might be more likely to comply with the survey request when given the chance to com-plete the survey in their own way and at their own pace, whereas other subgroups might prefer a personal interaction with an interviewer. Offering respondents a choice of modes at the onset of a survey has not always been shown to increase response rates, however (Griffin, Fischer, and Morgan 2001), and different sequential offerings might be required for different subgroups. The challenge is to be able to identify such subgroups ahead of time or, in the absence of data at the onset of the survey, to adapt the mode offerings as a function of data collected during fieldwork itself.

A statistical basis for such adaptive procedures has been developed and used in the context of clinical trials (Collins, Murphy, and Bierman 2004). In the sur-vey context, however, an optimal adaptive treatment regime is the sequence of treatments with the highest overall probability of achieving response (Wagner 2008). Research in this area is still quite new, and in its infancy in the field of survey methodology. Increased knowledge will be possible only if the survey community as a whole commits to continuously applying experimental designs as part of ongoing production surveys.

Beyond Nonresponse: Quality through Process Control

The foregoing examples demonstrate how survey methodologists and statisticians have reacted to the nonresponse challenge. More likely than not, however, the nonresponse challenge will not be the only challenge survey researchers will face. A more general shift in how investigators think about survey design and survey quality may also be warranted. Most likely, the entire survey process will need to shift in response to anticipated uncertainties before data collection begins and also to real-time information obtained in the process of data collection itself. Thus, understanding the survey process and being able to monitor and control what happens in the course of a survey might become even more important than it is now.

Survey quality—whether defined in terms of small error rates for specific sources of uncertainty or small mean squared errors for selected estimates—depends on the design and maintenance of the underlying processes that gener-ate the data products. Statistical agencies have always been concerned with quality, of course, but in many cases a clear recognition of the link between process and product has been missing. By controlling and possibly adjusting sur-vey processes, research goals might be reached more easily, and spending might be better managed. This approach to survey quality relies on embedding




continuous quality improvement measures into survey procedures, since a good process analysis can identify root causes of problems that, if corrected in a timely fashion, might contribute to decreased process variation and consequently less variation in product characteristics (Lyberg et al. 1997; Aitken et al. 2004).

Some survey organizations, including Statistics Canada (Mudryk, Bougie, and Xie 2002), the U.S. Census Bureau (Bushery et al. 2006), Westat Inc. (Morganstein and Hansen 1990), the Institute for Social Research at the University of Michigan (Couper and Lyberg 2005), and Statistics Sweden (Statistics Sweden 2004), have applied theories and methods from the quality control literature to their survey processes. The literature on statistical quality control discusses the use of statistical methods and other problem-solving tech-niques to improve the quality of data products (see Montgomery 2005). These theories and methods were originally developed for industrial applications but can be used in administrative settings as well.

The techniques for quality improvement range from the use of simple histo-grams, Pareto charts (or diagrams), scatter plots, and flow diagrams to more elaborate tools such as experiments, cause-and-effect diagrams, acceptance sam-pling, and control charts (Shewhart 1931). Pareto diagrams show the incidence of completed interviews over time in descending order using bars and the cumu-lative total interviews using a line. Control charts are used to distinguish between common-cause and special-cause variations in response rates, where common causes are those that affect all cases and special causes are those that influence cases on a person-to-person basis. The basic philosophy of this technique is to start process investigations by eliminating any special cause variation so that the process has only common cause variation. If the common cause variation is deemed too large, an attempt is made to improve the process so that the natural variability decreases.

With this framework in mind, one way of controlling the survey process is to identify key process variables that are monitored continuously and allow for tar-geted interventions (Groves and Heeringa 2006). This identification can be done for almost any process that directly or indirectly affects survey quality. The goal is to identify those variables that are most likely to affect the product character-istics that result from a particular process. Processes that easily lend themselves to this exercise include data collection, data editing, and data capture and coding but also administrative processes such as competence development for personnel and better project budgeting (Morganstein and Marker 1997). Examples of key process variables for editing and coding are given in Biemer and Lyberg (2003).

One of the key process variables used in the National Survey of Family Growth draws on interviewer observations about the presence of children in the household during the screener interviews. A key survey item asks respondents about the number of sexual partners they have had in the past year. From prior surveys, investigators know that there is a significant difference in the odds of cohabitating in the past year among respondents perceived to have young chil-dren in the household and respondents who do not (West 2011). Interviewer




observations about the presence of children can therefore be used to monitor the sample composition in terms of this variable; in general, these observations have been shown to correlate with several key variables in the survey (West 2011). When the nonrespondent pool becomes imbalanced based on interviewer obser-vations about the presence of children, an intervention can be launched to increase effort on those cases. Interviewers get those cases flagged as they down-load their daily assignments from the case management system and are directed to allocate more effort to those cases (Lepkowski et al. 2010).

Obviously the timely availability of process data and other paradata is crucial to successful process control. Unfortunately, in practice such timely availability of information appears to be a challenge for many survey organizations. Timeliness is a particular problem for those organizations relying on freelance interviewers who do not interact with the survey organization on a daily basis and for organiza-tions that lack electronic equipment in the field that is necessary for a timely uploading of the data.

Conclusion

The past 20 years of nonresponse research have clearly shown two things: that continuing to do things the old way is no longer sustainable and that to date there is no single solution to the nonresponse challenge. Survey response rates con-tinue to decline, costs are rising, and budgets are shrinking. As a result, research-ers must tackle the nonresponse challenge on a broad front. Efforts must go beyond simply trying to measure and reduce nonresponse bias and go one step further to manage and control quality throughout the entire survey process.

Perhaps survey researchers should go so far as to say that they will not worry about response rates themselves, any more than they worry about other survey errors. In this spirit, strong response rate standards discussed above are counter-productive because they focus all efforts on minimizing one potential source of error to the exclusion of all others. Strong guidelines with respect to the survey process and how it should be controlled would, in contrast, allow survey research-ers to trust surveys more than would be possible by simply knowing the value of any particular response rate.

With respect to adaptive and responsive designs, the field still has a lot to learn. knowledge about the effects of nonresponse on mean square error and costs is very limited, often because organizations lack good ways to measure partial costs. The ability to process and analyze available data will need to expand, and statistical tools from decision theory, risk analysis, loss functions, optimization, and process control will need to be brought into the survey methodologist’s toolbox, along with methods from economics such as cost functions and utility maximization. In addi-tion, experimental evaluations of procedures embedded within ongoing surveys will provide needed information about the sorts of adaptations surveys need to make. Last, survey clients need to learn not to rely any longer on unproductive summary measures of survey quality such as response rates.




Note1. See http://www.risq-project.eu/.

ReferencesAbowd, John M., and Julia I. Lane. 2004. New approaches to confidentiality protection: Synthetic data,

remote access and research data centers. U.S. Census Bureau Technical Paper TP-2004-03, Washington, DC.

Aitken, Alexis, Jan Hörngren, Nia Jones, Daniel Lewis, and Maria J. Zilhão. 2004. Handbook on improving quality by analysis of process variables. Luxembourg: Publications Office of the European Union.

Andridge, Rebecca R., and Roderick J. A. Little. 2011. Proxy pattern-mixture analysis for survey nonre-sponse. Journal of Official Statistics 27 (2):153–80.

Bates, Nancy, James Dahlhamer, and Eleanor Singer. 2008. Privacy concerns, too busy, or just not inter-ested: Using doorstep concerns to predict survey nonresponse. Journal of Official Statistics 24 (4): 591–612.

Biemer, Paul P., and Lars E. Lyberg. 2003. Introduction to survey quality. Hoboken, NJ: John Wiley.Bushery, John, Pamela McGovern, Paul S. Marck, and Howard R. Hogan. 2006. Quality audits at the U.S.

Census Bureau. In Proceedings of the European conference on quality in survey statistics. Newport, CT: Office for National Statistics.

Campanelli, Pamela, Patricia Sturgis, and Susan Purdon. 1997. Can you hear me knocking: An investiga-tion into the impact of interviewers on response rates. London: Social and Community Planning Research.

Collins, Linda M., Susan A. Murphy, and karen L. Bierman. 2004. A conceptual framework for adaptive preventive interventions. Prevention Science 5 (3): 185–96.

Couper, Mick P. 1998. Measuring survey quality in a CASIC environment. In JSM Proceedings, 41–49. Alexandria, VA: American Statistical Association.

Couper, Mick P., and Lars E. Lyberg. 2005. The use of paradata in survey research. In Proceedings of the 55th session of the International Statistical Institute. Sydney, Australia: International Statistical Institute.

De Leeuw, Edith D. 2005. To mix or not to mix data collection modes in surveys. Journal of Official Statistics 21 (2): 233–55.

Gonzalez, Jeffrey, and John Eltinge. 2008. Adaptive matrix sampling for the consumer expenditure inter-view survey. Paper presented at the Joint Statistical Meeting, American Statistical Association, Denver, CO. Available from http://www.bls.gov/osmr/pdf/st080060.pdf (accessed 14 May 2012).

Griffin, Deborah H., Donald P. Fischer, and Michael T. Morgan. 2001. Testing an Internet response option for the American Community Survey. Paper presented at the annual conference of the American Association for Public Opinion Research, Montreal, Quebec, Canada.

Griffin, Deborah H., and Sally M. Obenski. 2002. Meeting 21st century demographic needs: Implementing the American Community Survey, Report 2: Demonstrating survey quality. Washington, DC: U.S. Census Bureau.

Groves, Robert. 2006. Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly 70 (5): 646–75.

Groves, Robert, and Steve Heeringa. 2006. Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society, Series A 169 (3): 439–57.

Iannacchione, Vincent G. 2011. Research synthesis: The changing role of address-based sampling in sur-vey research. Public Opinion Quarterly 75 (3): 556–75.

king, Susan L., Boriana Chopova, Jennifer Edgar, Jeffrey M. Gonzalez, Dave E. McGrath, and Lucilla Tan. 2009. Assessing nonresponse bias in the Consumer Expenditure Interview Survey. In JSM Proceedings, 1808–16. Alexandria, VA: American Statistical Association.

kreuter, Frauke, and Ulrich kohler. 2009. Analyzing contact sequences in call record data: Potential and limitation of sequence indicators for nonresponse adjustment in the European Social Survey. Journal of Official Statistics 25 (2): 203–26.




kreuter, Frauke, Gerrit Müller, and Mark Trappmann. 2010. Nonresponse and measurement error in employment research. Public Opinion Quarterly 74 (5): 880–906.

kreuter, Frauke, kristen Olson, James Wagner, Ting Yan, Trena Ezzati-Rice, Carolina Casas-Cordero, Michael Lemay, Andy Peytchev, Robert Groves, and Trivellore Raghunathan. 2010. Using proxy meas-ures and other correlates of survey outcomes to adjust for nonresponse: Examples from multiple sur-veys. Journal of the Royal Statistical Society, Series A 173 (2): 389–407.

Lepkowski, James, William Axinn, Nicole kirgis, Brady T. West, Shonda kruger Ndiaye, William Mosher, and Robert Groves. 2010. Use of paradata in a responsive design framework to manage a field data collection. NSFG Survey Methodology Working Papers Report 10-012, Atlanta, GA.

Link, Michael, and Ali Mokdad. 2005. Use of alternative modes for health surveillance surveys: Results from a web/mail/telephone experiment. Epidemiology 16 (3): 701–4.

Little, Roderick J. A., and Donald B. Rubin. 2002. Statistical analysis with missing data. 2nd ed. New York, NY: John Wiley.

Lohr, Sharon. 1999. Sampling: Design and analysis. Pacific Grove, CA: Duxbury.Lundquist, Peter, and Carl-Erik Särndal. 2012. Aspects of responsive design for the Swedish Living

Conditions Survey. Research and Development Methodology Reports 2012:1. Stockholm: Statistics Sweden.

Lyberg, Lars, Paul Biemer, Martin Collins, Edith De Leeuw, Cathryn Dippo, Norbert Schwarz, and Dennis Trewin. 1997. Survey measurement and process quality. New York, NY: John Wiley.

Maitland, Aaron, Carolina Casas-Cordero, and Frauke kreuter. 2009. An evaluation of nonresponse bias using paradata from a health survey. In JSM Proceedings, 370–78. Alexandria, VA: American Statistical Association.

Montgomery, Douglas C. 2005. Introduction to statistical quality control. New York, NY: John Wiley.Morganstein, David, and Morris Hansen. 1990. Survey operations processes: The key to quality improve-

ment. In Data quality control. Theory and pragmatics, eds. Gunar E. Liepinks and V. R. R. Uppuluri, 91–104. New York, NY: Marcel Dekker.

Morganstein, David, and David A. Marker. 1997. Continuous quality improvement in statistical agencies. In Survey measurement and process quality, eds. Lars E. Lyberg, Paul Biemer, Martin Collins, Edith D. De Leeuw, Cathryn Dippo, Norbert Schwarz, and Denis Trewin, 475–500. New York, NY: John Wiley.

Mudryk, Walter, Bob Bougie, and Hansheng Xie. 2002. Some guidelines for data analysis in quality con-trol. Statistics Canada Technical Report. Ottowa, Ontario, Canada: Statistics Canada.

Raghunathan, Trivellore E., Jerome P. Reiter, and Donald B. Rubin. 2003. Multiple imputation for statis-tical disclosure limitation. Journal of Official Statistics 19 (1): 1–16.

Raghunathan, Trivellore, Dawei Xie, Nathaniel Schenker, Van L. Parsons, William W. Davis, kevin W. Dodd, and Eric J. Feuer. 2007. Combining information from two surveys to estimate county-level prevalence rates of cancer risk factors and screening. Journal of the American Statistical Association 102 (478): 474–86.

Rubin, Donald. 1987. Multiple imputation for nonresponse in surveys. New York, NY: John Wiley.Sakshaug, Joseph W., Mick P. Couper, and Mary Beth Ofstedal. 2010. Characteristics of physical measure-

ment consent in a population-based survey of older adults. Medical Care 48 (1): 64–71.Särndal, Carl-Erik, and Sixten Lundström. 2008. Assessing auxiliary vectors for control of nonresponse bias

in the calibration estimator. Journal of Official Statistics 34 (2): 167–91.Schmucker, Alexandra, and Martina Huber. 2009. Linking process generated employment data with survey

data: An assessment of the linkage and response quality. In FDZ Methodenreport 5/2009. Nuremberg, Germany: Institut für Arbeitsmarkt und Berufsforschung.

Schouten, Barry, Fannie Cobben, and Jelke Bethlehem. 2009. Indicators for the representativeness of survey response. Survey Methodology 35 (1): 101–13.

Shewhart, Walter A. 1931. Economic control of quality of manufactured product. New York, NY: Van Nostrand.

Smith, Tom W. 2011. The report of the international workshop of using multi-level data from sample frames, auxiliary databases, paradata and related sources to detect and adjust for nonresponse bias in surveys. International Journal of Public Opinion Research 23 (3): 389–402.




Statistics Sweden. 2004. Quality assurance of interview work at Statistics Sweden. Project Report SÄk. Stockholm: Statistics Sweden (in Swedish).

Wagner, James. 2008. Adaptive survey design to reduce nonresponse bias. PhD diss., Institute for Social Research, University of Michigan, Ann Arbor.

Wagner, James. 2010. The fraction of missing information as a tool for monitoring the quality of survey data. Public Opinion Quarterly 74 (2): 223–43.

West, Brady. 2011. The error properties of interviewer observations and their implications for nonresponse adjustment of survey estimates. PhD diss., Institute for Social Research, University of Michigan, Ann Arbor.



Documents

Facing the Nonresponse Challenge