Optimal Adaptive Survey Design

Lars Lyberg, Frauke Kreuter, and James

WagnerITSEW 2010

Stowe, VT, USA, June 16

The Survey Process

Research Objectives

SamplingDesign

Data Collection

Data Processing

Analysis/Interpretation

Concepts Population

Mode of AdministrationQuestions

Questionnaire

revi

se

revi

se

What Should Be Designed? Requirements+specifications+oper

ations Ideal goal+ Defined goal+Actual

results Good survey design means control

of accuracy through the specs (QA) and control of operations (QC)

Some Early Thinking

Hansen-Hurwitz-Pritzker 1967 Take all error sources into account Minimize all biases and select a minimum-

variance scheme so that Var becomes an approximation of (a decent) MSE

The zero defects movement that later became Six Sigma

Dalenius 1969 Total survey design

Some More Thinking Textbook on total survey design

Hansen-Hurwitz-Cochran-Dalenius Survey models and specific error sources Cochran’s comment from 1968

Alternative Criteria of Effectiveness

Minimizing MSE for a given budget while meeting other requirements

Maximizing fitness for use for a given budget

Maximizing comparability for a given budget

All these reversed Something else?

The Elements of Design Assessing the survey situation (requirements) Choosing methods, procedures, “intensities”, and

controls (specifications) Allocating resources Assessing alternative designs Carry out one of them or a modification of it Have a Plan B

So, What’s the Problem? No established survey planning

theory Multi-purpose, many users The information paradox Uninformed clients/users/designers Much design work is partial, not total Limited knowledge of effects of

measures on MSE and cost

More Problems Decision theory and economics

theory not used to their potential New surveys conducted without

sufficient consideration of what is already known

No one knows the proper allocation of resources put in before, during and after

The literature is small

Various Skills Needed Which Calls for a Design Team Survey methodology Subject-matter Statistics (decision theory, risk

analysis, loss functions, optimization, process control)

Economics (cost functions, utility) IT

The Adaptive Element The entire survey process should be

responsive to anticipated uncertainties that exist before the process begins and to real time information obtained throughout the execution of the process

or Use process data (paradata) to check, and

if necessary, adjust the process

We Should Assemble What We Know

Assessment methods Design principles Trade-offs and their effects The potential offered by other

disciplines We shouldn’t accept partial

designs

Apply Design Principles If pop is skewed then…. If pop is nested then…. If questions are sensitive then…. If a high NR rate is expected

then…

Apply SOPs, CBMs or Best Practices Part of the design is to use known,

dependable methods

Examples of Trade-offs Accuracy vs timeliness Response burden vs wealth of detail Conduct survey vs other information

collection Large n vs smaller n Mixed vs single mode NR bias vs measurement error NR vs interpretation by family members

Process view Upstream thinking (prevention) Understanding variation Measure cost of poor quality and

waste Intervention or improvement actions

should be based on good data and statistical analysis

Continuous monitoring

Tentative Course Syllabus The elements of design Real world examples (e.g., CPS

Technical Paper 63, PIAAC, the Monthly Retail Trade Survey, the Annual Survey of Hale Mountain Fish & Game Club, VT)

The literature on optimal decisions Theory for adaptive treatment design

and risk management

Course syllabus continued Data for monitoring and decision

making Analysis of such data Design lessons learned Examples of bad designs and not so

great trade-offs Student project with TSE perspective Student presentations