View
214
Download
1
Embed Size (px)
Citation preview
The Future of Survey Research:A Panel Discussion
Robert GrovesRonald Langley Burke Grandjean
March 6, 2008Survey Research Center
University of California, BerkeleyBerkeley, California
Homogeneity-Heterogeneity,Batch-Flow, Fixed-Adaptive,
Centralization-Decentralization
Robert M. GrovesUniversity of Michigan and
Joint Program in Survey Methodology
Four Powerful Social and Economic Forces
1. From homogeneity to heterogeneity
2. From batch to flow
3. From fixed to adaptive
4. From centralized to decentralized to… a third way?
Homogeneity to Heterogeneity: Views of Society
• Traditional assumption that democracies and justice systems demanded equal treatment to achieve equitable treatment
• Increasing observation that modern societies are far from homogeneous– Note: this itself is a finding of large scale
surveys
Inter-Nation Migration
• The past few years have seen large increases in migration across national boundaries
• These appear to be driven by macro-economic and demographic shifts– low fertility rates and aging of rich
countries– migrants from poorer countries attracted
by job markets of rich countries
Gross Inter-Nation Migration 1950-2040
United Nations Department of Economic and Social Affairs/Population Division 83World Population Prospects: The 2004 Revision, Volume III: Analytical Report
Social Forces
• This migration gives rise to changes– impact on educational systems– civic participation in “old” and “new”
social organizations– new political cleavages
Heterogeneity in Marketing
“Segmentation” partitions a general population with the goal of tailoring actions toward specific subgroups. It is a way of organizing customers into groups with similar traits, performance characteristics or expectations.
Edward J. Hass, Ph.D. An Overview of Segmentation: Why You Should Consider It And
a Thumbnail of Its Dynamics
http://www.icrsurvey.com/docs/segmentation_white_paper_final_111505.doc
Heterogeneity in Survey Sample Design
• Common practice in 1960-1980 to have equal probability samples in US surveys
• Common practice now to have special population surveys or full population surveys with disproportionate allocation
Heterogeneity in Respondent Recruitment
• Common in 1970’s-1980’s research to look for the single best argument to present to all sample units
• Common now to attempt to tailor argument to characteristics and concerns of the sample unit
Heterogeneity in Survey Measurements
• Common in the 1970-80’s to insist on uniform measurement for all respondents
• Suchman and Jordan’s (1990) attack that standardized wording may not produce consistent meaning
• Conversational interviewing (1997)
• Mixed-mode data collection
Note: CAI enabled this
Homogeneity to Heterogeneity: Conclusion
• The more we learn, the more we see heterogeneity
• The more we see heterogeneity, the more our survey tools must be suitable for the diversity we see
1. From homogeneity to heterogeneity
2. From batch to flow
3. From fixed to adaptive
4. From centralized to decentralized to… a third way?
Four Powerful Social and Economic Forces
Batch to Flow
• Much 20th century manufacturing– sequential– small steps– many “handoffs” across work groups– low skill levels, narrow focus
• Increasingly– system-thinking– project design, integrated skill sets– whole product development– simultaneous work processes
Batch to Flow – Commercial enterprises
• “Just in Time” supply chains– to create consistency in assembly schedule
• process flow production/ “lean manufacturing”– minimal interruptions in actual processing
between runs
– queue time is virtually eliminated by integrating the moving of the product into the actual operations of the resource performing the work
• Transaction-based processing systemsDaina Dennis; Jack Meredith (Aug., 2000), “An Empirical Analysis of Process Industry Transformation Systems,”
Management Science, Vol. 46, No. 8. pp. 1085-1099.
Batch to Flow: Survey Designs
• Move from discrete, repeated cross-sections to continuous interviewing– National Health and Nutrition Examination
Survey– National Survey of Family Growth– American Community Survey– UK Continuous Population Survey
Batch to Flow: Summary
• Computer-assistance gains its power by unifying former disparate batch steps
• Most CAI has focused on unifying question delivery, data input, editing, and data delivery
• This leaves sample design, sample unit recruitment, field administration, estimation
1. From homogeneity to heterogeneity
2. From batch to flow
3. From fixed to adaptive
4. From centralized to decentralized to… a third way?
Four Powerful Social and Economic Forces
Fixed Designs, Fixed Production
• Common for manufacturing and surveys to design all the details then construct the product in a unchanging fashion until the sufficient volume is achieved
Adaptive -- Clinical Trials
In a classical clinical trial, patients are allocated to one of two different treatment options with half being assigned to each therapy. At the end of the experiment a decision is made as to which treatment is more effective.
In contrast, in an adaptive clinical trial, patient outcomes can be used as they become available to adjust the allocation of future patients or some other aspect of the study design. This allows researchers to improve expected patient outcomes during the experiment, while still being able to reach good statistical decisions in a timely fashion.
Scott Gottlieb, 2006, 2006 Conference on Adaptive Trial Design,Washington, DC http://www.fda.gov/oc/speeches/2006/trialdesign0710.html
Adaptive -- Computer Science
Adaptive designs also arise naturally in computer-related areas such as computer systems resource allocation or parameter selection for repeated computer simulations, since the computer can both collect data and run a program to decide what to do next.
Adaptive -- Commercial products
Build-to-Order is the capability to quickly build standard or mass-customized products upon receipt of spontaneous orders without forecasts, inventory, or purchasing delays. These products may be shipped directly to individual customers, to stores or dealers, or as a response to assemblers’ "pull signals" (assemblers’ signals that certain parts are needed right away for assembly).
David M. Anderson, (2004)o-Order & Mass Customization; the Ultimate Supply Chain Management and Lean Manufacturing Strategy for Low-Cost On-Demand Production without Forecasts or Inventory,"
Adaptive – Responsive Survey Design
1. Preidentify a set of alternative features potentially affecting costs and errors of statistics
2. Identify a set of indicators of the cost and error properties of those feature
3. Monitor indicators in initial stages of data collection
4. Alter the active features of the survey based on cost/error tradeoff decision rules
5. Combine data from separate phases into a single estimator
Examples of Responsive Survey Design Features
• Double samples for nonrespondents without shutdown of field operations
• Dynamic adjustment of callbacks to sample units most underrepresented in the data set
• Dynamic assignment of incentive values
• Change of mode of data collection based on empirical triggers
From Fixed to Adaptive: Summary
• Adaptive requires information that identifies heterogeneous sets
• Surveys have developed over the years to– use sampling frames that offer good
coverage, whether or not they have rich variables
– limit data collected to substantive variables
1. From homogeneity to heterogeneity
2. From batch to flow
3. From fixed to adaptive
4. From centralized to decentralized to… a third way?
Four Powerful Social and Economic Forces
Centralization
• Hierarchies
• Command and control
• Downward communication
• Maximize information at the top of the hierarchy
Decentralization
• Flat structure
• Local autonomy
• Local knowledge communicated upward
• Only common information needed at the top
The Growth of “Third Ways”
• Outsourcing of responsibilities to experts
• Centralization of production information shared by all
• Separation of common good actions from local optimization actions
The False Dichotomy of Centralization and Decentralization• How can an organization be both
centralized and decentralized?– Centralized information about dispersed
production– Dispersed use of centralized information
Third Way: Surveys
• Use of local supervision for motivation and skill-transmission
• Use of centralized data bases to guide targeted deployment of resources to benefit the common good
Four Powerful Social and Economic Forces
1. From homogeneity to heterogeneity
2. From batch to flow
3. From fixed to adaptive
4. From centralized to decentralized to… a third way?
Challenges from these Forces
• The customized survey design needs new paradata to tailor designs to sample units
• Software functionality must be designed with responsive design in mind– real-time analysis of paradata– integrated statistical analyses to direct question
flow or recruitment step actions
• This requires centralization of information resources, but dispersed use of the resources
Technological and Legal Challenges: The View from a
Small Shop
Ronald E. LangleySurvey Research CenterUniversity of Kentucky
The University
of Kentucky
Four Powerful Social and Economic Forces
1. From homogeneity to heterogeneity
2. From batch to flow
3. From fixed to adaptive
4. From centralized to decentralized to… a third way?
Language Problems?
• Nearly 12% of US residents are born outside the US
• 18% speak a language other than English at home
• Nearly 5% do not speak any English at all
Source: National Health Interview Survey/ M. Link – The Neilsen Company
Language Problems?
• Solutions available with enough resources– Translation– Foreign language interviewers– Foreign Language supervisors/monitors?
Other Forces
• Human Subjects Protections and Right to Privacy
• Technological advances
• New laws and regulatory interpretations
Social Climate
• Increasing desire for privacy
• Increasing thirst for data
• Cell Phone more “private” than home phone?
Technology
• Help us adapt– CAI
• Create new challenges to which we must adapt– Privacy Managers
• Answering Machines• Caller ID/Call Blocking
– Cell phones
Legal Challenges
• Laws protecting privacy– FERPA– HIPPA– Human Subjects Protections
• Laws and regulatory interpretations in response to technological advances– Number Portability– TCPA 1991– DNC Implementation Act 2003– CAN SPAM Act
TCPA 1991
• Prohibited using automatic dialers or prerecorded or artificial voice to contact telephone numbers assigned to cell phones………
• Without prior express consent of the called party• Or to any service for which the called party is
charged for the call• Content neutral – includes research surveys
TCPA 1991
• Solicitors can only call between 8 a.m. and 9 p.m.
• Solicitors must maintain DNC list to be honored for 10 years
• Originally ineffective because burden on respondent to get on each telemarketers list
• Fixed by the DNC Implementation Act establishment of National DNC Registry
State Level Legal Challenges
• State DNC lists• Harassment Laws
– Utah– Missouri– Hawaii– Montana
• Time of day restrictions– May not distinguish between solicitation
and research
Cell Phones• Affect our coverage
– Cell only H/H’s
• Affect our Estimates?– BRFSS mixed mode results– Data Quality questions
• Lack of Knowledge about the frame– Little prescreening data– More difficult to “fix” by weighting
• Ethical issues– Respondent safety– Privacy re: sensitive questions– Cost to respondents
The cell phone problem: Trend in percentage ofU.S. households without landline telephones
Source
Source: National Health Interview Survey
Source: National Health Interview Survey/ M. Link – The Neilsen Company
02468
101214161820
1963
1970
1975
1980
1985-1
986199
7200
1200
3
Late 2
006
No Landline Cell Only
Cell Phone Only Households
• Cell phone-only highest among:– Adults with roommates (54%)– 18-30 year olds (~27%)– Renters (26%)
• Does it really make a difference?– Little change in overall estimates when cell
only added in;– BUT … are 18-34 year olds in households with landlines the same as those in cell only
households?
18-34 year olds: Binge drinking &HIV test by household phone access
40
23 22
52
35 34
0
10
20
30
40
50
60
Binge DrinkingPast 30 days
Ever tested for HIV
Cell Only Landline BRFSS
Source: 2006 BRFSS ABS mixed-mode survey (mail survey respondents only)Michael Link – The Neilsen Company
Solutions to Coverage Issue
• Mixed Mode• Collecting better (sub-national) information
about cell phone and cell only user population parameters
• Address Based Sampling– RTI– CDC/BRFSS– Neilsen
• More fully develop Web-based as access increases
The View from a Policy Analysis Shop
Burke D. GrandjeanWyoming Survey & Analysis Center
University of Wyoming
The Future of Our Industry
• Accosted at the factory gates
• Challenges or opportunities?
• Survey information in the Information Age– Data for economic decisions– Data for evidence-based policies
Shifts in Information Processing
1. From homogeneity to heterogeneity- Fine-grained info reveals heterogeneity
2. From batch to flow- Timely info allows “Just in Time” flows
3. From fixed to adaptive- Info feedback is essential for adaptation
4. From centralized to decentralized to …?- The “third way” seeks to minimize the
distance between info and decisions
Supply, Demand, and the Four Forces
• As information suppliers, we tend to focus on supply-side issues related to these four information processes
• Demand-side considerations reflect how our clients (and potential clients) are affected by these same forces
• Some examples …
Fine-Grained Information on Heterogeneity in Thermopolis, WY
Number of residents
Persons in the householdIncome greater than the cut-
off, or equalIncome less than the cut-
off Total 1 8 26 34*
2 26 56 82
3 9 18 27
4 4 24 28
5 0 15 15
6 0 6 6
Total
47 145 192
%
24.5% 75.5% 100.0%
Before and After Laramie’s Smoke-free Ordinance
Laramie Smokefree Attitudes
32%
51%
39%
61%
46%
65%
0%
20%
40%
60%
80%
Bars Restaurants
"Smoking should not be allowed in any areas"
Per
cen
t o
f S
amp
le A
gre
ein
g w
ith
S
tate
men
t
February 2005
July 2005
February 2006
NPS and EPA National Surveys• National Park Service
– Comprehensive Survey of the American Public– NPS and OMB reviews of survey instrument– 3500 telephone respondents– Seven regions, two field periods, dual frame
• Environmental Protection Agency– Willingness to Pay for Clean Air in National Parks– No EPA or OMB review of survey instruments– 3000 total respondents– Mode comparisons: telephone, mail, web