Embed Size (px)
DESCRIPTION
From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment. Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler United States Bureau of Economic Analysis. Overview – Redesign of BEA Form BE-605. - PowerPoint PPT Presentation
Citation preview
From Start to Pilot: Comprehensive Redesign of a
Questionnaire Measuring Foreign Direct Investment
Alfred D. Tuttle
Rebecca L. Morrison
United States Census Bureau
David H. Galler
United States Bureau of
Economic Analysis
2
Overview – Redesign of BEA Form BE-605
• Reimbursable project with BEA
• Multiple, intensive stages of research, design, testing, revision
• Qualitative and quantitative evaluations
• New BE-605 form mailed out in March 2007
3
Outline
• Background – BEA Form BE-605
• Summary of stages of research
• Findings
• Next steps
4
Background
• BE-605 (quarterly) and BE-15 (annual)
• Foreign Direct Investment (FDI) – >10% of voting rights in an incorporated entity (or equivalent interest in unincorporated entity)
• Sample – foreign-owned businesses in US
5
Research stages and methods
• Background investigations– Expert review of form– Focus groups with survey analysts– Observations of callbacks– Respondent debriefings
6
Research stages and methods
• Cognitive testing of mockups
7
Research stages and methods
• Pilot test – Respondent debriefings – Evaluation questions
8
Findings from background research
Data collection and design issues:
• Dense formatting
• Economic accounting concepts differ from US GAAP
• Separate instructions
9
Recommendations
• Formatting recommendations– Letter size pages (not legal)– More “open” visual design– Incorporate instructions into form:
• within questions• adjacent to questions on facing page• key instructions become questions
– Diagrams to augment questions
10
Original form
11
Recommendation: More “open” design
12
Recommendation: Embedded instructions
13
Recommendation: Adjacent instructions
14
Recommendations: Questions from instructions, and diagrams
15
Cognitive testing
• Problematic sections mocked-up and tested (not entire forms)
• 5 rounds of cognitive interviews
• Iterative process: – mock-ups revised between rounds – design features modified & re-tested
16
Findings from cognitive testing
• New design features were effective:– Majority of respondents liked the more
open format – Embedded instructions would likely
improve data quality– Some “InstructionsQuestions” and
diagrams required extensive modification, though most did not
17
Developing 605 pilot
• Redesigned entire BE-605 form
• Close collaboration with BEA survey staff
18
Pilot test
• Mailed 2nd quarter 2006
• Sent to sub-sample (n = 653; survey sample ~4,000)
• Completing pilot form optional
• Response rate: 53% (350 returned forms) during initial 8-week processing period
19
Pilot form evaluation
• Respondent debriefings
• Evaluation questions in form
20
Findings – R debriefings
• Favorable impressions of new form:– More pleasing to the eye– White reporting spaces easier to see
against colored background – Larger font, bold text, more space– Letter size pages easier to handle– Diagrams were helpful– Embedded instructions helpful
21
Findings – R debriefings
• Drawbacks:– More pages– Booklet format makes it harder to fax– Took longer to complete than original form
(disruption of reporting routine)
22
Limitations
• Stymied by experienced Rs, reporting routines…
• …though even experienced Rs were helped by new formatting and instructions.
23
Findings – Evaluation questions
• Four questions appended to survey
• Each item received between 279 and 290 responses
24
Findings – Evaluation questions
Instruction placement?
79.3% - Near questions
18.3% - In separate booklet
2.4% - Not applicable – I haven’t completed
the usual BE-605
25
Findings – Evaluation questions
Open space and number of pages?
48.7% - less open space, fewer pages
47% - more open space, more pages
4.3% - Not applicable – I haven’t completed the usual BE-605
26
Findings – Evaluation questions
Use of diagrams?
65.4% - very helpful or somewhat helpful
11.7% - only a little helpful
18% - did not need them
3.9% - confusing
27
Findings – Evaluation questions
Easy/difficult compared to usual form?
50.5% - harder
42.5% - easier
7% - Not applicable – I haven’t completed the usual BE-605
28
Findings – Evaluation questions
• Instructions preferred near questions
• Organization charts helpful
• Easier or more difficult?
• Preference for more/fewer pages?
29
Summary
• Redesigned BE-605 went into production in March 2007– Recommended analysis of edit checks
during form processing
30
Summary
• Multi-method approach to comprehensive survey redevelopment
