Embed Size (px)
Citation preview
Collaborating for Quality:A Cross-Discipline Approach to
Questionnaire Content Evaluationin Business Surveys
Diane K. Willimack
Peter GibsonU.S. Census Bureau
3
Outline
• Background – Business surveys – Response process
• Questionnaire development, pretesting & evaluation– Why collaborate?– An example– Benefits of the collaborative approach
4
• Nature of the collected data– Financial– Technical definitions
• Expect data to be in records
Business Surveys (1)
5
• Nature of response– Person answers survey questions for the
business– Cognitive response process
Business Surveys (2)
6
The Response Process in Business Surveys
Cognitive response model
PLUS
Organizational processes
7
Tourangeau’s (1984)
Cognitive Response Model
1. Comprehension
2. Retrieval
3. Judgment
4. Communication Survey
8
Organizational Aspects
4. Comprehension
5. Retrieval
6. Judgment
7. Communication
1. 2. 3.
8.
Business Survey
Response Process in Business Surveys (1) (Sudman et al., 2000)
9
Encoding in Memory / Record FormationSelection / Identification of Respondent(s)Assessment of Priorities (Motivation)
4. Comprehension
5. Retrieval
6. Judgment
7. Communication
8.
1. 2. 3.
Release of the Data
Business Survey
from Memory and / or Records
Response Process in Business Surveys (2) (Sudman et al., 2000)
10
Encoding in Memory / Record FormationSelection / Identification of Respondent(s)Assessment of Priorities (Motivation)
4. Comprehension
5. Retrieval
6. Judgment
7. Communication
8.
1. 2. 3.
Release of the Data
Business Survey
from Memory and / or Records
Response Process in Business Surveys (3) (Sudman et al., 2000)
11
• Management needs• Regulatory requirements• Accounting standards
• Knowledge domain of experts in accounting practices
Record Formation
12
RespondentRecords
Accountant
Putting the Pieces Together for Quality Response
13
• Answers survey questions for the business– Cognitive response process
• Expect respondent to know –– What is in records– How to get them
• Knowledge domain of cognitive survey methodologists
The Respondent in Business Surveys
14
Encoding in Memory / Record FormationSelection / Identification of Respondent(s)Assessment of Priorities (Motivation)
4. Comprehension
5. Retrieval
6. Judgment
7. Communication
8.
1. 2. 3.
Release of the Data
Business Survey
from Memory and / or Records
Response Process in Business Surveys (4) (Sudman et al., 2000)
15
Questionnaire Development, Pretesting & Evaluation
• Challenges– Technical definitions– Data availability – Labor-intensive response process
• Expertise– Subject matter– Accounting
practices– Cognitive survey
methodology
16
RespondentRecords
Cognitive Survey
Methodologist
Accountant
Putting the Pieces Together for Quality Response: Cross-Discipline Collaboration
17
Questionnaire Development & Pretesting
Methodologist role• Cognitive testing
– Non-directive probes get at cognitive steps
• Shortcoming:– Limited by labor-
intensive response process
– Unable to access records
Accountant role• Aid stakeholders in
specifying constructs– Improve initial
questions– Align with records
• Review draft form & protocol
• Shortcoming:– Directive approach
18
Questionnaire Evaluation
Methodologist role• Respondent
debriefings– Post-collection– Response
strategy• Shortcoming:
– Does not re-create construction of the response
– Does not validate the response
Accountant role• Review reported
data– Comparison to
public data sources
– Aid identifying & correcting reporting errors
• Shortcoming:– Lacks contact
with respondent
Cross-Discipline Collaboration:
Methodologist + Accountant
An Example
19
Example Draft Question
1. What is the amount of licensing fees or royalty income received by your company’s domestic operations?
2. Of the amount reported in item 1, how much was from:
a) Patents?
b) Copyrights?
c) Computer Software?
d) Other (specify _______________)20
Example Cognitive Protocol1. Tell me in your own words what the terms “licensing
fees” and “royalty income” mean to you in the context of this question.
2. Please explain how these are similar or different.
3. What sorts of “royalty income” or “licensing fees” does your company receive?
4. How would you go about reporting “royalty income” or “licensing fees” for your domestic operations? What records would you use?
5. How would you go about reporting the amounts earned from patents, copyrights, etc.? What records would you use?
21
Example Technical Issues Identified by the Accounting Expert
1. Forms of Intellectual Property (IP) tracked in the corporate legal entity that directly owns the IP.
2. The legal office structures the IP licensing arrangements (and not the Financial area).
3. Contracts consist of several forms of IP.
4. The company’s IP might be enhanced by the licensee and resulting income combines the original with the enhanced.
5. Attributing income to forms of IP (e.g., patents, copyrights) is a theoretical exercise. Actual pricing reflects opportunity costs, present value, profit potential, value of future development.
22
Example Enhanced Protocol with Technical Probes from the Accountant1. Do you have “patents, trademarks, copyrights?”
2. Are these IP owned directly by the domestic legal entity’s corporate or subsidiary business units?
3. Is any of this IP owned by a foreign subsidiary?
4. Do any of the “licensing fee or royalty income” accounts contain a coding for income tax purposes?
5. Would there be a difference between the amount you would report for “world-wide” or “domestic?”
6. Who handles licensing arrangements in sub-units?23
Example Outcomes from the Collaborative Interview• Burden:
– Reporting total domestic “licensing fees and royalty income” depends on data access.
– Analysis of each licensing arrangement needed to attribute income to individual forms of IP (e.g., patents, copyrights).
– Would not provide robust data.• Clarification of the construct being measured
– Stakeholder reduced requirements, dropped questions• Protocol
– Cognitive methodologist strengthened probes of technical issues vis-à-vis data availability
24
25
Methodologist + Accountant = Robust Pretest Interviews
Methodologist role• More comfortable
with technical nuances
• More effective probing
Accountant role• Gained insights
into variety of respondents and reporting issues
26
Methods• Probing
anticipated problems
• Prepared to investigate hypotheses about specific terms and concepts
Results• Respondent’s
initial response augmented with reaction to possible alternatives
• Insights into burden and data quality
Methodologist + Accountant = Robust Pretest Interviews
27
Benefits of Cross-Discipline Collaborative Approach
• Improved draft questionnaire for pretesting• More effective pretest interviews• More efficient pretesting strategy• More robust results and useable
recommendations• Additional expertise for review of reported
data
