Embed Size (px)
Citation preview
Arthur LUPIA University of Michigan
CHALLENGES AND OPPORTUNITIES
IN OPEN-ENDED CODING
Matthew Berent Matthew DeBell Jon Krosnick Arthur Lupia Language Logic ANES staff and several ANES expert committees
BASED ON WORK BY…
1. Background & Challenges
2. Example: “Political Knowledge”
3. General Attributes of Our Approach
4. Example: “Most Important Problem” If time permits
5. Conclusion
OUTLINE
The “gold standard” of election studies.
The empirical basis of many scholarly books and articles.
Founded at Michigan, now working with Stanford.
ANES OVERVIEW
FIELD PERIOD Pre-election: September 2 - November 3, 2008
(N=2323) Post-election: November 5 - December 30, 2008
(N=2102)
164 minutes of interview time Continues hundreds of core questions Adds hundreds of new questions
ANES TIME SERIES STUDY 2008
“most important problem”
candidate “likes-dislikes”
party “likes-dislikes”
ANES OPEN-ENDED QUESTIONS
“Now we have a set of questions concerning various public figures. We want to see how much information about them gets out to the public from television, newspapers and the like….
What about … William Rehnquist – What job or political office does he NOW hold?”
RECALL QUESTION
…expect ANES to convert OE answers to numbers
…use these numbers to draw inferences
…base inferences on beliefs about what each number means
ANES USERS…
Many users believe that open-ended coding is easy to do, generates valid measures, and is performed well by survey organizations
BELIEFS
We discovered a different reality at ANES
…and its practices were not unusual
THE PROBLEM
What is the correct inference to draw from open-ended responses to survey questions?
The answer depends on What we ask What they say Decisions that we make after an interview is conducted.*
OUR FUNDAMENTAL Q&A
DEFINITIONS OF PROGRESS
CREDIBILITY
the quality of being believable or trustworthy
Example “Social scientists seek to
offer credible explanations.”
LEGITIMATE
in accordance with recognized or accepted standards or principles
Example “Social science claims that
are inconsistent with the scientific method are less often seen as legitimate.
Increase Procedural transparency Documentational rigor Credibility of measures & inferences
GOAL
“political knowledge”
OUR FIRST SIGN OF TROUBLE…
“Close to a third of Americans can be categorized as “know-nothings” who are almost completely ignorant of relevant political information
which is not, by any means, to suggest that the other two-thirds are well informed….”
CRITICAL REVIEW (2006)
“The verdict is stunningly, depressingly clear:
most people know very little about politics…”
LUSKIN (2002, 284)
2004 ANES
“Now we have a set of questions concerning various public figures. We want to see how much information about them gets out to the public from television, newspapers and the like…. What about … William Rehnquist – What job or political office does he NOW hold?”
12% “correct.”
www.umich.edu/~lupia
GIBSON-CALDIERA (2009)
Recall questions asked in an OE format.
Codes released to users.
“Verbatim” responses never released… But can be accessed through RDA
ANES POLICY PRIOR TO 2008
2004 G-C Analysis
ANES 2004: Correct only if “chief justice” and “Supreme Court”
Another 30% identified him as a Supreme Court justice, but were marked “incorrect.”
www.umich.edu/~lupia
GIBSON-CALDIERA (2009)
2004 G-C Study
Respondents asked to state whether Rehnquist, Lewis F. Powell, or Byron R. White was Chief Justice.
71% correctly selected Rehnquist.
www.umich.edu/~lupia
GIBSON-CALDIERA (2009)
400 of the 1,555 respondents either
said that Rehnquist was a judge or said that he was on the Supreme Court
and yet were coded as having answered incorrectly
2000 ANES
Supreme Court justice. The main one. He’s the senior judge on the Supreme Court. He is the Supreme Court justice in charge. He’s the head of the Supreme Court. He’s top man in the Supreme Court. Supreme Court justice, head. Supreme Court justice. The head guy. Head of Supreme Court. Supreme Court justice head honcho.
“INCORRECT” ANSWERS (2000)
“…Tony Blair, What job or political office does he NOW hold?”
WE ALSO FOUND AN ERROR
“The reference must be specifically to ‘Great Britain’ or ‘England’ -- United Kingdom is *NOT* acceptable (Blair is not the head of Ireland), nor is reference to any other political/geographic unit (e.g. British Isles, Europe, etc.)
2004 CHANGE
did this happen? HOW
Interviewer transcribes response
Staff implements coding scheme weeks after interview
No record of instructions to staff
No documentation of reliability analyses
TYPICAL ANES CODING PRACTICE
OUR INITIAL RESPONSE
A basic expectation is to
document, archive, and share all data and methodology
so that they are available for careful scrutiny by other scientists.
PRINCIPLE
[Scientific integrity] corresponds to a kind of utter honesty—a kind of leaning over backwards….
In summary, the idea is to give all of the information to help others judge the value of your contribution; not just the information that leads to judgment in one particular direction...
RICHARD FEYNMAN (1974 – CALTECH COMMENCEMENT ADDRESS)
Redacted transcripts available
Conduct conference to discover best practices
Work with expert committees to develop coding schemes MECE replicable
ANES O-E NEW PRACTICES
Which responses are correct?
Which responses are incorrect?
Which responses constitute “partial knowledge?”
FIRST COMMITTEE
Did R give a correct answer to the question that was asked?
“What job or political office does he NOW hold?”
BREAKTHROUGH
Political Office Part of the title identified correctly Part of the title identified correctly and incorrect
statements about the title
Job Descriptions of the job
“Other” Responses not pertaining to job or political office
NEW ANES RECALL CODING FRAMEWORK
No judgments about truth-values.
Q not designed to elicit general knowledge
For general recall queries, different questions needed.
“OTHER” RESPONSES
Theoretically Defensible
High Inter-coder reliability
MECE
Scholars can use public data to compare other code frames.
ATTRIBUTES OF NEW CODING SCHEME
GENERAL ATTRIBUTES OF OUR APPROACH
Theoretical Framework Developed with expert committees
Code Frame Verified with expert committees
Chunking Developed in cooperation with vendor
Coding Executed by vendor with rigorous evaluation
HOW WE DID IT
We sought Written correspondence with working groups Written correspondence with coding vendor Written documentation of all decisions Written documentation of all conversations Multiple independent assessments of
decisions
To enhance legitimacy, we post everything.
IDEAL DOCUMENTATION
Documentation and validation are time consuming and expensive.
Premise: the ideal is worth approaching, even if it cannot be reached.
CHALLENGES
IF YOU EVER HAVE A QUESTION ABOUT WHAT YOU SHOULD DO, FILL OUT A “QUESTION FORM” AND GIVE IT TO YOUR SUPERVISOR. Your supervisor will get an answer to your question and pass it along to you.
A COMPLETE RECORD OF CORRESPONDENCE
Increased documentation at all stages
Evaluation at many stages
Increased procedural transparency
High inter-coder reliability
OUR CURRENT PRACTICES
“Most Important Problem”
SECOND EXAMPLE
What do you think is the most important political problem facing the United States today?
MIP
New categories added yearly. Categories modified from year-to-year.
2000 “EDUCATION; financial assistance for schools/colleges/students; quality of education/the learning environment/teaching”
2004: modified to include “the high cost of college”
2004: 154 categories.
MIP CHALLENGES
154 categories. No clear theoretical framework Not MECE No written instructions or validation statistics Users do not use original categories Only 14 categories attracted more than 5
answers
One category attracted 447 answers.
ANES MIP 2004
Matt Berent interviewed Gallup, Pew, Quinnipac, AP/IPSOS & NYT about the origin and maintenance of their MIP codes.
SURVEY OF MIP PRACTICES
Limited code frames
No set rule about addition & subtraction
Code frames often modified after data collected
No analysis of coding reliability or validity.
SURVEY OF MIP PRACTICES
A coding scheme must be defined with respect to a theory of language and meaning.
HOW TO CHOOSE CODES
We sought a MECE frame that is stable, replicable, and reflects common theoretical concerns
Base: Federal Budget Categories
Second: “Rule of Two”
NEW ANES MIP CODE FRAME
Stable since 1940
Every federal governmental program and activity is listed within this framework
Categorize all major federal government functions
FEDERAL BUDGET SUPERCATEGORIES
FEDERAL BUDGET CATEGORIES
“Any category used by two or more of ANES, AP/IPSOS, Gallup, Pew, Quinnipiac, and The New York Times
“Rule of two” defined subcategories within federal budget super-categories.
“RULE OF TWO”
MECE
Derivable from a transparent logic
High intercoder reliability achieved.
ADVANTAGES OF NEW CODE FRAME
Documentation and validation are time consuming and expensive.
Science and society benefits from rigorous public accounts of how we produce our data.
CONCLUSION
