Embed Size (px)
Citation preview
A Comparison of Cognitive Testing Methods and Sources:
In-Person versus Online Nonprobability and Probability
MethodsJessicaL.Holzberg,GersonMorales,Aleia ClarkFobia,
andJenniferHunterChildsU.S.CensusBureau
QDET2November11,2016
Disclaimer:AnyviewsexpressedarethoseoftheauthorsandnotnecessarilythoseoftheU.S.CensusBureau.
Motivation§ Wecangetvaluablecognitivetestingfeedbackusingunmoderated,onlineservices(e.g.,Edgar2013;Fowleretal.,2015;Cooketal.,2015)
§ Feedbackcanvarybynonprobabilitysamplesource(e.g.,Murphy,Edgar,&Keating,2014)
§ Onlineopt-innonprobabilitysamplessometimesdemographicallyskewed§ Maynotbeviableforsometypesofcognitivetesting
2
Motivation (cont.)§ Presentstudyisacomparisonbetween:
§ Twoonlinenonprobabilitymethods§ Traditionalin-personcognitiveinterviews§ Probabilitysample(coldcontact)
§ ResearchQuestions§ Howdoconclusionsaboutcomprehensionfromcognitivetestingfeedbackdifferbysamplesource?
§ Howdothedemographiccharacteristicsofrespondentsdifferbysamplesource?
3
Cognitive testing§ Assessedcomprehensionof36officialCensusBureaumessages*§ “TheCensusBureauwillneveruseyourresponsesforanythingotherthanstatisticalresearch.”
§ “Wewillnevershareyourinformationwithlawenforcementorallowittobeusedtodetermineyoureligibilityforgovernmentbenefits.”
*seeFobia andChilds,2016AAPOR
4
Instrument for online testing§ Respondentswererandomlyshown9messageson9screens,inrandomorder:§ Fivemessagesonprivacyandconfidentiality§ Onemessagefromeachoffoursub-sectionsonrequiredlanguage(burden,mandatoryresponse,OMBnumber,otherconfidentialityprotections)
§ Aftereachmessage,respondentswereasked,“Inyourownwords,whatisthismessagetellingyou?”withanopen-endedtextbox
§ Demographicquestions
5
Data sources (cont.)Nonprobab.Census opt-in
Nonprobab.MTurk
Prob. In-person
Description Emailaddressessigneduptobeinresearchoncensus.gov
Crowd-sourcing site;previouslyusedinonlinecognitivetesting
EmailaddressesmatchedtomasterlistofU.S.addresses
Think-aloud,concurrentlyprobedinterviewsonasubsetofthemessages
#Responses 303(8% RR) 200 330(3%RR) 30
Incentive - $0.50 - $40Datacollection
Twoweeks Twohours Twoweeks Overamonth
6
Analysis§ Comprehension?
§ Qualitativeassessmentoffeedback§ Codingofresponses
§ Diversityofrespondents?§ Demographicsofrespondents
7
Coding comprehension§ Understood§ Misunderstood
§ Misinterpretsthemeaningofthemessagebyparaphrasingincorrectly
§ Explicitlystatesthattheydonotunderstandthemessage
§ Requestsclarification§ Comprehensionofthemessage=
#𝑢𝑛𝑑. 𝑐𝑜𝑑𝑒𝑠#𝑢𝑛𝑑. 𝑐𝑜𝑑𝑒𝑠 + #𝑚𝑖𝑠𝑢𝑛𝑑. 𝑐𝑜𝑑𝑒𝑠
8
Findings: Comprehension
9
Online v. in-person generally§ Online
§ Morenegativity§ Moreoff-topicresponsesthatwecouldnotcodeforcomprehension
§ In-person§ Morerequestsforclarification
10
Comprehension§ Formostmessages,generalconsensusacrosssamplesourcesincomprehension§ Leadstosimilarconclusionaboutclarityofmessage
§ Sometimesunpredictablespikesinnegativityandoff-topicresponses
11
“Very few authorized individuals actually see your name or other personal information that could identify you. Most of the time, personal information that could identify you is removed from the file that contains your census or survey answers.”
12
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Und. Misund.
Census(nonprob.)
MTurk(nonprob.)
Prob.
Inperson
Mandatory response§ Short:Youarerequiredbylawtorespondtothecensus(Title13U.S.CodeSections141and193).
§ Wordy:CollectionoftheinformationismandatoryandiscollectedunderTitle13U.S.CodeSections141and193.
13
Mandatory response (cont.)
14
Short Wordy
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Und. Misund.
Census(nonprob.)
MTurk(nonprob.)
Prob.
Inperson
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Und. Misund.
Census(nonprob.)
MTurk(nonprob.)
Prob.
Inperson
Comprehension results (cont.)
§ Therewereafewinstanceswhereoneortwosourcesdifferfromtheotherincomprehension§ Noclearpatternacrosssamplesources§ Especiallyofconcernwhenin-persondiffersfromonline
15
“By law, we only allow access to data to conduct research that would help carry out the Census Bureau’s mission and benefit the public good.”
16
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Und. Misund.
Census(nonprob.)
MTurk(nonprob.)
Prob.
Inperson
Findings: Demographics of respondents
17
Age
18
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%
100.0%
18-24 25-34 35-44 45-54 55-64 65-74 75+ Samplesource
Census(non-prob.),n=228
Mturk(nonprob.),n=183
Prob.,n=224
Race
19
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%
100.0%
White Black AIAN Asian NHOPI 1+race Samplesource
Census(non-prob.),n=221
Mturk(nonprob.),n=170
Prob.,n=223
Education
20
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%
100.0%
Samplesource
Census(non-prob.),n=226
Mturk(nonprob.),n=183
Prob.,n=223
Sex
21
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%
100.0%
Male FemaleSamplesource
Census(non-prob.),n=228
Mturk(nonprob.),n=183
Prob.,n=224
Demographics of respondents: in-person
§ Obviously,abilitytobemoreselective§ Weneededtotestwith:non-white,lesseducated,men§ 22/30black§ 24/30somecollegeorless§ Stillendedupwith20/30women!
22
Conclusion§ Feedbackoncomprehension
§ Formostmessages,nomajordifferencesinconclusiononcomprehensionbyonlinesamplesource
§ Generallymorenegativityandoff-topicresponsesonline,fewerrequestsforclarification
23
Conclusion (cont.)§ Demographicsofrespondents
§ Useofprobabilitysamplenotaclearimprovementformostdemographics
§ Ifyou’reonlyusingoneonlinesample,demodifferencesmaybeusedtoguidechoice,dependingonwhatyouaretesting
24
Future Research§ Lengthofresponsesandotherqualitymeasures
§ Demographicdifferencesinresponses?§ Diggingintodiscrepanciesandusingothertypesofprobes
§ Inthemeantime:usein-person,too!
25
Thank you!AComparisonofCognitiveTestingMethods
andSources:In-PersonversusOnlineNonprobabilityandProbabilityMethods
JessicaL.Holzberg,GersonMorales,Aleia ClarkFobia,andJenniferHunterChilds
U.S.CensusBureau
26
