Issued March 2004 Census 2000 Alternative Questionnaire … · 2004. 4. 15. · Census 2000 Synthesis Report No.17 TR-17 Issued March 2004 Census 2000 Alternative Questionnaire Experiment

Census 2000 Synthesis Report No.17TR-17

Issued March 2004

Census 2000AlternativeQuestionnaireExperiment

U.S.Department of CommerceEconomics and Statistics Administration

U.S. CENSUS BUREAU

Census 2000 Testing, Experimentation, and Evaluation Program

The Census 2000 Evaluations Executive SteeringCommittee provided oversight for the Census 2000Testing, Experimentation, and Evaluations (TXE)Program. Members included Cynthia Z. F. Clark,Associate Director for Methodology and Standards;Preston J. Waite, Associate Director for DecennialCensus; Carol M. Van Horn, Chief of Staff; TeresaAngueira, Chief of the Decennial ManagementDivision; Robert E. Fay III, Senior MathematicalStatistician; Howard R. Hogan, (former) Chief of theDecennial Statistical Studies Division; Ruth AnnKillion, Chief of the Planning, Research and EvaluationDivision; Susan M. Miskura, (former) Chief of theDecennial Management Division; Rajendra P. Singh,Chief of the Decennial Statistical Studies Division;Elizabeth Ann Martin, Senior Survey Methodologist;Alan R. Tupek, Chief of the Demographic StatisticalMethods Division; Deborah E. Bolton, AssistantDivision Chief for Program Coordination of thePlanning, Research and Evaluation Division; Jon R.Clark, Assistant Division Chief for Census Design ofthe Decennial Statistical Studies Division; David L.Hubble, (former) Assistant Division Chief forEvaluations of the Planning, Research and EvaluationDivision; Fay F. Nash, (former) Assistant Division Chieffor Statistical Design/Special Census Programs of theDecennial Management Division; James B. Treat,Assistant Division Chief for Evaluations of the Planning,Research and Evaluation Division; and VioletaVazquez of the Decennial Management Division.

As an integral part of the Census 2000 TXE Program,the Evaluations Executive Steering Committee char-tered a team to develop and administer the Census2000 Quality Assurance Process for reports. Past andpresent members of this team include: Deborah E.Bolton, Assistant Division Chief for ProgramCoordination of the Planning, Research and EvaluationDivision; Jon R. Clark, Assistant Division Chief forCensus Design of the Decennial Statistical StudiesDivision; David L. Hubble, (former) Assistant DivisionChief for Evaluations and James B. Treat, AssistantDivision Chief for Evaluations of the Planning, Researchand Evaluation Division; Florence H. Abramson,Linda S. Brudvig, Jason D. Machowski, andRandall J. Neugebauer of the Planning, Research and Evaluation Division; Violeta Vazquez of theDecennial Management Division; and Frank A.Vitrano (formerly) of the Planning, Research andEvaluation Division.

The Census 2000 TXE Program was coordinated by thePlanning, Research and Evaluation Division: Ruth AnnKillion, Division Chief; Deborah E. Bolton, AssistantDivision Chief; and Randall J. Neugebauer andGeorge Francis Train III, Staff Group Leaders. KeithA. Bennett, Linda S. Brudvig, Kathleen HaysGuevara, Christine Louise Hough, Jason D.Machowski, Monica Parrott Jones, Joyce A. Price,Tammie M. Shanks, Kevin A. Shaw,

George A. Sledge, Mary Ann Sykes, and CassandraH. Thomas provided coordination support. FlorenceH. Abramson provided editorial review.

This report was prepared by Elizabeth Ann Martin ofthe Directorate for Methodology and Standards andEleanor Gerber and Cleo Redline (formerly) of theStatistical Research Division. The following authorsand project managers prepared Census 2000 experi-ments and evaluations that contributed to this report:

Demographic Statistical Methods DivisionSharon R. EnnisPhyllis Singer

Directorate for Methodology and StandardsElizabeth Ann Martin

Statistical Research DivisionAref N. DajaniEleanor GerberCleo RedlineMary Ann Scaggs

Independent ContractorDon Dillman

The authors would like to recognize Aref N. Dajaniand Mary Ann Scaggs, who prepared files, calculatedresponse rates, and assisted in analyses for the Census2000 Alternative Questionnaire Experiment. Thanks toDon Dillman for his comments on this report, and hisnever-flagging interest in, and enthusiasm for, ques-tionnaire design issues.

Greg Carroll and Everett L. Dove of the Admin-istrative and Customer Services Division, and WalterC. Odom, Chief, provided publications and printingmanagement, graphic design and composition, and edi-torial review for print and electronic media. Generaldirection and production management were providedby James R. Clark, Assistant Division Chief, andSusan L. Rappa, Chief, Publications Services Branch.

Acknowledgments

U.S. Department of CommerceDonald L. Evans,

Secretary

Vacant,Deputy Secretary

Economics and Statistics AdministrationKathleen B. Cooper,

Under Secretary for Economic Affairs

U.S. CENSUS BUREAUCharles Louis Kincannon,

Director

Census 2000 Synthesis Report No. 17Census 2000 Testing, Experimentation,

and Evaluation Program

CENSUS 2000 ALTERNATIVEQUESTIONNAIRE EXPERIMENT

TR-17

Issued March 2004

Suggested Citation

Elizabeth Ann Martin,Eleanor Gerber, and Cleo Redline

Census 2000 Testing, Experimentation, and Evaluation

Program Synthesis Report No. 17, TR-17, Census 2000 Alternative

Questionnaire Experiment.U. S. Census Bureau

Washington, DC 20233 ECONOMICS

AND STATISTICS

ADMINISTRATION

Economics and StatisticsAdministration

Kathleen B. Cooper,Under Secretary for Economic Affairs

U.S. CENSUS BUREAU

Charles Louis Kincannon,Director

Hermann Habermann,Deputy Director and Chief Operating Officer

Cynthia Z. F. Clark,Associate Director for Methodology and Standards

Preston J. Waite, Associate Director for Decennial Census

Teresa Angueira, Chief, Decennial Management Division

Ruth Ann Killion, Chief, Planning, Research and Evaluation Division

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov Phone: toll-free 866-512-1800; DC area 202-512-1800

Fax: 202-512-2250 Mail: Stop SSOP, Washington, DC 20402-0001

U.S. Census Bureau Census 2000 Alternative Questionnaire Experiment iii

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31.1. Experiment A: Effects of Altering the Design of Branching

Instructions on Navigational Performance in Census 2000, by Cleo Redline, Don Dillman, Aref Dajani, and Mary Ann Scaggs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

1.2. Experiment B: An Experiment to Improve Coverage Through Revised Roster Instructions, by Eleanor Gerber, Aref Dajani, and Mary Ann Scaggs . . . . . . . . . . . . . . . . . .3

1.3. Experiment C: Questionnaire Effects on Reporting of Race and Hispanic Origin: Results of a Replication of the 1990 Mail Short Form in Census 2000, by Elizabeth Martin . . . .4

2. Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.1. Questionnaire development . . . . . . . . . . . . . . . . . . . . . .52.2. Sample design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.3. Experimental treatments . . . . . . . . . . . . . . . . . . . . . . . . .52.4. Additional processing and sources of data . . . . . . . . . . . .6

3. Major Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 3.1. Effects of Altering the Design of Branching Instructions

on Navigational Performance . . . . . . . . . . . . . . . . . . . . .73.2. An Experiment to Improve Coverage Through Revised

Roster Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83.3. Questionnaire Effects on Reporting of Race and Hispanic

Origin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

4. Implications and Recommendations . . . . . . . . . . . . . . . . . . .15

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Contents

iv Census 2000 Alternative Questionnaire Experiment U.S. Census Bureau

LIST OF TABLES, CHARTS, AND FIGURES

Chart 1. Alternative Questionnaire Experiment Panels . . . . . . . . .6

Table 1. Error Rates For All Census Long Form Items With Branching Instructions . . . . . . . . . . . . . . . . . . . . . . . . .7

Table 2. Omission Rates for Hispanics by Panel and Strata . . . . .8

Table 3. Frequency of Reasons Given for Erroneous Enumerations of People Age 18 to 35 . . . . . . . . . . . . . . . . . . . . . . . . .9

Table 4. Reporting of Hispanic Origin in Mail Questionnaires by Form Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Table 5. Race Item Nonresponse Rates by Form Type and Hispanic Origin . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Table 6. Detailed Hispanic Reporting by Form Type . . . . . . . . . .10

Table 7. Race Distributions for Hispanics by Form Type . . . . . . .11

Table 8. Percentage Who Reported in a Race "Example Group" by Form Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Table 9. Percent Writing in Each "Example Group" by Form Type 12

Figure 1. Illustration of the Five Branching Instruction Treatments21

Figure 2. Census 2000 Roster Instructions (Experiment B) . . . . .22

Figure 3. Roster Instructions in Experimental Form (Experiment B) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

Figure 4. Census 2000-style Form: Race and Hispanic Questions(Experiment C) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

Figure 5. 1990-style Form: Race and Hispanic Questions (Experiment C) . . . . . . . . . . . . . . . . . . . . . .25

U.S. Census Bureau Census 2000 Alternative Questionnaire Experiment v

The Census 2000 Testing, Experimentation, and Evaluation Programprovides measures of effectiveness for the Census 2000 design,operations, systems, and processes and provides information on the value of new or different methodologies. By providing measuresof how well Census 2000 was conducted, this program fully sup-ports the Census Bureau’s strategy to integrate the 2010 planningprocess with ongoing Master Address File/TIGER enhancements andthe American Community Survey. The purpose of the report that follows is to integrate findings and provide context and backgroundfor interpretation of related Census 2000 evaluations, experiments,and other assessments to make recommendations for planning the 2010 Census. Census 2000 Testing, Experimentation, andEvaluation reports are available on the Census Bureau’s Internet siteat: www.census.gov/pred/www/.

Foreword

This page intentionally left blank.

This report summarizes the find-ings of three experiments includedin the Census 2000 AlternativeQuestionnaire Experiment. Thepurposes of these experimentswere diverse:

• The skip instruction experi-ment examined respondent per-formance in following skipinstructions in the Census longform. It compared differentways of aiding respondents'navigation through the ques-tionnaire. One design incorpo-rated instructions and visualfeatures to help respondentsprevent errors before theyoccurred, and another wasdesigned to help respondentsdetect errors after theyoccurred. In addition to theseprevention and detectiondesigns, other potential designimprovements, such as usingreverse print to attract respon-dents' attention to instructions,and rewording the standard"skip" instruction, were alsotested.

• The residence instructionsexperiment involved the pres-entation of residence rules inthe Census 2000 short form.This research aimed to improvewithin-household coverage byrewording the roster instruc-tions to be understandable torespondents, encouragingrespondents to read themthrough appropriate placementand formatting, and by present-ing the instructions so as toincrease respondents' willing-ness to follow them.

• The race and Hispanic originexperiment compared the1990-style race and Hispanicorigin questions with the newquestions in Census 2000 shortform. It examined the effects ofchanges mandated by the Officeof Management and Budget toallow the reporting of more thanone race and reverse thesequence of the race andHispanic origin items. Otherchanges in format, categoriesand wording were also intro-duced in Census 2000, and thenet effects of all the changes onrace and Hispanic reportingwere analyzed.

All three experiments were limitedto the mailout-mailback universe.

The results of the three experi-ments include the following:

• Skip instruction experiment:Errors of commission (whichoccur when respondents incor-rectly answer questions theyshould have skipped) were sig-nificantly reduced in all of theexperimental treatments, sug-gesting that the design changesimproved respondents' percep-tion and comprehension of theinstruction. Errors of omission(which occur when respondentsskip questions they should haveanswered) decreased for theDetection Treatment, but signifi-cantly increased for every otherexperimental treatment. Eithertype of error indicates respon-dent difficulty navigating thequestionnaire, but their impactis different. Errors of omissionresult in missing data. Errors of

commission increase respondentburden and frustration. We rec-ommend that the CensusBureau adopt the Detectionmethod in mail question-naires, since it significantlyreduces both types oferrors.

• Residence instructionsexperiment. The changes informat, presentation, and word-ing of the residence instructionsresulted in a significantly higherresponse to the householdcount question (which serves asan important indicator of miss-ing data and flags large house-hold follow-up.) The experimen-tal panel also producedsignificantly fewer omissionsamong Hispanics in the low cov-erage stratum. We recom-mend additional testing ofthe graphical and wordingchanges that led to theseimprovements, to betterunderstand their effects andto further improve the quali-ty of household count data.

• Race and Hispanic originexperiment. Overall, the ques-tionnaire revisions substantiallyimproved the completeness ofrace and Hispanic origin report-ing in mail short form question-naires. In addition, Hispanicswere less likely to report theirrace as Some other race, andmore likely to report as White,in the 2000-style question-naires.

Although there were no appar-ent questionnaire effects on thefraction reported as Hispanic,

U.S. Census Bureau Census 2000 Alternative Questionnaire Experiment 1

Executive Summary

there were effects on the report-ing of detailed Hispanic origingroups. The 1990-style ques-tionnaire obtained more detailedreports of Hispanic origin thanthe 2000-style questionnaire,probably due to the effects ofquestion wording differences aswell as examples.

Unexpectedly, there were threetimes as many reports of theexample groups for Asian andPacific Islander groups in the2000-style form, which did notlist examples, as there were inthe 1990-style form, whereexamples were listed. Theexperiment demonstrates that

some questionnaire designchanges made in Census 2000resulted in substantial improve-ments in data quality, but thatother changes had unintendedconsequences. We recom-mend careful pretesting andfield testing of all changesto the questionnaire, andthat similar but larger repli-cation studies be conductedin future censuses to evalu-ate the effects of question-naire changes on the compa-rability of data from onecensus to the next.

The results of all three experi-ments point to interactions

between question format and con-

tent, suggesting that we must

attend to the complex relation-

ships between format and meaning

in self- administered question-

naires. These factors have been

demonstrated here to have meas-

urable effects on the data. These

experiments demonstrate that for-

mat affects performance on

branching instructions, affects

response/nonresponse on the

household count question and indi-

rectly affects coverage and that

format differences between 1990-

style and Census 2000 forms

affect race and ethnicity reporting.

2 Census 2000 Alternative Questionnaire Experiment U.S. Census Bureau

An Alternative QuestionnaireExperiment (AQE) to test theeffects of variations in the designof census mail questionnaires uponresponse rates and data qualityhas been conducted in each of thepast three decennial censuses.The 1980 Alternative QuestionnaireExperiment tested two FOSDIC(Film Optical Sensing Device forInput to Computer) matrix styleforms, and a non-FOSDIC form thatwas intended to be attractive andeasily understood (Mockovak1984). The 1990 AQE experimen-tally compared five long formquestionnaires that involved suc-cessively more radical departuresfrom the traditional design, includ-ing wording and format changes, akit containing individual question-naires for each household member,and anonymous census question-naires (Bates 1991, 1992).

The 2000 AQE incorporates threeseparate experiments, one involv-ing census long forms and theother two involving short forms,with different objectives. Twoexperiments look forward, seekingimprovements in the design ofmail forms that may lead toimproved data quality in the nextcensus. A third looks backward,replicating the 1990 short form inCensus 2000 in order to documentthe effects on the data of changesin the design of the Census 2000short form.

All three experiments test combi-nations or "packages" of designfeatures, rather than testing eachdesign change separately in a con-trolled fashion that would permitinferences about their individual

effects. Thus, firm conclusions canonly be drawn about the combinedeffect of multiple design features,and this is an important limitationof all three experiments. However,previous research and testing oftenprovides insights into the effectsof particular design features.Another limitation is that the find-ings are only generalizable to themailout/mailback universe. Thisexcludes certain populations ofinterest, such as Indians living onreservations and Alaska Natives (ofinterest to the analysis of question-naire effects on race reporting) orrural populations not enumeratedby mail (of interest to the analysisof roster completeness).

The three experiments are:

1.1 Experiment A: Effectsof Altering the Design ofBranching Instructions onNavigational Performancein Census 2000, by CleoRedline, Don Dillman, ArefDajani, and Mary AnnScaggs

This experiment took as its start-ing point the difficulty manyrespondents have navigating thecensus long form, causing them tomistakenly skip questions they aresupposed to answer or answerquestions they are supposed toskip. Hypotheses derived fromresearch on visual perception wereapplied to develop new strategiesfor helping respondents navigatetheir way through the question-naire. One design incorporatedinstructions and visual features tohelp respondents prevent errorsbefore they occurred, and another

was designed to help respondentsdetect errors after they occurred.In addition to these prevention anddetection designs, other potentialdesign improvements, such asusing reverse print to attractrespondents' attention to instruc-tions, and rewording the standard"skip" instruction to clarify it, werealso tested. The larger purpose ofthis experiment was to develop abetter understanding and generalprinciples of how graphical designfeatures of a questionnaire influ-ence respondents' ability to navi-gate through it.

1.2 Experiment B: AnExperiment to ImproveCoverage Through RevisedRoster Instructions, byEleanor Gerber, ArefDajani, and Mary AnnScaggs

This experiment focused on theproblems respondents have fillingout household rosters correctly.Erroneously including a personwho does not live in a household,or omitting one who does, result incensus coverage errors. The prob-lem is exacerbated by counter-intu-itive and complex census residencerules, which are often ignored byrespondents who may have theirown firm ideas about who belongsin their household. In addition,census residence rules do not fol-low any simple logic which is easi-ly communicated to respondents.This research aimed to improvewithin-household coverage byimproving the roster instructions inthree ways: first, by rewordingthem to be understandable torespondents, second by encourag-ing respondents to read them


1. Introduction

through appropriate placement andformatting, and third, by present-ing the instructions so as toincrease respondents' willingnessto follow them.

1.3 Experiment C:Questionnaire Effects onReporting of Race andHispanic Origin: Results ofa Replication of the 1990Mail Short Form in Census2000, by Elizabeth Martin

This experiment replicated a 1990-style mail short form during

Census 2000 and compared the

results to data from Census 2000

short forms in order to evaluate

how the questionnaire changes

affected reporting of race and

Hispanic origin reporting. The

questionnaire changes introduced

in Census 2000 included allowing

the reporting of more than one

race and reversing the sequence of

the race and Hispanic origin items,

as well as other changes in format,

categories and wording.

This synthesis report summarizes

the results of the three experi-

ments, and seeks to draw more

general conclusions from them.

Section 2 describes the methods

used in the AQE, and section 3

summarizes the major findings of

each separate experiment (as well

as additional analysis of one that

did not appear in the final report).

More detailed discussions of the

methods and findings of each

experiment are found in their

respective final reports.



2. Methods

2.1 Questionnairedevelopment

Experiment A. Conventionalbranching (skip) instructions mayeasily be overlooked. Alternativeforms were designed to manipulateverbal (revising "skip to" to "go to")and graphic (reverse print) fea-tures, and to implement errordetection or error preventionstrategies. Five alternative formswere developed and tested in aclassroom experiment and in cog-nitive tests. The testing was usedto revamp and refine the designswhich were tested in the AQE. Thequestions were chosen specificallyso that content would not providecues to navigation. Each panelused identical content, ensuringthat the length of each panel wasthe same. See Fig. 1 for the fiveformats that were included in theAQE.

Experiment B. Several differentroster formats were designed, andtwo rounds of cognitive testingwere conducted to select andrefine the best format, whichbecame experimental panel 7 (seechart 1, below) in this experiment.The experimental roster includedseveral design features, includingplacing the instructions before theroster question, rewording theinstruction to read "our guidelines"before answering the question,enclosing the instructions andanswer box in a black outlined boxwith a slightly brighter backgroundthat the surrounding questionnaire,double-banking theinclude/exclude rules and reword-ing them to make them more inclu-sive and easier to understand (see

Fig. 2 for the Census 2000 rosterquestion and Fig. 3 for the experi-mental version).

Experiment C. A 1990-style formwas developed which preserved1990 question wording, cate-gories, order, type size, matrix for-mat, and other essential designfeatures which might influenceresponses. The 1990-style formwas not identical to the 1990 cen-sus form, however, because itchanged dates and incorporatedseveral publicized and familiar fea-tures of the 2000 design (color,logo, "Start here" instruction, enve-lope and letter). Any questions notincluded in the Census 2000 shortform, such as marital status, weredropped. The form was notpretested. See Figs. 4 and 5 forthe two versions of the race andHispanic questions which werecompared.

2.2 Sample design

The AQE included eight panels ofapproximately 5,000 households(10,000 for one panel) each. Thesample was stratified into highcoverage areas (HCAs) which hadlow proportions of minorities andrenters in the 1990 census, andlow coverage areas (LCAs) withhigh proportions of minorities andrenters. Sample cases were dis-tributed equally between strata,implying that households in LCAswere oversampled. All results areweighted to reflect sampling prob-abilities.

Addresses on the Decennial MasterAddress File in the mailout/mail-back areas of the country at the

time sample selection took placeserved as the universe for sampleselection (Woltman, 1999).Addresses in non-mailback areas(mostly rural areas) were excludedfrom sample. This excludes cer-tain population groups of interest,including American Indians livingon reservations and AlaskaNatives. Addresses that wereadded later as a result of coverageimprovement operations wereexcluded, as were addresses in thesample for the Accuracy andCoverage Evaluation survey. A sys-tematic sample by state, stratum,and treatment was selected.

2.3 Experimentaltreatments

The AQE included eight panels(three short form and five longform panels). Chart 1 shows theexperimental and control panelsand number of households in each.Letters in parentheses indicatewhether the panel is part ofExperiment A, B, or C of the AQE.

For all panels, questionnaires weremailed out according to the Census2000 schedule, with every sam-pled address mailed an advanceletter, a questionnaire, and a fol-low-up postcard. For respondentsin the AQE, the responses providedon the mail forms were their cen-sus data. Telephone QuestionnaireAssistance operators were trainedto answer questions from respon-dents who received an experimen-tal questionnaire (e.g., about theinstruction in the 1990-style formto select one race category fromrespondents who wanted to reportmore than one).

Questionnaires were mailed backto the National Processing Centerin Jeffersonville, Indiana, wherethey were keyed and processedseparately from production Census2000 data, which were imaged.Households which did not return amail questionnaire were followedup as part of the Census 2000nonresponse operation and are notincluded in this analysis.Response rates for the eight panelsare shown in the second column ofChart 1. The rates are weightedand exclude undeliverable address-es, duplicate forms, and blankforms. Response rates for the twoexperimental short form panels donot significantly differ from thecontrol2. As was true in the cen-sus, response rates for the longform panels are significantly lowerthan for the short form panels.There are no significant response

rate differences among the longform panels. (Significances werecalculated using Bonferroni adjust-ments for multiple comparisons;see Dajani and Scaggs, 2001.)

2.4 Additional processingand sources of data

Experiment B. To analyze theeffects of roster variations, cover-age was measured by a speciallydeveloped telephone coveragereinterview, conducted by Westatabout four months after the cen-sus, in late July of 2000. The rein-terview sample consisted of cases

that had completed and returnedthe census form, had phone num-bers, and were not sent to largehousehold follow-up. These caseswere subsampled randomly at arate of 50 percent in the high cov-erage area stratum. The total sam-ple size for the reinterview was4,218 households. There were2,958 completed interviews: 1,497completed cases in the control and1,461 in the experiment. This rep-resents a response rate of 70.35percent in the control and 69.90percent in the experiment, with anoverall response rate of approxi-mately 70 percent.

Experiment C. To increase samplesize and improve reliability for theanalysis of questionnaire effects onrace and Hispanic reporting, theshort form control (panel 6) wassupplemented with mail returnsfrom the control panel for theResponse Mode and IncentivesExperiment (RMIE) (Guarino, 2001).The RMIE control group sample ofapproximately 20,000 addresseswas selected from the same uni-verse using the same stratificationas AQE, except the sample wasallocated proportionately to theHCA and LCA strata. All addressesin the RMIE control group receivedCensus 2000 short form question-naires, which were processed sepa-rately, like the AQE. The responserate was 71.5 percent.

Race and Hispanic origin data forpanels 6 and 8 (and the RMIE con-trol data) were coded and pre-edit-ed by applying a simplified versionof pre-edits used in Census 2000production. (See Martin 2002 fordetails.) Missing data were notimputed or allocated, as theywould be in fully edited censusdata. Results may differ for fullyedited census race and Hispanicorigin data.


2 Although overall short form responserates do not differ, panel 8 has a significant-ly lower response rate (57.63 percent) thanthe control panel 6 (60.78 percent) in thelow coverage area stratum (p < .05). Thisdifference suggests that the Census Bureau'sinvestment in developing a "user-friendly"design for the Census 2000 short form didachieve a higher response rate in theseareas, compared to a 1990-style form.

Panel Mailing Response sample rate

size

Long form panels

1. Census 2000 long form questionnaire 5,257 63.95(Control panel, experiment A)

2. "Go to" questionnaire (A): identical to panel 1 5,248 64.34questionnaire, except "skip to" is replaced with "go to" throughout.

3. Reverse print questionnaire (A): identical to 5,251 61.82panel 2 questionnaire, except the "go to" instruction appears in reverse print (yellow letters on black background).

4. Prevention questionnaire (A): adds an instruction 5,241 63.13alerting respondents to look for branching instructions; adds arrows and other features to attract attention to instructions.

5. Detection questionnaire (A): adds an arrow to 5,238 63.25guide respondents away from branching instruction when appropriate; adds information to the next question to allow respondents to judge if they have correctly skipped.

Short form panels

6. Census 2000 short form questionnaire 5,252 73.07(Control panel, experiments B, C)

7. Revised roster questionnaire (B) 5,256 73.52

8. 1990-style short form questionnaire (C): 10,499 72.60replicates 1990 question wording, categories, order, type size, matrix format, and other essential design features. Incorporates Census 2000 color, logo, envelope and letter. Drops questions not included in the Census 2000 short form (marital status)

Chart 1. Alternative Questionnaire Experiment Panels

3.1 Effects of Altering theDesign of BranchingInstructions onNavigational Performance

The analysis examined two typesof errors. Commission errorsoccur when respondents incorrect-ly provide an answer to questionsthey should have skipped, andomission errors occur whenrespondents skip questions theyshould have answered. Either typeof error indicates respondent diffi-culty navigating the questionnaire,but their impact is different. Errorsof omission result in missing data.Errors of commission increaserespondent burden and frustration.

To control for differences in thenumber of questions that respon-dents answered, analysis was limit-ed to Person 1 responses (thequestionnaire provides space fordata to be provided for up to sixhousehold members). Error rateswere calculated for questions withbranching instructions (because

only their designs differed betweenform types) and with validresponses (because only then wasit evident whether a respondentshould branch or not).

The results indicate that simplychanging the wording of branchinginstructions from "skip to" to "goto" (panel 1 versus 2) did not sig-nificantly affect either errors ofcommission or omission. Probablyrespondents did not notice eitherinstruction, and no amount ofrewording will help if the problemis that respondents are not readingthe information in the first place.

Panel 2 was adopted as the controlfor comparison with the remainderof the experimental panels, sinceall shared the "go to" instructionwording. All three experimentaltreatments resulted in significantlyfewer errors of commission thanpanel 2. The reverse print instruc-tion (panel 3) had significantlyfewer errors than the instructionwithout reverse print (panel 2).

The prevention questionnaire hadsignificantly fewer errors than thereverse print instruction. Therewas no further significant reduc-tion in commission errors with thedetection instruction overall(although there was in the LCAstratum). The fact that the com-mission error rate decreases acrossthe Go To, Reverse Print,Prevention and DetectionTreatments suggests that thechanges made from one design tothe next improved respondents'perception and comprehension ofthe instruction.

A different pattern occurs forerrors of omission. While errors ofomission decreased for theDetection Treatment, they signifi-cantly increased for every otherexperimental treatment. Possibly,the attention-getting features ofthe experimental treatment dis-tracted respondents' attention fromother cues about which questionsthey were supposed to answer.The better performance of thedetection method may be due totwo of its features. First, it includ-ed a visual cue (arrow) to directrespondents to the next question.Second, the feedback mechanism(the left-hand arrow that came offthe non-branching response optionand pointed to a parentheticalfeedback phrase) may have helpedrespondents avoid errors of omis-sion.

Thus, commission errors weredecreased in this treatment with-out the omission errors increasing.The level of omissions in theDetection treatment was signifi-cantly less than in the "Go to"


3. Major Findings

Table 1.Error Rates for all Census Long-Form Items WithBranching Instructions

Instruction Treatment Errors ofcommission

Errors ofomission

1. Census 2000 (Skip to instruction) . . . . . . . . . 19.7 5.02. (Go to) Control . . . . . . . . . . . . . . . . . . . . . . . . . 20.8 5.43. (Go to) Reverse Print. . . . . . . . . . . . . . . . . . . . 17.9 7.64. (Go to) Prevention . . . . . . . . . . . . . . . . . . . . . . 14.7 7.05. (Go to) Detection . . . . . . . . . . . . . . . . . . . . . . . 13.5 4.0

Statistical Comparison1 vs. 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . n.s. n.s.2 vs. 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p<.01 p<.012 vs. 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p<.01 p<.012 vs. 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p<.01 p<.013 vs. 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p<.01 n.s.3 vs. 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p<.01 p<.014 vs. 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . n.s. p<.01

Note: Lack of statistical significance is denoted by n.s.

control. Interestingly, some cogni-tive respondents and respondentsin a debriefing expressed the opin-ion that the many arrows in thedetection method were "confus-ing"–even though the results sug-gest these arrows improved theirperformance! This indicates aneed for care about what evidenceshould be used in making deci-sions, and also indicates that per-haps the effects of the arrows needto be tested separately.

Wide variations existed in errorrates for individual questions. Thereasons were not explored in thisresearch. However, the resultssuggest that respondents do notunderstand the questions or theunderlying response task (that theyneed not answer every question).As a result the respondent burdenis greater than necessary (on theaverage, respondents are answer-ing 20 percent more questionsthan they need to). Furtherresearch is needed to improverespondents' understanding of thequestions and the response task toreduce respondent burden.

Thus, manipulating the verbal sym-bolic and graphic languages ofbranching instructions significantlyinfluences whether the instructionsare followed. Further research isneeded into the ways that graphicfeatures interact with reading com-prehension–that is, what respon-dents read, the order in which theyread it and their consequent inter-pretation of that they read. This isan area of questionnaire designthat is clearly emerging as criticalto data quality and in need of fur-ther research.

3.2 An Experiment toImprove Coverage ThroughRevised Roster Instructions

The revised roster design was eval-uated using several error meas-ures: nonresponse for the house-

hold count box, where respon-dents recorded the number of per-sons in their household; omissionrates; and erroneous enumerationrates. Omission rates were calcu-lated as the number of Census Dayresidents omitted from the censusroster (but identified during rein-terview) divided by the number ofcorrect enumerations, using infor-mation both from the census formand from the reinterview.Erroneous enumeration rates werecalculated as the number of per-sons on the census roster whowere identified as Census Day non-residents in reinterview, dividedby the number of people on thecensus roster. In addition, thedemographic characteristics ofpeople enumerated using theredesigned roster and Census2000 roster design were com-pared.

Item Nonresponse. One of thealterations in the experimental ver-sion of the form was the place-ment of the box where respon-dents were to record the numberof persons in their households. Itis critical that the box whererespondents record the number ofpersons in their households becompleted, since it serves to flagmissing person-level data and tocue large household follow-up. Anyincrease in item nonresponse inthis item would be unacceptable.The item nonresponse for this itemis significantly lower in the experi-mental form (.80 percent) than inthe control (1.78 percent) at the

0.01 level of significance. Thus,the item nonresponse rate was cutin half, and the difference was sig-nificant in both strata. Thisdemonstrates that the wordingchanges and/or the format inte-grating the instructions with thefirst question were effective in get-ting respondents to fill in the box.

Omissions are persons who shouldhave been listed on the census butwere not. Such persons were iden-tified only in the reinterview (sinceby definition, they were not on theCensus form.)

There is no significant form differ-ence in the omission rates, whichwere 1.13 percent for the controland 1.21 percent for the experi-mental form. However, the experi-mental form had a significantlylower omission rate for Hispanicsin the low coverage stratum, asshown in Table 2.

Erroneous enumerations are per-sons who were included on thecensus forms although they werenot legitimate census day resi-dents. They included persons whospent most of their time else-where, or who were in group quar-ters where they should have beencounted on Census Day. Examplesare college students living awayfrom home and persons in the mili-tary stationed elsewhere.

No significant panel differenceswere found in erroneous enumera-tion rates, which were 0.40


Table 2.Omission Rates for Hispanics by Panel and Strata

PanelStratum

Total Percent HCA Percent LCA Percent

Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.54 3.90 3.23Experiment . . . . . . . . . . . . . . . . . . . . . . . . . 2.55 4.26 1.00Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.00 4.09 2.02Control vs. Experiment:

Statistical Comparison Not Significant Not Significant p<.05. . . . . . . . . . .


percent for the control and 0.39percent for the experimental form.

Although the rates did not differby panel, the characteristics of thepeople who were erroneously enu-merated did, for the low coveragestratum: in the experimental panel,the fraction who were young peo-ple 18-35 years old was 58 per-cent, compared to 30 percent inthe control panel.

The reasons why the 57 peopleage 18 to 35 were erroneouslyenumerated were further exam-ined. Table 3 shows that collegestudents account for a substantialportion of the erroneous enumera-tions in both forms (and in boththe LCA and HCA strata; this resultis not shown). However, thelargest number of erroneous enu-merations is not accounted for bythe categories of erroneous enu-merations which are usually includ-ed in residence rules research,such as college, military, and vari-ous group quarters institutions.This suggests that highly mobilepeople account for a substantial

proportion of erroneous enumera-tions. This confirms previouslyconducted qualitative research thathas demonstrated that youngadults in this age group may behighly mobile for reasons otherthan college.

Conclusions. The results suggestthat the revisions were effective.We do not know which of thedesign and wording changes areresponsible for the improvements.Further research is needed to bet-ter understand how the differentdesign features affect responses,how respondents naturally readquestionnaires, and the relation-ship between graphical presenta-tion and meaning in them. We rec-ommend additional research thatbuilds on these encouraging find-ings in further cognitive and fieldtests of wording and graphicaldesign changes in the roster ques-tions and instructions.

Differences between the effective-ness of these techniques in highand low coverage areas are encour-aging, since coverage improve-

ments occurred for a group(Hispanics) with relatively highrates of omission in the census.The differences also suggest thatdemographic, cultural, or socialfactors may influence the effective-ness of one or more of the graphi-cal and wording changes. Forinstance, perhaps the new formatwas easier to understand and fol-low by respondents who speakSpanish rather than English as theirnative language, or by those whoare less educated. The experimentdoes not allow us to identify thefactors which may have played arole, but the results suggest thatfurther research is needed toexpand our understanding of theirinfluence.

3.3. Questionnaire Effectson Reporting of Race andHispanic Origin

Item nonresponse rates and differ-ences in reporting of Hispanic ori-gin and race were comparedbetween the 1990- and 2000-stylequestionnaires and assessed usingVPLX (Fay, 1998).

Item Nonresponse Rates. Overall,the questionnaire revisions sub-stantially improved the complete-ness of race and Hispanic originreporting in mail short form ques-tionnaires. Item nonresponse (i.e.blank or uncodable responses) forHispanic origin was 3.33 percent in2000-style questionnaires, com-pared with 14.46 percent in 1990-style questionnaires, as seen inTable 4.

Item nonresponse for race was3.27 percent for 2000-style formsand 5.95 percent for 1990-stylequestionnaires, as shown in Table5. For Hispanics the reduction inrace item nonresponse was verylarge, from 30.53 to 20.79 percentin the 2000-style questionnaires,but remained quite high.

Table 3.Frequency of Reasons Given for Erroneous Enumerationsof People Age 18 to 35

Total Control Experiment

College . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 9 14Military . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3 1Institution (GQ). . . . . . . . . . . . . . . . . . . . . . 2 0 2Unknown . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 12 16Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 24 33

Table 4.Reporting of Hispanic Origin in Mail Questionnaires byForm Type

Form type

t2000-19902000-style 1990-style

Total Percent . . . . . . . . . . . . . . . . . . . . . . . . . 100.00 100.00Hispanic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.17 11.14 .05Non-Hispanic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85.50 74.39 *15.8Hispanic item blank, uncodable . . . . . . . . . . . . . 3.33 14.46 *–21.9N. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40,723 16,616

*p<.05


Hispanic Origin Reporting. Resultsin Table 4 show that the same per-centage (slightly over 11.1 per-cent) was reported as Hispanic inboth 1990- and 2000-style forms,despite the reversed sequence ofHispanic origin and race and ques-tion wording differences. In pastcensuses, most people for whomorigin was missing were non-Hispanic. On this assumption,these results suggest that ques-tionnaire changes reduced itemnonresponse but did not otherwiseaffect reporting as Hispanic.

However, the assumption may nolonger hold with the new question.In 1990, Hispanics and non-Hispanics were equally likely toleave the Hispanic item blank(McKenney et al., 1993), implyingthat Hispanics were equally repre-sented among people whoresponded and those who did notrespond to the item in 1990.Recently available evidence fromthe Content Reinterview Survey for

Census 2000 suggests that 25 per-

cent of people who did not

respond to the Hispanic item in

Census 2000 reported as Hispanic

in the reinterview (Singer and

Ennis, 2003). This suggests that

Hispanics were overrepresented

among nonrespondents to the

Hispanic item in this census. The

high nonresponse rate (esp. for

1990-style forms), uncertainty

about what fraction of nonrespon-

dents to the item are Hispanic, and

the possibility that the fraction

varies by form type, creates uncer-

tainty about whether there might

be differences between forms in

the fraction identified as Hispanic,

after the data were fully edited and

imputed. Any such differences are

not measurable using this sample.

Effects of Examples: Detailed

Hispanic Reporting. Although

there were no apparent question-

naire effects on the fraction report-

ed as Hispanic, there were effects

on the reporting of detailedHispanic origin groups.

It has been suggested that drop-ping examples from the Hispanicorigin question in the Census 2000mail form may have resulted in aloss of detail in Hispanic reporting.To examine this possibility,Hispanic origins were classifiedinto four categories, as shown inTable 6:

• Groups with check boxes(Mexican, Puerto Rican, Cuban)in both 1990- and 2000-styleforms;

• Groups listed as examples in the1990- but not the 2000-styleform (Argentinian, Colombian,Dominican, Nicaraguan,Salvadoran, Spaniard);

• All other specific groups with nocheck boxes and not listed asexamples in either form; and

• General descriptors: "Hispanic,""Latino," or "Spanish" were writ-ten in, rather than a specificgroup.

The 1990-style form elicited moredetailed reports of Hispanic originthan the 2000-style questionnairefor all three categories: Hispanicgroups with separate check boxes,those listed as examples in 1990but not 2000, and the remainingdetailed groups. The differencesare significant for the latter twocategories. Overall, about 93 per-cent of Hispanics reported a specif-ic group in 1990-style forms, com-pared with 81 percent who filledout 2000-style forms. In the 2000-style forms, Hispanics tended todescribe their ethnicity in generalrather than specific terms. About12 percent gave Hispanic, Latino,or Spanish as their "group," com-pared with 2 percent in 1990-styleforms.

Table 5.Race Item Nonresponse Rates by Form Type and HispanicOrigin

Hispanic originPercent of people missing data on race

2000-style 1990-style t2000-1990

Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.27 5.95 *–7.34Hispanics. . . . . . . . . . . . . . . . . . . . . . . . . . . 20.79 30.53 *–4.42Non-Hispanics . . . . . . . . . . . . . . . . . . . . . . .60 1.53 *–5.03Hispanic origin missing. . . . . . . . . . . . . . . 13.18 9.72 *2.00

*p<.05

Table 6.Detailed Hispanic Reporting by Form Type

2000-style 1990-style t2000-1990

Percent of all people identified as Hispanic . . . . . . . . . . . 100.00 100.00Check box groups: Hispanic groups with separate

check boxes in both forms. . . . . . . . . . . . . . . . . . . . . . . . . 70.25 73.23 –1.37Example groups: listed as examples in 1990-style

form but not Census 2000 . . . . . . . . . . . . . . . . . . . . . . . . 6.41 11.16 *–3.58All other detailed Hispanic groups . . . . . . . . . . . . . . . . . . . 4.20 8.68 *–3.38Write-in is general descriptor (Hispanic /Latino / Spanish) 11.90 1.90 *10.32Hispanic, no write-in (or write-in uncodable) . . . . . . . . . . 7.25 5.03 *2.15N. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5,163 3,091

*p<.05


These results suggest that theexamples helped respondentsunderstand that the intent of thequestion was to elicit a detailedHispanic origin, and therebyimproved reporting of both exam-ple groups and non-examplegroups. However, the reportingdifferences are likely due to theeffects of question wording differ-ences as well as examples. Therewas a significant difference inreporting for one of the checkboxcategories, the wording of whichwas identical in both forms ("Yes,Mexican, Mexican-Am., Chicano").About 54 percent of Hispanicschecked the Mexican box (or wrotein Mexican) in 2000-style forms,compared to 59 percent in the1990-style forms. This differencemay result from dropping the word"origin" from the question. A sub-sequent experiment has confirmedthat question wording differencesaccount for most of the differencein detailed Hispanic reporting,although the examples also con-tributed (Martin et. al., 2003).

Race Reporting. Race reportingwas also affected by differencesbetween the questionnaires.Overall, reports of two or moreraces more than doubled (.82 per-centage points to 2.03 percentagepoints) in response to the "markone or more" instruction, therewere significantly more reports ofNative Hawaiian and Other PacificIslander (NHOPI), and significantlyfewer reports of "Some other race."(These results are not shown.)Contrary to what might have beenexpected, there is little evidencethat allowing respondents to reportmore than one race reduced singlerace reporting in the five majorrace categories.

The effects of the questionnairedifferences on race reporting byHispanics were marked. Reportingas White was higher by about 10percent, while reporting as Someother race was lower by about thesame amount, in 2000-style forms,as shown in Table 7. Missing oruncodable responses are excluded,

so the distribution approximatesthe distribution that would beobtained were missing data imput-ed. The form differences in report-ing of Some other race are consis-tent with prior research, and areprobably due to the effects ofreversing the order of Hispanic andrace items, as well as the new "oneor more" option.

Example Effects: Race Reporting3

In the 1990-style form, examplesof "other Asian or Pacific Islander"groups were placed in the leftmostcolumn of the matrix, below therace question (see Fig. 5). Theseexamples (Hmong, Fijian, Laotian,Thai, Tongan, Pakistani,Cambodian) were dropped in the2000-style form. Table 8 showsthe fraction who reported in a raceexample group in 1990- and 2000-style questionnaires. (People forwhom the race question was leftentirely blank are dropped fromthe table.)

The table shows a highly signifi-cant form difference (t=3.58) butits direction is unexpected: thereare three times as many reports ofthe example groups in the 2000-style form, which did not listexamples. One would expect theuse of examples to be associatedwith higher, not lower, reporting ofexample categories.

Table 9 shows the fraction report-ing in each specific "examplegroup." In general, the 2000-styleform elicited more reports of boththe Asian and the Pacific Islanderexample groups, although only theoverall differences for Asians andfor Pacific Islanders are statisticallysignificant at the .05 level. Notethat there were no write-ins of theexample Pacific Islander groups in1990-style forms. Clearly, for thepurpose of assessing example

Table 7.Race Distributions for Hispanics by Form Type

Form type t-statistic

2000-style 1990-style t2000-1990

Percent of all people identified as Hispanic . . . 100.00 100.00White . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48.98 39.88 *3.23Black . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.07 2.32 –.34American Indian and Alaska Native . . . . . . . . . . 1.48 .72 1.61Asian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58 .88 –.60Native Hawaiian and Other Pacific Islander. . . .01 .15 –1.14Some other race . . . . . . . . . . . . . . . . . . . . . . . . . . 39.03 51.47 *–4.32Two or more races . . . . . . . . . . . . . . . . . . . . . . . . 7.84 4.59 *2.88

*p<.05

Table 8.Percentage Who Reported in a Race "ExampleGroup" by Form Type

1990-style 2000-style

Wrote in Hmong, Fijian, Laotian, Thai, .106 .356Tongan, Pakistani, or Cambodian . . . . . . . . . . . . . . . (.0351) (.0606)

Checked or wrote in another race . . . . . . . . . . . . . . . 99.894 99.644

Total percent. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100.000 100.000

3 This section was not included inMartin (2002).


effects for Pacific Islanders, a larg-er sample is needed. Nevertheless,the difference is consistent for allthe groups, and marginally signifi-cant for several (t=1.645 is signifi-cant at p<.10 with a 2-tailed test),despite very small cell sizes.

It is useful to interpret theseresults in the context of overallquestionnaire differences in report-ing of major race groups (fromTable 5, Martin, 2002):

• There was no differencebetween questionnaires in theoverall fraction reporting anAsian race: 4.04 percent report-ed as Asian in 2000-style ques-tionnaires, compared to 4.06percent in 1990-style question-naires.

• There was a significant ques-tionnaire difference in the over-all fraction reporting as NativeHawaiian and Other PacificIslander: 0.17 percent reportedas NHOPI in 2000-style forms,compared to 0.05 percent in1990-style forms. Thus, the dif-

ference in reporting of examplegroups is consistent with anoverall reporting difference forthis group.

Other questionnaire features, suchas splitting the Asian and PacificIslander (API) category into twoseparate categories, undoubtedlyinfluenced the results for PacificIslanders. The Pacific Islander cat-egory is probably more populatedin 2000-style forms because it iseasier for Pacific Islanders toreport with the Pacific Islanderboxes grouped together ratherthan interspersed among Asianboxes, as they are in the 1990-style form, and with their own"Other Pacific Islander" check boxassociated with a write-in space(Cf. Figs. 4 and 5). For these rea-sons, and because of the verysmall sample size, we cannot draweven tentative inferences about theeffects of the Pacific Islanderexamples.

The evidence appears stronger thatthe Asian examples may have

affected reporting. The greaterreporting of Asian example groupsin the 2000-style form is not con-sistent with an overall increase inreporting for Asians as a whole, asis the case for Pacific Islanders.Moreover, the form differences areconsistent and statistically signifi-cant overall, as well as for threeindividual example groups(Hmong, Pakistani, Lao). Theresults suggest, although they donot prove, that the use of Asianexamples in the 1990-style ques-tionnaire somehow reduced report-ing in the example groups.

The contrast between the effectsof examples in the Hispanic originand race items merits furtheranalysis and research. The exam-ples in the 1990 Hispanic originquestion seem to have helped toclarify the intent of the question tocollect detailed Hispanic origin,while the examples in the racequestion did not help reportingand may have adversely affectedreporting of Asian example groups.Possibly, the different placement ofthe examples was a factor. In the1990-style form, the Hispanicexamples were prominently placedabove the write-in space, justbelow the "other Spanish/Hispanic"response option. The race exam-ples were off to the left, below thequestion and remote from thewrite-in space. Respondents maynot have understood how to inter-pret the meaning of the arrow thatconnected them. The physical dis-tance between the examples andthe write-in space may have meantthat many respondents never sawthe examples, while those who didmay not have realized they weremeant to be associated with awrite-in space. The examples mayhave distracted respondents anddisrupted the response process.Perhaps respondents were con-fused by the label "Other API" for

Table 9.Percent Writing in Each "Example Group" by Form Type

1990-style 2000-style t2000-1990

Asian examples . . . . . . . . . . . . . . . . . . . . . .106 .300 *2.99(.0351) (.0545)

Cambodian . . . . . . . . . . . . . . . . . . . . . . . . . .014 .041 1.27(.0139) (.0164)

Hmong. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .002 .044 †1.69(.0023) (.0243)

Pakistani . . . . . . . . . . . . . . . . . . . . . . . . . . . .029 .095 †1.72(.0217) (.0317)

Thai . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .051 .054 .09(.0218) (.0188)

Lao . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .010 .066 †1.92(.0097) (.0276)

Pacific Islander examples . . . . . . . . . . . . 0 .057 *2.16(.0264)

Fijian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 .019 1.60(.0119)

Tongan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 .038 1.61(.0236)

Another race written in or checked . . . . 99.894 99.644 *–3.58(.0351) (.0606)

Total percent . . . . . . . . . . . . . . . . . . . . . . . . 100.000 100.000

*p<.05,2-tailed test.

†p<.10. 2-tailed test


the write-in box in the 1990-styleform, which they may not haverealized applied to them, soattempted to find some way ofreporting their race other thanwriting in a response. Possibly,respondents in the example groupswho found the examples circledthem, without writing in aresponse. Testing these hypothe-ses and speculations would requireexamining the forms, which wehave not undertaken, or additionalexperiments. Alternatively, it ispossible that features of the ques-tionnaire other than examplessomehow explain these results.

In any case, these results suggestthe possibility that in some circum-stances, examples may interferewith responding in the example

groups. More controlled experi-ments are needed to understandthe effects of examples on raceand Hispanic reporting, and howquestionnaire features such astheir location influence theresponse process. Meanwhile, con-siderable caution in their useappears to be warranted.

Conclusions. Additional research isneeded into methodological influ-ences on race reporting byHispanics and non-Hispanics(including experimental research toevaluate mode differences inreporting, which are not exploredhere but are troubling; see Martinand Gerber, 2003) .

The results raise doubts about themeaningfulness of race data, espe-

cially for Hispanics, for whom race

reporting is highly vulnerable to

methodological influences.

Research is needed to develop

more robust race measurement

methods that are less vulnerable to

methodological effects, especially

for Hispanics, and to evaluate the

effects of examples on race and

Hispanic origin reporting.

Coding, pre-editing, editing, and

imputation procedures may sub-

stantially affect the quality and

comparability of race and Hispanic

origin data, and their effects are

largely hidden. They need to be

documented and evaluated in con-

junction with questionnaire design

changes.



The results of the three experi-ments share certain themes incommon. In this section, we dis-cuss these themes and recommendnew avenues for research.

The central lesson of this researchis that we must attend to the com-plex relationships between formatand meaning in self administeredquestionnaires. These factors havemeasurable effects on the data.Our experiments demonstrate theeffects of questionnaire format onperformance on branching instruc-tions, item nonresponse on thehousehold count question and cov-erage of Hispanics, race and eth-nicity reporting, and responserates. In addition, the same ques-tionnaire design feature (e.g.,examples) may have complex andvarying effects, depending on howit is used and integrated into thequestionnaire as a whole.Examples apparently helpedrespondents understand the intentof the Hispanic origin question,and provide more detailed data.But Asian examples in the racequestion may have actuallyreduced reporting of the examplegroups. Our results provide acaution that seemingly minorquestionnaire variations in thedesign of a questionnaire canhave substantial effects on thedata.

We recommend against intro-ducing any questionnairechanges without fully pretest-ing and field testing them,since they may have complexand unexpected consequences.

We also note that cognitive testingalone is not sufficient to under-stand the effects of questionnairedesign changes on the data.Cognitive testing alone may provemisleading, and must be supple-mented by field experiments. Asdiscussed above, some respon-dents in cognitive tests, as well asexpert reviewers, found the arrowsin the Detection method inExperiment A distracting, yet thismethod resulted in significantlylower error rates.

These results should focus ourattention on the importance of thevisual aspects of self-administeredquestionnaires. As the Redlineexperiment points out, these visualaspects impart meaning in them-selves, in that they guide basicquestionnaire behavior of respon-dents. Two factors appear to be atwork. One is the attention ofrespondents. There is little valuein altering the wording of a ques-tion or instructions if the visualformat does not draw the respon-dent's attention. The second ispattern recognition, or respon-dents' perception that certain visu-al elements of a questionnaire areassociated, and the inferences andmeaning they draw from thegroupings they perceive. Bymanipulating visual features of thequestionnaire, Redline succeeds inreducing the frequency with whichrespondents answer questions theyshould have skipped, or skip ques-tions they should have answered.We recommend that the CensusBureau use the Detectionmethod developed by Redlinein mail questionnaires because

it results in fewer errors. TheMartin experiment also suggeststhat visual elements of the designinfluence respondents' behavior incomplex ways, since the effects ofexamples appear to depend ontheir placement on the question-naire (and possibly on other, as yetunidentified, questionnaire designfeatures as well). Gerber attemptsa holistic design that graphicallyintegrates the list of residenceinstructions and the householdcount box. While we cannot besure whether the graphical featuresor the wording changes in herdesign were responsible for theimprovements she obtained, herexperiment also reduces errors,supporting its use as well as fur-ther research and development ofher innovations. We recommendthat the Census Bureau con-duct further tests of graphicaldesign and wording changes inthe residence instructions,which show promise of improv-ing coverage.

All three experiments suggest thatvisual presentation and organiza-tion affect respondents' perception,comprehension and performanceon self-administered question-naires. We recommend furthertheoretical and empirical ques-tionnaire design research thatdraws on relevant scientificdisciplines to better under-stand how graphical designfeatures affect the responseprocess. This includes additionalresearch on the reading of ques-tionnaires. New methodologies,such as eye movement research,are necessary to establish a base-

4. Implications and Recommendations


line for how questionnaires arenormally read, in order to be ableto better understand how ques-tionnaire design features affect thereading and response process. Wealso recommend that theCensus Bureau add the analy-sis of errors of commissionand omission to its arsenal ofdata quality measures, asthese are more direct meas-ures of navigational perform-ance than item non-responserates or edit changes. Usingmeasures of data quality that arerelevant both to questionnairedesigners and to statisticians mayimprove their communicationabout data quality.

Some format effects appear to bemore relevant for one group thanothers. Gerber found that therevised residence rules hadstronger effects for Hispanics thannon-Hispanics in low coverageareas, perhaps because thechanges in wording and organiza-tion made the question easier tounderstand and follow. Martinfound response rate differences forthe low coverage stratum thatincluded more Black and Hispanichouseholds. In both cases, designchanges intended to be "user-friendly" had a bigger effect forrespondents who otherwise wereless likely to respond or have diffi-culty with the questionnaire. Inaddition, questionnaire design dif-ferences sometimes affected racereporting by Hispanics and non-Hispanics in opposite ways (seeMartin, 2002).

We have not explored the basis forthese differences, so cannotaddress their causes. Possibly, dif-ferences in learning and expecta-tions, and training in the form-fill-ing task, underlie them. Thissuggests that additional basicresearch should be carried outto understand how cognitive

as well as sociocultural factorsaffect questionnaire perform-ance.

The Martin experiment identifiesquestionnaire effects that con-found comparisons of 1990 and2000 census data. The degree ofconfounding cannot be inferreddirectly from the experimental evi-dence, which is restricted to mailshort forms and does not employfully edited data. However, we caninfer that the differences in thedesign of 1990 and 2000 mailshort forms would have resulted inan increase from the 1990 to the2000 census in Hispanics' report-ing of White race, and a decline intheir reporting of detailed Hispanicgroups, in the absence of truechange in the racial or ethnic com-position or identifications of thepopulation. The percentage ofHispanics who reported as White(alone) was 51.7 in the 1990 cen-sus and 47.9 in Census 2000 (U. S.Census Bureau, 2001). Becausethe questionnaire effect wouldhave led more Hispanics to reportas White, we infer that the declinein White reporting would havebeen even larger had the 2000-style questionnaire not increasedHispanics' reporting as White, com-pared to a 1990-style question-naire. We can also infer that anymeasured decline from the 1990 tothe 2000 census in reporting ofdetailed Hispanic origins is over-stated; the decline would be less ifthe 2000-style questionnaire hadnot resulted in less detailed report-ing. The confounding effectsof questionnaire differencesshould be taken into accountwhen comparing 1990 and2000 census data.

This panel of the AQE had as itsmain purpose calibrating the meas-urement properties of the 1990and 2000 mail short form ques-tionnaires. By replicating the 1990

census form in Census 2000, it ispossible to evaluate whether differ-ences in reporting are attributableto the questionnaire, and must betaken into account in interpretingpopulation trends from one censusto the next. The results demon-strate that replicating a prior cen-sus's questions in the current cen-sus can help shed light on possibleerrors and reporting differencesthat otherwise would be the sub-ject of unchecked conjecture andspeculation. For example, the AQEdata made it possible to under-stand how changes in the ques-tionnaire caused an unexpecteddecline in detailed Hispanic report-ing between 1990 and 2000 cen-suses. Without the AQE data, ana-lysts might have been tempted toexplain the apparent trend as theresult of an increasing pan-Hispanic identification in theHispanic population, when itappears to be an artifact of thequestionnaire design. We recom-mend that a similar replicationbe carried out in 2010, andthat the long form be included,to calibrate the different versionsof the long form used in the cen-sus in 2000 and in the AmericanCommunity Survey (ACS) in 2010.

The AQE sample was not largeenough to permit us to test designchanges separately. Rather, theexperiments (particularly theMartin and Gerber experiments) alltested "packages" which confoundthe effects of format and content,and which do not permit us to iso-late the factors responsible foreffects on particular populations.More sophisticated experimentaldesigns might permit future exper-iments to disentangle theseeffects. In addition, where cover-age is measured, larger reinterviewsamples are needed, since omis-sions and erroneous enumerationsaccount for a very small fraction of


the entire population. These limi-

tations were implicit in the size

and scope of the current AQE.

Both for the purposes of calibrat-

ing measurements (as in the Martin

experiment), and the purpose of

measuring small improvements

with high precision (as in the

Gerber experiment), we recom-

mend much larger sample

sizes and more sophisticated

experimental designs for

Alternate Questionnaire

Experiments of the future.



References

Bates, N. A. (1991), "TheAlternative QuestionnaireExperiment: Preliminary Report ofthe 100-percent Items." 1990Decennial Census PreliminaryResearch and EvaluationMemorandum No. 108.

Bates, N.A., (1992), "AdditionalResults from the 1990 AlternativeQuestionnaire Experiment." 1990Decennial Census PreliminaryResearch and EvaluationMemorandum No. 146.

Dajani, A. and Scaggs, M. A.(2001) "AQE 2000 Response RateAnalysis." Unpublished memoran-dum, Census Bureau.

Fay, R. E. (1998) VPLX ProgramDocumentation, Vol. 1. CensusBureau.

Gerber, E., Dajani, A., and Scaggs,M. A. (2002) An Experiment toImprove Coverage ThroughRevised Roster Instructions. Finalreport of Census 2000 AlternativeQuestionnaire Experiment. U.S.Census Bureau.

Martin, E. (2002) QuestionnaireEffects on Reporting of Race andHispanic Origin: Results of aReplication of the 1990 Mail ShortForm in Census 2000. Final reportof Census 2000 AlternativeQuestionnaire Experiment. U.S.Census Bureau.

Martin, E., Sheppard, D., Bentley,M., and Bennett, C. 2003. “Resultsof 2003 National Census Test ofRace and Hispanic Questions.”Report presented to Census BureauRace and Ethnic AdvisoryCommittees, October 1, 2003.

Martin, E. and Gerber, E. (2003)"Methodological Influences onComparability of RaceMeasurements: Several CautionaryExamples." Paper presented at theAmerican Statistical Association,San Francisco, CA.

McKenney, N. D., Fernandez, E. W.and Masamura, W. T. (1985) "Thequality of the race and Hispanicorigin information reported in the1980 census." Proceedings of theAmerican Statistical Association(Survey Research Methods Section),pp. 46-50.

Mockovak, W. (1984) AlternativeQuestionnaires Experiment.Preliminary Evaluations ResultsMemorandum No. 93, 1980Census. U.S. Census Bureau.

Redline, C., Dillman, D., Dajani, A.and Scaggs, M. A. (2002) Effectsof Altering The Design ofBranching Instructions OnNavigational Performance inCensus 2000. Final report ofCensus 2000 AlternativeQuestionnaire Experiment. U.S.Census Bureau.

Singer, P. and Ennis, S. R. (2003)Content Reinterview Survey of theCensus 2000: Accuracy of Data forSelected Population and HousingCharacteristics as Measured byReinterview. Census 2000Evaluation B5. U. S. CensusBureau.

U. S. Census Bureau (2001)"Population by Race and Hispanicor Latino Origin for the UnitedStates: 1990 and 2000." Census2000 PHC-T-1.



Appendix

Figure 1. Illustration of the five branching instruction treatments


Figure 2. Census 2000 Roster Instructions (Experiment B)


Figure 3. Roster Instructions in Experimental Form (Experiment B)


Figure 4. 2000-style form: Race and Hispanic Questions (Experiment C)


Figure 5. 1990-style form: Race and Hispanic Questions (Experiment C)

Census 2000 Topic Report No. 7TR-7

Issued March 2004

Data Processing in Census 2000

U.S.Department of CommerceEconomics and Statistics Administration

U.S. CENSUS BUREAU

Census 2000 Testing, Experimentation, and Evaluation Program