Journal of Clinical Epidemiology 64 (2011) 1262e1265

Numbering questionnaires had no impact on the response rateand only a slight influence on the response content of a patient

safety culture survey: a randomized trial

Francois Kundiga,*, Anthony Stainesa,b, Thompson Kingea, Thomas V. PernegercaHopital neuchatelois, La Chaux-de-Fonds, Switzerland

bIFROSS, Universit�e Lyon 3, Lyon, FrancecDivision of Clinical Epidemiology, University Hospitals of Geneva and University of Geneva, Geneva, Switzerland

Accepted 21 February 2011; Published online 8 June 2011

Abstract

Objectives: In self-completed surveys, anonymous questionnaires are sometimes numbered so as to avoid sending reminders to initialnonrespondents. This number may be perceived as a threat to confidentiality by some respondents, which may reduce the response rate, orcause social desirability bias. In this study, we evaluated whether using nonnumbered vs. numbered questionnaires influenced the responserate and the response content.

Study Design and Setting: During a patient safety culture survey, we randomized participants into two groups: one received an anon-ymous nonnumbered questionnaire and the other a numbered questionnaire. We compared the survey response rates and distributions of theresponses for the 42-questionnaire items across the two groups.

Results: Response rates were similar in the two groups (nonnumbered, 75.2%; numbered, 72.8%; difference, 2.4%; P5 0.28). Five ofthe 42 questions had statistically significant differences in distributions, but these differences were small. Unexpectedly, in all five instances,the patient safety culture ratings were more favorable in the nonnumbered group.

Conclusion: Numbering of mailed questionnaires had no impact on the response rate. Numbering influenced significantly the responsecontent of several items, but these differences were small and ran against the hypothesis of social desirability bias. � 2011 Elsevier Inc.All rights reserved.

Keywords: Numbered questionnaire; Anonymity; Response rate; Response content; Epidemiologic research; Patient safety culture

1. Introduction

Self-completed questionnaires are commonly used inhealth research. Such surveys are often hampered bya low response rate, which reduces statistical power andmay cause selection bias. Several strategies are availableto increase response rates [1e4], including the use of re-minder mailings [5]. Reminders can be sent to everyone, in-cluding early respondents. This method does not require theidentification of respondents but results in more mailings,hence higher costs, and along with a high consumption ofpaper, which in a context of growing sensitivity to environ-ment issues may irritate early respondents. Another possi-bility is to track respondents by means of an identifying

* Corresponding author. Service de M�edecine, Hopital neuchatelois,

CH-2300 La Chaux-de-Fonds, Switzerland. Tel.: þ41-32-967-27-28;

fax: þ41-32-967-27-29.

E-mail address: francois.kundig@ne.ch (F. Kundig).

0895-4356/$ - see front matter � 2011 Elsevier Inc. All rights reserved.

doi: 10.1016/j.jclinepi.2011.02.008

number and send reminders only to current nonrespondents.The volume of reminders will be lower. Also, those who donot wish to participate can inform researchers of their deci-sion, thus avoiding repeat reminders. The downside of anidentification number is that it makes the participant’sresponses identifiable, which may cause some people to de-cline participation, or to provide socially desirable answers.Whether and to what extent these reactions occur is notwell known.

Most studies examining the impact of respondents’ iden-tifiability on survey results were conducted in settings, suchas teaching [6], psychology [7], or business research[8e10]. Most found little or no impact on the response rateor response patterns [6,8,9], but one found different re-sponse patterns for signed and nonsigned questionnaires[10]. However, the impact may be different for sensitivetopics, which are common in health research. In particular,patient safety and handling of errors are sensitive topics, aspatient harm and professional identity are at stake. We only

1263F. Kundig et al. / Journal of Clinical Epidemiology 64 (2011) 1262e1265

What is new?

� Numbering survey questionnaires caused no signif-icant reduction in the response rate and modifiedthe pattern of responses for a minority of items, al-though in an unexpected direction.

� Numbering questionnaires may be useful for logis-tical purposes, but its impact on the responses andresponse rate is not well known. This study showsthat such impacts are small or nonexistent.

� Both numbered and nonnumbered questionnairescan be used in culture surveys.

found one previous health-related study: in postal surveysof AIDS knowledge conducted in 1986e1987 informingparticipants that the questionnaires were numbered and thata reminder would be forthcoming (vs. anonymous question-naire) resulted in similar response rates [11]. We found nostudy of health care professionals.

In this article, we examined the impact of numbered vs.nonnumbered questionnaires on the response rate and con-tent in a patient safety culture survey conducted in a Swisshospital.

2. Methods

2.1. Participants and setting

The study was conducted in 2009 as part of the qualityimprovement program at a regional public hospital locatedin French-speaking Switzerland, which serves a populationof about 170,000 inhabitants. The hospital includes sepa-rate facilities for acute medical care and rehabilitation. Ithas a capacity of 450 beds distributed in seven locations.

The purpose of the survey was to assess the patientsafety culture among hospital staff. All employees wereinvited to participate as long as they had worked in thehospital for at least 6 months, were directly involved inpatient care, or belonged to managerial staff.

2.2. Study design

Eligible employees were randomly assigned to twogroups: one received a numbered questionnaire and theother a nonnumbered questionnaire. The randomizationwas performed by the firm that conducted the survey(Stat’Elite, Bussigny, Switzerland), through the assignmentof random numbers.

2.3. Questionnaire

We used the Hospital Survey on Patient Safety Cultureestablished by the Agency for Healthcare Research and

Quality [12]. The questionnaire included 42 items that mea-sured 12 dimensions of patient safety culture. The esti-mated time of completion was 20 minutes. Items wereanswered on a five-point agreement scale (from ‘‘stronglyagree’’ to ‘‘strongly disagree’’) or frequency scale (from‘‘always’’ to ‘‘never’’). Half of the questionnaires werenumbered, and half were not. All questionnaires werestamped with a bar code necessary for optical scanning.

2.4. Data collection

A week before questionnaire distribution, hospital staffreceived a prenotification letter describing the study, its ob-jectives, and the reason for testing two versions of the ques-tionnaire. The study was also described in the hospital’sinternal journal, organized meetings, and on the Intranet.

One week later, hospital staff received the questionnairedirectly in the units, with cover letters. They were asked toreturn filled questionnaires in ballot boxes placed in severallocations in every building. The time allowed for answeringthe questionnaire was considered as part of work time. Twoweeks after the distribution of the questionnaires, a follow-up letter was sent to all hospital staff who had received anyof the questionnaires. After 1 month, a second reminderattached to the questionnaire was sent to everyone whoreceived the nonnumbered questionnaire and to nonrespon-dents only among those who received the numbered ques-tionnaire. Data entry was performed by optical scanner.

2.5. Sample size and statistical analysis

The primary objective of the survey (assessment of pa-tient safety culture) required the recruitment of all eligibleemployees (N5 1,583), and all were included in this ran-domized study.

The two groups were compared to check that they hadsimilar baseline characteristics. The response rate wasdetermined for both groups of questionnaires. Eighteennegatively formulated items were coded in reverse, so thathigh scores represented a favorable opinion or attitude. Toexamine possible information bias, differences in responsedistributions were compared for the 42 items of the ques-tionnaire, using the ManneWhitney test. Ordinal logisticregression was used to adjust each comparison for respon-dent characteristics.

3. Results

3.1. Response rate

Of 1,583 questionnaires that were distributed, 1,221were received in response. Fifty returned questionnaireswere considered as nonresponses, 25 in each group, be-cause they were empty, only partially answered (less thanone entire section in the survey or fewer than half of theitems throughout the entire survey in different sections),

Table 1. Characteristics of study respondents

Characteristics

Numberedquestionnaires,

n (%)

Nonnumberedquestionnaires,

n (%)

GenderFemale 431 (77.8) 479 (80.4)Male 123 (22.2) 117 (19.6)

Age (yr)!25 16 (2.9) 31 (5.2)26e35 133 (23.9) 149 (24.9)36e45 176 (31.6) 184 (30.7)46e55 166 (29.8) 175 (29.2)56e65 65 (11.7) 58 (9.7)O65 1 (0.2) 2 (0.3)

Staff positionNurse 214 (38.4) 224 (37.6)Nurse assistant 67 (12.1) 70 (11.8)Head nurse 27 (4.8) 28 (4.7)Resident physician 5 (0.9) 22 (3.7)Attending physician, head

of service or department52 (9.4) 39 (6.6)

Paramedical staffa 88 (15.8) 88 (14.7)Pharmacist 7 (1.3) 8 (1.3)Administration/management 40 (7.2) 38 (6.4)Other 58 (10.4) 78 (13.1)

Professional experience (yr)!1 7 (1.3) 9 (1.5)1e5 98 (17.5) 122 (20.4)6e10 128 (22.9) 128 (21.4)11e15 81 (14.5) 93 (15.5)16e20 75 (13.4) 96 (16.0)O21 170 (30.4) 151 (25.2)

Work areaMedicine 64 (11.4) 92 (15.5)Surgery 61 (10.9) 63 (10.6)Operating room 29 (5.2) 30 (5.1)Anesthesiology 17 (3.0) 12 (2.0)Palliative care 10 (1.8) 11 (1.9)

1264 F. Kundig et al. / Journal of Clinical Epidemiology 64 (2011) 1262e1265

or because all questions of one part of the questionnairewere given the same answer. The global response ratewas 74.0% (1,171/1,583); 75.2% (605/805) in the nonnum-bered group and 72.8% (566/778) in the numbered group.This difference of 2.4% (95% confidence interval5�1.9,6.7) was not statistically significant (P5 0.28). The posthoc power estimation for the observed difference was20%; more than 5,000 observations per group would havebeen necessary to achieve a power of 80%.

3.2. Respondent characteristics

Respondents were predominantly women, aged between26 and 55 years (Table 1). Almost half of the respondentshad worked in one of the affiliated hospitals for at least10 years. The most represented profession was nursing (in-cluding auxiliary nurses). About three-quarters of respon-dents worked in acute care units. Nine of 10 respondentsdeclared having direct contact with patients. Medical resi-dents represented only a small proportion of the respon-dents, but as they frequently worked only for 1 year atthis hospital, many were ineligible because they had beenat the hospital for less than 6 months.

3.3. Response patterns

Mean responses to the 42 items were similar for the twogroups, but 34 scored higher in the nonnumbered group(i.e., in a direction opposite to our hypothesis), and fiveof these differences were statistically significant (Fig. 1).The mean difference was 0.04 of a standard deviation(range, �0.11 to 0.13) in favor of the nonnumbered group.Adjustment for respondent characteristics changed theseresults very little, but all differences became statisticallynonsignificant (results not shown).

Pediatrics 17 (3.0) 14 (2.4)Intensive care unit 24 (4.3) 31 (5.2)Emergency 31 (5.5) 24 (4.1)Rehabilitation 43 (7.7) 41 (6.9)Oncology 12 (2.1) 17 (2.9)Technician diagnostics 1 (0.2) 2 (0.3)Pharmacy 7 (1.3) 10 (1.7)Radiology 24 (4.3) 33 (5.6)Laboratory 30 (5.4) 30 (5.1)Obstetrics and gynecology 41 (7.3) 46 (7.8)Many different hospitals/units 71 (12.7) 56 (9.5)Other 78 (13.9) 80 (13.5)

a Includes radiologist technician, therapist, lab assistant, dieti-tian, psychologist, occupational therapist, and social worker.

4. Discussion

In our study, numbering questionnaires had no impact onthe response rate. The observed difference was 2.4% and isnot significant. Either questionnaire numbering reduces theresponse rate by a small amount or it does not, and the ob-served difference was because of chance. The direction ofthe observed difference is consistent with our a priori hy-pothesis that the possibility of a breach in confidentialitymay incite some potential participants not to respond whenquestionnaires are numbered. A much larger study wouldbe necessary to detect a difference such as we have ob-served with sufficient power.

Furthermore, numbering survey questionnaires had onlya small impact on response content. Of the 42 questions in-cluded in the questionnaire, five had different means in thetwo groups. Unexpectedly, the means of all these five itemswere more favorable in the nonnumbered group. This iscontrary to our hypothesis that the guarantee of anonymitywould enable more negative or less desirable responses in

the nonnumbered group. This result argues against our hy-pothesis that social desirability bias can be caused by thenumbering of questionnaires. We do not see what causalmechanism might explain this paradoxical finding. Previ-ous studies have found either no bias or bias in the expecteddirection. For example, nonanonymous conditions lead tounderreporting of socially sensitive behaviors [13]. In anycase, the differences in responses between the two groups

Fig. 1. Scatter plot of 42 item means from numbered and nonnum-bered questionnaires. Black dots indicate statistically significant dif-ferences (P! 0.05).

1265F. Kundig et al. / Journal of Clinical Epidemiology 64 (2011) 1262e1265

were small, of about oneetwenty-fifth of a standard devia-tion on average. In practical terms, such differences arenegligible.

A possible explanation for our finding is type 1 error.The randomization procedure may have resulted in studygroups that were uneven in their attitudes and perceptionsof patient safety. Randomization does not guarantee equal-ity between groups in any given instance; rather, one can becertain that sooner or later, randomly allocated groups willdisplay substantial differences. Replication of our resultswould help resolve this issue.

Our study has both strengths and limitations. We useda randomized design to avoid confounding. Second, we as-sessed both the response rate and response patterns. Third,this is one of few studies of questionnaire coding in healthresearch. On the other hand, the sample size was not suffi-cient for the detection of small effects, such as a 2% or 3%difference in response rates. Furthermore, we did not probenonrespondents about their reasons for declining participa-tion or participants about the reasons for a given response.Therefore, we can only speculate about the possible causalmechanisms.

The prenotification letter included an explanation aboutthe two versions of the questionnaire to provide full disclo-sure to participants. This may have drawn the respondents’attention to whether their questionnaire was numbered.Drawing attention to the experiment would have increasedthe contrast between the groups. Because our results wereglobally negative, we believe that our conclusions are valid,despite the disclosure. Another limitation is that the pres-ence of a bar code required for optical scanning may haveled some respondents to believe (erroneously) that ques-tionnaires could be traced to the respondent’s identity; this

may have diluted the effect of questionnaire numbering.The only way to determine if the bar code played a rolewould be to repeat the study without a bar code. Finally,the applicability of our results to other contexts is by defi-nition debatable.

Consequently, we believe that both numbered and non-numbered questionnaires can be used in culture surveys,without concern about lowering unduly response rates, orinfluencing response content. The choice between the twomethods should be guided by logistical concerns, such asthe cost of sending unnecessary reminders, and the relativeburden of monitoring numbered returns.

Acknowledgment

The authors thank Estelle Martin, PhD, Stat’Elite, forperforming the data collection and initial data analysis.

References

[1] Nakash RA, Hutton JL, Jørstad-Stein EC, Gates S, Lamb SE. Maxi-

mising response to postal questionnairesda systematic review of

randomised trials in health research. BMC Med Res Methodol

2006;6:5e13.[2] Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R,

et al. Increasing response rates to postal questionnaires: systematic

review. BMJ 2002;324:1183e91.

[3] Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I,

et al. Methods to increase response to postal and electronic question-

naires. Cochrane Database Syst Rev 2009;(Issue 3). Art. No.:

MR000008.

[4] Mallen CD, Dunn KM, Thomas E, Peat G. Thicker paper and larger

font increased response and completeness in a postal survey. J Clin

Epidemiol 2008;61:1296e300.

[5] Perneger TV, Etter JF, Rougemont A. Randomized trial of use of

a monetary incentive and a reminder card to increase the response

rate to a mailed health survey. Am J Epidemiol 1993;138:714e22.

[6] Mason WS, Dressel RJ, Bain RK. An experimental study of factors

affecting response to a mail survey of beginning teachers. Public

Opin Q 1961;25:296e9.

[7] King FW. Anonymous versus identifiable questionnaires in drug us-

age surveys. Am Psychol 1970;25:982e5.

[8] Albaum G. Do source and anonymity affect mail survey results? J

Acad Market Sci 1987;15(3):74e81.

[9] McKee DO. The effect of using a questionnaire identification code

and message about non-response follow-up plans on mail survey re-

sponse characteristics. J Market Res Soc 1992;34(2):179e91.

[10] Futrell CM, Stem DE, Fortune BD. Effects of signed versus unsigned

internally administered questionnaires for managers. J Bus Res

1978;6:91e8.[11] Campbell MJ, Waters WE. Does anonymity increase response rate in

postal questionnaire surveys about sensitive subjects? A randomised

trial. J Epidemiol Community Health 1990;44:75e6.

[12] Available at http://www.ahrq.gov/qual/patientsafetyculture/hospsurv

index.htm. Accessed November 21, 2010.

[13] Durant LE, Carey MP, Schroder KE. Effects of anonymity, gender,

and erotophilia on the quality of data obtained from self-reports of

socially sensitive behaviors. J Behav Med 2002;25:438e67.