Commentary

114 Academic Psychiatry, 27:2, Summer 2003

Survey Research: Think . . . Think Again

Sarah Hudson Scholle, Dr.P.H.Harold Alan Pincus, M.D.

The authors are with the Department of Psychiatry, Univer-sity of Pittsburg, and the Western Psychiatric Institute and Clinic.Dr. Pincus is also associated with RAND. Send correspondence toDr. Pincus at the Western Psychiatric Institute and Clinic, Depart-ment of Psychiatry, University of Pittsburgh, 3811 O’Hara Street,Pittsburgh, PA 15213. E-mail, pincusha@msx.upmc.edu

Copyright � 2003 Academic Psychiatry.

Surveys are ubiquitous. All of us, almost daily, re-ceive questionnaires in the mail and telephone

calls at dinnertime by marketing surveyors. The highprevalence and incidence of surveys in modern lifetends to trivialize the serious nature of survey re-search. Also, the low cost and the apparent accessi-bility of survey methods can make such approachesseem like low-hanging fruit to junior faculty seekingto produce scholarly publications for professional ad-vancement.

Obviously, the concrete tools of survey researchare available to anyone with access to a telephone orpostal services. However, like psychotherapy, surveyresearch may seem like an easy thing to do but ac-tually requires a great deal of highly specific expertiseto do it right. Also, although the technology of con-ducting surveys is more accessible and less expensivethan, say, a positron emission tomography scannerand a cyclotron, surveys do expend real resources—by both researchers and respondents. A survey of1,000 psychiatrists that takes each an hour to fill out“costs” $100,000 in respondents’ time (at $100 perhour). Survey research is serious business.

Detailed knowledge of sampling, design, andanalysis are critical to the proper conduct of studiesinvolving survey research. It would be expected thatindividuals undertaking them either possess the nec-essary methodological knowledge and skills or have

colleagues with the appropriate expertise as part oftheir team.

Perhaps even more important, though, are thefundamental approaches and values of science thatunderlie research across all fields, from molecular ge-netics to clinical trials to health services and educa-tional research. The article by Sierles in this issue pro-vides useful suggestions for conducting surveyresearch on educational issues, but we are concernedthat some readers may get the impression that surveyresearch is a casual process, lacking the seriousnessof purpose and need for scrupulous attention to sci-entific process of other, “harder” research.

There are four essential steps that individualsshould consider before undertaking a survey project:

1. Explicitly define the question. As with any re-search study, the first steps are to define the indepen-dent and dependent variables. In survey research,choosing the right items to capture these variables iscritically important. Experts in survey developmentuse both qualitative and quantitative research meth-ods to determine whether survey questions ade-quately capture the content of interest and to ensurethat survey respondents and researchers have thesame understanding of the questions. The careful de-velopment of tools for assessing consumer satisfac-tion with health plans is an example of extensive cog-nitive laboratory testing and piloting (1). Wheneverpossible, it is preferable to search for questionnaireitems that are well validated, because it saves the in-vestigator time, makes the research comparable toprevious studies, and helps guard against problemsthat arise from poorly worded questions.

2. Determine whether a survey is the best way to an-swer the research question. Survey research is well suitedto research on knowledge, attitudes, and beliefs. How

COMMENTARY

Academic Psychiatry, 27:2, Summer 2003 115

else but to ask them can you capture individuals’ opin-ions or beliefs? Surveys may also be useful for cap-turing behaviors and practices, especially those thatare rarely documented otherwise, as Sierles showedwith his study of cheating by medical students. How-ever, as Sierles pointed out, individuals may know-ingly or unknowingly misrepresent their behaviors oractions to “look good” in the research. It is importantto determine whether other, more reliable datasources might exist to capture the information of in-terest. For example, using medical records can be amore accurate way of assessing physicians’ prescrib-ing behavior than asking physicians about their usualpractices.

3. Figure out whether your survey, as designed, willactually answer the research question. Every question-naire item and instruction should be scrutinized forerrors, from typographical mistakes to incorrect re-sponse categories. Make sure that each item is prop-erly framed. If the questionnaire asks about experi-ences that happen over time, the time period shouldbe specified and consistent throughout the survey.There is no substitute for sufficient piloting of the in-strument and the process. It is also essential to planthe analysis necessary to answer the research ques-tion before the survey is administered. Statistical anddata management consultation should occur as earlyas possible in the development of the survey, and cer-tainly before the survey is conducted. Plans for datacoding and entry as well as the database constructioncan identify questions that may need to allow re-spondents to choose multiple responses. (For exam-ple, many patients have multiple types of health in-surance, so any “type of insurance” question shouldallow respondents to endorse multiple plans, such asMedicaid, Medicare, employer-sponsored, and soon.) Preparing the analysis plan and drafting dummytables often identifies missing information that iscritical in defining independent or dependent vari-ables as well as important confounders or covariates.

4. Ruthlessly consider and reveal all potential threatsto the validity of your study. In survey research, re-sponse bias and inadequate measurement of potentialconfounding variables are common threats to astudy’s internal and external validity. Since briefersurveys generally obtain higher response rates, these

factors are related. Investigators should strive to cre-ate a questionnaire that adheres closely to the pre-specified research questions while assessing impor-tant potential confounding factors. A thoroughliterature review and a clearly articulated conceptualmodel simplify the task of item selection and providea rationale for the inclusion of each item. Similarly,this conceptual model and consideration of potentialconfounders should also help the investigator in as-sessing whether the planned survey method willlikely yield a representative sample of the populationof interest. An e-mail survey may be an excellent wayto ask students about the quality of Internet educa-tional materials. E-mail surveys may be less usefulfor gauging patients’ views of educational materialsif many patients do not have Internet access—par-ticularly if the researcher is interested in reachinglow-income populations that are less likely to haveInternet access (4). It is possible to get an adequateresponse rate but still have a study sample that is notrepresentative of the population of interest; efforts toanticipate threats to generalizability and ways toreach populations that often do not participate in sur-vey research should be considered.

We certainly do not wish to discourage junior fac-ulty who have research questions they wish to an-swer within an evidence-based framework. However,such projects should not be undertaken lightly. Asthese principles suggest, researchers should thinkthrough all of the scientific and practical issues beforecollecting any data, and they should identify any po-tential flaws after the process is completed.

People usually like to be asked about their opin-ions and experiences. They are more likely to be in-terested in divulging personal information when asurvey has a well-developed rationale and when theinstrument shows that the investigator is knowledge-able about the topic, has planned the questionnairecarefully, and poses questions that make sense. Sur-veys that have a strong conceptual framework are abetter value for the research dollar and also show re-spect for the participants’ time.

The authors are supported by the Mental Health In-tervention Research Center (MH 30915; David Kupfer, PI)at the University of Pittsburgh.

SURVEY RESEARCH: THINK . . . THINK AGAIN

116 Academic Psychiatry, 27:2, Summer 2003

1. Hays RD, Shaul JA, Williams VS, et al: Psychometric prop-erties of the CAHPS 1.0 survey measures. Consumer Assess-ment of Health Plans Study. Med Care 1999; 37(3 suppl):MS22–MS31

2. Oxman TE, Korsen N, Hartley D, et al: Improving the pre-cision of primary care physician self-report of antidepressantprescribing. Med Care 2000 38(7): 771–6

References

3. Harns-Kojetin LD, Fowler FJ Jr., Brown JA, et al: The use ofcognitive testing to develop and evaluate CAHPS 1.0 coresurvey items. Med Care 1999 37(3 suppl): MS 10–21

4. Newburger EC: Home Computers and Internet Use in theUnited States, August 2000. Current Population Reports,September 2001. Washington DC: U.S. Census Bureau.http://www.census.gov