Quality Assurance and Quality Control in Longitudinal Studies...QUALITY ASSURANCE Overview As stated above, quality assurance consists of those activities undertaken prior to data

Epidemiologic ReviewsCopyright © 1998 by The Johns Hopkins University School of Hygiene and Public HealthAll rights reserved

Vol. 20, No. 1Printed in U.S.A.

Quality Assurance and Quality Control in Longitudinal Studies

Coralyn W. Whitney, Bonnie K. Lind, and Patricia W. Wahl

INTRODUCTION

Many elements of quality control for cohort studiesare similar to those for other types of studies, i.e.,standardization of protocol (study procedures), goodcommunication among all study staff, clear expecta-tions of requirements, and monitoring to ensure thatrequirements are met. However, cohort studies havethe added dimension of longitudinal data collection,which adds issues related to drift over time, changes inequipment (including both degradation due to wearand upgrades due to new technology), and staff turn-over. If the study uses central laboratories or readingcenters to process data, these sites are particularlysubject to quality issues related to changes over time.Cohort studies involving multiple centers have extraissues related to comparability of data collected atdifferent locations. This presentation will provide ageneral overview of quality control elements commonto many types of medical research studies, but primar-ily focuses on issues related to cohort studies. Empha-sis will be placed on multicenter cohort studies, asthey face additional quality control issues comparedwith single-center cohort studies.

Much of the literature on quality assurance andquality control has arisen in the context of clinicaltrials, focusing on maximizing the quality of the datathrough standardized study-wide and local protocols,training of study personnel, and data managementsystems (1-9). Quality assurance and control proce-dures are generally described in the context of a spe-cific study, such as the Multiple Risk Factor Interven-tion Trial (10-12), the Hypertension PrimaryPrevention Trial (3, 13), or the Optic Neuritis Treat-ment Trial (14, 15), with some articles providing gen-eral guidelines (4, 16-18). There is less literature inthe context of nonclinical trial studies, although somestudy-specific approaches (8, 19-22) and generalquality assurance/quality control guidelines have been

Received for publication June 26, 1997, and accepted for pub-lication May 8, 1998.

From the Department of Biostatistics, School of Public Health,University of Washington, Seattle, WA.

Reprint requests to Bonnie K. Lind, Department of Biostatistics,Box 35-8223, University of Washington, Seattle, WA 98195.

addressed (5, 23-25). The literature specifically ad-dressing quality control issues in longitudinal studiesis limited (10, 16, 26).

Clinical trials often have a large number of sites,each collecting a small amount of data on a fewindividual patients, while cohort studies generally in-volve fewer sites collecting a much broader set of data.This facet of cohort studies can lead to unique qualitycontrol challenges due to the sheer bulk of the datacollected (2, 10, 27, 28). Therefore, quality controlliterature based on clinical trials is often not relevant tocohort studies. By having fewer sites, the cohort modelis more conducive to the development of long-termrelationships between central quality control supervi-sors and site personnel, and to the supervision ofindividual site performance. However, it also meansthat the consequences of a site with quality controlproblems (e.g., failure to follow protocol, sloppy datahandling procedures) can be much more grave for thestudy as a whole.

Quality control is one of the most important aspectsof any study, as the integrity of the conclusions drawnby a study are in large part determined by the qualityof the data collected. Data of poor quality, containinga great deal of random noise, decrease the power of astudy and can cause a type II error. An even worseresult is the collection of data that are biased due tofaulty instruments or errors in implementing the pro-tocol, leading to an incorrect report of relations (typeI error). Many aspects of data collection can impactthe quality of the data, including completeness andclarity of questionnaires, the interviewer's delivery,the accuracy of mechanical instruments, and techni-cians' measurement techniques (16, 19, 20, 23). Thevalidity of the study depends on the interviewers andtechnicians from all centers consistently applyingstudy protocol (3, 16). Other errors can be introducedinto the data after the data are collected, during transcrip-tion, at data entry, and data manipulation for analysis (1,2, 29). Minimizing all of these potential sources of erroris of paramount importance in the planning and imple-mentation of any study. This presentation will suggestways of minimizing these potential sources of error.

There are actually two basic components to quality

71

Dow

nloaded from https://academ

ic.oup.com/epirev/article-abstract/20/1/71/400230 by guest on 26 M

arch 2019

72 Whitney et al.

control: quality assurance and quality control. Qualityassurance consists of those activities that take placeprior to data collection while quality control consistsof those activities that take place during and after datacollection to identify and correct any errors or discrep-ancies in the data that have been collected (table 1). Eachof these components will be discussed separately.

QUALITY ASSURANCE

Overview

As stated above, quality assurance consists of thoseactivities undertaken prior to data collection to ensurethat the data are of the highest possible quality at thetime they are collected. These activities include thedevelopment of the study protocol, development ofthe data entry and data management systems, trainingand certification of data collection personnel, and test-ing of data collection procedures.

Development of the study protocol

The major component of quality assurance is thedevelopment of the study protocol and the creation ofmanuals documenting the protocol. Aspects of proto-col development that are common to most study de-signs are summarized below (table 1).

Design of data collection instruments, includingcontent, format, and step-by-step instructions forcompletion of the instrument. Procedures should bedesigned to ensure that the data being produced are infact reliable, valid, on the appropriate scale, and do notreflect bias of the instrument or bias that may arise insubgroups of the population being studied (13, 14, 20,23, 26, 30). For example, older participants tend togive responses that are "socially desirable" rather thanstrictly accurate (19). Limitations of various types ofinstruments should be considered during developmentso that potential problems can be identified and miti-gated to the extent possible. For example, issues in-

TABLE 1. Summary of quality assurance and quality control activities

Quality assurance activities Quality control activities

Development of protocol and recruitment

Protocol development and design Checking for participant eligibility

Questionnaires and data collection instalments

DesignPilot testingCoding and editing proceduresTrouble shootingMissing valuesUpdating

Development/pilot testingTraining of interviewersTraining of techniciansCertification proceduresRecertification procedures

Development/pilot testingTraining of personnel

Development of local quality assurance/quality control

Data cleanupLocal and central evaluationProblems with self-report data

Interviewer and technician protocol

For each interviewer and technician:Observe application of protocolMonitor certification maintenanceCompleteness of dataRetraining when neededRegular feedback on quality controlTraining of new personnel

Data entry packages

Updating systemsDouble entry

Study wide

Monthly quality control reportsReliability/reproducibility studiesSite visits

Use and maintenance of study equipment

Develop and pilot equipment useTraining of personnelMaintenance and schedules

Routine maintenance of equipmentTrouble shootingQuality of equipment data

Epidemiol Rev Vol. 20, No. 1, 1998

Dow



arch 2019

Quality Assurance and Quality Control 73

herent in self-administered questionnaires have beendescribed in the literature, such as the lack of reliabil-ity of self-reported data and the potential for questionsto be misunderstood (23). Adequate testing of self-administered questionnaires on volunteers similar tothe anticipated cohort are crucial for identifyingany problematic questions before the study begins.Interviewer-administered questionnaires also havepotential problems; for example, participants may re-spond differently to interviewers of different age, gen-der, or ethnic background. In addition, it is vital thatthe protocol outlines how interviewers are to respondwhen participants ask for clarification of a question(i.e., is the interviewer allowed to rephrase the ques-tion in his/her own words, and if so, how does thestudy ascertain that interviewers are rephrasing a ques-tion in comparable ways). During planning, investiga-tors should also consider any problems that may arisewith regard to cohort members who are illiterate or donot speak English.

Cohort studies typically involve many interviewers,creating the potential for differences between inter-viewers or centers and for differences over time (dueeither to drift or staff turnover). This makes the estab-lishment of a documented, standardized interviewingprotocol of paramount importance (2, 8, 10-12, 31). Inaddition, investigators must decide during the plan-ning phase whether questionnaires will be allowed tochange over time and how the study will ascertain thatresponses given over time are comparable.

All aspects of the protocol must be documented in amanual of operations (2). This document should beviewed as the official study reference document fordata collection staff and, as such, should contain alldetails of data collection procedures. Once the manualis written, it should be reviewed so that any sectionsthat are ambiguous or subject to misinterpretation canbe clarified (32). The manual should help to ascertainthat data collection is consistent across field centersand over time. Many good manuals of operations existin ongoing multicenter cohort studies, with examplesusually available from funding sources (33-35).

Selection of standard equipment for all measureddata. All sites should begin the study with identicalequipment to minimize data variability due to collec-tion by different equipment. A protocol should beestablished to maintain and recalibrate equipment overtime, and forms should be developed to document thatthese activities have occurred. Ways of identifying andreplacing equipment that is worn out or failing must beagreed upon, and procedures for dealing with the de-velopment of new technologies must be discussed.

Development of procedures for reviewing and up-dating the protocol as needed, and for communicating

changes to all study personnel. Once data collectionactually begins, minor adjustments to the protocol areusually needed. Oversight of this function should beassigned to a group of investigators, and a protocolshould be developed for implementing and communi-cating such changes to all study personnel (see thesubsection on "Communication," below).

Development of procedures for obtaining and main-taining certification to perform study procedures, andprocedures for monitoring that requirements are met.An integral component of quality assurance is thetraining and certification of study personnel in accor-dance with study protocol (2, 3, 10-14, 16, 32). Pro-cedures must be established for training and certifyingdata collection personnel prior to the start of the study,as well as for training and certifying new staff hiredthroughout the course of the study. In addition, re-quirements for maintaining certification must be set(both in terms of number of procedures performed andthe quality of the performance), and procedures mustbe developed for recertification of any technicianswho fail to maintain certification.

Hiring individuals to serve as field center techni-cians is also a crucial part of quality assurance. Thepersonalities and skills of these individuals will have adirect bearing on the final quality of study data. Somesites have found that data quality is better if techni-cians are hired who have little or no medical back-ground. A person who has been performing proce-dures such as blood pressures in a clinical setting formany years may be more difficult to retrain to followa standard protocol than a person who has never per-formed blood pressures.

Communication. Communication is a key featureof successful cohort quality control systems (2), andpart of the protocol development process should bedevoted to setting up the structure of the communica-tion system in the study (figure 1). Many large studiesuse a committee structure to ensure that tasks areaccomplished and problems addressed, and in multi-center studies, the coordinating center plays the keyrole of facilitating communication between commit-tees and field centers. For example, an operationscommittee may be formed to address questions thatarise about implementation of the protocol, to ensurethat the protocol is followed at all sites, and to addressany problems that arise. A quality control committeemay be formed to monitor the actual data quality andto address any problems that arise. For such commit-tees to be effective, communication routes must beestablished. These communications often take theform of quality control reports, typically produced bythe coordinating center, which are distributed to prin-cipal investigators. It is equally important that proce-


Dow



arch 2019

74 Whitney et al.

QC CommitteeOperations Committee

i

\

( Reading >̂\ . Center J

Principal Investigator

i >

Study Coordinator

CoordinatingCenter

IData Interviewers Technical

Personnel Personnel

•* • : Indicates direction of responsibilities

FIGURE 1. Organization and communication among study personnel.

dures be developed for responding to quality controlreports, including identification of the person who is torespond to any problems identified in a quality controlreport, what steps are to be taken, and what the dead-line is for response (table 2). During the developmentphase, the study leadership may also want to delegateresponsibility for quality control at each center to aspecific individual designated as the quality controlsupervisor, who could be an investigator, the studycoordinator, or another staff member.

Development of data editing, transcription, andentry procedures

Collection of accurate data is only the first step. It isequally important that errors are not introduced in theprocess of converting the data to electronic format. Forsome forms it may be desirable to have the formreviewed and edited before data entry. The purpose ofthis review would be to ascertain that the form wascomplete, that skip patterns were followed, and thatany data values that seemed inconsistent or looked likepossible errors were checked before being entered.

Once the data are ready to be converted to electronicformat, several options are available. In the past, moststudies have relied upon key-entry of data into thecomputer. Software packages exist that allow dataentry to be customized to limit the range of entereddata, to check for internal consistency, and to catcherrors in key fields such as participant identificationnumbers. Many studies use double-entry verificationto catch and correct any data entry errors from theoriginal entry (16, 36-38). While this system mayseem cumbersome, it is more efficient than trying toidentify and correct errors at a later point in time (38).

New technologies are continually being developed

which offer more efficient ways of converting data toelectronic format. Some studies have interviewers en-ter data directly into a computer without first writingthe data onto a paper form (21). In this type of system,it is important that the data entry software include asmany ways of checking data accuracy at entry aspossible, as any errors identified later cannot be com-pared with a paper form for correction. Some of thenewest technologies being used involve scanning pa-per forms directly into the computer, or faxing formsto a central data center where the fax machine servesas a scanner to convert the data to electronic format.These systems are also very efficient, but it is im-portant that scanned files be visually reviewed foraccuracy.

Training and certification

Once all protocols have been developed and docu-mented, the next step is to train study personnel toimplement the procedures. Training and certificationactivities result in standardization and can lead toreduced costs over time (16). Such training is key, asthe interviewer's or technician's perceived value of thedata being collected can have a direct impact on thecare taken in following the protocol and the likelihoodthat discrepancies are introduced (27, 32). In a multi-center study, a central training session is often the bestway to ensure that all personnel are trained in a stan-dardized manner, providing the added benefit of in-stilling in local personnel the wider scope of the studyand the importance of their work.

Testing of procedures

The final step in the quality assurance process is totest all procedures that have been developed. This is


Dow



arch 2019

Epid

em

iol R

ev

Vol. 2

0, r

p

, 1998

TABLE 2. Quality control and recruitment reports—responsibilities and distribution

Report

Recruitment and data completenessreport

Failed studies

Quality ot studies

Technical data distributions

Protocol adherence report

Recruitment by demographics

Data cleanup

• F = sent by fax; M = sent by mall.

Purpose

Tabulates recruitment andstudy data receivedat coordinating centerby center

Tabulate no. of failedstudies

Evaluation of studies bycenter, personnel, andany reading centers

Analysis of technicalprocedure distributionsby center andtechnicians

Adherence to protocol:No. of technical proceduresEquipment maintenanceSupervisor checklist

Analysis of demographicsby center

By center, identification ofoutliers and discrepan-cies In all data sent tocoordinating center

Frequency/distribution*

Weekly (F)

Monthly (F)

Monthly (M)

Monthly (M)

Bimonthly (M)

Monthly (F)

Quarterly (M)

Committeeresponsible

Operations committee


Quality control committee

Quality control committee



Coordinating committee/quality controlcommittee

Distribution

Principal investigators,program office, studycoordinators

Principal investigators,program office, studycoordinators

Principal Investigators,program office, studycoordinators

Study coordinators

Principal investigators,quality controlcommittee, studycoordinators

Principal investigators,program office,study coordinators

Study coordinators

Response required

From

Principal investigators

Principal Investigators


Study coordinators

Study coordinators


Study coordinators

Schedule

As needed

Last day ot the monthdistributed

Last Irlday of the monthdistributed

Last day of the monthdistributed

15th of the monthfollowing the monthdistributed

Calls/meetings otInvestigators

1 month after date otdistribution

QualH

y A

ssura

inee and

Quality

Contro

l 75

Dow



arch 2019

76 Whitney et al.

often done by means of a pilot study in which theentire protocol is performed on volunteers who aresimilar demographically to the anticipated cohort. Thepilot study should encompass all aspects of the proto-col, including all interviews and measured variables,entry and transmission of data to the coordinatingcenter, sending of samples to any reading centers orlaboratories involved in the study, processing at thereading centers or laboratories, and creation and dis-tribution of reports by the coordinating center. Sometime should then be allowed for implementing refine-ments to the protocol between the end of the pilotstudy and the beginning of recruitment.

QUALITY CONTROL

Once data collection begins, the quality control pro-cedures developed as part of the quality assuranceprocess must be implemented. The goal of these qual-ity control procedures is to identify and correct sourcesof either bias or excessive noise in the data both duringand after data collection. Some of the quality controlactivities are procedures common to most study de-signs and will be outlined briefly. The complexity ofquality control monitoring introduced by the longitu-dinal nature of cohort studies will be addressed inmore detail. Quality control activities will be dividedinto those relevant to field centers and those relevantto reading centers or laboratories. The role of thecoordinating center will be addressed.

Field center quality control

The quality assurance procedures developed duringthe planning phase for monitoring technician perform-ance must be implemented. As noted above, it iscrucial that reports be timely and that responsibilitiesfor responding to reports and taking actions to correctproblems are clear. Frequently, studies establish pro-cedures for identifying problems but fail to establishadequate procedures for ensuring that the problems areresolved. Good quality control reports can be used asan incentive at the field center to encourage top qualitywork by the data collection staff, serving as a reminderthat they are part of a bigger study and that otherpeople outside of their own center are interested in thequality of their work. In cohort studies, typical items tomonitor include the following:

Recruitment. Frequent reports should be sent toeach center showing how well the site is doing towardmeeting recruitment goals, in terms of total numbersrecruited and any targets regarding gender or minorityenrollment. Data should also be reviewed to ensurethat enrolled participants meet all eligibility criteria(e.g., not outside specified age range).

Technician performance. Data should be analyzedregularly to assess technician performance. Evidencethat a technician's data deviate significantly from thegroup as a whole should be reported to the site andinvestigated by the quality control coordinator. Forexample, if the overall average systolic blood pressuremeasured by all technicians during a certain timeperiod was 122.3 mmHg, and the average for onespecific technician was 129.7 mmHg, the quality con-trol coordinator should be notified to investigate anypotential problems in that technician's technique. Inthe Cardiovascular Health Study, this type of reportuncovered the problem that a certain technician wasroutinely using a pediatric blood pressure cuff ratherthan an adult cuff to measure blood pressure on adults,resulting in bias in the data collected by that technician(B. K. Lind, Department of Biostatistics, University ofWashington, Seattle, WA, personal communication,1991). This systematic type of problem can often becorrected using a statistical adjustment. In this exam-ple, the effect of using the wrong cuff was quiteconsistent across all measurements taken with thatcuff, so once the size of this effect was determined, astatistical correction factor was applied to the errone-ous data.

Monitoring interviewers can be more problematic

Having a quality control supervisor watch an inter-view is not the best method, as this may impact boththe technique that the interviewer uses on that occa-sion and the participant's responses. Some studiesroutinely tape all interviews (with the participant'spermission), and a random sample is then reviewed bythe quality control supervisor for adherence to proto-col. In a multicenter study, it is important that tapes arealso reviewed by quality control supervisors at othersites to make sure that interviewing techniques areconsistent between sites as well as within sites.

In addition to cross-sectional analyses of perform-ance, data should be analyzed by field center and bytechnician over time to look for drift. Plots and graphscan be especially useful in this type of analysis.

Data clean-up. Field center data should be ana-lyzed routinely for problems such as extreme or in-consistent values. In this effort, longitudinal data arevaluable in detecting errors. For example, if a partic-ipant's weight is entered as 58 kg one year and 101 kgthe next, it is likely that one of these values is an error.Comparing values longitudinally can detect thesetypes of errors, which are easily corrected by askingthe field centers to check participant charts.

Replicate measures. Multiple measures taken atone point in time can be used both to identify possibledata errors and to calculate more accurate measures of


Dow



arch 2019


exposure. For example, multiple blood pressures areoften taken during one study visit and averaged inorder to give a more accurate value of the participant'sblood pressure. Combining multiple measurements byaveraging serves to provide more accurate estimates oftrue data values and to minimize measurement error.For detection of potentially erroneous data, data entrysoftware can be used to flag the data entered if suc-cessive values differ by more than a predeterminedamount.

For measurements such as blood pressure, it is easyand inexpensive to perform multiple measurements onall participants. In some cohort studies, important out-come or exposure measures are obtained using expen-sive or invasive technology (e.g., magnetic resonanceimaging, blood draws) which cannot reasonably berepeated on all participants. In such situations, repli-cate measurements can be performed on a subset ofparticipants to quantify variability in the measurementand to obtain a more accurate measure of the variable.For example, in the Sleep Heart Health Study (39),sleep apnea and other sleep-related breathing measure-ments were obtained using overnight monitoring ofparticipants in their homes. In order to assess howmuch the values obtained reflected a "first night ef-fect," a second study was performed on a subset ofparticipants. This repeat allowed the Sleep HeartHealth Study investigators to determine how accuratetheir estimates were.

Aspects of field center quality control unique tocohort studies

The longitudinal nature of cohort studies introducesseveral additional potential quality problems whichneed to be addressed. These include staff turnover,technician drift, degradation of equipment, technologychange, and inconsistencies in participant responsesover time.

Problems resulting from staff turnover and techni-cian drift can be addressed in several ways. Estab-lished certification requirements will ensure that newstaff are trained similarly at all sites, and establishedrequirements for a minimum number of procedureswhich must be performed each week or month tomaintain certification will help minimize drift.

Frequent central training and retraining sessions,held throughout the course of data collection, are pow-erful tools to use to minimize this problem. Thesetraining sessions can have a great impact on dataquality in several ways: quality control supervisorsfrom all sites work together and can ascertain that theyare all training new staff in comparable ways; all sitescan make sure that everyone is interpreting details ofthe protocol in comparable ways; all staff members get

a refresher on protocol details (preferably from some-one outside their own site); and staff members canshare ideas for retaining participants in the study andfor maintaining enthusiasm. In addition to sessions forretraining and recertifying of technicians on studyprocedures, these sessions should include presenta-tions from study scientists. Scientific sessions on studyresults are always greatly appreciated by techniciansand help to remind them of the importance of then-jobs, which can seem mundane on a day-to-day basis.

In order to reduce the costs associated with training,some studies have conducted regional training ses-sions utilizing computer instruction (16). Other studieshave completed certification via telephone for certainprocedures (15), or used video cassettes or teleconfer-encing (30, 32). All of these are valuable tools and areless expensive than central training; however, they alllack some of the benefits of face-to-face training.

Degradation of equipment and technology changesprovide another set of unique challenges. It is impor-tant that equipment be maintained and calibrated reg-ularly and comparably at all sites, to minimize anymeasurement error due to equipment. For example,scales should be calibrated regularly and the temper-ature in freezers used to store samples should bemonitored and recorded in a log. In some studies,equipment has a direct impact on data quality. In theSleep Heart Health Study, overnight sleep data wascollected on participants in their homes using a porta-ble monitor. Over time these monitors were subjectedto a great deal of wear and tear. Statistics were kept onthe proportion of studies scored as "good" or betterquality for each monitor, and when a monitor wasfound to have less than 85 percent "good" or betterstudies, the sites were notified to have the monitorrefurbished. As data collection equipment ages or newtechnology appears, it can be tempting to replace oldequipment with more up-to-date models. However,data must first be collected using both the old and newequipment to ensure that the data are comparable.Changes in either the hardware or software (if appli-cable) in new equipment can introduce variability orbias in data collected with the new equipment whencompared with data collected with the old model.

Data entry and management systems can provide areal challenge in this regard. In cohort studies collect-ing a broad spectrum of data, a great deal of effort isinvested in creating systems for entering, tracking, andmanaging data. Often the system will become obsoletelong before the cohort study ends. The coordinatingcenter or other group responsible for these aspects ofthe study must decide whether it is worthwhile toinvest the effort to move to a more up-to-date andefficient system. An added complication is that in


Dow



arch 2019

78 Whitney et al.

some cases field staffs are not highly sophisticatedcomputer users, so updating the systems causes com-plications not only at the coordinating center but forthe users at remote sites as well.

A final aspect of field center data quality that mustbe considered in cohort studies is inconsistent partic-ipant response over time. Some data are best collectedby self-report, even though the limitations of self-reported data are well-known (23). For example, cur-rent smoking status is collected at each visit in manycohort studies, usually by self-report. A certain pro-portion of participants will give inconsistent answersto smoking questions, e.g., reporting one year that theyare current smokers and reporting the next year thatthey have never smoked. The quality control protocolshould recognize that such inconsistencies will occurand have a method in place for handling them. Possi-ble solutions are to recontact participants with incon-sistent responses and ask for clarification, to set in-consistent responses to missing values, or, if possible,to use an independent source (such as blood chemistryvalues) to verify the data.

Quality control of reading centers andlaboratories

Many cohort studies send some types of data tocentral reading centers or laboratories for processing.Reading centers and laboratories present unique qual-ity control challenges because the data processing thatthey perform adds another source of variability. Forexample, if echocardiography is performed at fieldcenters, variation is introduced both by the field centertechnician performing the echocardiogram and by theperson at the echocardiogram reading center where thedata are interpreted. Often the data processed at suchcenters or laboratories are the key exposures or out-come variables for the study. Since most cohort stud-ies are interested in measuring real change in exposurevariables over time, one goal of quality control proce-dures should be to assess and minimize extraneousvariability in these important exposure measures.

It can be difficult to separate random biologic vari-ability, measurement error, and true change. There-fore, it is essential to get good estimates of biologicvariability and measurement error so that true changecan be assessed. There are several methods availablefor doing this.

First, at some point early in the study, a set ofsamples or tapes should be designated as the "calibra-tion set." This set should be processed at the readingcenter or laboratory in a blinded fashion at regularintervals, with the results tabulated at the coordinatingcenter. This process serves to identify overall driftover time or the introduction of bias into the data. This

calibration set would also be used to train and certifyany new readers at the reading center. Second, a care-fully designed substudy should be carried out to assessinter- and intrareader variability. In such a study, thecoordinating center identifies a set of studies to bereprocessed and assigns multiple readers to each one.These studies are reread by the assigned readers in ablinded fashion, and the results give estimates of theinter- and intrareader variability.

Another important substudy involves having thefield center technician perform repeat studies on asubset of participants. Once estimates of inter- andintrareader variability are available, as describedabove, this type of study allows the estimation ofadditional variability due to field center technician andbiologic variability. (These two aspects of variabilityusually cannot be estimated separately.) Combiningthe reader effect and the technician effect gives anestimate of overall measurement error.

Measurement error can lead to either type I or typeII errors. For example, when a continuous outcomevariable in longitudinal data is measured with error, atype I error can occur. In such a case, a regressionanalysis looking at the relation between the outcomevariable and a set of exposure variables, adjusted forthe baseline value of the outcome variable, can show arelation between the observed change in the outcomeand the exposure variables even when no associationexists between these variables and the true change inthe outcome variable (N. David Yanez, Department ofBiostatistics, University of Washington, Seattle, WA,personal communication, 1997). High levels of ran-dom measurement error can lead to a type II error,obscuring a true relation that may exist.

In a cohort study, quality control analyses of dataprocessed at a reading center or laboratory over timeare crucial, even when processes seem automated. Onestudy (the Cardiovascular Health Study) found thatcholesterol levels had drifted significantly over a3-year period even though the laboratory used stan-dard processing and maintained Centers for DiseaseControl and Prevention certification during that timeperiod. In another case, a study found that certainelectrocardiography values jumped during 1 year inthe study, due to the hiring of a new reader at thereading center (Mary Ann McBurnie, Departmentof Biostatistics, University of Washington, Seattle,WA, personal communication, 1996). Often, simpleplots and graphs can quickly identify these types ofproblems.

As in the field centers, staff turnover in a readingcenter can cause quality problems, in particular prob-lems of drift. When such a problem is identified (usingone of the techniques described above, such as the


Dow



arch 2019


calibration set or longitudinal plots), several steps areneeded to solve the problem and correct the data.First, the reader must undergo a complete retraining/recertification process and must perform up to anacceptable level before processing additional studydata. Second, the data processed by the technicianshould be analyzed to see if a statistical correction ispossible. Sometimes drift can be corrected using sta-tistical adjustment. For example, in one acquired im-munodeficiency syndrome study, blood CD4 countsdrifted over time, so in order to analyze changes inCD4 levels in seropositive individuals, their valueshad to be adjusted according to the time point andcenter at which the sample was collected (40).

If statistical adjustment is not possible, completereprocessing of those data may be required. In theCardiovascular Health Study, all baseline ultrasounddata were reread by the set of readers who processedthe follow-up ultrasonograms collected 3 years later,because it was determined that both the software andthe reading techniques had changed too much in theintervening time to compare the two sets of data (LynnR. Shemanski, Department of Biostatistics, Universityof Washington, Seattle, WA, personal communication,1998).

Finally, quality control of reading center and labo-ratory data must include complete tracking proce-dures. Often, reading centers start a study with littleexperience processing the volume of data generated bya large, multicenter cohort study. The reading centermust be monitored to make sure that it has adequateinternal data processing and tracking systems in place,and the coordinating center must make sure that it hasadequate systems in place to track data between fieldcenters, reading centers, and the coordinating center.This will minimize data lost due to mishandling, mis-labeling, or other problems.

Role of the coordinating center

The coordinating center is generally responsible forperforming all of the quality control activities de-scribed above. Performing these activities requires amajor commitment of personnel, which should be an-ticipated during planning. The coordinating centermust be well-organized, with adequate systems inplace to ensure that all reports and quality controlanalyses are completed and distributed in a timely way(table 2) and that all sites respond to the reports asrequired. It is also responsible for making sure thatstudy communications are adequate and that all in-volved investigators are kept up-to-date on any qualitycontrol issues that exist. Thus, the value of the coor-dinating center is not to be underestimated and should

receive high priority in its support and maintenance(6), staffed with personnel experienced in data man-agement, quality control, and statistical support.

SUMMARY

As we have presented, it is evident that cohortstudies are confronted with their own special, non-trivial issues of quality assurance and quality control.Such studies are typically large-scale designs and in-volve an extensive amount of data to be collected andprocessed, the quality of which depends on a variety offactors related to study personnel and equipment. Thefact that data are collected over an extended period oftime and at several centers greatly increases the mag-nitude of the data processing task, significantly in-creasing the likelihood of discrepancies and measure-ment error in the data.

As presented in tables 1 and 2, the quality assuranceand quality control procedures span the entire courseof the study and include a multitude of tasks. Suchtasks are delegated to various committees and/or areundertaken by participating centers, all of which musttake responsibility for understanding, implementing,and following through on all procedures that maximizedata quality. The quality of the quality assurance/quality control process is highly correlated with thequality of the communication within and between cen-ters and all researchers. Maintaining standardization ofprocedures across centers and long-term stability ofequipment and analytic procedures are integral com-ponents of quality control.

In conclusion, the magnitude of the quality controlprocess in a multicenter longitudinal study should notbe underestimated, requiring a significant commitmentof study resources. The quality control process is keyto the integrity of the study, and an integral part of thedesign of the study. In a well-designed study, with agood quality control process and dedication to theprocess by the research team, the validity of the con-clusions of the cohort study can be established.

ACKNOWLEDGMENTS

This publication was supported by research grants 5U01-HL53940-04 and NOlHC85079from the National Heart,Lung, and Blood Institute.

REFERENCES

1. Freedland KE, Carney RM. Data management and account-ability in behavioral and biomedical research. Am Psychol1992;47:640-5.

2. Gassman JJ, Owen WW, Kuntz TE, et al. Data quality assur-


Dow



arch 2019

80 Whitney et al.

ance, monitoring, and reporting. Control Clin Trials 1995;16(2Suppl):104S-36S.

3. Prud'homme GJ, Canner PL, Cutler JA. Quality assurance andmonitoring in the Hypertension Prevention Trial. Hyperten-sion Prevention Trial Research Group. Control Clin Trials1989;10(3 Suppl):84S-94S.

4. Karrison T. Data editing in a clinical trial. Control Clin Trials1981;2:15-29.

5. Marinez YN, McMahan CA, Barnwell GM, et al. Ensuringdata quality in medical research through an integrated datamanagement system. Stat Med 1984;3:101-ll.

6. Bagniewska A, Black D, Molvig K, et al. Data quality in adistributed data processing system: the SHEP Pilot Study.Control Clin Trials 1986;7:27-37.

7. Severe JB, Schooler NR, Lee JH, et al. Ensuring data qualityin a multicenter clinical trial: remote site data entry, centralcoordination and feedback. Psychopharmacol Bull 1989;25:488-90.

8. Sforza VA. Quality data: what are they? Ann 1st Super Sanita1994;30:439-43.

9. Hohnloser JH, Puerner F, Soltanian H. Improving coded dataentry by an electronic patient record system. Methods Inf Med1996;35:108-11.

10. Tillotson JL, Gorder DD, DuChene AG, et al. Quality controlin the Multiple Risk Factor Intervention Trial Nutrition Mo-dality. Control Clin Trials 1986;7(3 Suppl):66S-9OS.

11. Dischinger P, DuChene AG. Quality control aspects of bloodpressure measurements in the Multiple Risk Factor Interven-tion Trial. Control Clin Trials 1986;7(3 Suppl):137S-57S.

12. Cooper GR, Haff AC, Widdowson GM, et al. Quality controlin the MRFIT local screening and clinic laboratory. ControlClin Trials 1986;7(3 Suppl):158S-65S.

13. Canner PL, Borhani NO, Oberman A, et al. The HypertensionPrevention Trial: assessment of the quality of blood pressuremeasurements. Am J Epidemiol 1991;134:379-92.

14. Keltner JL, Johnson CA, Beck RW, et al. Quality controlfunctions of the Visual Field Reading Center (VFRC) for theOptic Neuritis Treatment Trial (ONTT). Control Clin Trials1993; 14:143-59.

15. Anderson MM Jr, Boly LD, Beck RW. Remote clinic/patientmonitoring for multicenter trials. Optic Neuritis Study Group.Control Clin Trials 1996; 17:407-14.

16. Neaton JD, Duchene AG, Svendsen KH, et al. An examinationof the efficiency of some quality assurance methods com-monly employed in clinical trials. Stat Med 1990;9:115-24.

17. O'Connor RW, Miller AK. Managing data quality throughautomation. Toxicology 1987;47:109-18.

18. Fleming TR. Data monitoring committees and capturing rel-evant information of high quality. Stat Med 1993;12:565-73.

19. Sherbourne CD, Meredith LS. Quality of self-report data: acomparison of older and younger chronically ill patients. JGerontol 1992;47:S204-ll.

20. Habig RL, Thomas P, Lippel K, et al. Central laboratoryquality control in the National Cooperative Gallstone Study.Control Clin Trials 1983;4:101-23.

21. Christiansen DH, Hosking JD, Dannenberg AL, et al.

Computer-assisted data collection in multicenter epidemio-logic research. The Atherosclerosis Risk in CommunitiesStudy. Control Clin Trials 1990; 11:101-15.

22. Teppo L, Pukkala E, Lehtonen M. Data quality and qualitycontrol of a population-based cancer registry. Experience inFinland. Acta Oncol 1994;33:365-9.

23. Correa A, Stewart WF, Yeh HC, et al. Exposure measurementin case-control studies: reported methods and recommenda-tions. Epidemiol Rev 1994; 16:18-32.

24. Cook RR. Overview of good epidemiologic practices. J OccupMed 1991;33:1216-20.

25. Lepore P. The 10 essential activities of the quality assuranceunit. Qual Assur 1995;4:29-33.

26. Bergman LR. Measurement and data quality in longitudinalresearch. Eur Child Adolesc Psychiatry 1996;5(Suppl 1):28-32.

27. Arndt S, Tyrrell G, Woolson RF, et al. Effects of errors in amulticenter medical study: preventing misinterpreted data.J Psychiatr Res 1994;28:447-59.

28. Vantongelen K, Rotmensz N, van der Schueren E. Qualitycontrol of validity of data collected in clinical trials. EORTCStudy Group on Data Management (SGDM). Eur J CancerClin Oncol 1989;25:1241-7.

29. Blumenstein BA. Verifying keyed medical research data. StatMed 1993; 12:1535-42.

30. Johstone FD, Brown MC, Campbell D, et al. Measurement ofvariables: data quality control. Am J Clin Nutr 1981;34(Suppl4):804-6.

31. Knatterud GL. Methods of quality control and of continuousaudit procedures for controlled clinical trials. Control ClinTrials 1981;l:327-32.

32. van der Putten E, van der Velden JW, Siers A, et al. A pilotstudy on the quality of data management in a cancer clinicaltrial. Control Clin Trials 1987;8:96-100.

33. Atherosclerosis Risk in Communities (ARIC) manual 1-12.Chapel Hill, NC: National Heart, Lung, and Blood Institute ofthe National Institutes of Health, Collaborative Studies Coor-dinating Center, Department of Biostatistics, University ofNorth Carolina, 1987.

34. Cardiovascular Health Study manual of operations. CHS Co-ordinating Center, Seattle, Washington, 1989.

35. Sleep Heart Health Study manual of operations. SHHS Coor-dinating Center, Seattle, Washington, 1995.

36. Mullooly JP. The effects of data entry error: an analysis ofpartial verification. Comput Biomed Res 1990;23:259-67.

37. Reynolds-Haertle RA, McBride R. Single vs. double dataentry in CAST. Control Clin Trials 1992;13:487-94.

38. Cummings J, Masten J. Customized dual data entry for com-puterized data analysis. Qual Assur 1994,3:300-3.

39. Quan S, Howard BV, Iber C, et al. The Sleep Heart HealthStudy: design, rationale, and methods. Sleep 1997;20:1077-85.

40. Muiioz A, Carey V, Saah AJ, et al. Predictors of decline inCD4 lymphocytes in a cohort of homosexual men infectedwith human immunodeficiency virus. J Acquir Immune DeficSyndr 1988;l:396-404.


Dow



arch 2019

Documents

Quality Assurance and Quality Control in Longitudinal Studies...QUALITY ASSURANCE Overview As stated above, quality assurance consists of those activities undertaken prior to data