c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 1 0 9 ( 2 0 1 3 ) 86–91

j o ur nal homep age : w ww.int l .e lsev ierhea l th .com/ journa ls /cmpb

QAIT: A quality assurance issue tracking tool to facilitatethe improvement of clinical data quality

Yonghong Zhanga,1, Weihong Suna,1, Emily M. Gutchell b,2, Leonid Kvechera,Joni Kohra, Anthony Bekhasha, Craig D. Shriverb, Michael N. Liebmana,3,Richard J. Murala, Hai Hua,∗

a Windber Research Institute, Windber, PA 15963, United Statesb Walter Reed National Military Medical Center, Bethesda, MD 20889, United States

a r t i c l e i n f o

Article history:

Received 22 July 2011

Received in revised form

18 May 2012

Accepted 15 August 2012

Keywords:

a b s t r a c t

In clinical and translational research as well as clinical trial projects, clinical data collection

is prone to errors such as missing data, and misinterpretation or inconsistency of the data.

A good quality assurance (QA) program can resolve many such errors though this requires

efficient communications between the QA staff and data collectors. Managing such com-

munications is critical to resolving QA problems but imposes a major challenge for a project

involving multiple clinical and data processing sites. We have developed a QA issue track-

ing (QAIT) system to support clinical data QA in the Clinical Breast Care Project (CBCP).

Quality assurance

Issue tracking

Data acquisition

Data management

This web-based application provides centralized management of QA issues with role-based

access privileges. It has greatly facilitated the QA process and enhanced the overall quality

of the CBCP clinical data. As a stand-alone system, QAIT can supplement any other clinical

data management systems and can be adapted to support other projects.

customized clinical data management systems due to restric-

Clinical data

1. Introduction

Collecting and ensuring high quality clinical data is critical tothe success of clinical and translational research projects aswell as clinical trials. For clinical and translational research,a mixture of paper questionnaires, electronic questionnaires,or electronic medical records is being used for clinical datacollection. For clinical trials, historically such collection was

performed using a special paper form called Case Report Form(CRF, similar to a questionnaire), and the data were entered,cleaned, and stored for later use. Machine-readable forms are

∗ Corresponding author at: Windber Research Institute, 620 7th Street, Wfax: +1 814 467 6334.

E-mail address: h.hu@wriwindber.org (H. Hu).1 Current address: H. Lee Moffitt Cancer Center, Tampa, FL, United St2 Current address: Risk Management Department, MedStar Health, C3 Current address: Strategic Medicine, Kennett Square, PA, United Sta

0169-2607/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights reshttp://dx.doi.org/10.1016/j.cmpb.2012.08.010

© 2012 Elsevier Ireland Ltd. All rights reserved.

also used, e.g. Teleform [1]. Recently, electronic data collection(EDC) has increased considerably [2,3], using a variety of self-developed, commercially available, or open source software.For about half of clinical trials, clinical data are now cap-tured electronically [2,4]. Commercial software such as OracleClinicalTM [5] and MedidataTM [6] are more likely to be usedby industry-sponsored trials, but academic clinical researchcenters are still more likely to develop and deploy their own

indber, PA 15963, United States. Tel.: +1 814 361 6903;

ates.olumbia, MD, United States.tes.

tions of available funds and trial-specific needs [2,3].It is the general trend that collection of clinical data is

becoming more electronic instead of being mediated by paper

erved.

s i n

fd(pfisatpbedSm

cadEcnrcn

tpimtBPbmQdimld[tt

neiiOmquWQttaa

n

c o m p u t e r m e t h o d s a n d p r o g r a m

orms. With either approach, quality improvement of clinicalata after data collection, here termed as quality assurance

QA), is very important. Nahm et al. reviewed 42 availableublications containing quantitative source document veri-cation (SDV) of recorded data that compared original datauch as medical records to the data on the CRF; they reportedn average error rate of 976 errors per 10,000 data points [7]. Onhe contrary, the error rate of CRF-to-database is only 14 errorser 10,000 data points [7,8]. When using a paper form, doublelind data-entry is effective in reducing the CRF-to-databaserror [9,10]. Other QA measures, such as logic checks, outlieretection, unusual data pattern identification, and targetedDV, are also applied; study monitoring is an important QAeasure in clinical trials [11].Correction of errors identified by the QA measures typi-

ally requires coordinated efforts from data collection sitesnd data processing centers, which are often geographicallyistributed in a translational research or clinical trial project.rror correction cannot be always completed in one round ofommunication, thus an effective communication channel iseeded to keep track of the whole QA procedure from erroreporting to the completion of error correction. In our specificase, the Clinical Breast Care Project (CBCP), we faced such aeed.

CBCP is a multi-site clinical and basic research program ini-iated in 2000, with a long term goal of improving the care ofatients with breast cancer [12]. Patients are enrolled follow-

ng HIPAA-compliant, IRB approved protocols at participatingedical centers. Currently, participating medical centers are

he Walter Reed National Military Medical Center (WRNMMC),ethesda, MD, the Joyce Murtha Breast Care Center, Windber,A, and the Anne Arundel Medical Center, Annapolis, MD. Aroad range of clinical data is collected from participants usingultiple, stand alone, paper-based questionnaires. The Coreuestionnaire, completed by a trained nurse case manageruring a patient interview, covers a wide range of information

ncluding demographics, family history, reproductive history,edical history, lifestyle, risk factors, etc. A Pathology Check-

ist, which is completed by a pathologist covering detailediagnosis, disease characteristics, and tissue characteristics

13–15] is generated for patients undergoing a biopsy. Thesewo questionnaires alone contain 799 data fields capturedhrough hundreds of questions.

By definition, clinical data QA is performed after question-aire completion. The CBCP project has made it a priority tonsure the completeness and accuracy of its clinical data bydentifying and correcting errors when possible. The effortsnclude (1) special training of the staff following the Standardperating Procedures (SOPs) which are updated as needed, (2)anual review by the clinical data QA team at WRNMMC after

uestionnaire completion for obvious errors and missing val-es, (3) double data entry by two data entry specialists at theindber Research Institute (WRI) to reduce data entry error, (4)

A by the lead data entry specialist who communicates withhe clinical data QA team for error correction, and (5) execu-ion of a computer program termed QAMetrics which applies

ccumulated QA rules to the data committed to the databasend reports anomalies (errors) for resolution.

With our QA efforts we were able to capture a significantumber of errors before they were finally loaded to our data

b i o m e d i c i n e 1 0 9 ( 2 0 1 3 ) 86–91 87

warehouse for subsequent research use. Resolving QA issuesdemands intensive communications between the data pro-cessing team and the clinical data QA team who have access tonurses, pathologists, medical records, and patients if needed.Historically the staff relied on phone calls and emails for com-munications, which were often inadequate, especially whenmultiple people from multiple sites were involved in resolv-ing a complicated QA issue that required multiple rounds ofcommunication imposing a bottleneck for CBCP clinical datacollection. Resolving QA problems demands an effective com-munication channel between multiple domain experts, as wellas proper access and storage of relevant information. Ourneeds in this project are: (1) a web-based system to enableconvenient access from multiple sites; (2) a systematic track-ing capability to track the problem resolving process involvingmultiple data collection and processing sites; (3) role manage-ment capability to enable different users to perform differentresponsibilities; (4) reporting capability to enable generation ofspecific reports for improvement of data collection; (5) light-weight to enable easy deployment; (6) minimal, if any, licensefees.

Tools have been developed for issue tracking and manage-ment in many other areas such as software engineering. Wefound these insufficient for our needs because they addressdifferent problems than the ones we face. Discrepancy flag-ging and management are provided as a function in someclinical data management systems [5,16,17], and developmentof tools to supplement and enhance discrepancy managementfor Oracle ClinicalTM has been reported [18]. However adoptionof the entire system is usually required and even this does notmeet all of our needs. In the end, we were unable to identifyan existing light-weight, free application that could meet all ofour needs. Therefore, we developed our own system, called QAissue tracking system (QAIT), which we believe will be usefulfor other projects facing similar problems.

2. Methods

QAIT is a web application consisting of a PHP applicationembedded with Javascript functions supporting a MySQLdatabase. Through several user-friendly interfaces, it providesusers with role-based access to information and functionsfor system administration, creation and management of QAissues, and search, report and export of QA issues. QAIT canbe accessed by a user from anywhere there is a network. Fig. 1Ashows the architecture of the system.

2.1. Information tracked

The system tracks all necessary information about a QA issue,its solution, and the process for resolving it. For each QAissue, the tracked information includes the action and event(open and close of issue); the questionnaire from which the

issue is identified (questionnaire/version, date, barcode, sub-ject ID, etc.); information about the issue (date, title, reviewer);response to the issue and final decision (date, user, and detailsof response).

88 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 1 0 9 ( 2 0 1 3 ) 86–91

Fig. 1 – (A) Diagram of system architecture; (B) Flowchart of QAIT functions with color scheme showing accessibility tofunctions by user roles. (For interpretation of the references to color in this figure legend, the reader is referred to the web

version of this article.)

2.2. User role and control of access

In QAIT, access to information and functions are controlledby user roles. There are four user roles: manager, question-naire reviewer, issue responder and guest user. Questionnairereviewer and issue responder roles are associated with indi-vidual questionnaire forms. The questionnaire reviewer isresponsible for creating issues and entering questionnaireinformation. The issue responder provides information toresolve issues. Modification to an issue is also dependent onits status. The QAIT manager is responsible for questionnaireand user management (Fig. 1B).

2.3. Implementation of logical procedure for errorcorrection

The logical error correction procedure is implemented in QAITthrough the status of the questionnaire as well as the QA issue.The status of a questionnaire includes; in-review, ready for QA,response to issue, issue(s) with response, and issue free. Thestatus of an issue can be open, open with response, or closed.When a questionnaire reviewer identifies a QA issue, the ques-tionnaire is added in QAIT and its status is “in-review”. Theissue is then added to the questionnaire with an issue status of

“open”. Multiple issues may be recorded for one questionnaire.A questionnaire status of “in-review” alerts all users that thequestionnaire is still under review. The reviewer sets the ques-tionnaire status to “ready for QA” after all identified issues are

recorded. An issue responder can then claim the question-naire and provide information to resolve each issue and thequestionnaire status is changed to “response to issue”. Thestatus of the questionnaire will be changed to “issue(s) withresponse” when all issues have responses. The questionnairereviewer reviews each responded issue and will close it asappropriate. The questionnaire becomes “issue free” when allissues are closed. A “closed” issue cannot be further modifiedbut can be re-open by a QAIT manager.

2.4. Query and report

QAIT provides a simple interface to search the database forquestionnaires by barcode or subject ID (Fig. 2A). A more pow-erful interface for searching QA issues with multiple criteriais available, generating a PDF (Fig. 2B and C). Summary reportscan also be generated based on the user or the time (by week,month, quarter, and year). Such reports, in the form of a flatfile, can be imported to Excel Spreadsheet or SAS or othersoftware for analysis.

2.5. System requirement, security, and availability

QAIT requires a web server that supports PHP, such as

Microsoft IIS or Apache. Generating a report in PDF formatrequires FPDF [19], a freeware application for creating ad hocPDF documents in PHP. Information in QAIT is secured by useraccount and roles within a firewall. No information in the

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 1 0 9 ( 2 0 1 3 ) 86–91 89

Fig. 2 – QAIT user interfaces. (A) QA issue summary report for questionnaires. (B) Advanced issue query and report. (C) QAissue query result reported in PDF.

90 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 1 0 9 ( 2 0 1 3 ) 86–91

Table 1 – Representative sections of questions in the Core Questionnaire with highest number of QA issues.

# of issues # of questionsinvolved

# of issues/question

Error rate/questionnairea

DOB/Age (Q3, Q4) 119 2 59.50 0.031Biopsy section (Q141–Q151) 569 12 47.42 0.025Menopausal section (Q53–Q59) 124 7 17.71 0.009Alkaline phosphatase section (Q160–Q162) 114 3 38.00 0.020Medication (Q138, Q139) 147 2 73.50 0.038

r the

QAIT provides a centralized platform for systematic informa-tion collection and management for QA error identificationand correction. It generates reports on QA issues which can

a The error rate was calculated by dividing # of issues/question oveQuestionnaire contains 225 questions.

system can be deleted via the user interface. QAIT can beaccessed by users outside of the network through Virtual Pri-vate Network (VPN). The underlying database is backed updaily. QAIT is available upon request.

3. Application to the motivating CBCP case

QAIT has greatly enhanced the clinical data QA efficiencyin CBCP, by eliminating communications via phone callsand emails, and reducing spreadsheet and word documentbookkeeping efforts. The staff can readily access relevantinformation via the web without having to refer to notes oremails or other records associated with the QA issue, andchecking the status of an issue and assigning/claiming anissue became straight forward. The time-saving benefit of theQAIT is not obvious for resolving simple QA problems, whenonly one QA staff and one round of communication is needed.However, for many QA problems, multiple staff members andmultiple rounds of communications are needed, and the clin-ical staff is often not immediately available to address theidentified problems. Digging into accumulated documents andemail tracks as well as properly forwarding relevant infor-mation to the responsible QA staff member became a realinformation management challenge for the QA team leader.QAIT resolved this communication bottleneck of CBCP clini-cal data QA management, and enabled the QA team to cleara six month backlog of QA issues in several weeks. The sys-tem hosted 4617 QA issues from 1910 questionnaires spanning2005 Q4 to 2008 Q2, the period for which the QA data wereanalyzed and reported here. The problem-resolving process isdocumented; periodic reports are generated and reviewed.

QAIT effectively identified questions or sections of ques-tions that had high error rates. Identified problems aresometimes a confirmation but now with hard evidence, andother times are previously unknown. Table 1 shows represen-tative sections of questions with the highest number of QAissues in the Core Questionnaire that we derived using flatfile reports generated by QAIT. For example, the “Biopsy sec-tion” had the highest number of issues, and the error rate perquestion per questionnaire was also among the highest. Sub-sequent analysis show that the original design of this sectionwas to capture historical biopsies in both breasts, but somenurses mistakenly counted in the current biopsy. This finding

supported a systematic auditing of all the biopsy data, a revi-sion of the question and the SOP, and re-training of nurses toensure the accuracy of the captured biopsy data. Date-of-birth(DOB) and Age also bear high error rate, which was somewhat

total of 1910 questionnaires with errors tracked in QAIT. The Core

surprising. Further analysis shows that some patients, inten-tionally or unintentionally, reported a younger age than whatwas derived from the date-of-enrollment and the DOB. Thisfinding re-affirms the need to build in redundant questions inclinical data collections.

With drastically improved communication efficiency, theQA team was able to spend more time focusing on resolvingQA problems and developing new methods. Using the datacaptured in the QAIT, we were able to monitor the error rate asthe CBCP QA program improves with time. For questionnaireswith errors, the number of errors per questionnaire reducedfrom 3.9/questionnaire when QAIT was implemented in 2005Q4, to 1.9/questionnaire in 2008 Q2 (Fig. 3). Such hard evi-dence would not have been derivable without the systematictracking of the QA issue resolving process.

Another important benefit of the QAIT is that with thehistory of the QA issue resolving process tracked, an organi-zational memory is maintained which is not gone with thedeparture of a staff member. QAIT also plays an important rolein training new QA team members with the availability of con-crete QA issues and the track record how they were resolved,as a supplemental to the training on the SOP.

4. Discussion and conclusions

Fig. 3 – Total number of issues identified (bar) and numberof issues per questionnaire (line) for different quartersfollowing the deployment of QAIT.

s i n

besip

ticiqbswti

C

N

A

TwHMtrDTS[

r

c o m p u t e r m e t h o d s a n d p r o g r a m

e used as a reference for questionnaire improvement or SOPnhancement. This system has greatly helped the CBCP QAtaff to improve QA issue management, and is playing a crit-cal role in CBCP clinical data collection and managementrocesses.

Although the development of the QAIT was motivated byhe CBCP, the system was designed for general QA issue track-ng involving multiple data collection and processing sites. Itan be applied to any clinical data collection settings. The QAssue reports can be generated in flat file format for subse-uent statistical analysis. QAIT currently uses MySQL as theackend database but the design was not technology-specifico it can be adapted to Oracle or other RDBMS. As a stand-aloneeb-based application, the system can be adopted or adapted

o supplement the capability of academic or commercial clin-cal data management software.

onflict of interest

one.

cknowledgments

his work was supported by the Clinical Breast Care Projectith funds from the US Department of Defense through theenry Jackson Foundation for the Advancement of Militaryedicine, Rockville, MD. The opinions and assertions con-

ained herein are those of the authors and do not necessarilyeflect the official positions of the United States Army, theepartment of Defense, or the United States government.his work has been presented as a poster at the 27th Annualymposium of the American Medical Informatics Association

20].

e f e r e n c e s

[1] J.M. Hardin, L.L. Woodby, M.A. Crawford, R.A. Windsor, T.M.Miller, Data collection in a multisite project: teleform, PublicHealth Nursing 22 (2005) 366–370.

[2] K. El Emam, E. Jonker, M. Sampson, K. Krleza-Jeric, A. Neisa,The use of electronic data capture tools in clinical trials:web-survey of 259 Canadian trials, Journal of MedicalInternet Research 11 (2009) e8.

[3] W. Kuchinke, C. Ohmann, Q. Yang, N. Salas, J. Lauritsen, F.Gueyffier, A. Leizorovicz, C. Schade-Brittinger, M.Wittenberg, Z. Voko, S. Gaynor, M. Cooney, P. Doran, A.Maggioni, A. Lorimer, F. Torres, G. McPherson, J. Charwill,

b i o m e d i c i n e 1 0 9 ( 2 0 1 3 ) 86–91 91

M. Hellstrom, S. Lejeune, Heterogeneity prevails: the state ofclinical trial data management in Europe – results of asurvey of ECRIN centres, Trials 11 (2010) 79.

[4] Website, Center Watch: EDC adoption in clinical trials: a2008 analysis. <http://www.bio-itworld.com/EDC-Adoption.aspx?terms=health>.

[5] Website, Oracle Clinical. <http://www.oracle.com>.[6] Website, Medidata. <http://www.mdsol.com/>.[7] M.L. Nahm, C.F. Pieper, M.M. Cunningham, Quantifying data

quality for clinical trials using electronic data capture, PLoSOne 3 (2008) e3049.

[8] B.A. Blumenstein, Verifying keyed medical research data,Statistics in Medicine 12 (1993) 1535–1542.

[9] R.A. Reynolds-Haertle, R. McBride, Single vs. double dataentry in CAST, Controlled Clinical Trials 13 (1992) 487–494.

[10] S. Day, P. Fayers, D. Harvey, Double data entry: what value,what price? Controlled Clinical Trials 19 (1998) 15–24.

[11] C. Baigent, F.E. Harrell, M. Buyse, J.R. Emberson, D.G. Altman,Ensuring trial validity by data quality assurance anddiversification of monitoring methods, Clinical Trials 5(2008) 49–55.

[12] Website, The Clinical Breast Care Project.<http://www.cbcp.info/>.

[13] H. Hu, H. Brzeski, J. Hutchins, M. Ramaraj, L. Qu, R. Xiong, S.Kalathil, R. Kato, S. Tenkillaya, J. Carney, R. Redd, S.Arkalgudvenkata, K. Shahzad, R. Scott, H. Cheng, S. Meadow,J. McMichael, S.L. Sheu, D. Rosendale, L. Kvecher, S. Ahern, S.Yang, Y. Zhang, R. Jordan, S.B. Somiari, J. Hooke, C.D. Shriver,R.I. Somiari, M.N. Liebman, Biomedical informatics:development of a comprehensive data warehouse forclinical and genomic breast cancer research,Pharmacogenomics 5 (2004) 933–941.

[14] L. Bronfman, C.D. Shriver, E. Gutchell, Chapter 2. The clinicalperspective, in: H. Hu, R.J. Mural, M.N. Liebman (Eds.),Biomedical Informatics in Translational Research, ArtechHouse, Boston, London, 2008, pp. 11–29.

[15] J. Hooke, L. Fantacone, C.D. Shriver, Chapter 3. Tissuebanking: collection, processing, and pathologiccharacterization of biospecimens for research, in: H. Hu, R.J.Mural, M.N. Liebman (Eds.), Biomedical Informatics inTranslational Research, Artech House, Boston, London, 2008,pp. 31–41.

[16] Website, OpenClinica. <http://www.OpenClinica.org>.[17] Website, ClinAccess/PowerServer.

<http://www.majaro.com>.[18] N. Trocky, M. Fontinha, Quality Management Tools:

Facilitating Clinical Research Data Integrity by UtilizingSpecialized Reports with Electronic Case Report Forms,AMIA Annual Symposium Proceedings, 2005, p. 1142.

[19] Website, FPDF. <http://www.fpdf.org>.[20] Y. Zhang, W. Sun, E.M. Gutchell, H. Hu, M.N. Liebman, C.D.

Shriver, R.J. Mural, An issue tracking system to facilitate theenhancement of clinical data quality in the clinical breastcare project, in: AMIA Annual Symposium Proceedings,2007, p. 1173.