The use of handheld computers in clinical trials

Controlled Clinical Trials 23 (2002) 469–480

0197-2456/02/$—see front matter © 2002 Elsevier Science Inc. All rights reserved.PII: S0197-2456(02)00224-6

ARTICLE IN PRESS

The use of handheld computers in clinical trials

Andreas Koop, M.Sc.*, Ralph Mösges, M.D., Ph.D.

Institute for Medical Statistics, Informatics, and Epidemiology, University of Cologne, Cologne, Germany

Manuscript received October 3, 2001; manuscript accepted April 22, 2002

Abstract

A recently completed, randomized, double-blind placebo-controlled clinical trial is presented inwhich Palm handheld computers were used as a substitute for normal paper-based patient diaries. In thisnasal provocation study, a common antihistamine approved for the treatment of seasonal allergic rhinitiswas tested against placebo for evidence of additional properties. In addition to their medical examina-tions, the 12 study volunteers rated subjective complaints in a diary program on 4 examination days, fora duration of 4.5 hours every 15 minutes at each visit. This resulted in 903 data sets consisting of fivequestions each, or 4515 data points total. In this study the use of handheld computers resulted in an in-crease in data quality and shortened the time needed to close the database. Moreover, the benefit of elec-tronic reminders for protocol compliance is clearly demonstrated. Our findings support the results foundin the literature we reviewed. For more than 16 years, mobile computers have been supporting the imple-mentation of clinical trials. Our review of 27 articles out of more than 100 clinical trials in which mobilecomputers have been used elaborates on the advantages and problems of this technology. We give acomprehensive overview of the various technologies as used in different settings, and then discuss themethodology of using mobile devices in comparison to traditional methods, the considerations that needto be made and things to be avoided in order to conduct a successful clinical trial with mobile tools. Weconclude that mobile devices are very useful in most cases, especially when design and software valida-tion aspects have been taken into account. © 2002 Elsevier Science Inc. All rights reserved.

Keywords:

Pen-based computing; Handheld electronic diary; Software validation; Clinical trials; eCRF; Comput-

erized data collection; Remote data entry; Mobile data capture; Automatic reminders

Introduction

Technology has been used to support clinical trials for many years. Databases and statisti-cal software packages were the first technological aids to be utilized, followed by computers

* Corresponding author: Andreas Koop, University of Cologne, Institute for Medical Statistics, Informatics, and

Epidemiology, Joseph-Stelzmann-Str. 9, 50931 Cologne, Germany. Tel.:

�

49-221-4783460; fax:

�

49-221-4783465.E-mail address: [email protected]

470

A. Koop, R. Mösges/Controlled Clinical Trials 23 (2002) 469–480

ARTICLE IN PRESS

for on-site data entry. Thereafter, network solutions such as the French Minitel or the Ger-man BTX (Bildschirmtext) were employed. In the meantime, the Internet and especially theWorld Wide Web have increasingly been utilized in clinical trials as well as in many otherfields of business and science [1,2]. Interestingly, a similar development can be observedwith mobile devices. Nearly every mobile technology capable of data capture or data entryhas and will be employed in clinical trials. A wide range of devices has been used, from theearly programmable calculators and organizers to sophisticated pen-based devices with wire-less data transmission capabilities [3,4]. We used Palm personal digital assistants instead ofpaper patient diaries in a randomized, placebo-controlled, double-blind clinical trial and de-scribe here our methods and experience.

Objectives

A monocentric randomized two-phase cross-over study was carried out to verify whetherFexofenadine 120 mg tablets compared to placebo relieves nasal obstruction in subjects with ahistory of seasonal allergic rhinitis. In the following description of our experience with thehandheld devices, we focus on data quality, patient acceptance, study settings, software valida-tion, problems, and protocol compliance. The medical results are to be published elsewhere.

Setting and examinations

The study was approved by the local ethics committee. Twelve volunteers with a history ofseasonal allergic rhinitis received Fexofenadine 120 mg once daily or placebo during a first pe-riod of 13 days. On the last day of that period immediately following drug intake, the volun-teers underwent nasal provocation with their respective allergen. The same procedure had beencarried out at baseline before drug intake. Thereafter, a washout period of 8 days followed. Thevolunteers were then crossed over to the alternate medication for 13 days. All participants vis-ited the study center and were seen by an ears, nose and throat (ENT) specialist for examina-tions on the following dates: January 27, February 10, February 17 and March 3, 2001, all ofwhich did not fall within the local pollen season for the volunteers’ individual allergens. Theschedule of each of the visits included an initial ENT examination and intake of the drug/pla-cebo 15 minutes before provocation with allergens. Besides undergoing repeated nasal air flowand endoscopic examinations of the nasal mucosa 15, 45, 135 and 270 minutes after provoca-tion, all volunteers used a handheld computer with the patient diary program “ClinDiary” torecord their ratings of itchiness of the nose, runny nose, obstruction of the nose, sneezing andremote symptoms such as itchiness of the eyes, ears and the throat. These subjective data wereto be recorded every 15 minutes for 4.5 hours after receiving the computers.

Materials and methods

We used four different models of Palm handheld computers: PalmPilot (operating system:PalmOS 2.0, memory: 1 MB), Palm III (PalmOS 3.0, 2 MB), Palm IIIe (PalmOS 3.3, 2 MB)and Palm m100 (PalmOS 3.5.1, 2 MB). In spite of different operating system versions, thehandling was identical for all models. Of the four keys for starting programs, the rightmost


471

ARTICLE IN PRESS

three were set to start ClinDiary; the default was not altered for the leftmost key, which startsthe datebook application.

ClinDiary was developed using Waba 1.0, a Java version from Wabasoft (

��

http://www.wabasoft.com

��

) with its own class library and additional classes, called the Waba Extras1.30 (

��

http://www.cygnus.uwa.edu.au/~rnielsen/wextras/

��

). Both can be downloadedfor free via the Internet. In order to run ClinDiary the Waba Virtual Machine must be in-stalled as well. We chose Waba because it was one of the first Java versions available forhandheld computers. Currently, other Java versions are also available such as the Java MicroEdition from Sun Microsystems and many other programming languages (e.g., C, C

��

orVisual Basic) from different vendors. In addition, one can use form-generating tools witheasy database connectivity such as Satellite Forms from Pumatech or Pendragon Forms fromPendragon Software. We recommend using such software provided all functionality needsare met; otherwise a higher programming language such as Java is more appropriate. Shouldboth of these methods be insufficient, C or C

��

are good alternatives.Fig. 1 shows the possible program sequence of ClinDiary. After being started with a pro-

gram key or via the graphical user interface ClinDiary asks the volunteer to answer the re-quired questions. While changing to the data entry screen, the data entry schedule (every 15minutes from the start for the next 4.5 hours) is written into the internal datebook applicationdatabase. The device is designed to give visual and acoustic reminders. Data entries arechecked for completeness, and the volunteer receives a warning message if an entry is miss-ing. It is important to stress that a normal PalmOS application is closed every time a newprogram is started and can always be closed by tapping on the home/applications icon. In or-der to protect supplied data, the data are saved every time the program is closed, even if va-lidity checks have not been carried out. Thus, missing values (unanswered questions) can betransferred to the database. If the program is restarted within 10 minutes, the user can chooseto revise the last data entry or to make a new entry.

After data entry has been completed and the volunteer leaves the site, a medical assistantcollects the device and synchronizes the data with a personal computer (PC). The problem ofhaving different docking stations for three of the four different models was solved with a me-chanical switch. The data are transmitted in a simple table format (Table 1), which can beeasily imported into data analysis programs such as Excel or SPSS.

At the time of transmission to the PC not only are the ClinDiary data synchronized, butalso the data of every other application (e.g., the datebook program). Thus, we are able to an-alyze the datebook entries made by ClinDiary and all other changes the volunteer made usingother programs. Subsequently, a backup of the whole device is performed using the softwareBackupBuddy (BlueNomad Inc.), so even months later the state of the device when the vol-unteer stopped using it can be restored. This can be regarded as a “certified copy,” as statedin the Food and Drug Administration (FDA) recommendations, if an access rights manage-ment does not allow alteration of this data later [5].

Results

During the test phase of ClinDiary before its first use in a study, two problems arose thatcould not be solved without altering the programming tools. First, changing data in the data

472


ARTICLE IN PRESS

entry screen in rapid succession resulted in a crash of the program. Second, activation ofmore than one answer per question was possible when using the pen very quickly, whereasthe program design allows only one selection. These possible errors were accepted becausethe effort of optimizing the programming tools, which were available in source code, was notworth the expense. Most notably, none of these errors occur when the user waits the 0.5 sec-onds necessary to update the screen after each touch of the pen.

On the first visit the program developer distributed the devices to the volunteers to witnesstheir acceptance as required by the method of “usage testing” [6]. The developer served as acontact person in case of technical problems at the first visit. Interestingly, entirely differentproblems arose than in the previous tests, whereas none of the known errors occurred. Clin-

Fig. 1. Possible program sequence.


473

ARTICLE IN PRESS

Diary crashed six times out of 204 data set entries at five questions at a time, because the in-ternal event handler failed when a volunteer tried to answer all five questions within 1 sec-ond. The data entry screen needed about 0.5 second to refresh after an entry had been made.If graphic elements of the screen were used while it was refreshing, the above-mentionedproblem rarely happened. After requesting that the volunteers enter the data more slowly, theproblem did not occur again. A program crash such as this did not have any consequencesother than that ClinDiary had to be restarted and the last data entry repeated. The internaldatebook application crashed once, so the Palm handheld had to be reset, with no data loss.All things considered, the volunteers used the computers diligently, having no problems be-sides the ones mentioned above. An initial explanation lasting 5–10 minutes seemed to suf-fice and was given upon distribution of the devices at Visit 1. A study nurse was available toanswer questions at all visits.

Missing values

The analysis of the data received at Visit 1 (which is also the case for Visits 2 to 4) re-sulted in no missing values within the data set of five questions. In five cases, however, awhole data set was left unanswered. Three cases were similar to that in Table 1 in which thevolunteer recognized the error and repeated the data entry immediately afterward.

The cause for these missing values could come from inadvertently quitting ClinDiary(e.g., by pressing a program key before entering the required data). This kind of missingvalue was found six times at Visit 2, of which four were corrected immediately as in Table 1and four occurred with the same volunteer. The data for Visit 3 showed no missing values,and the data for Visit 4 showed only one, which was also amended right away.

The experience gained at Visit 1 resulted in program improvements for reducing the al-ready low number of missing values and above all for minimizing the consequences of a cer-tain error not detected during program testing. The testing procedure itself, which includedmodification of the datebook and synchronization with the PC, did not allow the error to sur-face in contrast to a full-length test in the study setting: Due to missing internal program in-terrupts, the Palm handhelds gave neither an acoustic alarm nor the associated visual mes-sage every 15 minutes according to the schedule of the study protocol. These procedureswere fully understood only after Visit 2, resulting in only two Palms giving alarms at Visit 1and three at Visit 2. The other volunteers were asked to make their data entries every 15 min-utes according to the schedule in the datebook application. One volunteer used a type of egg

Table 1. Structure of the data sets

Patient ID Q1 Q2 Q3 Q4 Q5 Time Date stamp

Pat2 0 0 1 0 0 11:11 27.01.2001Pat2

�

1

�

1

�

1

�

1

�

1 11:25 27.01.2001Pat2 0 0 1 0 0 11:26 27.01.2001Pat2 0 0 1 0 0 11:40 27.01.2001

Coding:

�

1

�

question not answered (missing value), 0

�

none, 1

�

mild, 2

�

moderate, 3

�

severe, se-quence in order of questions (e.g., Q1

�

question 1

�

itchiness of the nose).

474


ARTICLE IN PRESS

timer, while two others programmed the datebook application of their cellular phones to 15-minute intervals. Because the Palm itself contains a clock and the datebook application withthe appointments, volunteers looked at their datebooks. The volunteers suggested improvingClinDiary so it would automatically delete previous data entry requests in the datebook ap-plication. It was interesting to see that the volunteers themselves developed these problem-solving strategies.

Program improvements

Before Visit 2 ClinDiary was updated to version 1.1 with the following modifications andadditions:

• ClinDiary’s request to make a decision (Fig. 1), which appeared if the last data entrywas made within 10 minutes of the previous one, was supplemented with a quit button.Thus, the volunteers who did not know how to stop a palm program and unintentionallystarted ClinDiary did not return to the data entry screen again and were therefore unableto produce missing values.

• A report function was introduced to log all user actions within the datebook application.This way the volunteers knew which data entry request was the current one. The follow-ing list depicts all reports: (a) Data received. Thank you. (b) Data entry from [date,time] has been corrected. Thank you. (c) At least one question was not answered. Pleasecorrect it. (d) Last data entry from [date, time] still not complete. Please correct it!

• If data were corrected in ClinDiary 1.0, the logged time of the data entry was changed tothe time the data were amended. In version 1.1 the original time of data entry was pre-served.

The latter of the three modifications raises the question as to how often entered data hadbeen corrected: At Visit 2 the volunteer who compensated for three of four missing valueswith new data entries corrected four additional missing values. At Visit 3 one missing valuewas corrected and at Visit 4 four missing values were corrected. Only one complete and val-idated data set was changed later. The datebook application was used to manage one volun-teer’s medical examination appointments. The logging of the volunteers’ actions in the date-book application is a first step toward an “audit trail” as stated in the FDA recommendations,but without fulfilling all mentioned conditions (e.g., the volunteer would have been able tomanipulate the datebook entries) [5]. In most cases, however, this would have led to discrep-ancies between the ClinDiary data, which cannot be manipulated, and the logged data, so amanipulation could have been detected.

Analysis of compliance with the protocol

The lack of computer reminders at Visits 1 and 2 permits a comparison with Visits 3 and4, where all devices worked according to plan. The question arises as to how significantly thePalm datebook alarm affects the data entry interval of 15 minutes for 4.5 hours, as requiredby the study protocol. Fig. 2 shows the deviation from the data entry schedule as stated in thestudy protocol. The score used is the sum of the absolute differences between the requiredtime of a data entry and the time of the actual data entry per volunteer. If a data entry is miss-


475

ARTICLE IN PRESS

ing completely, a penalty of 15 minutes was added. In Table 2 the mean and standard devia-tion of each visit is listed and the

p

-values resulting from a Wilcoxon test to the significancelevel

��

0.05 comparing Visit 1 with Visits 2–4. A significant difference in protocol compli-ance can be observed between Visit 1 and the Visits 3–4.

The positive effect of computer reminders on protocol compliance is clearly shown in Fig.2. At Visit 1 the procedure and the handheld computers were new to the volunteers and thereminder function did not work as planned. At Visit 2 the volunteers knew they could notrely on the reminder function, thus finding other ways to keep track of the data entry times.At Visits 3 and 4 the reminders functioned correctly. In order to avoid confusion, each devicewas labeled with the volunteer’s number and name.

Study timing

The data obtained by the handheld computers were ready for statistical analysis immedi-ately after the last computer was turned in at Visit 4, whereas it took more than 2 months toanalyze the data of the medical examinations, to type them into a database, and finally tocarry out the statistical evaluation.

Fig. 2. Deviation to the given schedule of data entries (protocol compliance).

Table 2. Mean, standard deviation and

p

-values with comparison to Visit 1

Mean SD

p

-value

Visit 1 83.67 45.71 —Visit 2 65.67 43.48 0.209Visit 3 26.00 9.89 0.002Visit 4 26.42 13.34 0.004

476


ARTICLE IN PRESS

Discussion

Mobile devices have been used in clinical trials for more than 16 years, utilizing differenttechnologies (e.g., programmable calculators such as the Sharp PC-1500A) [3], devices builtfor special purposes such as the MEMS-4 Medication-Event Monitoring System [7] or theMiniDoc electronic patient diary [8–10], in-house developments such as PIPER (PromptingIntensity of Pain Electronic Recorder) [11] or Compu-Void [12], early pen-based devicessuch as the Go Corporation Prototype with the PenPoint-System [13] and the wide range ofmultipurpose key-based or pen-based handheld devices such as the Psion family [4,14–18],the Atari Portfolio [19], the Apple Newton family [20–22], the PalmOS family [23–26] andothers). At the beginning of 2002 the leading operating systems for handheld devices arePalmOS (Palm), Windows CE and Pocket PC (Microsoft), Epoc (PSION) and Linux (Open-Source). These devices have been used in different medical fields such as diabetes[3,16,19,27], nutrition [17,21,28], pain [4,10,11,20], asthma [14,22,29], oncology [13,26],allergy [8], pharmacology [25], coronary artery disease [7], nosocomial infections [24], den-tistry [30], urology [12], psychology [15], occupational medicine [18], burns [23] and differ-ent surveys [9,31]. The variety of applications ranges from electronic patient diaries [3,8–11,14,16,17,21,27,29] to electronic case report forms (eCRFs) [13]. The sizes of the studiesrange from small (e.g., 19 patients [27]) to very large (e.g., 469 patients in 258 centers [8],1400 patients in one center [15] or a survey with 99,598 people [31]), and the durations ofthe studies range from 2 weeks [12] to 1 or more years [15,20]. Thus, the question arises asto what we can learn from the experience gained in earlier studies.

Balas et al. emphasize the importance of time in the form of reminders or time-stampeddata when using computers and the setting in which the computers are used [32]. The superi-ority to not being reminded and to manual reminders has been shown for desktop computers,which corresponds with our findings using mobile computers [33,34]. Longitudinal analysisand trend recognition have been used as well [11,25]. Handheld computers can greatly en-hance the ease and quality of time-verified data capture, continuous quality improvement ac-tivities and adaptive therapies [3,21,26].

The quality of data is improved in comparison to paper because the data are more com-plete [14,25,30], have fewer errors [23,28], are more consistent [13,31], and there are fewerprotocol violations [4]. Data editing, retyping, proofreading, and the process of clarificationcan be eliminated [13,22,30,31]; irrelevant questions can be omitted [9,31].

Compliance is another important aspect. Time-stamped data contribute to improved com-pliance control or allow for the assessment of compliance [8,14,20,28,29]. Results suggestthat use of electronic data entry shows promise for improving compliance with long-termdata collection [20]. Even the least restrictive approach is a great improvement over paper,where there is no information at all concerning entry times, nor any control of them [22].

Looking at the whole study process, an increase in efficiency is found in most of the stud-ies through easy storage and downloading of the data in combination with fast and easy dataprocessing [4,9,22,28,30]. The sponsor can more easily perform analyses while the study isin progress. This facilitates the use of adaptive study designs. The devices and systems canbe reused in several studies, where a single unit can be used repeatedly by different patientsand investigators, and other applications can be used simultaneously (e.g., food diaries and


477

ARTICLE IN PRESS

psychometric data collectors) [9,21]. Thus, if the initial program is written in such a way thatthe questions asked can be changed in content and number to suit the study, it can result ingreater adaptability and versatility of the tool; these features make it an ideal data capturetool for use in natural environments, the laboratory, and clinical settings [21].

Besides the above-mentioned advantages of the use of mobile computers in clinical trialsand in the clinical setting, a few disadvantages must be noted: Study centers and sponsorshave to deal with technology (hardware, software and data exchange) aspects [35]. Some-times there are large investment costs, but over a period of time the advantages outweigh thehigh initial costs (investments were: handheld computers, their peripherals, converting thequestionnaire into computer programs) [31]. Data must be downloaded daily (or at leastsometimes) for backup [21]. A few patients found the screen difficult to see [22]. Finally, ad-ditional effort is necessary to ensure electronic signatures and certified copies of electronicdocuments [5].

A comprehensive overview from the sponsor’s and investigator’s perspective regardingcomparisons of different data capturing and transmission techniques (e.g., pen and paper,handheld computers, Fax, optical mark readers, voice, remote data entry, etc.) can be foundelsewhere [35]. If a decision is made in favor of handheld devices, the following tips can beuseful.

Two-way messaging available through the palmtop computer seemed to encourage con-tinued use of the device. Patients stated that they felt that someone was closely monitoringtheir progress and they reported feeling as if someone cared about the information they werereporting [20,27]. Electronic diaries made the patients feel involved; written diaries were te-dious [12]. The use of other electronic devices can be monitored [11]. After protecting thedevice and the software so that no other implemented functions could be utilized, the use ofthe handheld computers generally posed no problems [15]. It is well recognized by medicalstaff that additional functionality such as the connection to the hospital information system(e.g., over a wireless connection) or lists of drug contraindications and interactions is advan-tageous. On the other hand, the additional functionality should not be confusing. If patientsare to be the users, it could be helpful to encourage them to use the handheld computers formanaging their calendars or address books, still the attractive applications of the devices.This could motivate them to accept the devices for daily use. That is only feasible, however,for studies running on the patient’s private hardware or if the devices are dispensed to the pa-tients for a long period of time, as with studies regarding chronic diseases where longitudinalanalysis is important [36]. Another important aspect is ensuring that the ability and willing-ness to use an electronic diary is part of the inclusion criteria of the study, especially wherethe diaries are the main source of data [22]. It is important to identify the team members whoassume primary responsibility for survey administration [26]. Members of nursing and med-ical staff should contribute to the development of the survey instrument to ensure that thenecessary components are included [26]. Use of a previously validated (pen and paper) sur-vey instrument is recommended as a model for an electronic version [26]. While desktopPCs interfere with the doctor/investigator–patient relationship, eCRFs on flat handheld com-puters, like paper, do not [4,13]. Patients’ and investigators’ acceptance of mobile devicesare widely documented, which corresponds with our findings [4,8,11,13,18,22,24,31]. Whencompared to previous paper and pencil methods, those of mobile devices have been proven

478


ARTICLE IN PRESS

to be equal or superior [3,7,9,11,12,14,15,20,21,23,27–30]. Finally, the most importantpoints are brevity and simplicity of the user interface and input from those who rely on thecollected data [26].

Surely, the above-mentioned advantages do not come without a price. Even as a study canhave badly designed questionnaires or case report forms, there is no guarantee that there arewell-designed user interfaces nor that the software or hardware is free of errors and fool-proof. This leads to the necessary reflection on software validation and standards. Interest-ingly, only two of the above-listed articles mentioned their software tests and validations[4,25]. In the mid-1990s there was a discussion whether devices used in clinical trials shouldbe treated as medical devices or not. This discussion led to recommendations for responsiblemonitoring and regulation of clinical software systems, influencing the point of view theFDA later took [37]. Their document entitled “Computerized systems used in clinical trials”gives a good insight into the considerations that need to be made and the processes that mustbe implemented in order to set up and use computerized systems [5]. Biebel stresses that riskmanagement and software validation may appear to be a hindrance to product development,but they are in fact tools that support the development process [38]. Used in the correct man-ner, they can even help speed up the process of producing better and safer software. Moreabout such models and software testing approaches can be found elsewhere [6,39]. Shouldthe new electronic system be audited, Stinchcomb describes in detail (e.g., using lists ofquestions) the necessary steps for carrying out such an audit [40].

For mobile devices used for data capture, biased data or data loss are the main concerns [41].These problems can be avoided or minimized using the procedures mentioned previously.

Conclusions

The many years of experience using mobile computers in clinical trials have resulted in afew very important advantages besides those listed above: time, compliance, quality, andcontrol. Time stamping of data and the use of reminders enhance the quality of the rendereddata significantly. Several studies show that one cannot rely on the patient’s memory andself-assessment, especially when real-time data gathering is essential [7,42]. The better thedata exchange between examination centers, patients and the study headquarters, the betterthe control of the study and the faster the response to aberrations and adverse events. From 8KB data packs (memory cards) used in 1988, which were sent by regular mail to the sponsorand back, to collecting or sending floppy disks or compact disks, to the use of computer net-works, Internet and wireless networks, the purpose of communication is more than just trans-ferring the data to the statistician [4]. If the patients have the impression that someone caresabout them, as with two-way messaging, their compliance will be better as well [20,27].Communication technology is mature and stable and can be used even from the most remoteareas of the world, and it is well known that mobile computers are capable of collecting datafrom patients in their natural environments [43,44]. Last but not least, the study is morelikely to be carried out faster and better than with traditional methods.

Patients and medical staff usually have no problems using the devices if the program de-velopers are familiar with basic software and user interface design issues, such as minimiz-ing free text entry or knowing how to manage text entry, and the system is explained well to


479

ARTICLE IN PRESS

the end user [45]. It is important to undertake the final test and validation with real patients inthe study environment, putting the program designer into the clinical setting. This can lead toimprovements of the clinical trial software and in turn enhance the data quality of the study.

Acknowledgments

The authors gratefully acknowledge linguistic support by Gena Kittel, B.A., in the prepara-tion of this article. They acknowledge the efforts of Robert Matesic, Karin Funke, M.Sc., Pe-ter Kiencke, Ph.D., Hartmut Stützer, Ph.D. and Kai Goetze for discussions and support withthe score statistics.

References

[1] Santoro E, Nicolis E, Franzosi MG, Togoni G. Internet for clinical trials: past, present, and future. Control Clin Trials1999;20:194–201.

[2] Kelly MA, Oldham J. The internet and randomised controlled trials. Int J Med Inf 1997;47:91–99.[3] Schrezenmeir J, Achterberg H, Bergeler J, et al. Controlled study on the use of hand-held insulin dosage computers en-

abling conversion to and optimizing of meal-related insulin therapy regimens. Life Support Syst 1985;3(Suppl 1):561–567.

[4] Tattersall AB, Ellis R. The use of a hand-held computer to record clinical trial data in general practice: a pilot study. J IntMed Res 1989;17:185–189.

[5] FDA. Guidance for industry: computerized systems used in clinical trials. U.S. Department of Health and Human Services,

Food and Drug Administration. April 1999

��

http://www.fda.gov/ora/compliance_ref/bimo/ffinalcct.htm

��

or .pdf.[6] Kit E. Software testing in the real world. New York: Addison-Wesley, 1996.[7] Straka RJ, Fish JT, Benson SR, Suh JT. Patient self-reporting of compliance does not correspond with electronic monitor-

ing: an evaluation using isosorbide dinitrate as a model drug. Pharmacotherapy 1997;17:126–132.[8] Bousquet J, Wouters FR, van Cauwenberge P, Lund V, El Akkad T. Electronic patient diary is well accepted by patients

and investigators in recording daily data in seasonal allergic rhinitis trials. Allergy 2000;55(Suppl 63):194–195.[9] McBride JS, Anderson RT, Bahnson JL. Using a hand-held computer to collect data in an orthopedic outpatient clinic.

Med Care 1999;37:647–651.[10] van Gerven JM, Schoemaker RC, Jacobs LD, et al. Self-medication of a single headache episode with ketoprofen, ibupro-

fen or placebo, home-monitored with an electronic patient diary. Br J Clin Pharmacol 1996;42:475–481.[11] Lewis B, Lewis D, Cumming G. Frequent measurement of chronic pain: an electronic diary and empirical findings. Pain

1995;60:341–347.[12] Rabin JM, McNett J, Badlani GH. “Compu-Void II”: the computerized voiding diary. J Med Syst 1996;20:19–34.

[13] Beinlich I, Bokemeyer C, Räth U, et al. Pen-based remote data entry system. A pilot clinical trial. Drug Res 1993;43:399–404.[14] Hyland ME, Kenyon CAP, Allen R, Howarth P. Diary keeping in asthma: comparison of written and electronic methods.

BMJ 1993;306:487–489.[15] Rose M, Hess V, Horhold M, Brahler E, Klapp BF. Mobile computer-assisted psychometric diagnostics — economic ad-

vantages and results of test stability [in German]. Psychother Psychosom Med Psychol 1999;49:202–207.[16] Clarke WL, Cox DJ, Gonder-Frederick LA, Kovatchev B. Hypoglycemia and the decision to drive a motor vehicle by per-

sons with diabetes. JAMA 1999;282:750–754.[17] Greeno CG, Wing RR, Shiffman S. Binge antecedents in obese women with and without binge eating disorder. J Consult

Clin Psychol 2000;68:95–102.[18] Totterdell P, Spelten E, Smith L, Barton J, Folkard S. Recovery from work shifts: how long does it take? J Appl Psychol

1995;80:43–57.[19] Harris ND, Baykouchev SB, Marques JL, et al. A portable system for monitoring physiological responses to hypoglycae-

mia. J Med Eng Technol 1996;20:196–202.[20] Jamison RN, Raymond SA, Levine JG, et al. Electronic diaries for monitoring chronic pain: 1-year validation study. Pain

2001;91:277–285.[21] Stubbs RJ, Hughes DA, Johnstone AM, et al. The use of visual analogue scales to assess motivation to eat in human sub-

jects: a review of their reliability and validity with an evaluation of new hand-held computerized systems for temporaltracking of appetite ratings. Br J Nutr 2000;84:405–415.

480


ARTICLE IN PRESS

[22] Tiplady B, Jamieson AH, Crompton GK. Use of pen-based electronic diaries in an international clinical trial of asthma.Drug Information Journal 2000;34:129–136.

[23] Lal SO, Smith FW, Davis JP, et al. Palm Computer demonstrates a fast and accurate means of burn data collection. J BurnCare Rehabil 2000;21:559–561.

[24] McLaws ML, Caelli M. Pilot testing standardized surveillance: Hospital Infection Standardised Surveillance (HISS). Am JInfect Control 2000;28:401–405.

[25] Reilly JC, Wallace M, Campbell MM. Tracking pharmacists interventions with a hand-held computer. Am J Health SystPharm 2001;58:158–161.

[26] Holdsworth MT, Adams VR, Raisch DW, Wood JG, Winter SS. Computerized system for outcomes-based antiemetictherapy in children. Ann Pharmacother 2000;34:1101–1108.

[27] Tsang MW, Mok M, Kam G, et al. Improvement in diabetes control with a monitoring system based on a hand-held,touch-screen electronic diary. J Telemed Telecare 2001;7:47–50.

[28] Kos J, Battig K. Comparison of an electronic food diary with a nonquantitative food frequency questionnaire in male andfemale smokers and nonsmokers. J Am Diet Assoc 1996;96:283–285.

[29] Chowienczyk PJ, Lawson CP, Morris J, Kermani A, Cochrane GM. Electronic diary to record physiological measure-ments. Lancet 1992;339:251.

[30] Berthelsen CL, Stilley KR. Automated personal health inventory for dentistry: a pilot study. J Am Dent Assoc2000;131:59–66.

[31] Gupta PC. Survey of sociodemographic characteristics of tobacco use among 99,598 individuals in Bombay, India usinghandheld computers. Tob Control 1996;5:114–120.

[32] Balas EA, Austin SM, Mitchell JA, et al. The clinical value of computerized information services. A review of 98 random-ized clinical trials. Arch Fam Med 1996;5:271–278.

[33] Shea S, Du Mouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-basedclinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996;3:399–409.

[34] Cannon DS, Allen SN. A comparison of the effects of computer and manual reminders on compliance with a mental healthclinical practice guideline. J Am Med Inform Assoc 2000;7:196–203.

[35] Lampe AJ, Weiler JM. Data capture from the sponsors’ and investigators’ perspectives: balancing quality, speed, and cost.Drug Information Journal 1998;32:871–886.

[36] Koop A, Gatermann C, Mösges R. Benefit of handheld computers and WWW-technologies in clinical multicenter trialsand the care of patients with chronic diseases [in German]. Zentralbl Gynakol 2000;3:183–186.

[37] Miller RA, Gardner RM. Recommendations for responsible monitoring and regulation of clinical software systems. J AmMed Inform Assoc 1997;4:442–457.

[38] Biebel J. Software validation for medical devices and in vitro diagnostics. Med Device Technol 2000;11:18–21.[39] Sanders J, Curran E. Software quality. New York: Addison-Wesley, 1994.[40] Stinchcomb WB. Auditing new electronic systems. Drug Information Journal 1998;32:1073–1080.[41] Neumann PG. Computer related risks. New York: Addison-Wesley, 1995.[42] Smith WB, Safer MA. Effects of present pain level on recall of chronic pain and medication use. Pain 1995;55:355–361.[43] Otto C, Pipe A. Remote, mobile telemedicine: the satellite transmission of medical data from Mount Logan. J Telemed

Telecare 1997;3(Suppl 1):84–85.[44] Stone AA, Shiffman S. Ecological momentary assessment (EMA) in behavioral medicine. Ann Behav Med 1994;16:199–

202.[45] Wright P, Bartram C, Rogers N, et al. Text entry on handheld computers by older users. Ergonomics 2000;43:702–716.

Documents

The use of handheld computers in clinical trials