A Comparative Study of Nurses Accessing Electronic Patient Record Systems

with PDAs and Tablet PCs

Gilberto Crespo Pérez

Electrical & Computer Engineering DepartmentUniversity of Puerto Rico at Mayagüez

December 2005

Hospitals in Puerto Rico & United States keep their patients’ records in paper form.– Problems with information management effectiveness.– Handwriting may produce errors due to misunderstanding of

written information. Compromise patients’ health.

PDAs & Tablet PCs are two technological devices with potential for accessing and capturing clinical data at bedside. – Allow greater speed and effectiveness to compile patient’s

information.– Reduce the risk of errors during treatment of patients.– Improve physicians and nurses’ effectiveness and productivity

at point-of-care, as well as the quality of patient care.

Introduction

A review of the literature reveals that there are no formal studies comparing the use of PDAs and Tablet PCs for accessing electronic patient records.

We believe that such study could help determine which of the two technologies fits better for collecting and accessing clinical information at the point-of-care.

Introduction (Cont.)

Previous Work From 44,000 to 98,000 people die annually in U.S.

hospitals as a result of medical mistakes -- more than half of them preventable [Rosenbloom03].

Trying to correct some of these errors, the Veteran’s

Administration (VA) has taken effective measures through electronic system controls and implementation [Rosenbloom03]

– Over a five-year test period, the medication error rate dropped 70%

Previous Work (Cont.)

The Use of Portable Devices in Health Care: E-prescribing, ordering, checking labs tests, dictation

notes among others - [Fischer03, Ying03, Rosenbloom03, Barret02,].

Clinical Applications for Portable Devices: Drug references, pharmacopoeias, medical

calculators, and patient trackers among others -[Adatia03, Rosenbloom03, Choi00, Kimura03, van der Velde01, Berner04].

Previous Work (Cont.)

Usability Studies of Portable Devices in Health Care: PDA vs Laptop & PDA vs Paper Based [Rodríguez02,

Rodríguez03, Rodríguez04, Staggers00, Stausberg03]. Users performance, subjective satisfaction, and preference were

dependable variables.

Usability of Tablet PC [Narayan04, Andon04]. Physical Aspects.

No more studies related to other usability issues of Tablet PCs have been found, which indicates that this field is in its early stages of research.

Objectives

Conduct a usability study to compare the use of PDA and Tablet PC-based applications to support nurses’ tasks at point-of-care.– These studies compare both systems in terms of

performance parameters: Time to complete tasks.

Number of tasks completed. Subjective user satisfaction.

– Identify: Potential usability problems. Interaction differences. Advantages and Disadvantages.

Systems Tablet PC version

– Implemented on a Gateway tablet.– Windows XP Tablet PC OS.– Uses a Stylus as a pointing device.– A soft keyboard provided by OS was used for text input.

PDA version– Implemented on a HP iPaq 5500.– Windows Pocket PC 2003 OS.– Uses a small stylus as a pointing device.– A soft keyboard provided by OS was used for Text input.

Systems (Cont.)

The Tablet PC version– Developed in Java.

The PDA version – Developed in C Sharp.

Both applications use:– MSSQL to store and retrieve the patient’s clinical

information in a database.– The standard IEEE 802.b to communicate with the

database server.

Graphical User Interface

Patient List Windows

Graphical User Interface (Cont.)

Medicine Windows

Graphical User Interface (Cont.)

Intake/Output Windows

Participants

20 staff nurses– Selected on a first-come first-served basis from those that

responded a call for participation.– Experience as staff nurses ranged from 1 to 27 years

(Mean=12.3 years).– Experience with computers ranged from 0 to 9 years

(Mean=4.6 years).– On average, they used computers on their job for 3.8 hours

per day.– None of them had prior experience with the systems used

for the study or with any similar application.– None of them had experience using PDAs or Tablet PCs.

Experimental Design

Participants were asked to fill out a questionnaire about their work experience and their experience using computers.

They were asked to sign a letter of consent. An orientation script was read to each participant

explaining the objective of the test. Nurses were given a short tutorial session on the graphical

system.– 18 minutes for the PDA version.– 13 minutes for the Tablet PC version.– They were allowed to use the system by themselves and were

guided to practice using each of the functions of the system.

Half of the nurses performed the tasks first on one system and the other half started on the other system.

Once the participants complete all the task in both versions, they were asked to fill a user satisfaction questionnaire.

Experimental Design (Cont.)

Tasks1. Indicate patient’s age and weight. Say them aloud.2. Indicate the most recent registered patient’s temperature.3. Look for the most recent nurse note and read it aloud.4. Acknowledge any pending medication order as administered.5. Enter the following set of vitals signs:

Temp: 37.0o C; BP: 130/90; Pulse: 71; RR: 18; O2 Sat: 96%. 6. Indicate the total balance of intake/output of fluids in the last

24 hours.7. Look for a Dr. Colón note.8. Enter the following patient assessment information provided.9. Enter the following I/O information:

Intake PO: 50 ml; Output Urine: 650 ml;

10. Enter the following text as a note:Patient presents fever and a large lung mass on left upper lobe consistent on CT with multi-focal pneumonia

11. Enter the following patient’s pain information.Classification: Four; Location: Right frontal shoulder; Description: Hurt; Therapy: Massage; Administered medicine.

12. Look for most recent physician consult order and acknowledge it as that was notified by phone.

13. Show where is indicated in the record the reason why a dose of Roboxine was omitted.

Tasks (Cont.)

User Satisfaction Questionnaire

Interaction Aspects:– Look up patient information– Acknowledge medication orders– Acknowledge consult order– Enter vitals signs– Document intake/output– Document patient assessment– Document pain assessment– Read a note– Write a note

Nurses were asked to rate on a 1 – 7 scale (1 being poor & 7 being excellent)

System Aspects:– Record organization– Trustworthiness of information– Precision of Information – Accessibility of information– System security

Physical Aspects:– Use of the stylus– Use of the screen keyboard– The screen– The weight– The portability

Level of Satisfaction with each system The system they would prefer for doing their

nursing documentation.

User Satisfaction Questionnaire (Cont.)

Statistical Analysis

Dependent variables were:– Task completion time.– Number of tasks completed.– Subjective user satisfaction.

Time analysis – Paired t Test User Satisfaction – Wilcoxon Signed Ranks Number of Task Completed – Wilcoxon Signed

Ranks

Results: Overall Completion Times

656.2 667.3

191.7215.6

0.0

100.0

200.0

300.0

400.0

500.0

600.0

700.0

Seco

nds

Mean Std. Deviation

PDATablet PC

Dependent Sample t test did not reveal significant difference between average time it took the participants to complete all the tasks on both system.

Results: Overall Completion Time (Cont.)

0

50

100

150

200

250

Com

plet

ion

Tim

e (

Seco

nds)

1 2 3 4 5 6 7 8 9 10 11 12 13Tasks

PDATablet PC

Results: Completion Times for Each Task on Both Versions of the System

Dependent Sample t test did not reveal significant difference in the time it took the participants to complete each of the tasks on both versions of the system.

Results: Completion Times for Each Task on Both Versions of the System (Cont.)

Results: Number of Participants that Complete Each Task on Each Version of the System

0

2

4

6

8

10

12

14

16

18

20N

umbe

r of

Par

ticip

ants

1 2 3 4 5 6 7 8 9 10 11 12 13Tasks

PDA

Tablet PC

Wilcoxon Signed Ranks did not reveal significant difference in the total number of tasks completed by the participants.

The participants completed an average of 12.0 tasks on the PDA and 11.7 on the Tablet PC

Results: Number of Participants that Complete Each Task on Each Version of the System (Cont.)

Results: Average Satisfaction Ratings for Individual Interaction Aspects

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0

Rating

Look up patient informationAcknowledge medication orders

Acknowledge consult orderEnter vitals signs

Document intake/outputDocument patient assessment

Document pain assessmentRead a noteWrite a note

Inte

ract

ion

Aspe

cts

PDA Tablet PC

Wilcoxon test did not reveal significant a difference on each individual interaction aspects considered.

Also, Wilcoxon Test did not reveal a significant difference in the overall average satisfaction rating given to the 9 interaction aspects.

Results: Average Satisfaction Ratings for Individual Interaction Aspects (Cont.)

Results: Average Satisfaction Ratingsfor Individual System Aspects

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0

Rating

Recordorganization

Trustworthinessof information

Precision ofInformation

Accessibility ofinformation

Systemsecurity

Syst

em A

spec

ts

Tablet PCPDA

Wilcoxon Test did not reveal a significant difference in the overall average satisfaction rating given to the 5 system aspects.

The overall average satisfaction rating for the PDA and the Tablet PC was 6.8

Results: Average Satisfaction Ratingsfor Individual System Aspects (Cont.)

Results: Average Satisfactions Ratings for Individual Physical Aspects

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0

Rating

Use of the stylus

Use of the screen keyboard

The screen

The weight

The portability

Phys

ical

Asp

ects

Tablet PCPDA

Wilcoxon Test revealed a significant difference in overall satisfaction with the five physical aspects considered.

Wilcoxon Test also revealed significant user satisfaction rating for:– The use of the Stylus– The weight of the device– The portability of the device

Results: Average Satisfactions Ratings for Individual Physical Aspects (Cont.)

Participants were more satisfied with:– The PDA computer (Average Rating = 6.6) than with the

Tablet PC computer (Average Rating = 5.2).– The use of the stylus on the PDA (Average Rating = 6.7)

than on the Tablet PC (Average Rating = 6.2). – The weight of the PDA (Average Rating = 6.5) than with

the weight of the Tablet PC (Average Rating = 4.1).– The PDA portability (Average Rating = 6.6) than with the

Tablet PC portability (Average Rating = 3.5).

Results: Average Satisfactions Ratings for Individual Physical Aspects (Cont.)

Overall Results

No significant differences were found in the No significant differences were found in the overall satisfaction rating given by the overall satisfaction rating given by the participants to each version of the system.participants to each version of the system.

However, the majority of the participants However, the majority of the participants preferred the PDA over the Tablet PC version preferred the PDA over the Tablet PC version for performing their duties.for performing their duties.

Discussion No significant difference was found in the overall time it

took nurses to complete tasks on both portable devices.– It contradicts our expectations that nurses would be faster using

a larger user interface than a smaller one. Our results are consistent with the results of the study by

Rodríguez et. al. [Rodríguez03]– When the task of writing a note was not considered, nurses were

able to complete nursing tasks in similar overall times on the PDA and on Laptop.

The use of a stylus, the screen size, and resolution is not a factor in nurses’ performance.

Discussion (Cont.) Nurses were as fast on a PDA as they would on a

Tablet PC.

Younger nurses tend to take shorter times in completing tasks.

Due to the short training given to nurses to teach them how to use the systems and the high percentage of tasks completed:– Results indicate that both nursing documentation systems

are very easy to learn .– Similar results were found in the [Rodríguez03] study.

Discussion (Cont.)

The word “document” in tasks T4, T6, T8, and T12 seems to create confusion in some of the nurses to the point that they did not find the way to complete the task successfully.

Nurses expressed similar high levels of satisfaction with the documentation activities performed in both, the PDA and Tablet PC systems.

They were significantly more satisfied with the physical aspects of the PDA than those of the Tablet PC

Discussion (Cont.) Many of the participants (12 out of 20) expressed

discomfort in holding the Tablet PC in a standing position during the test session. – This result is consistent with the studies conducted by:

Michael A. Narayhan [Narayhan04] Christopher L. Andon [Andon04]

Fourteen of the nurses indicated that they preferred the PDA version for performing nursing documentation tasks while only four preferred the Tablet PC.

Conclusions Our results supports that it is possible to design PDA-

based applications that allow nurses to achieve similar performance and satisfaction levels as with a Table PC-based.

Screen size and display resolution of the PDA are no factors that limit nurses’ performance and satisfaction in comparison to Tablet PCs.

Nurses were significantly more satisfied with the physical aspects of the PDA than with those of the Table PC.– Aspects such as the use of the stylus, the weight, and portability

are factors that influence nurses’ satisfaction with the system.

Conclusions (Cont.) These applications exhibited a high degree of

learnability. Differences on user interfaces have not affected users

satisfaction. Usability engineering principles and guidelines

– Are critical part of the graphical user interface success.– Proved to be powerful and important tools for measuring the

efficiency of the system. The age seems to be a factor that affects nurses’

performance..

Conclusions (Cont.)

Considering that:Considering that:– Satisfaction with the physical aspects of the systems Satisfaction with the physical aspects of the systems

was the only dependent variable for which a was the only dependent variable for which a significant difference was found in favor of the PDAsignificant difference was found in favor of the PDA

– That 14 out of 20 participants selected the PDA over That 14 out of 20 participants selected the PDA over the Tablet PC the Tablet PC

– And the relatively low costs of the PDAAnd the relatively low costs of the PDA

We can conclude that PDAs are a better alternative We can conclude that PDAs are a better alternative for supporting nursing documentation tasks at for supporting nursing documentation tasks at bedside than Tablet PCs.bedside than Tablet PCs.

Future Work

Other usability studies should be conducted:– After the users have had a year of experience with the system.– With physicians interacting with each version of the system used

on this research.

Application improvement:– Incorporation of printing and supplemental language options

capabilities.– Speech recognition functionality.– Convert to a web based application.

Questions/CommentsQuestions/Comments

Usability Concepts Definitions

Usability: the extent to which the intended user can meet his or her goals using the system being tested.

Learnability: easy to learn -> time a novice user takes to complete tasks using the system.

Efficiency: high level of productivity -> time that users with certain expertise take to complete typical tasks.

Memorability: easy to remember. Errors: Low error rate and easy to recover. Satisfaction: users are subjectively satisfied -> ask users opinions.

When replies from multiple users are average, the result is objective. -questionnaire with Likert scale.

General Concepts Definitions

Ethical Aspects with Human Subjects:– Respect– Comfortable environment– Confidentiality– Early success experience

Statistical Tools Definitions Dependent-samples t test: used to compare the

time to complete the tasks -> same users, 2 measurements.

Wilcoxon Signed Rank test: compare differences in user satisfaction -> dependent samples.

Correlation analysis: determine associations between time to complete the tasks, computer literacy, typing skills, age, and eye glasses, among others.

Linear regression: evaluate the learning effect between 2 systems.