Leading a Horse to Water:Using Automated Reminders to Increase

Use of Online Decision Support

James J. Cimino and Dmitriy Borovtsov

NIH Clinical Center

andDepartment of Biomedical Informatics

Columbia University College of Physicians and Surgeons

Knowledge Resources are Underused

Clinicians (physicians) with questions in clinical practice:

– sought answers only 45% of the time

– found answers only 34% of the time

– used online resources 18% of the time

* Ely JW, Osheroff JA, Maviglia SM, Rosenbaum ME. Patient-care questions that physicians are unable to answer. J Am Med Inform Assoc. 2007 Jul-Aug;14(4):407-14.

Health Resources

• Available from main menu

• Static list of resources

• No assistance with retrieval

Infobuttons

• Links inserted next to clinical information

• Resources selected based on likely information need

• Links are customized to assist with retrieval

Log File of HR vs IB 2006

2006 Health ResourcesInfobuttons

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

7000

6000

5000

4000

3000

2000

1000

0

Monthly Usage

Log File of HR vs IB 2006, 2007

2006 2007Health ResourcesInfobuttons

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

7000

6000

5000

4000

3000

2000

1000

0

Monthly Usage

Educational Interventions

• Orientation for new housestaff

• Orientation for new medical students

• Infobutton exercises for new medical students

Educational Interventions

2006Medical StudentsHousestaff

Jun Jul Aug Sep Oct Nov Dec

Monthly Usage

450

400

350

300

250

200

150

100

50

0

Educational Interventions

2006 2007Medical StudentsHousestaff

Jun Jul Aug Sep Oct Nov Dec

450

400

350

300

250

200

150

100

50

0

Monthly Usage

Usage of IB vs HR in Inpatient Meds

Health ResourcesInfobuttons

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

1400

1200

1000

800

600

400

200

0

Monthly Usage

Usage of IB in Outpatient Meds

Health ResourcesInfobuttons

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

800

700

600

500

400

300

200

100

0

Monthly Usage

Hypothesis

Users of the Health Resources page in a particular context will be especially receptive to a suggestion to use Infobuttons in the same context.

SendReminder

No

StopYesReminder

Sent in past2 months?

No

StopYesIB Used

in past2 months?

InterventionHR used in OM

HR = Health Resources PageOM = Outpatient Medications

E-MailDate: Fri, 2 Mar 2007 06:00:11 -0500 (EST)From: James Cimino <jjc7@columbia.edu>To: xxx@columbia.eduSubject: Getting Drug Information while Using WebCIS

Dear WebCIS User:

I am writing to let you know about a handy feature in WebCIS called the "infobutton". You will see it in places like the Outpatient Medication list - it is a little purple circle with a white letter "i". If you click on it, it will give you a list of topics that you can select to get more information about the drug next to the infobutton.

I hope you'll find it useful.

--Jim Cimino, Infobutton Project Manager

Reminders Sent

• 522 messages• 371 users

– 80 attending physicians– 189 housestaff– 29 nurses– 24 students– 49 other/unknown

• 173 users received two messages• 1 user received three messages• Average 89.9 days between messages

Usage of Infobuttons by Reminder Recipients

• 371 recipients of 552 reminders• 111 (20.1%) eventually used IBs in OM• 52 (9.4%) “early responders”

– 26 housestaff– 18 attending physicians– 8 other/unknown

• 201 used either IB or HR within one month• 25.9% “early responders with opportunity”• 9 early responders used IBs 8 to 97 times in

subsequent months

Intervention versus Control

552 Intervention525 Control

Days to First Use of Infobuttons after First Use of Health Resources Page

P<0.05 by Day 4

Effect of delay

0

100

200

300

400

1-2Days

3-5Days

>5Days

RemindersSent

EarlyResponders

Adverse Effects

• 522 reminders

• 12 replies– All were positive– Most were thanking us for infobuttons

Discussion

• Traditional training method appears to have limited effect

• Simple, spam-like reminder appears to have an effect on a substantial fraction (26%) of users

• Sustained effect in some (17%) of the early responders

Limitations

• Impersonal message

• Timing of reminder less-than ideal

• However: Logfile+e-mail approach is application independent

• Quasi-experimental design

• Convenience sample of controls

• However: Strong temporal association between intervention and outcome

Conclusions

• Automated, context-specific reminders are technically feasible, requiring few resources

• Reminders to use online information resources appear to be effective

• Timing of message is critical to success

Acknowledgments

This work was supported in part by:National Library of Medicine Grant R01LM07593

NIH Clinical Center intramural research funds

National Library of Medicine intramural research funds

The authors also thank:Dr. Jianhua Li for technical support

Dr. Rick Gallagher for log files

Dr. Krystl Haerian for statistical support