International Journal of Medical Informatics (2005) 74, 535—543

Understanding technology adoption in clinicalcare: Clinician adoption behavior of apoint-of-care reminder system

Kai Zhenga,∗, Rema Padmana, Michael P. Johnsona, Herbert S. Diamondb

a H. John Heinz III School of Public Policy and Management, Carnegie Mellon University,Department of Medicine, Pittsburgh, PA 15213-3890, USAb Department of Medicine, The Western Pennsylvania Hospital, 4800 Friendship Avenue,

Pittsburgh, PA 15224, USA

Received 8 November 2004; received in revised form 21 March 2005; accepted 22 March 2005

KEYWORDSClinical decision supportsystems;

SummaryBackground: Evaluation studies of clinical decision support systems (CDSS) havetended to focus on assessments of system quality and clinical performance in a

Usage analysis;Qualitative research;Evaluation studies

laboratory setting. Relatively few studies have used field trials to determine if CDSSare likely to be used in routine clinical settings and whether reminders generatedare likely to be acted upon by end-users. Moreover, such studies when performedtend not to identify distinct user groups, nor to classify user feedback.

Aim: To assess medical residents’ acceptance and adoption of a clinical reminder sys-tem for chronic disease and preventive care management and to use expressed pref-erences for system attributes and functionality as a basis for system re-engineering.Design of study: Longitudinal, correlational study using a novel developmental tra-jectory analysis (DTA) statistical method, followed by a qualitative analysis basedon user satisfaction surveys and field interviews.Setting: An ambulatory primary care clinic of an urban teaching hospital offeringcomprehensive healthcare services. 41 medical residents used a CDSS over 10 monthsin their daily practice. Use of this system was strongly recommended but not manda-tory.Methods: A group-based, semi-parametric statistical modeling method to identifydistinct groups, with distinct usage trajectories, followed by qualitative instrumentsof usability and satisfaction surveys and structured interviews to validate insightsderived from usage trajectories.

∗ Corresponding author.E-mail addresses: kzheng@andrew.cmu.edu (K. Zheng), rpadman@andrew.cmu.edu (R. Padman), johnson2@andrew.cmu.edu

(M.P. Johnson), hdiamond@wpahs.org (H.S. Diamond).

1386-5056/$ — see front matter © 2005 Elsevier Ireland Ltd. All rights reserved.doi:10.1016/j.ijmedinf.2005.03.007

536 K. Zheng et al.

Results: Quantitative analysis delineates three types of user adoption behavior:‘‘light’’, ‘‘moderate’’ and ‘‘heavy’’ usage. Qualitative analysis reveals that cliniciansof distinct types tend to exhibit views of the system consistent with their demon-strated adoption behavior. Drawbacks in the design of the CDSS identified by users ofall types (in different ways) motivate a redesign based on current physician workflows.Conclusion: We conclude that this mixed methodology has considerable promise toprovide new insights into system usability and adoption issues that may benefit clinicaldecision support systems as well as information systems more generally.© 2005 Elsevier Ireland Ltd. All rights reserved.

1. Summary of research results

What was known before the study?

• Clinical decision support systems can enhancethe clinical performance in a variety of areas.

• Many CDSS fail in real-life installations regardlessof the technological quality of the application.

• Research on the effects of CDSS on user adoptionand organization workflows in realistic settings islacking.

What the study has added to the body of knowl-edge?

• Introduced a new statistical methodology drawnfrom the social sciences for analyzing revealedbehavior versus perceived behavior in medical in-formatics evaluation studies.

• Illustrated its successful application in under-standing the user adoption behavior and catego-rizing users into distinct user groups.

• Physician qualitative impressions of the CDSS areconsistent with their classification according tothe statistical clustering model for user adoption.

whether a CDSS shown to be effective in laboratorysettings will be effective over time in routine clin-ical settings with real patients.

Despite efforts to optimize CDSS as much as tech-nically possible, practitioners may not view use of aCDSS as being beneficial, and thus may be reluctantto incorporate it into their daily practice [10]. Aprevious study estimates that 45% of computerizedmedical information systems fail because of user re-sistance, even though these systems are technolog-ically sound [12]. A variety of factors can contributeto this resistance, such as insufficient computerability, diminished professional autonomy, lack ofawareness of long-term benefits of system use, andlack of desire to change conventional behaviors.As systematic reviews have indicated, few evalu-ation studies have considered these attitudinal andcontextual factors of system acceptance and adop-tion, and little use has been made of qualitativetechniques in studying effectiveness of CDSS, leav-ing unanswered questions, such as why some sys-tems worked while some others failed. Researchers,therefore, have called for methodological plu-ralism for evaluating informatics applications

• A CDSS, especially designed for outpatient clinicpractice, must incorporate workflows consistentwith care processes during the physician—patientencounter.

2. Introduction

Clinical cueing systems (CCS) are a class of clinicaldecision support systems (CDSS) that send just-in-time alerts to clinicians when potential errors or de-ficiencies in the patient management are detected.Significant research evidence shows that CDSS canenhance the clinical performance in drug dosing,preventive care, and other aspects of medical care[1—9]. However, most evaluations of CDSS empha-size clinical performance and diagnostic accuracy;few studies address user acceptance and adoptionof such tools in the ambulatory care practice settingthat reflects specific characteristics of the usersand/or the environment [8,9]. It remains unclear

[8,9,10].With these perspectives in mind, we have de-

veloped and deployed a CDSS called clinical re-minder system (CRS) for chronic disease and pre-ventive care management using evidence-basedmedicine principles for an outpatient primary careclinic of an urban teaching hospital. Our pri-mary objectives in the long-term study includeevaluation of adoption and diffusion of informa-tion technology interventions at the point-of-care,and impacts on patient and organizational out-comes. In this paper we assess clinicians’ accep-tance and adoption of the clinical reminder sys-tem using mixed methods: a quantitative analy-sis identifying distinctive developmental trajecto-ries of user acceptance and adoption of the sys-tem over time, and a qualitative examination of theattitudinal and contextual influences on adoptionof the system, providing insights into interpretingthe adoption trajectories generated by specializedmodels.

Understanding technology adoption in clinical care 537

3. Methods

3.1. Clinical reminder system

Clinical reminder system (CRS) uses patients’medical status data to provide ‘‘just-in-time’’reminders to clinicians at the point of the careconsistent with the latest evidence-based medicineguidelines for chronic disease and preventive caremanagement. CRS is made available to physiciansand clinic staff via desktop computers installedin every examination room of the clinic. The ap-plication integrates the hospital’s administrative,laboratory, and clinical records systems into asingle database. Patient’s data essential to specificguidelines, but not stored in the database, arecollected and entered into the system by clinicstaff and physicians during the encounter.

CRS is adapted to the workflow of a typicalclinic by supporting the appointment scheduling,patient check-in, recording of vital signs, browsingpatient information, generating physician-directedreminders, and check-out activities. Remindersgenerated by CRS take the form of on-screen rec-ommendations to have tests scheduled or per-formed, review abnormal test results, receive vac-ccCopsntip

oomiaiqpwassu

3d

Cc

second- and third-year residents. System imple-mentation was completed 3 months prior to thestudy, and individual training was provided to allthe users. The adoption study was conducted be-tween 1 February and 30 November 2002. Forty-one internal medicine residents used the systemto treat, approximately, 4500 patients. Their usagedata, from system login to logout (which patientrecords were accessed, what data was entered,whether reminders were generated, what actionswere taken on various reminders, and so on), wererecorded for each user in the log files. Results ofsystem use in this clinic setting provided the datafor usage analysis that forms the basis of furtherqualitative assessments.

3.3. Qualitative data collection frommultiple sources

We conducted structured interviews with 16 res-idents who used the system continuously duringthe 10-month evaluation period. Based on initialusage analysis indicating three distinct user adop-tion groups, we enrolled users from each of the us-aghi

sTiTos1ovttc

vstcsFasdrcto

inations, or follow-up on patients with medicalonditions that require unscheduled interventions.RS is currently designed to improve the qualityf care for two chronic diseases: diabetes and hy-erlipidemia, and five preventive care categories:teroid-induced osteoporosis, influenza, pneumo-ia, breast cancer, and cervical cancer. Details ofhe architecture of CRS and the algorithms thatmplement evidence-based medicine guidelines areresented elsewhere [13].The data used in this study are the combinations

f quantitative data from log files of actual usagef CRS, results of a statistical trajectory analysisethod, and qualitative data from surveys, field

nterviews and textual notes (detailed below). Wenalyze these data using a combination of methods,.e. statistical models to identify user groups andualitative analysis methods to understand usageatterns in the context of defined user groups. Thisas done not only to understand who the usersre, and how and why they do or do not use theystem, but also to define high-priority areas forystem redesign and deployment based on actualser feedback.

.2. Intervention and quantitative usageata collection

RS was installed in the hospital’s primary carelinic, which serves as a rotation-site for first-,

ge groups to ensure representation of the clinicianroups who demonstrated all types of adoption be-avior. Choice of participants was based on the clin-cian’s availability.

We also administered two surveys to assess theystem usability and user satisfaction, respectively.he two survey instruments are based on IBM sat-sfaction questionnaire, developed by Lewis [14].he survey response rates were 78% for both (25ut of 32 and 29 out of 37, respectively). The useratisfaction survey, conducted at the end of the0-month evaluation period, included a number ofpen-ended questions for residents to express theiriews, experiences, and expectations of the sys-em. This feedback was compared to the themeshat emerged from the interviews to check on theonsistency of the qualitative assessments.

The qualitative data from interviews and sur-eys were analyzed by the authors using the con-tant comparative method—–an inductive methodhat ‘‘involves the continuous comparison of in-idents and interviewees’ remarks, and then con-tructs categories and themes from the data’’ [15].inally, we compared the themes we identifiedgainst the user types generated using a specializedtatistical method (described below) to identify theiscrepancies among the user adoption groups. Weeported these results to administrators and pre-eptors of the clinic to verify the findings, as well aso solicit their interpretations and comments basedn their experiences.

538 K. Zheng et al.

Fig. 1 CRS study framework.

In addition to the data collected via surveys andinterviews, we analyzed electronically filed bug re-ports and the text notes, stored by the system inconjunction with reminder responses, as well asother types of data entry. We also made a num-ber of random on-site observations to determinehow clinicians actually used the reminder systemand how their patient interactions were affectedby the system. These assessments are incorporatedin our qualitative analysis.

3.4. Developmental trajectory analysis

A ‘‘developmental trajectory’’ describes the courseof a developmental behavior over age or time. De-velopmental trajectory analysis (DTA) is a semi-parametric, group-based approach for identifyingdistinctive groups of individual trajectories withinthe population, and for profiling the character-istics of group members [16,17]. Given the ob-servations of users in discrete time periods, andwell-defined user characteristics, DTA maximizes aBayesian information criterion, by which users canbe assigned to distinct groups following distinct tra-jectories. This method optimally determines the

4. Results

4.1. Distinct user adoption groups

Our application of DTA identified three distinctuser adoption types. The developmental trajecto-ries and the group compositions are depicted inFig. 2. Solid and dashed lines denote the observedand predicted trends, respectively. Observed datavalues are computed as the mean use rate of usersassigned to each of these groups identified by es-timation. Predicted values are computed using DTAmodel coefficient estimates. Numbers above eachtrendline denote the percentage of residents thatdemonstrated coherent behavior consistent withthat trend.

Fig. 2 also reveals distinct trends in adoption pat-terns. Clinicians later confirmed that these trendsreflect actual experiences of system use. For in-stance, residents classified as ‘‘light’’ (41.46%) ini-tially used the system for about 35% of all patientencounters, and this rate remained steady overthe 10-month evaluation period. ‘‘Moderate’’ users(36.59%) had the highest initial usage rate, about70%, but this rate consistently decreased over thes‘ts

‘t

ajec

number and membership of groups. DTA has beensuccessfully applied in many areas, such as stud-ies of physical aggression among the youth [17]. Adetailed description of this model can be found in[18].

Our approach to this study can be summarized inFig. 1.

Fig. 2 Developmental tr

tudy period to a level comparable with that of the‘light’’ users. ‘‘Heavy’’ users (21.95%) had an ini-ial usage rate of about 50%, which increased con-istently to almost 100%.

Based on actual usage data, we found that‘light’’ users tended to limit their interaction withhe system to accessing patient records,entering

tories of distinct groups.

Understanding technology adoption in clinical care 539

minimal new patient care data and rarely review-ing and responding to system-generated reminders.‘‘Medium’’ users tended to enter more patientcare data and generated a higher volume of more-relevant reminders, but not to act on them con-sistently. ‘‘Heavy’’ users generally used all the sys-tem functionality, as intended for nearly all patientencounters (exceptions are described below). Theyalso often added comments that assisted in follow-up care. Additional details regarding usage-baseduser acceptance and adoption analysis are providedelsewhere [18].

4.2. Qualitative assessments via surveysand interviews

‘Clinicians’ comments on the reminder system aregrouped according to the generally positive or neg-ative nature of the feedback, and presented in de-creasing order of consensus among respondents.

4.2.1. Positive feedbackMost of the positive feedback is grouped underthe theme of ‘‘practice implications’’. These com-ments suggested that CRS had positive implica-tivwtuiqtsuafiit

4Mteibmcdt‘n

r

fort required for entering the patient data. Com-ments that support this conclusion include, for in-stance: ‘‘it is very tedious to put in all of thework’’, ‘‘takes too much time to enter patient’’.Technical difficulties such as lack of typing skillshave proven to be irrelevant, based on observationsmade on-site and confirmed by the interviewees.Residents were also reluctant to perform the dataentry duty that is considered a traditional role ofsupport personnel. One user commented: ‘‘I thinknurses should enter it. I don’t have time in an ap-pointment of 20min to fill all reports’’.

Another major type of negative feedback isgrouped under the theme of ‘‘soliciting for onesingle integrated system’’, reflecting the resi-dents’ desire to have a single system perform allencounter-related tasks. One example of commentsin this connection is: ‘‘we should only have onecomplete system with labs/meds/notes, etc.’’.

Clinicians expressed concerns about time and ef-ficiency impacts in another set of negative com-ments. This theme is distinguished from the dataentry issue because it specifically targets the re-minder generation process. Examples of commentssupportive of this theme include: ‘‘takes too muchtmtudIbqrsa1t3

rcdsaottpFfttpqb

ions for the medical practice. Typical commentsnclude: ‘‘it does move me to think about some pre-entive measurements’’ and ‘‘hard to miss thingse usually tend to’’. These comments are indica-ive of a general consensus among residents thatse of this reminder system has the potential tomprove clinician performance, leading to betteruality of care. The other major category of posi-ive feedback is ‘‘ease of use’’; typical commentsuch as ‘‘Easy to use, time efficient’’ suggest thatse of the system required very minimal trainingnd skills. Difficulty of use or lack of computer pro-ciency appears not to be a significant barrier forncorporating the system into their routine prac-ice.

.2.2. Negative feedbackost of negative feedback is grouped under theheme of ‘‘iterative advisories’’, criticizing the rel-vance of reminders issued for follow-up visits. Typ-cal comments include: ‘‘it does not take feed-ack from us’’ and ‘‘it generates the same re-inders every time’’. The textual notes that ac-ompanied reminder responses also provide evi-ence regarding the potential lack of relevance ofhe reminders: nearly 30% of such notes asserted‘the suggested action has already been taken butot yet recorded’’.The theme of ‘‘heavy and hard data entry duty’’

elates to the considerable amount of time and ef-

ime to review reminders’’ and ‘‘responding to re-inders can cut in on our time with patients’’. Fur-

her investigation revealed that comments groupednder this theme originated primarily from resi-ents classified as ‘‘light’’ or ‘‘moderate’’ users.n contrast, this theme did not emerge from feed-ack of the ‘‘heavy’’ users who used the system fre-uently and worked to integrate system-generatedeminders into their practice. A quantitative analy-is based on recorded usage logs indicate that lightnd moderate user groups spent, approximately,8.5 s on average on reviewing and responding tohe reminders, whereas the heavy user group spent7.1 s.Some physicians also believed that CRS dis-

upted physician—patient communication. Typicalomments in this vein include: ‘‘using the system isisruptive during patient encounters’’. To help re-earchers learn how the reminder system was actu-lly being used, we conducted on-site observationsf a number of randomly selected patient encoun-ers. Observational descriptions reveal that use ofhe system during encounter did, in fact, impedehysician—patient communication to some extent.or instance, residents repeatedly turned back andorth between the computer and their patient. Al-hough most of the interviewees agreed that pa-ients did not object to the presence of the com-uter, some residents did express concerns thatuality of physician—patient communication coulde diminished.

540 K. Zheng et al.

As was the case for the ‘‘time-consuming anddetrimental to efficiency’’ theme, this feedbackprimarily originated from the users of light or mod-erate user groups. We conclude, this is due to thesame fact that ‘‘heavy’’ users had adapted theirbehavior accordingly. We will discuss this modifiedbehavior in the later section.

Finally, residents complained that the remindersystem lacks guidance in the application of work-flow. In contrast to the history and physical exami-nation forms that residents typically use, the inter-face of CRS appeared to provide little guidance asto a preferred order of data entry.

4.2.3. Discrepancy among user adoption groupsDiscrepancies among user adoption groups areclarified by performing the constant comparativemethod within groups. Perspectives found to be dis-tinct across groups include the themes of ‘‘time-consuming and detrimental to efficiency’’ and‘‘disruptive to physician—patient communication’’.The fact that ‘‘heavy’’ users did not comment neg-atively on these two themes is noteworthy; we findthat these users had adapted their behavior to usethe system and act upon reminders at the end of en-

5.2. Results

Comments and related system data related to itera-tive advisories indicate that information gatheringand reminder-generating functions of the system,particularly in situations that involve multiple en-counters, are problematic. We have investigatedthis problem in great detail with respect to the sys-tem’s fundamental architecture, coding integrity,system integration, and user practices. We con-clude that the problem has its origin in the ‘‘heavyand hard data entry duty’’ issue. Due of the effortrequired for data entry, we observed that clinicianstend to avoid entering newly received patient infor-mation that is essential for the reminder algorithmsto work, which in turn affects the relevance of re-minders the system generates.

For physicians who desire a single integrated sys-tem, the lack of a comprehensive electronic med-ical record system incorporating physician orderentry undoubtedly created ambiguity regarding thesearch for and recording of relevant clinical data.It also challenged the initial perception of systemdesigners that the reminder component could betreated as an ‘‘add-on’’ to the clinicians’ tradi-tional workflow.

totfatcbhsomd

wnidtd‘‘wmcfrsa

counters, or even after patients left. We do not fa-vor such a modification to the intended system usebecause it is contrary to the rationale of remindingclinicians at the point-of-care to improve compli-ance with evidence-based guidelines. However, weview this behavior as an indication of these users’strong desire to use the system. In contrast, ‘‘light’’and ‘‘moderate’’ users lacked such initiative, andthus gave up use of the system before any fur-ther adaptation attempts. These results are consis-tent with the patterns observed in the trajectoriesthat we depicted from usage data by DTA analysis;although ‘‘moderate’’ users initially showed evenstronger enthusiasm than ‘‘heavy’’ users, their us-age level underwent a gradual decline, and the us-age of ‘‘light’’ users remained at a constant andvery low level.

5. Discussion

5.1. Methods

The mixed methods model we have used in this pa-per has enabled us to successfully distinguish dis-tinct groups of CRS users, to validate the results ofstatistical analysis with actual user comments, andto finally identify key user concerns that have en-abled system redesign towards more efficient andmore effective use.

Regarding comments on the use of CRS beingime-consuming and detrimental to efficiency, webserve that our CDSS was intended to replace theraditional hand-written case notes. Despite theact that hand-written notes are often hard to readnd search and are bulky, many users believed thathe CDSS decreased the efficiency of the patient en-ounter. We think that this particular problem maye exaggerated due to the pre-existing perceptionseld by ‘‘light’’ or ‘‘moderate’’ users that use ofuch systems would severely slow the things down,r amenable to straightforward solutions such asodifications to practice routines that accommo-ate the presence of reminders.The comments on CRS’ unguided application

orkflow have resulted in a number of re-engi-eering requirements. The CRS client applications currently coded in Visual Basics, as a stan-ard Microsoft Windows application. The user in-erface is composed of several tabs that lead toifferent functionality areas. These tabs, labeled‘current reminders’’, ‘‘visit details’’, ‘‘lab test’’,‘diagnosis’’ and so on, did not provide physiciansith the specific guidance as to the order of dataanagement; data to be reviewed prior to the en-ounter, data to enter during the encounter be-ore generating reminders, data to be entered inesponse to reminders generated, and encounterummary forms to be printed for the patient chartnd for patient check-out.

Understanding technology adoption in clinical care 541

5.3. Comparison with existing literature

In the absence of systems that are deployed andregularly used in the field, researchers tend to useuser perception as a means of studying technologyacceptance and adoption. For instance, a widelyaccepted method in IS research, the technologyacceptance model (TAM), relates perceived useful-ness to use [21]. However, as other studies suggest,perceived usefulness is poorly correlated with ac-tual use. In addition to this, self-reported usage, orintent to use, may not be an appropriate surrogatefor actual use because users are poor estimators ofaspects of their own behaviors [20].

Users of an IT application differ in many ways. Ithas been well recognized that individual users playa crucial role in that their experiences, and theiropinions or reactions to a technology make a dif-ference in whether or not the technology will beadopted [19]. Nevertheless, few studies have usedindividual-level data to measure the magnitude ofuser differentiation, and the impact of such dif-ferentiation on technology adoption. Furthermore,voluntary use has not been adequately addressedin the existing informatics evaluation studies dueto the prevalent experimental designs, while thelti

5

Rttcorp

mvttttt

ttaaiosP

icant impacts on the system adoption. ‘‘Light’’users, for instance, stayed away from the systemfrom a very early stage.

5.5. Implications for re-engineering thereminder system

The studied system implementation clearly did notmeet the goals of the designers regarding systemuse. However, our experiences with this processhave helped us re-engineer the reminder system ina number of specific ways.

In response to the ‘‘iterative advisories’’ theme,we have redesigned the reminder generation mech-anism to ensure that a reminder generated repet-itively due to lack of feedback information isrephrased or temporarily suppressed. This is not apermanent or comprehensive solution; we assumedthat the data entry policy would be rigorously fol-lowed, ensuring that records of actions would befed back to the system in a timely manner. Solv-ing this problem more permanently requires a moreseamlessly integrated system and more pertinentresident education of the importance of the dataentry.

otCnfmraoeefa

recmtofp

rplotbc

evel of voluntary adoption has received close at-ention in evaluation studies in other disciplines forts value in assessing IT impacts [11,12].

.4. Lessons learned

esistance exists to use of the reminder system. Al-hough there is general consensus among clinicianshat use of this system can potentially enhance thelinical performance, leading to improved qualityf care, a significant proportion of users remainedeluctant to incorporate the system into their dailyractice.One explanation for the low usage of the re-

inder system is that the clinicians primarilyiewed the use of reminders as an ‘‘add-on’’ toheir work instead of an intrinsic part of the pa-ient encounter. Use of the system and the efforthey need to commit to activities such as data en-ry were viewed as a cumbersome addition to theirraditional patterns of care delivery.

Different users responded to this perceivedhreat in different ways; some rejected to usehe system after very minimal attempts, and somedapted their practice routines in a way that en-bled them to use the system, though not alwaysn a manner consistent with the system’s principlef generating reminders and storing physician re-ponses to these reminders at the point-of-care.re-existing perceptions are found to have signif-

Tediousness of recording data and ambiguity ofwnership of the data entry duty impose a severehreat to the implementation of systems such asRS. We have observed that certain clinical data areot captured at all by CRS due to a lack of systemunctionality. In response to discussions with ad-inistrators and preceptors of the clinic, we have

edesigned CRS to incorporate the clinical historynd physical examination forms that were previ-usly kept in paper form. We expect that this willnable the residents to enter a wide variety of rel-vant clinical data electronically. Such effort alsoulfils the desire of users to have a single system toccommodate all the clinical tasks.To further encourage the users to generate and

espond to reminders, the new system also mergesvidence-based reminders into the encounter pro-ess via history and physical examination forms. Re-inders currently appear primarily in the order en-

ry section in the form of ‘‘recommended orders’’;thers appear in the history or physical examinationorms so they can receive attention while physicianserform other routine tasks.The presence of computers in examining

ooms is found to be a distraction to efficienthysician—patient interaction. Although this prob-em is not specific to the reminder system, we rec-mmend that examination rooms be reconfiguredo better accommodate the system, for exampley placing computers in such a way that cliniciansan view the monitor while preserving eye contact

542 K. Zheng et al.

with patients. Longer-term technological solutionsinclude delivery of reminders via tablet PC or hand-held devices; such considerations are beyond thecurrent scope of this project, however.

The traditional Windows-based user interfacecomposed of hierarchical ‘‘forms’’ and parallel‘‘tabs’’ does not provide adequate guidance forusers’ workflow. To remedy this problem we haveconverted the system into a fully web-based ap-plication that largely resembles the paper-basedforms familiar to users. An intuitive navigationtool now indicates the appropriate steps of work-flow. This platform transition also makes possi-ble remote access of patient data via the weband redirecting reminders to other devices, suchas personal digital assistants. Nevertheless, it willbe crucial to train physicians in the appropriateuse of information technology in the examinationroom.

6. Conclusion

In this study, we assess clinician users’ acceptanceand adoption of a clinical reminder system. We ap-

of heart failure, J. Am. Med. Inform. Assoc. 2 (5) (1995)316—322.

[2] E.A. Balas, S. Weingarten, C.T. Garb, D. Blumenthal,S.A. Boren, G.D. Brown, Improving preventive care byprompting physicians, Arch. Intern. Med. 160 (2000) 301—308.

[3] D.L. Hunt, R.B. Haynes, S.E. Hanna, K. Smith, Effects ofcomputer-based clinical decision support systems on physi-cian performance and patient outcomes, JAMA 283 (2000)1875—1879.

[4] I. Sim, P. Gorman, R.A. Greenes, R.B. Haynes, B. Kaplan, H.Lehmann, P.C. Tang, Clinical decision support systems forthe practice of evidence-based medicine, JAMIA 8 (2001)527—534.

[5] C.L. McDonald, Protocol-based computer reminders, thequality of care, and the non-perfectability of man, N. Engl.J. Med. 295 (24) (1975) 1351—1355.

[6] C.P. Friedman, A.S. Elstein, F.W. Wolf, Enhancement ofclinicians’ diagnostic reasoning by computer-based consul-tation: a multisite study of 2 systems, JAMA 282 (1999)1851—1856.

[7] R.M. Gardner, H.P. Lundscaarde, Evaluation of user accep-tance of a clinical expert system, JAMIA 1 (1994) 428—438.

[8] B. Kaplan, Evaluating informatics applications—–some alter-native approaches: theory, social interactionism and call formethodological pluralism, Int. J. Med. Inf. 64 (2001) 39—56.

[9] B. Kaplan, Evaluating informatics applications—–clinical de-cision support systems literature review, Int. J. Med. Inf. 64

[

[

[

[

[

[

[

[

[

ply a mixed methods approach that combines quan-titative methods to identify distinct developmentaltrajectories of user adoption with qualitative in-struments to examine the causal processes of suchadoption behaviors.

We find that a significant level of resistance ex-ists to the use of the reminder system; a large pro-portion of users demonstrated a consistently lowor decreasing level of usage over time. Even thoseusers who were labeled as ‘‘heavy’’ had adaptedtheir behavior to use the system posterior to thepatient encounter; though such alteration is indica-tive of their desire to use the system, it violatedthe principle of improving compliance to evidence-based guidelines at the point-of-care.

The lessons learned and experiences gained havehelped system designers to re-engineer the re-minder system for future implementation. We alsoconclude that this mixed approach of quantitativeand qualitative methods has considerable promiseto provide new insights into the system usabilityand adoption issues that may benefit clinical deci-sion support systems as well as information systemsmore generally.

References

[1] W.M. Tierney, J.M. Overhage, B.Y. Takesue, L.E. Harris, M.D.Murray, D.L. Vargo, C.J. McDonald, Computerizing guide-lines to improve care and patient outcomes: the example

(2001) 15—37.10] N. Rousseau, E. McColl, J. Newton, J. Grimshaw, M. Eccles,

Practice based, longitudinal, qualitative interview study ofcomputerized evidence-based guidelines in primary care,Br. Med. J. 326 (7384) (2003) 314.

11] H.A. Heathfield, J. Wyatt, Philosophies for the design anddevelopment of clinical decision-support systems, MethodsInform. Med. 32 (1993) 1—8.

12] A.F. Dowling Jr., Do hospital staff interfere with computersystem implementations? Health Care Manage. Rev. 5 (1987)23—32. Reprinted in J.G. Anderson, S.J. Jay (Eds.), Use andImpact of Computers in Clinical Medicine, Springer-Verlag,New York, 1987, pp. 302—317.

13] H. Diamond, M.P. Johnson, R. Padman, Physician remindersystem: a relational database application for evidence-based medicine practice, Working paper series 2000-19, TheH. John Heinz III School of Public Policy and Management,CMU, Pittsburgh, PA 15213.

14] J.R. Lewis, IBM computer usability satisfaction question-naires: psychometric evaluation and instructions for use,Int. J. Human—Comput. Interact. 7 (1995) 57—78.

15] S.B. Merriam, Qualitative Research and Case Study Applica-tions in Education, Jossey-Bass Publishers, San Francisco,1998.

16] D.S. Nagin, Analyzing developmental trajectories: a semi-parametric, group-based approach, Psychol. Methods 4(1999) 139—157.

17] D.S. Nagin, R.E. Tremblay, Analyzing developmental tra-jectories of distinct but related behaviors: a group-basedmethod, Psychol. Methods 6 (1) (2001) 18—34.

18] K. Zheng, R. Padman, M.P. Johnson, J.B. Engberg, H.S. Di-amond, An adoption study of a clinical reminder systemin ambulatory care using a developmental trajectory ap-proach, in: M. Fieschi, et al. (Eds.), Proceedings of EleventhWorld Congress on Medical Informatics, MEDINFO 2004, SanFrancisco, USA, IOS Press, Amsterdam, 7—11 September,2004, pp. 1115—1120.

Understanding technology adoption in clinical care 543

[19] T.A. Massaro, Introducing physician order entry at a majoracademic medical center. I. impact on organizational cou-ture and behavior, Acad. Med. 68 (1) (1993) 20—25.

[20] C.E. Aydin, R.E. Rice, Social worlds, individual differencesand implementation: predicting attitudes toward a med-

ical information system, Info. Manage. 20 (1991) 119—136.

[21] F.D. Davis, Perceived usefulness, perceived ease of use, anduser acceptance of information technology, MIS Q. 13 (1989)319—339.