Comput.• Environ. and Urban Systems. Vol. 21. No .5. pp. 317-333 . 1997iC 1998 Elsevier Science Ltd . All rights reserved

Printed in Great Britain0198-9715/98 SI9.00 + 0.00

po: 50198-9715(97)10009-6

INTERACTIVE IMPLEMENTATION: PROMOTINGACCEPTANCE OF EXPERT SYSTEMS

Anne ShepherdCity Planning Program, Georgia Institute of Technology,

Atlanta, GA 30332-0155, U. S.A.

ABSTRACT. Successful implementation depends both on a computer system'scharacteristics and on the process by which the system is integrated into theuser's work routine. The importance of organizational involvement in expertsystems development has been relatively neglected in previous studies. Moreover,relatively few expert systems progress past the prototype stage and still/ewer areaccepted by the users they are designed to assist. This article examines how andwhy an expert system is implemented and accepted by a public works agency. Itanalyzes the process and the results ofan in-depth case study ofan expert systemfor water supply system operations. The focus in this study differs from muchprevious work on computer implementation : instead ofexamining just the processof technology development, this study analyzes the critical interactions betweenthe system developer, the technology, and the organization's members. A keyfinding is that the systems developer can engage in a process of "interactiveimplementation " to develop the expert system with the organization rather thanmerely for the organization. This process can catalyze organizational support,reduce barriers, and shape a system that suits user and organizational needs,which ultimately can improve acceptance. ~) 199B Elsevier Science Ltd . Allrights reserved

INTRODUCTION

"We built an expert system - so, why don't they use it?" This is a common lament.Expert systems are computer programs that seek to provide expert-quality advice indecision making. The promise of expert systems sparked the development of numerousprototypes over recent years . However, despite the resources devoted to expert systemdevelopment, relatively few systems go beyond the prototype stage and into the user'swork routine.

A fundamental reason that expert systems are not widely used is that little attention ispaid to socio-organizational factors in expert system implementation . Traditionally, farmore attention has been given to technology development than the process ofimplementation (Campbell, 1996; Kimberly, 1981; Leonard-Barton, 1988). However,expert system acceptance depends not only on system attributes, but also on the process by

317

318 A. Shepherd

which the system is integrated into the organization (Eason, 1988; Heikkila & Blewett ,1992). Organizational issues are at least as important as technical know-how (Handy,1993; Masser & Onsrud, 1993), for even the most technically sophisticated or benefici alsystem might face rejection if organizational members are either unwilling or unable to useit.

This article investigates how an expert system is integrated in a public works .organization and why certain activities influenced user acceptance. In particular, thearticle examines the crucial relationship between the expert system developer and theorganizational members in the process of system development and implementation. Itanalyzes the process and the results of a 4-year research project to implement an expertsystem for operations of the San Francisco water supply system . The analysis suggests thatthe expert system developer can engage in a process of "interactive implementation" tocatalyze organizational support, reduce barriers, and develop a system to meet user andorganizational needs. The implementation process also affected the organization in waysthat may have been unexpected but regarded as nonetheless beneficial, such as increasedcomputerization and communication. This study contributes to the theoretical andempirical understanding of technology implementation, and it provides practical insightson ways to improve expert system acceptance.

EXPERT SYSTEMS IMPLEMENTATION IN PUBLIC AGENCIES

Expert systems arc computer programs that apply artificial intelligence programmingtechniques to narrowly defined problems. They fit into the broad family of computer­based decision support systems, but differ from their siblings in several ways . For example,expert systems: usc heuristic reasoning rather than numeric computations, focus onaccepta ble rather than optimal solutions, separate knowledge and control, incorporatehuman expertise, and tend to be suitable for ill-structured and semistructured problems(Buchanan & Shortliffc, 1984; Dym & Levitt, 1991 ; Masri & Moore, 1993; Waterman,1986). In addition, expert system development differs from that of other computer-baseddecision support systems. Expert systems arc usually developed for specific domains, usingspecialized expertise, rather than for generic applications. Expert system development alsorequires a degree of interaction between the system developer and organizational members,particularly the expert(s) whose knowledge will be represented. I

Few have previously studied expert system implementations in public agencies for twopr imary reasons. First, expert system research has focused more on technologydevelopment and the adoption decision, rather than the process of implementation.? Thisemphasis overlooks the process that is actually responsible for integrating these systemsinto practice (Leonard-Barton, 1988). That is, the mere development of an expert system,or even the decision to adopt an expert system, docs not guarantee that members of anorganization will accept and use the system . This provides an important motivation forthis study: to better understand the relationships between the system developer and theorganizational members in the implementation process . Second, the paucity of literaturereflects, perhaps, the paucity of successful implementations in public agencies. Theliterature abounds with reports of implementations in the private sector; however, expertsystems have not been widely accepted in the public sector. They arc still regarded as a newtechnology with many possibilities, but one that has lagged behind other computer

Acceptance of Expert Systems 319

applications) (Davis & McDonald, 1993; Denning, 1992; Han & Kim , 1989; McKinney,Maidment, & Tanriverdi, 1993). This provides a second motivation: the need to study andlearn from public sector expert system projects.

Indeed, a review of the literature on expert systems for this particular domain - watersupply system operations - was telling. Although a number of expert systems weredeveloped and tested as prototypes (Armijos, Wright, & Houck, 1990; Clarkson &Hartigan, 1989; Goforth & Floris , 1991 ; Nishida, Palmer, & Siaughterbeck, 1990; Palmer& Tull , 1987; Sirnonovic, 1992), none of them were used on a regular basis by water supplysystem operators. Interviews with operators and expert system developers reveal reasonsfor the lack of use. In one case, operators "don't like being told what to do '" they have abetter feel for the water supply system than the expert system" (D. Rice, personalcommunication, 17 December 1993). In another case, the intended user does not use thesystem because "he prefers to make decisions in his own way" (R. Palmer, personalcommunication, II March 1994). In two other cases, the operators did not use the expertsystem because they felt that the "system was imposed upon them" and "the system wasalien to their way of working.'?'

Studies of expert systems for public sector applications, in particular urban systems,reinforce these findings; there are far more prototypes than accepted systems, andacceptance ultimately depends on the intended system users (Campbell, 1996; Davis& McDonald, 1993; Heikkila & Blewett, 1992; Kim, Wiggins, & Wright, 1990; Masri &Moore, 1993; Ortolano & Perman, 1990; Wiggins, Kim, & Jain , 1993; Wright, Wiggins,Ja in, & Kim, 1993). Davis & McDonald (1993) suggest that user acceptance could beincreased if expert system developers focused more on ways to support users as they makea decision, and less on the sophisticated algorithms to make the decision. This resonates inOrtolano and Perman (1990) who suggest that expert systems can assist users, but a moredifficult question is whether expert systems will be acceptable to the users they are designedto assist. As Heikkila and Blewett (1992) note, an expert system will be useful only if itconforms to and enhances existing practice. Wiggins et al, (1993) found that expert systemusers who became more productive became expert system advocates. A champion - astrong vocal advocate in technology adoption - played an important role in encouragingthe usc of the expert system. These previous studies, while instructive, have not specificallyanalyzed the interactive involvement between the system developer and the organizationalmembers, and the resulting effects on user acceptance, whieh is an objective of this study.

This interplay between technology, organizations, and user acceptance draws upon theliterature on innovations and the diffusion of technology (Leonard-Barton, 1995; Rogers,1995; Tornatzky ct aI., 1983). This literature is voluminous; nearly 4,000 articles have beenpublished since the 1940s (Rogers , 1995). In addition, the literature has been characterized,as Meyer and Goes (1988, p. 897) point out, as fragmentary (Kelly & Kranzberg, 1978),contradictory (Kimberly & Evan isko, 1981), beyond interpretation (Downs & Mohr,1976), and without a single theory that will permit researchers to predict success(Mohr, 1982). Despite the lack of consensus on an appropriate theory, well-specifiedmodels are none the less an important starting point on which to ground case studies ofexpert system projects (Eliot, 1992). An important model for this research is put forth byMeyer and Goes (1988, p. 903), which parallels Rogers (1995, p. 163), for examining theimplementation process (Table I). This process characterizes the degree to which theexpert system has progressed through the stages of acceptance." Within this process, this

320 A. Shepherd

Table 1. Implementation Process of Expert Systems in Public Agencies

Knowledge-awareness stageI. Consideration - individual organization members learn of existence of expert systems and

consider its applicability for their organization.2. Discussion - organization members discuss possible uses of expert systems in their organization.

Evaluation-choice stage3. Proposal - unit within the organization writes a proposal to develop a prototype expert system

for a particular task or set of tasks.4. Fiscal evaluation - proposal for prototype is evaluated by individuals, units, or organizations

controlling resources needed for system development.S. Political evaluation - proposal for prototype is evaluated by individuals, units, or organizations

whose political support is needed for system development.

Adoption-implementationstage6. Conceptual model - organization members and/or consultants develop a conceptual model and

detailed plan for developing an expert system prototype.7. Prototype development - conceptual model is operationalized in the form of a prototype expert

system.8. Verification and validation - prototype is tested to determine correctness of computer coding

and validity of expert system outputs.9 . Tri al implementation - prototype is refined and introduced into workplace on a trial basis.10. Acceptance - prototype is further refined and introduced widely for routine use with provisions

for maintenance and support.

Adapted from Meyer and Goes (1988. p. 903).

study devotes particular attention to the interactions between the system developer andorganizational members that influenced implementation and user acceptance.

The analytic focus of this study addresses what Rogers (1995, p. 157) stated as animportant research area that should be addressed to improve our understanding of thesystem development process: the key interrelationships among the various actors in thedevelopment-implementation process. In particular, how does the system developerin terface with change agents" in the implementation of an innovation? Why didorganizational members become involved and how did that involvement influence useracceptance? This article pursues these questions as follows. First, the expert system projectis presented. Then, the implementation process is examined; specifically, the interactionsbetween the system developer, organizational members, and technology development. Theexpert system attained a relatively high degree of aeceptance, even though expert systemshave not been widely adopted by public agencies in the U.S.A. This leads to an analysi s ofhow and why interactive involvement influenced the implementation process andorganizational acceptance. The article concludes with lessons learned for promotingacceptance of expert systems.'

AN EXPERT SYSTEM FOR WATER SUPPLY SYSTEM OPERATIONS

An expert system was developed to assist operators of the San Francisco WaterDepartment (SFWD).8 The SFWD supplies water to more than 2 million people in theCity and County of San Francisco, the San Francisco Peninsula, the South Bay and EastBay regions (Figure I). The water supply system is highly complex: several large reservoirs,hundreds of valves, and thousands of miles of pipelines . The operation of the water supply

Acceptance of Expert Systems

(not to scale)

FIGURE I. The San Francisco watcr supply systcm.

321

system is handled chielly by operators stationed at the San Andreas Water TreatmentPlant. These operators base their decisions on a "feel" for how the water supply networkbehaves and responds to changes. That is, they do not rely on formal procedures,mathematical models, or optimization techniques - they rely on heuristics developedthrough years of experience with the water supply system.

The expert system was developed in response to several concerns of the SFWD. First,expert operators are scarce, operating expertise is valuable, but this expertise is elusive andnot documented. Operating expertise resides in the memory of expert operators who haveworked with the system for decades. The SFWD was concerned that the tacit,undocumented knowledge required to operate the water supply system might be lostwhen personnellcave. Indeed, the SFWD recently lost over 100 years of collective expertisedue to personnel changes and retirements. The expert system was seen as a way todocument, retain, and make available the expertise required for operations.

Second, trained operators and training resources are scarce. There is a high turnover innovice operators and a high demand for trained operators. A substantial amount of experttime is required to train novices. However, because experts have many other demands, andbecause operating guidance and expertise is not well documented, novice operator trainingis often deficient. Training and operations manuals exist - but they are deemed "useless"

322 A. Shepherd

by many staff members. SFWD personnel expressed the need for training that wouldimprove novice operators' techniques and would reduce demands on experts. The expertsystem could, conceivably, help to train novice operators using the expertise of senioroperators.

Third, operating mistakes can have serious and adverse consequences. Novice operatorsare customarily placed on the graveyard shifts with little , if any, guidance or expertfeedback. They need to wake up a senior, day-shift operator if they have an importantquestion. The ramifications of operating mistakes can be serious, such as water mainbreaks or reservoir spills. Novice operators expressed that their primary concern is"goofing up" - they fear making mistakes and therefore desire expert feedback to helpprevent or reduce the incidence and impacts of mistakes. SFWD personnel saw the expertsystem as a way to alert novice operators to the undesirable consequences of their decisions.

At the outset of this project, it appeared that an expert system could help to addressthese concerns, but that the conventional expert system approach would likely encounterproblems of user acceptance (Shepherd & Ortolano, 1994). For instance, a conventionalexpert system generally provides directive advice, based on what the expert would do.However, operators did not want to be told "what to do" by a computer, nor be forced toemulate the expert's approach. Moreover, conventional expert systems have difficultyaccommodating multiple problem-solving approaches and solutions. However, there aremany acceptable approaches to water supply system operations, and operators wanted toretain, improve, and obtain feedback on their own problem-solving approach.

These user concerns prompted the development of an alternative approach to expertsystem development: a critiquing approach. In a conventional expert system, the usersubmits data to the system and receives a system-generated solution. In a critiquing expertsystem, the user submits both data and a proposed solution or plan of action (Figure 2).The system evaluates the user's plan and provides a critique. The critique includesalternatives, explanations, justifications, warnings, and additional information (Table 2).

The development of a critiquing expert system, rather than a conventional expertsystem, was appealing to the SFWD for several reasons') (Shepherd & Ortolano, 1996).First, the critiquing approach is well-suited for training. The consultation session is user­focused rather than expert-focused: it can provide feedback on an operator's plan, ratherthan just the expert's plan. Second, critiquing can accommodate multiple problem-solvingapproaches. SFWD personnel generally agree that there is no single best approach to

Input data data proposellplan

Output

ConventionalExpert System

plan

CritiquingExpert System

critique ofproposedplan

FIGURE 2. Conventional versus critiquing approaches to expert systems.

Acceptance of Expert Systems

Table 2. Sample Inputs and Outputs for the SFWD Critiquing Expert System

Inputs: Data and Plan

What is the state of the Balancing Reservoir? High

What is the state of the Pulgas Tunnel? High

Are you trying to spill to Crystal Springs Reservoir? No

Do you want to bring water into the main system from the Balancing Reservoir? No

Do you want to fill Crystal Springs from the East Bay supply? No

What do you plan to do? Increase East Bay supply

Outputs:

The Balancing Reservoir is high, the Pulgas Tunnel is high. You answered that you do not want tospill to Crystal Springs Reservoir; you do not want to bring water into the main system from theBalancing Reservoir, and you are not trying to fill Crystal Springs from the East Bay supply.

Your plan is to increase the East Bay supply.

It is preferable to decrease the East Bay supply in this case.

It is not advisable to increase the East Bay supply because it would cause the Balancing Reservoir levelto become dangerously high. Also, increasing the East Bay supply would result in an excess supply inthe Low Zone region, since you are not planning to transfer water to Crystal Springs. By decreasingthe East Bay supply, you could keep the Balancing Reservoir and Pulgas Tunnel at safe levels.

Since you do not want to bring water into the main system from the Balancing Reservoir, make surethat it fluctuates for water quality purposes.

To decrease the East Bay supply, you could contact Sunol pump station and have them go down inrate.

In addition to decreasing the East Bay supply, you may consider increasing the Low Zone demand tosupplement the remedial effects of decreasing the East Bay supply. To increase the Low Zone demand,you could take the following actions:

I. light ofT Baden to supply the High Zone;2. decrease or slop the San Andreas filter plant rate;3. increase the pumpage of the Sunset pumps at Lake Merced.

323

operations; different operators' styles can yield acceptable results. Third, critiquingsupports active learning. The operator must think through the problem and develop theirown plan, rather than rely on the system-generated solution. This could reduce problemswith user-passivity or over-reliance on the system. Fourth, critiquing allows operators todirect the consultation session. Operators can override the system or obtain feedback onjust a portion of their plan. Critiquing, unlike the conventional expert system approach,can be used with partial proficieney or knowledge in an area; operators do not need toanswer all system queries in order to obtain a critique of their plan.

The expert system was developed, incrementally and iteratively, over a period of 3 years(1989-91). The development process proceeded, in general, according to the rapidprototyping paradigm: (a) perform knowledge acquisition - observe and interview theexpert(s); (b) implement the acquired knowledge in a computer program (referred to as anexpert system prototype); (c) demonstrate performance of prototype to the expertsby running test cases; (d) obtain a critique of system performance from the experts;(e) demonstrate prototype again to obtain additional feedback from the experts; and(f) repeat the process. This incremental style of expert system development uses a workingversion of the system, however incomplete, throughout the process.

However, this development process included a second, and important, dimension torapid prototyping: end-users. In each of the steps above, "experts" included not only the

324 A. Shepherd

expert operator, but also the intended end-users and other organizational members. In thisperspective, end-users are also "expert" when it concerns the ways in which they would orcould use the expert system. This point is crucial: the merit of an expert system depends notonly on what the system produces as output, but on how that output is received by theend-users. Put another way, system development needs to focus on both knowledgetransmission and reception. to Because the critiquing approach to expert systems seeks tohelp users learn, it is essential that users participate in the development of the expertsystem.

Developing a critiquing system required the acquisition of two types of knowledge:domain knowledge and critiquing knowledge. Domain knowledge is expertise aboutoperations of the San Francisco water supply system. Critiquing knowledge is expertiseabout how to offer feedback on possible operating plans; knowledge on how to helpoperators improve their methods of operating the system. The critique itself is how theexpert would articulate feedback to the user. That is, the expert operator, when faced witha novice operator's proposed action in a particular operations' scenario, would offer aparticular set of reactions and advice. This constitutes the critique, and one such critiquewas acquired for every meaningful combination of operations' scenarios and operators'plans. (This critiquing system included over 200 combinations of operations scenarios,operating plans, and critiques.)

The expert system was installed on an IBM-compatible PC at the San Andreas WaterTreatment Plan (the primary site of operations), in Millbrae, California. in October 1992.The system was formally evaluated during a period of 6 months. In particular, empiricalstudies were performed to determine ways in which the operators used, or would usc, theexpert system. Operators evaluated the system on a set of criteria related to useracceptance and desirability' I using questionnaires with Liken-type response categories.The expert system was rated favorably on each criterion of user acceptance and desirability(averaging 4.0 on a 5-point scale). The expert system received its highest ratings on criteriarelated to practice-based training and instruction. which were also the attributes that weremost important to the operators. (Shepherd, in press, provides details on the results ofthese empirical studies.) The SFWD used the system on a trial basis, and allocated funds toexpand the system's capabilities and to integrate the system into a long-termcomputerization and operator training program. Based on Table I, the degree ofassimilation of the expert system at the SFWD was rated, by SFWD personnel, as 9 on alO-point scale (i.c., "trial implementation").

EXPERT SYSTEM IMPLEMENTATION PROCESS

This section explores the implementation process within the framework outlined inTable I. The first part examines how the system developer involved SFWD organizationalmembers and how that involvement shaped the expert system. The second part examineshow the expert system is regarded to have shaped the SFWD organization. This is followedby an analysis of how and why this interaction influenced implementation and useracceptance.

Acceptance of ExpertSystems 325

Influence of Organizational Involvement on System Development

Knowledge-Awareness Stage

The idea for this project sparked when the SFWD General Manager heard about expertsystems at a conference, and wanted to learn more. Through a network of telephone callsand connections, the SFWD General Manager was put in contact with the author. I metwith a group of SFWD managers to discuss possible applications of expert systems. Evenat this early stage, the managers were interested in pursuing an expert system project, asevidenced by their comments, such as "ways we see an expert system being used in thewater department" and "this could really help US.,,12

I then met with each of the managers to learn how an expert system could help them.These meetings helped to identify organizational needs and to build consensus. They alsogave the managers a chance to "buy into" the system by contributing to crucialdevelopment decisions. After the initial meetings, I reconvened with all of the managers,and presented an overview of the system's design and objectives in order to obtain theirfeedback and approval. This process helped to define expert system development goals thatwere widely supported by managementY

Indeed, the consensus-building process yielded a group of system advocates within top­level management at SFWD. The managers' support was important because it providedaccess to the organization and to the operators. For example, one manager not only gavehis approval for me to work with the expert and novice operators - he made projectparticipation a part of the operators' job descriptions. Without this cachet for this project,the operators might have been reluctant to spend time with an "outsider" instead of ontheir regular duties.

Evaluation-Choice Stage

The decision was made to go forward with the expert system project. The GeneralManager circulated a memorandum that documented the need for the expert system andthe perceived benefits to the SFWD. It should be noted that, at this stage, the SFWD wasnot required to put up funding for the expert system project, although they would beproviding resources in terms of equipment and personnel time.

Top-level management was on board, but a managerial directive can only go so far inimplementation. The next crucial step was to obtain support from the end-users: theSFWD operators. I went to the operations facilities to meet with the operators and, inmuch the same way as with the managers, to identify the operators' problems and the waysthat an expert system could help them. Even though managers were enthusiastic about theexpert system, the operators were skeptical. First, operators had serious reservations abouthow an outsider (the system developer) might affect their work procedures. The operatorswere already busy; they did not want more interference. Second, the operators wereconcerned that "another useless computer model" would be developed that "just sitsthere." They explained that, in previous years, the managers had hired consultants todevelop computer-based systems "for the operators." These computer systems were basedon simulation and optimization models. The operators did not usc them because "theydon't work the way that we work" and because "they built these things without ever askingus." 14

Reducing barriers and creating incentives is more of an art than a science. The operatorsseemed to become more receptive when they were assured that the system would be

326 A. Shepherd

developed with them and for them. Operators also became more interested when theylearned more about expert system technology; in particular, the critiquing systemapproach. This approach appeared to meet some of the operators' immediate concerns: thedemands on operators' time, the need for training operators, lack of expert-qualityguidance, and the fears associated with operating mistakes. I also assured operators thatinterference would be kept at a minimum. The initial stages of knowledge acquisition couldbe performed through observation, which would reduce demands on their time. In laterstages of knowledge acquisition, operators appeared far less concerned about interference,as discussed in the next section.

Adoption-Implementation Stage

System development proceeded according to the rapid prototyping paradigm, as earlierdescribed. The primary source of expertise for the knowledge base was the individualregarded by SFWD managers as the "super expert." He participated in knowledge­acquisition and system-validation exercises, and served as a champion of the expert systemproject. He also seemed flattered that management selected him and regarded him as asuper expert. When asked if he had any reservations about sharing his knowledge (forinstance, job security issues), he responded that he had "none at all." Instead, he hopedthat by formalizing what he had learned from running the water supply system that itmight be helpful to othcrs.P

An additional and vital source of knowledge came from the SFWD operators - theoperators who were the intended end-users of the expert system. I spent considerable timewith the operators; observing them while they worked, asking them about their jobs, andlearning about SFWD operations. This activity was important in order to understand howoperators worked in practice, and to understand the context in which the system wasintended to be used. This activity also had an unexpected yet beneficial side-effect: Ibecame regarded as "one of them" rather than an "outsidcr.r '" I made an effort to learnabout operators and their work. In turn, operators appeared to like to be asked about theirwork, and to teach me about their work. This enabled me to become, in some ways, anovice operator, and to engage in a process of mutual adaptation: to refine the system tofit user needs, and to understand how users could take advantage of the new system. 17

I met regularly with the SFWD personnel to demonstrate the system, and to obtainfeedback on the system's design and performance. The demonstrations served threeimportant purposes. First, SFWD personnel were able to sec, first-hand, ways in which theexpert system could benefit them. The SFWD General Manager remarked that other wateragencies would view this project as "the way of the future for water agencies." 18 A noviceoperator said it was "very helpful" and "easy to usc and understand." Another operatorcommented, "Can't you leave it here tonight - we could really usc it!" Second, throughprototype demonstrations, I was able to obtain valuable feedback from the expert and theintended users on how to improve the system. Because the system incorporated theknowledge and needs of organizational members, it could be made more suitable to theintended working environment. Third, the process of involving personnel in prototypedevelopment led to their feeling of ownership. One barrier seemed to be diminished: thesystem was frequently referred to as "our system"!" rather than a system developed by anoutsider and imposed upon them.

Moreover, operators appeared to genuinely enjoy being involved in the expert systemproject, and they willingly devoted their time (often after-hours) to participate.I'' An

Acceptance of Expert Systems 327

example of this occurred during a prototype demonstration and evaluation. After an hour,I thanked the operators, and added, "I'm grateful for your help, but I don't want to takeup any more of your time. " One operator responded, "Oh, no, keep going, this is fun . I gotoff work two hours ago, but I stayed around to be able to see this." Another operator said ,"It's great to see what we know and what we do put in a computer like this. It can reallyhelp."

In the 4th year of the research project, several events occurred. First, the GeneralManager, the individual who first supported the idea of an expert system, left the SFWD.His replacement appeared to be equally interested in the expert system project, andarranged a series of meetings to discuss ways in which the expert system could beintegrated with other planned computing facilities at SFWD. Notably, the new GeneralManager dedicated funds to the ongoing development, maintenance, and expansion of thesystem. The funds were approved by the SFWD and were pend ing approval by the PublicUtilities Commission (the oversight board for the SFWD). However, after repeatedattempts , the SFWD was not able to release the funds because the Public UtilitiesCommission could not justify expenditures on something other than "infrastructureimprovements," despite the attestations of SFWD personnel regarding the system'sbenefits. Second, the "super expert" operator, the primary source of expertise for thesystem, left his position and was unable to continue with the expert system project. Third,the system developer (the author) was preparing to move out of the area. The systemdeveloper cautioned the SrWD personnel against using the expert system unless thesystem could be subjected to regular maintenance and validation, as it had been over thepast 4 years. However, SFWD was not able to ident ify or hire a suitable replacement forthe system developer to carryon the process. Anyone of these factors might have led to thedecline of the expert system's usc within the SFWD. All three of them together proved tobe too much to overcome. So, the decision was made to wait until more resources wereavailable to expand the expert system's capabilities and integration with SFWD computingfacili tics.

Influence of System Development on the Organization

An interesting aspect of the expert system project concerns the ways in which the systemdevelopment process influenced the SrWD. The changes in the srWD provide a strikingexample of the impact of a new computer technology on an organization. This sectionexamines two types of changes: increased computerization and increased communicationbetween SFWD managers and operators. Numerous interviews conducted with SFWDmanagers and operators substantiate the observat ions of the changes in organizationalbehavior and operations reported below. 21

First, the SFWD appears to be more receptive to computer technology than at thebeginning of the project in 1989. This receptivity can be attributed, in part, to the expertsystem . In particular, the expert system appears to have increa sed both computerautomation and autonomy for operators' work . For example, data acquisition has becomeautomated as a result of needs highlighted by the expert system development process.Before the start of the project, operators had to make several telephone calls every hour inorder to obtain data, for example, on reservoir levels and pumpage rates. Interestingly,these data were routinely and automatically printed out at the SFWD managers' offices.Operators expressed that "they [the managers] don't even usc that data but they have itbecause it's hard for them [the managers] to admit that we [the operators] really arc the

328 A. Shepherd

ones running the system." Even though the data printouts were not used by managers, thedata appeared to be a means for managers to retain control and to signal authority.However, the expert system project highlighted the need for, and the advantages of,automated data retrieval as a source of input for the expert system. That way, operatorswould not have to make the telephone calls for each piece of input data; operations couldbe made more efficient. Indeed, since the expert system project, operators can now obtaindata automatically from computerized monitoring equipment and data acquisitionfacilities. As a second example, before the start of the expert system project, to make avalve adjustment, an operator had to submit a request for a maintenance technician todrive to the valve location and manually adjust the valve. This was cumbersome andinefficient, especially because valves need to be adjusted every few hours. However, therequirement that an operator perform an intermediary step in order to make changes onthe water supply system appeared to restrict the operators' ability to run the systemdirectly and on their own. After the expert system project, however, valve adjustmentsbecame automated: operators can now adjust valves directly from a computer at theoperations facility.

That the system development process led to increased computerization at SFWO wascorroborated by one SFWO manager:

Thinking of how the computer [the expert system) was going to work at the San Andreas[Water] Treatment Plant made us think about other computing facilities that needed to be builtin. It called our attention to the fact that we needed to give the operators data, in addition tothe kind of help it [the expert system] would give by bringing an operator's attention tosomething. It caused us to pursue further computerization. What has happened may havehappened anyway ... well, maybe not ... but if it had, it would have taken much longer if theexpert system project hadn't been there.22

Operators perceived the expert system, and increased computerization, as a means forgreater autonomy. One operator said, "If we use the [expert] system, we could makedecisions rather than follow directions ...23 A systematic study of whether the expert systemproject in and of itself catalyzed these computer innovations and increased autonomy foroperators was not conducted. Nevertheless, there is wide agreement at the SFWO that theprocess of developing the expert system increased discussion on the need for computerinnovations to provide operators streamlined access to data.

Second, the expert system project appears to have encouraged, and necessitated,communication between operators and managers. The managers depended on theoperators not only to provide the expertise for the system, but also to provide feedbackon the usefulness of the system in operations. When managers heard positive responsesfrom the operators, they were encouraged to continue their support. Similarly, operatorsdepended on the managers for their support of the system, and to "not let it get away fromthe department. ..24 One manager said that, in contrast to before the expert system project:

We now ask questions of operators - we have much more frequent and much more relaxedcommunication. Before [the expert system project) we really didn't communicate. Now we [themanagers] admit that no one knows everything, and we need the operators' knowledge and totalk with them about operations and how we can help each other to improve them. The expertsystem was a major reason that we started communicating with the operators and appreciatingthe operators.25

Acceptance of Expert Systems 329

Similar impressions hold true on the operators' side. Before the expert system project,operators expressed that "the managers don't view our work as important as it really is tothe entire operations of the water department." After the expert system project, operatorssaid that "the expert system shows them [the managers] how much someone has to knowto run this system" and that "the managers now realize that if anything happened to us,they'd be in trouble.,,26 The expert system appears to have validated the operators' roles inthe eyes of management. The expert system project also seems to have increased, andimproved, communications between managers and operators at the SFWD.

ANALYSIS OF FACTORS THAT INFLUENCED ORGANIZATIONALINVOLVEMENT AND USER ACCEPTANCE

This section analyzes the interrelationships between the system developer, the SFWDmanagers and operators, and expert system implementation and acceptance. In particular,it looks at the involvement of the managers, the operators, and the system developer, andthe factors that influenced user acceptance.

Why did managers lend support, and how did managers' involvement contribute tosystem development and implementation? There is little question that managerial support,at the beginning, enabled the system developer to work within the SFWD. (Later stagesdepended not only on managerial commitment, but on operator involvement.) Theongoing managerial commitment enabled the project to go forward, and appears to havebeen motivated by several factors. First, the SFWD managers were interested in exploringa new technology, an expert system, to assist in operations. They saw it as something thatcould both address their needs and be a source of prestige for the agency. Second, themanagers were instrumental in defining the goals of the expert system; that is, the waysthat an expert system could assist them. The system development process was shaped byorganizational needs, rather than by just the technology or a research agenda. Third, themanagers were involved throughout the project. This interactive involvement contributedboth to system development decisions, as already noted, and to the managers' favorableperception of the expert system. Prototype demonstrations were a way for managers to seethe benefits of the expert system and to offer suggestions for improvement. Fourth,admittedly, the SFWD did not have much at risk. The SFWD was not required to outlayany funds for the project during the first 4 years. The SFWD did, however, commitpersonnel time spent by managers and operators who participated in system development.While this cost is notable, it is largely hidden. That is, the SFWD would not be heldaccountable by the Public Utilities Commission or others outside the organization if thesystem development project did not achieve its anticipated goals.

Why did operators willingly participate in the project, and how did their involvementinfluence implementation and acceptance? It could be argued that operators were requiredto participate, based on the managerial instruction. That might have been the reason in theinitial stages, but it was not a significant factor in the later stages. There appeared to beother reasons for the operators' cooperation. First, the operators' immediate concernswere alleviated. The system developer was able to address their fears and provide thepromise of a system that might be able to help them. Second, the promise was backed upby the product. During prototype development, the operators could see the ways in which

330 A. Shepherd

the system could assist them. They could make suggestions and they would see theirsuggestions implemented. Third, the operators tried the system and they liked it. Itprovided them with much-needed expert-quality assistance, particularly in training.Fourth, the operators appeared to enjoy the process of system development. They believedin the expert system's merits and willingly spent their time, often after-hours, to participatein the expert system project. The operators also, basically, liked to be asked about theirwork. Fifth, operators saw the expert system as a way to advance in their jobs, and toincrease their decision-making autonomy. Sixth, the expert system appeared to validateoperators' roles and responsibilities in the eyes of management. Seventh, the expertoperator played a pivotal role; the system relied on his expertise, and systemimplementation relied upon his support of the project. His wholehearted cooperationwas a major reason that the expert system project went forward.

How did the system developer promote involvement, and how did that involvementinfluence implementation and acceptance? The system developer took deliberate steps toinvolve SFWD personnel before and during expert system development. In contrast totraditional prescriptions for expert system development that involve just the expert, thisprocess involved other personnel, including the intended end-users. As evidenced inprevious studies, a new technology that is developed external to an organization, and thenimposed on an organization and its personnel, can be subject to resistance. Therefore, Isought to work with - and within - the organization. This interactive involvementpromoted acceptance in a number of ways. First, involvement created a sense of ownershipor buy-in among the personnel, which appears to have led to a sense of responsibility,commitment, and advocacy. Second, involvement helped to create a system that was"better" because the system could incorporate the expertise, feedback, and needs of theorganizational members. Third, involvement helped to reduce barriers. Conflicts andskepticism could be resolved before they turned into deep dissatisfaction or a pre hocrejection of the system. Fourth, users felt the system was developed with them and forthem, rather than imposed upon them. This was a proactive step toward acceptance, ratherthan reactive attempts to try to figure out "why won't they usc the expert system?" Fifth,because of involvement, the system developer gained in-depth knowledge of theorganization and how operators worked in practice. The operators started to regard thesystem developer as "one of them" rather than an "outsider." Because of this involvement,the system developer could shape the system to fit user needs, and could sec ways foroperators to take advantage of the system.

What could have been improved? Greater attention could have been given to ensuringongoing maintenance and validation. The system was developed, the organization seemedenthusiastic, the users were using it, but no one expected key personnel to leave. Eventhough funds were committed to system maintenance and expansion, they could not bereleased. It could be argued that these administrative difficulties were characteristic of apublic agency. However, it could also be argued that because this was a public agency, andnot a private firm, the organization had the freedom and flexibility to pursue this expertsystem project in the first place. Moreover, the system developer might not have gainedthis degree of access to the organization and its members if it had been a private firm.

In conclusion, interactive implementation can help to promote user acceptance. Thisprocess recognizes that acceptance depends not only on system benefits, or on managerialstrategies, but on the system developer's involvement with members of the organization,particularly the intended end-users. There is no generic recipe for success; technology

Acceptance of Expert Systems 331

implementation is both art and science. However, results of this study strongly suggest thatexpert system developers should seek to work with an organization in an interactive,adaptive approach to implementation.

ACKNOWLEDGEMENTS- I thank Leonard Ortolano and John Steinemann for theirvaluable suggestions on this article and this research. Christi Bowler, Jake Gilmer, andJeann Greenway reviewed previous drafts. I am grateful to the San Francisco WaterDepartment for their cooperation throughout the expert system project.

NOTES

I However, as this article will show, the standard rapid prototyping paradigm for expert systemsprescribes expert involvement, but not necessarily end-user involvement. This study extends thisparadigm by explicitly involving end-users in prototype development and implementation.2This article considers implementation as the process of user acceptance; the expert system'sassimilation into the user's work routine. In this sense, implementation is distinguished fromdiffusion (to spread throughout the organization) and innovation (to introduce or effect a change),although the literature tends to use these terms interchangeably.3 That public agencies have not adopted expert systems to the extent of private firms is an interestingand complex issue, and one that is beyond the scope of this immediate study. Nonetheless, this studydoes provide insights on aspects of a public agency that, in this case, influenced expert systemimplementation.4 Personal interview by telephone with four individuals who requested confidentiality, March 1994and October 1997.5 The focus in this article is on implementation rather than diffusion because of the nature of theorganization, the technology, and the operators' work tasks. The expert system, fully implemented.will be used at only one location. rather than diffused throughout the organization.«A change agent is an individual who influences innovation-decisions in an organization. The changeagent usually seeks to obtain the adoption of new ideas, but can also attempt to slow down or impedethe process (Rogers. 1995, p. 27).7 The research approach for this case study was developed according to the principles of Yin (1994).To increase reliability and validity, the data collection effort used (a) multiple sources of evidence,(b) a case study database, and (c) a chain of evidence. Multiple sources of data included: in-personand telephone interviews with more than 20 water managers and operators and 12 expert systemdevelopers; surveys; correspondence; and review of documentary evidence. Quotations herein aretaken from interviews with water agency personnel and regulators who. unless named, requestedconfidentiality.HThe system developer was the author of this article.9 For the remainder of the article. the "critiquing expert system" developed for this application willbe referred to as simply the "expert system."In Sec Shepherd (in press) and Waem et a!. (1992) for a discussion of these aspects of critiquingsystems.II The criteria were the following: (a) provides me useful feedback on my operating plan, (b) allowsme flexibility in my operating style, (c) lets me feel in control making the decision, (d) helps me toimprove and refine my own skills, (e) is focused to my needs, (I) asks relevant questions, (g) lets mecarry out practice as usual, (h) could be a positive innovation in the workplace, (i) is a form ofdecision support whieh I could rely on, U) could improve the quality of my operating decisions,(k) could enhance my professional image, (I) improves the efficiency of my decision making,(m) allows me to feel active rather than passive, (n) gets me to think thoroughly about my planbeforehand, (0) gets me to think thoroughly about my plan afterwards. (p) provides advice which Itrust, (q) is a good teaching tool, (r) is a good learning tool, (s) is something I would usc in dailyoperations.

332 A. Shepherd

12Personal interview with SFWD managers, 2 October 1989, Millbrae, CA.IJ The goals of a proposed expert system project were the following: (a) to assist in meeting SFWD'soperational goals and have the water system run more efficiently, (b) to alert operators to theramifications of the proposed decisions and changes on the water supply system, (c) to train andassist less-experienced operators, (d) to provide a context for the operators to work within whileallowing for individual style, experimentation, and flexibility, (e) to preserve and formalize historicaloperating expertise in order to communicate that expertise to less-experienced operators, and (0 toapply historical expertise to attain a higher quality of operations.14 Quotations in this paragraph are from personal interviews with SFWD operators, November andDecember 1989, Millbrae, CA.tSPersonal interviews with the SFWD expert operator, Winter 1990, Millbrae, CA.t6These observations and direct quotes are based on interviews with three novice operatorsconducted in Winter 1992, Millbrae, CA.17 This is supported by Leonard-Barton and Sinha (1993) who found that the process of mutualadaptation was a determinant of success in internal technology transfer.18 Personal interview with the SFWD Deputy General Manager, November 1992, San Francisco,CA.19Personal interviews with SFWD operators, Winter 1992, Millbrae, CA.20 Personal interviews with SFWD operators, January to June 1991 and Winter 1992.21 Personal interviews with SFWD operators, Winter 1991, Autumn and Winter 1992, Millbrae, CA,and with three managers, Winter 1992, Millbrae and San Francisco, CA. In addition, manycomments are not included here because they were offered to the author as "off the record."22 Personal interview by telephone, Manager of Water Resources, November 1992.2J Personal interviews with SFWD operators, Winter 1992, Millbrae, CA.24 Personal interview by telephone, Manager of Water Resources, November 1992.2S Personal interview by telephone, Manager of Water Resources, November 1992.26 Personal interviews with SFWD operators, Winter 1992, Millbrae, CA.

REFERENCES

Armijos, A., Wright, J. R., & Houck, M. (1990). Bayesian inferencing applied to real time reservoir operations.Journal of Water Resources. Planning. and Management. ASCE. 166(1), 3ll-51.

Buchanan, B. G., & Shortliffe, E. H. (19ll4). Rule-based expert systems: The Mycin experiments of the StanfordHeuristic Programming Projects. Reading, MA: Addison-Wesley.

Campbell, H. (1996). A social interactionist perspective on computer implementation. Journal of the AmericanPlanning Association. 62(1), 99-107.

Clarkson, C, & Hartigan, J. (1989). Expert systems approach to water supply management decisions. In ASCEConference on Computing in Civil Engineering (pp. 101-107), New York, NY: ASCE. Atlanta, GA.

Davis, J. R., & McDonald, G. (1993). Applying a rule based decision support system to local governmentplanning. In J. R. Wright, L. L. Wiggins, R. K. Jain, & T. J. Kim (Eds.), Expert systems in environmentalplanning (pp. 23-46). New York: Springer-Verlag.

Denning, J. (1992, June). Expert systems: Ready to hit the road? Civil Engineering, pp. 71-74.Downs, G. W., & Mohr, L. B. (1976). The adoption of radical and incremental innovations: An empirical

analysis. Management Science. 32, 1422-1433.Dym, C. L., & Levitt, R. (1991). Knowledge-based systems in engineering. New York: McGraw-Hili.Eason, K. (1988). Information technology and organizational change. London: Taylor & Francis.Eliot, L. B. (1992). Case analysis of expert systems projects: Strategies and examples. Journal ofSystems Software.

19,153-157.Goforth, G. F., & Floris, V. (1991). OASIS: An intelligent water management system for South Florida. Al

Applications, 5(1), 47-55.Han, S-Y., & Kim, T. J. (1989). Can expert systems help with planning? Journal of the American Planning

Association. 55, 296-308.Handy, C. B. (1993). Understanding organizations. Harmondsworth, England: Penguin.Heikkila, E. J., & Blewett, E. J. (1992). Using expert systems to check compliance with municipal building codes.

Journal oJ the American Planning Association. 58, 72-80.

Acceptance of Expert Systems 333

Kelly. P., and Kranzberg, M. (eds .) (1978) . Technological innova tion: A critical review of current knowledge. SanFrancisco. CA : San Francisco Press .

Kim . T. J.• Wiggins, L. L.. & Wright, J . R. (Eds.). (1990). Expert systems: Applications to urban planning. NewYork: Springer-Verlag .

Kimberly, J. R. (1981) . Managerial innovation. In P. Nystrom & W. Starbuck (Eds.). Handbook ofOrganizationalDesign (Vol. 1, pp . 84-104). London: Oxford Un iversity Press.

Kimberly, J . R., & Evanisko, M. J. (1981). Organizational innovation: The influence of individual, organizationaland contextual factors on hospital adoption of technological and administrative innovations. Academy ofManagement Journal . 24, 689-713.

Leonard-Barton, D . (1988). Implementation as mutual adaptation of technology and organization. ResearchPolicy, 17, 251-267.

Leonard-Barton. D. (1995) . Wellspr ings of knowledge: Building and susta ining the sources of innovation. Boston.MA : Harvard Business School Press .

Leonard-Barton. D.• & Sinha, D. (1993) . Developer-user interaction and user satisfaction in internal technologytransfer. Academy of Management Journal . 36(5), 1125-1139.

Masri. A., & Moore II. J . E. (1993) . Integrated planning information systems: Context, design requirements. andprospects. Computers. Environment. and Urban Systems. 17(6),491-511.

Masser , I., & Onsrud, H. J. (Eds .). (1993) . Diffusion and use of geographic information technologies. Dordrecht,The Netherlands: Kluwer.

McKinney, D. c., Maidment, D. R., & Tanriverdi, M. (1993). Expert geographic information system for Tex aswater planning. Journal of Water Resources. Planning, and Management. ASCE. 119(2), 170-183.

Meyer, A. D., & Goes. J. B. (1988) . Organizational assimilation of innovations: A multilevel contextual analysis.Academy of Management Journal. J1(4),897-923.

Mohr, L. B. (1982) . Exploring organizational behavior: The limits and possibilities of theory and research. SanFrancisco: Jessey-Bass.

Nish ida, S., Palmer, R., Slaughtcrbcck, c., & Walker. S. (1990). Expert system for control curve evaluation duringdrought. In Optimi:ing the resources f or water management, Proceedings of the ASCE 17th Annual Nat ionalConference (pp. 294-297), New York . NY : ASCE. Fort Worth. TX .

Ortolano . L.. & Perman. C. D. (1990) . Appl ications to urban planning: An overview. In T . J . Kim, L. L. Wiggins.& J. R. Wright (Eds .), Expert systems: Applications to urban planning (pp , 3-14). New York: Springer-Verlag.

Palmer. R. N., & Tull, R. M. (1987) . Expert system for drought management planning. Journal of Computing andCivil Engineering, ASCE. 1(4), 284 -297.

Rogers, E. (1995) . Diffusions of innovations (4th cd .). New York : The Free Press .Sirnonovic, S. (1992). Reservoir systems ana lysis: Closing gap between theory and practice. Journal of Water

Resourc es. Planning, and Management. ASCE. 1/8(3), 262-280.Shepherd , A., & Ortolano, L. (1994) . Critiquing expert systems for planning and management. Computers.

Environment, and Urban Systems, /8(5), 305-314.Sheph erd . A.. & Ortolano. L. (1996) . Water supply system operations: A cr itiquing expert system approach .

Journal of Waler Resources, Planning, and Management ASCH. 111(S), 3411-JS5.Shepherd. A . (in press) . Knowledge-based expert systems: Crit iqu ing versus conventional approaches. Expcr!

Sy stems with Applications.Tornatzky, L. G ., Eveland, J . D., Boylan, M. G .. Hctzner, W. A., Johnson, E. D., Reitman. D.• & Schne ider , J .

(1911 3). Innovation processes and their management: A conceptual, empirical and po/icy review of innovationprocess research. Washington. DC : National Science Foundation.

Waern, Y., Hagglund, S., Lowgren, J., Rankin, I., Sokolnicki, T ., & Steinemann, A. (1992). Water supply systemoperations: A critiquing expert system approach. Communication Knowledge for Knowledge Communication.37,215-239.

Waterman, D. A. (1986). A guide to expert systems. Reading, MA : Addison-Wesley.Wiggins. L. L., Kim, T. J ., & Jain, R. (1993). Evaluating an expert system in thc field: Experience with the CORA

expert system. In J . R. Wright, L. L. Wiggins, R. K. Jain, & T . J. Kim (Ed s.), Expert systems in environmentalplannin g (pp . 299-311). New York : Springer-Verlag.

Wr ight , J. R., Wiggins, L. L., Jain, R. K., & Kim, T . J. (Ed s.). (1993). Exp ert sys tems in environmental planning.New York: Springer-Verlag .

Yin, R. K. ( 1994). Case study research: Design and methods. Thousand Oaks. CA : Sage Publications.