Contrasting successful and unsuccessful expert systems

122 European Journal of Operational Research 61 (1992) 122-134

North-Holland

Contrasting successful and unsuccessful Expert Systems

Peter Duchessi

School of Business, State Unversity of New York at Albany, Albany, NY 12222, USA

Robert M. O'Keefe

Department of Decision Sciences and Engineering Systems, Rensselaer Polytechnic Institute, Troy, NY 12180, USA Received March 1991; revised September 1991

Abstract: Many Expert Systems are unsuccessful in that they are never fully implemented, or when implemented quickly fall into disuse, or are not used to the extent originally envisioned. Alternatively, some systems have become very successful, supposedly saving or generating considerable amounts of money for the organizations that developed them. This paper reports on research aimed at investigating why some Expert Systems are more successful than others. It considers four cases which describe Expert Systems developed in (or under the auspices of) two leading technological companies; two are very successful, two less so. The systems deal with problems in manufacturing scheduling, computer customer support, engineering design and commercial lending. The authors employ a factor-oriented analysis where discussions with developers were used to ascertain the importance of user motivation, software and hardware platform, project planning methods, and other factors generally considered as relevant to success. The cases reveal that six key factors differentiate the successful from the less successful applications, namely demonstrable benefits, adequate development resources, commitment to implement, rapid turnaround of users' requests, understanding and control of data, and growth in the underlying application.

Keywords: Expert systems; implementation; technology management

I. Introduction

Expert Systems (ES) are computer software that contain a model of an expert's knowledge and reasoning strategies. During execution, they mimic the expert's judgement and capabilities. Successful applications result in numerous benefits, including reduced decision-making time, improved service levels and better use of expert time (Leonard-Barton and Sviokla, 1988; Meyer and Curley, 1991), and discussions of successful systems have reached book status (Andrews, 1989; Feigenbaum et al., 1988). Companies that want to attain the benefits spend heavily on projects and the commercially available languages, shells and tools that facilitate ES development (Coats, 1988).

Not every ES project, however, meets with success. There have been many failures or near failures in implementing the technology with con- sequent disappointments, financial losses and organizational disruptions. Despite this, there are few reports about less successful projects, giving the impression that most every attempt meets with success. In this paper, we attempt to identify factors associated with successful implementations, or conversely less successful implementations, so as to better understand how to deploy the technology. We contrast four ES projects, two successful and two less successful efforts. The applications range in size, scope, purpose and sophistication, so some of our conclusions can be generalized.

0377-2217/92/$05.00 © 1992 - Elsevier Science Publishers B.V. All rights reserved

P. Duchessi, R.M. O'Keefe / Contrasting successful and unsuccessful Expert Systems 123

2. Research approac h

To date, considerable research has been undertaken that attempts to look at the successful adoption, development and implementation of new technology. Examples include research on Management Information Systems (MIS) (Ginz- berg, 1981), Management Science and Operations Research ( M S / O R ) (Schultz and Ginzberg, 1984), MRP systems (Duchessi et al., 1989) and C A D / C A M (Currie, 1989). Whereas numerous theories and research methods have been used, the vast majority of the research can be divided into two camps: process-oriented and factor-oriented.

Process-oriented research typically follows the process of development, adoption and implementation of technology with the goal of understanding how the process can be better managed. Typi- cal studies include Currie (1989) and Leonard- Barton (1987). Alternatively, factor-oriented research attempts to identify critical success factors that can be viewed across multiple applications and organizations. An example of an extreme factor-oriented study is Duchessi et al. (1988). Although these basic approaches are not com- pletely exclusive (process research can uncover common factors, for example, and the process and management of implementation can them- selves be factors), our work is essentially factor- oriented. Based upon an analysis of four cases, we uncover critical factors that appear to be keys to successful ES implementation.

2.1. Critical implementation factors

Using the opinions of the managers, developers and project leaders that we interviewed, and our own experience, we developed a list of possible critical implementation factors and organized them into eight classes. Table 1 lists several factors for each class. Within the factor-oriented literature on M S / O R and MIS implementation (for example, Schultz and Ginzberg, 1984), four factors consistently emerge (Kwon and Zmud, 1987): management commitment, quality design, user involvement and user motivation. Our factor classes cover these, and also include other factors typically prevalent in the ES literature, for example the availability of expertise and the nature of the problem (Waterman, 1986). Our research is

Table 1 Some potentially critical implementation factors

Organizational caliditity: Organizational culture

Pioneering climate Participative management style No mistrust of new technologies Technology-oriented climate Intraorganization communication

Education Management education User education and hands-on training Experienced developers

Benefits Quantifiable costs and benefits Tangible payoffs

Managerial commitment and organizational support Financial support Identified sponsor or 'champion' User motivation Support from lower-level gatekeepers

Technical validity: Experts

Willingness to cooperate Experienced instructors Good user perspective

Project team and project planning Technically skilled developers Experienced developers User-oriented project team Formal project proposal Detailed requirements analysis Formal project planning methods

Problem structure Available data Existence of expertise Manageable domain Limited uncertainty

Technical management Adequate hardware/software Appropriate representation scheme Prototyping User participation Test cases available

not deductive (in that we developed a model from these factors and then tried to test the model), but inductive in that we try to generalize findings from the cases. Using factors was simply a conve- nient way to organize our analyses and discuss our findings.

We relate our class factors to both technical and organizational validity. Technical validity refers to timely completion of the project on schedule and within budget constraints, and the ES's technical capability to satisfy performance

124 P. Duchessi, R.M. O'Keefe / Contrasting successful and unsuccessful Expert Systems

requirements. Organizational validity refers to the degree of fit between the ES and the organization's duties, responsibilities, operations, power and authority relations, and cultural realities. It is worth noting that the ES literature has consis-

Reasoning methods: tently focused on technical validity, with occas- Goal-driven x x sional reference to education requirements. The Data-driven x x implementation process is frequently viewed as software: technology transfer (for example, Prerau, 1990). Language x However, as discussed elsewhere (Cox et al., Shell

Tool x x 1981), successful implementation of information Hardware." systems is dependent upon both technical and Microcomputer x organizational validity. Mini/mainframe x

System integration: Data base x Other software x 2.2. Collecting the cases

Table 2 CARE, CLASS, LMS and CXPRO general features

General features

CARE CLASS LMS CXPRO

X X

X X

For each of the four ES cases, we conducted unstructured, face-to-face interviews with the developers and managers of the ES projects. These individuals are at various levels of their respective organizations. The four projects were undertaken in (or under the auspices of) two companies that are leaders in the development and application of the technology: General Electric (GE) and IBM.

We conducted the interviews in late 1990, and used follow-up phone interviews as necessary to clarify details. We asked subjects to consider what factors affected implementation and to discuss factors that we 'brought to the table'. The results of our effort were detailed descriptions and histo- ries for each implementation, along with varied anecdotal information. We now present synopses of the ES prior to reviewing them in the light of our factors.

3. The systems

The four systems presented here are as follows:

CARE (Computer-Aided Requisition Engi- neering), which develops electric motor designs and models to satisfy customer orders,

CLASS (Commercial Loan Analysis Support System), which supports the evaluation of commercial loan applications,

LMS (Logistics Management System), which schedules shop floor activities in a semi-conductor manufacturing environment, and

CXPRO (Control Executive Prototype), a sys-

tem that diagnoses operating system problems in a series of IBM computers.

They vary considerably in terms of size and domain. Further, whereas LMS and CLASS have both been written about in some detail (respectively, Sullivan and Fordyce, 1990; Duchessi et al., 1988), to our knowledge this is the first pub- lished descriptions of CARE and CXPRO.

Table 2 compares the systems according to reasoning methods, software and hardware, and integration with conventional systems software. All of the systems use rule-based programming and goal-driven reasoning; three also use data- driven reasoning. Whereas LMS is written in APL2, PASCAL and Assembler, the developers of the other systems used ES shells or tools, such as GEN-X and ESE (Expert System Environ- ment). CXPRO developers incorporated external routines written in PASCAL to build data base search strings, and CARE developers used C to build an intelligent interface. CLASS requires a microcomputer, while the other applications run on mini or mainframe computers. Finally, most systems incorporate conventional systems software, including data base and spreadsheet software: CARE, LMS and CXPRO interface with data base packages, while CLASS interfaces with LOTUS 1-2-3.

3.1. C A R E

CARE allows salesmen to search a data base for a design or model that meets customer speci-


fications, and automatically performs the requisition engineering activities of finding existing models or designing new ones from scratch. (The combination of the rotor and strator configura- tion constitute an electrical design, and the design plus all remaining components represent a model.) CARE consists of several modules under the control of an ES referred to as the Executive, including an intelligent interface that checks user input for consistency and completeness and dis- plays the results of ES activity. If the user-di- rected search fails to return a design or model, the Executive activates Expert Search, an ES that builds queries and uses product structure knowledge to evaluate the possibility of using existing models that may exceed the requirements, but are cost effective. If Expert Search fails, the Executive executes Expert Design, an ES that modifies one or more standard designs to produce a new design that satisfies the desired electrical characteristics. CARE combines the design with the overall mechanical requirements to de- fine a new model and adds the design and model to its data base. Through a bill of materials generator, CARE also supplies information for manufacturing operations.

In 1987, following a review of technology at GE Corporate Research and Development (CR& D), the Vice President of GE Motors saw a feasibility demonstration of a system that could automate the requisition function. Recognizing the system's potential to reduce cycle time and improve customer service he provided financial support for one year. The problem was well suited to an ES approach: there were a large number of existing rules and heuristics, there were identified experts and there was no negative cost associated with a failure to find a solution.

C R & D formed a development team that consisted of the C R & D developer, a GE Motors Information Systems Management Program technical specialist, the Manager of Requisition Engi- neering and an expert motor designer. The team wrote a project plan that contained milestones and tasks. Notably the team excluded salesmen, the system's targeted users, because the development team felt that salesmen could not specify requirements for the system, and felt it knew how salesmen performed their tasks. Technical specialists rotated on the project team. The developer trained GE Motors' personnel who were

responsible for supporting the system after implementation, including the technical specialists.

During the first six months of the project the development team designed CARE's basic archi- tecture, data base and interface. In the next six months technical specialists interviewed the expert to acquire substitution and design rules that were subsequently added to ES modules. Sales- men attended design overviews and demonstrations, providing valuable input for the system's design and interface, and prototyping was used to discover time, cost and technology requirements. The developer and technical specialists modified the basic design as apparent weaknesses emerged. A formal requirements analysis document was prepared more than half way through the project solely for management information.

The data base component was delivered to users first. This component gave salesmen an- swers to pertinent questions, including in stock motors, which helped them quickly respond to customers' queries. It was highly successful: salesmen reduced customer response time, customers and G E Motors' managers could see the system, and sales successes were reported to management. The development team subsequently ap- pended CARE's ES components, which were transparent to the salesmen, and these components significantly improved CARE's functionality. During implementation, salesmen could recommend enhancements and modifications, via software change requests, which were implemented by the development team.

CARE handles routine designs and models which constitute 80% of all orders; the remaining orders require human engineering. Currently, there are 350 salesmen using CARE. CARE made significant contributions to the project's three objectives: reduce base costs, reduce engineering errors and reduce cycle time. Additionally, it gave salesmen and managers a new perspective that technology could improve business performance.

3.2. C L A S S

CLASS provides commercial loan officers in lending banks a system to support the evaluation of a lendee's financial position, recommend loan covenants and document the commercial loan analysis. Using three to five years of data from a company's financial statements and industry data


sources, CLASS performs general trend, credit, collateral, capital and capacity analyses. When it encounters a weakness, CLASS spawns additional analyses to identify the source of the weakness and generate loan covenants, and combines the results of all analyses to arrive at an overall evaluation of the company's financial situation. CLASS provides substantial opportunity to view financial trends, respond to queries, execute options and obtain reports, and supports extensive sensitivity analysis that can incorporate pro forma financial statements.

In 1987, as part of a strategy to move its computer systems mix away from engineering applications to financial/business applications, GE CR&D decided to fund the CLASS project. The project director (one of the authors) formed a development team consisting of himself, another developer and an expert who had considerable experience in evaluating commercial loans and educating commercial loan officers. There was no project plan, however, the development team had a goal of providing a system that simulated the expertise of the expert. Over a six-month period the developers interviewed the expert, had the expert solve numerous problems, recorded his rules and heuristics and added them to the system. Using a prototyping approach the developers and expert evaluated each prototype, including the system's interface, design, reasoning approach and analysis capabilities. To validate CLASS, the development team used unit testing on general trends, credit and other analyses, and complete system testing with actual commercial loan cases. Validation continued until the expert's and CLASS's evaluations converged on all cases examined.

In 1988 CLASS was implemented in the commercial lending office of a large New York bank. The system was demonstrated to two senior vice presidents; one had the commercial lending of- rice reporting directly to him. Both officers were impressed with CLASS's capabilities and agreed to Beta test the system. The development team was expanded with the addition of seven commercial loan officers and five financial analysts from the bank. The financial analysts were adept at using PCs, whil e the commercial loan Officers were not. The development team educated its new members in ES and CLASS, including capabilities and limitations.

The financial analysts were responsible for en- tering financial data into spreadsheets which were then used by the commercial loan officers to aid lending decisions. Because the spreadsheet in use at the bank differed from the one used by CLASS, the developers created a flexible data entry interface to handle all cases, which took two months to complete. The financial analysts' department was understaffed and its priorities were constantly changed by the commercial loan officers and management. Over time, the department's priorities relegated CLASS's Beta test to the bottom of the priority list. The financial analysts on the development team were not preparing cases for CLASS and commercial loan officers returned to using traditional analysis methods.

All attempts to revive CLASS, including direct appeals to bank management, failed. Although CLASS provided support in a number of actual commercial loan evaluations, it failed to reveal any Surprises. Additionally, the developers could not demonstrate productivity gains within the commercial lending office. There are no plans to reimplement CLASS in the bank.

3.3. LMS

LMS is a shop floor scheduling and management system implemented at an IBM plant in Burlington, Vermont. The Burlington plant pro- duces various semi-conductor products for inclu- sion in a large range of IBM products. It provides access to shop floor information, monitors shop floor transaction streams, issues alarms when anomalies and problems occur (called 'alerts') and recommends responses to alerts. The base LMS component, called GATEWAY, records the shop floor transaction stream, including lot loca- tion and machine status. The ES modules, collec- tively known as ALERT, monitor the transaction stream, generate alerts and route mail messages about the alerts. The Dispatch Decision Maker (DDM) component of LMS contains ES modules that automatically respond to shop floor anomalies and make dispatch decisions.

In 1985, the manager of the Industrial Engi- neering Group (lEG) at the Burlington plant proposed LMS as a tool to improve line flow, utilization, and reduce cycle time. Although manufacturing and the information systems group had doubts about lEG's ability to make contributions,


lEG's manager had considerable modeling experience and a good understanding of plant operations. In 1986, Burlington plant management and IBM's Advanced Engineering and Manufacturing group (which provides financial support to worthy project across all of IBM) commissioned lEG to build LMS and provided the requisite funds.

The development team consisted of lEG's manager, several people from lEG and manufacturing, two industrial engineers and an internal software development consultant. The plant is split into various buildings with each building under the management of a superintendent: a building was chosen for initial implementation and the building superintendent participated in the development and implementation of LMS.

The development team implemented GATE- WAY first, which provided manufacturing with a macro perspective of the line and allowed them to specify their own so-called 'views' (essentially tailored reports). Requests for new views were satisfied very quickly - in some cases overnight. GATEWAY provided valuable functionality and tangible results that management could see. Based upon an enhanced understanding of the manufacturing process, gained during the development of GATEWAY, the contents of shop floor paper and experts from task forces and research teams, the development team wrote ALERT modules to signal alarms. The growth of the development team's collective shop floor knowledge coincided with the building superintendents' inability to meet manufacturing performance requirements stipulated by management, and thus in response to these needs the development team built and validated DDM using a trial-and-error approach. Over time they executed DDM, implemented its dispatch recommendations, examined the results and modified the rules until the system demonstrated positive results. LMS was subsequently implemented in other buildings.

Currently, various LMS configurations are running in several IBM facilities. The system has demonstrated that it can make positive contributions to improving manufacturing performance.

3.4. CXPRO

CXPRO diagnoses problems with the Dis- tributive Processing Control Executive (DPCX) operating system for IBM's 8100 series comput-

ers. It helps technical specialists identify the causes of problems such as 'hard wait', 'slow wait', 'program abnormal', incorrect output and other problems. Technical specialists maintain a data base of historical and active problem records, and when a customer problem emerges they search the data base to identify a set of records that may explain the problem. Based upon a technical specialist's description of the problems, CXPRO builds a symptom string that it uses to search the data base. If the search returns promising matches, CXPRO queries the technical specialist to gain additional information which it uses to make recommendations. The problem records and recommendations constitute CX- PRO's output.

In 1985 the manager of the DPCX support group decided to fund the development of CX- PRO as a means to primarily build ES development expertise in the group and to improve customer service. DPCX represented a stable product, the support group was shrinking (initially from 50 to a target of 20 members) as the customer base stopped growing, customers reported fewer problems and the current staff had sufficient expertise to develop the system.

The development team consisted of technical specialists from the support group: one full-time technical specialist (who became the principal developer and was an expert in several problem areas) and several part-time technical specialists who were experts in nonoverlapping areas. In areas where the development team lacked expertise, it appealed to other experts in the support group and performed knowledge acquisition activities. There was no formal project justification, requirements analysis or project plan. The development team presented the CXPRO project as part of a much larger effort to build internal skills and produce software products to support customer service.

Iterative prototypes were built until the system achieved certain defined performance objectives. Prior to releasing CXPRO, the development team provided the entire support group with generic ES education, distributed user manuals, and sat with support group members to help them learn about CXPRO and how to use it. During implementation, technical specialists could recommend changes and enhancements via feedback options that were constantly displayed and that activated


an electronic mail utility for sending messages to the development team. Shortly after deployment, the development team found that only the technical specialists who made contributions to CX- PRO were using the system. The absense of electronic mail messages, informal contact and electronic logging of CXPRO usage revealed that the remaining technical specialists were either not using the system or using it very lightly. The development team attributed low usage to a general perception among technical specialists that the system had low utility and a reluctance to change current procedures and methods of problem solving.

Appeals to management to help revive CX- PRO failed. Over time, initial CXPRO contribu- tors left the support group, and new management had no interest in CXPRO. There are no plans to reimplement CXPRO.

4. Analyses of critical implementation factors

CARE and LMS are resounding successes that are highly regarded, both within and outside, respectively, GE and IBM. Each system is implemented, regularly used and maintained. CLASS and CXPRO are inactive - tangible and nontan- gible development costs are considered sunk and the implementing organizations received few discernable benefits. Here we apply our critical implementation factors to all four systems.

4.1. Organizational validity

4.1.1. Organizational culture Organizations that have a more pioneering

(rather than traditional) climate, have a more participative (rather than authoritarian) management style, are flatter (i.e., less hierarchical) and have no mistrust of new technologies are more likely to consider implementations of new technology (Leavitt, 1965; Woodward, 1965; Greiner and Barnes, 1976). Both development organizations here (GE and IBM) are pioneering technological companies, and thus both climates encour- aged use of new technologies. It is possible that implementation of CLASS in the bank encoun- tered cultural problems since commercial loan

officers are not accustomed to using computers, and it was the first deployment of an ES in that organization.

4.1.2. Education Education provides managers and users with

an understanding of the basic technology and of its capabilities. It fosters management commitment and organizational support by developing favorable attitudes and keeping expectations reasonable. The LMS and CXPRO development teams did not provide generic education and demonstrations to top and middle managers, primarily because management was aware of the technology and prototypes had not been developed prior to project approval. The CARE team provided a feasibility demonstration, and CLASS was demonstrated to bank management with an actual case. Their efforts increased management's awareness of the technology and its potential benefits. All development teams provided users with hands-on training and trained developers who were unfamiliar with development software. Thus, education was adequate for all four projects.

4.1.3. Benefits The anticipated benefits, including improved

decision making, improved customer service and increased productivity, generate management interest and commitment. After implementation, demonstrable positive benefits promote organizational support and operational use. In all four projects, managers and developers recognized that the applications could result in positive business benefits, such as reduced cycle time (LMS) and improved customer service (CXPRO). The anticipated benefits and an enhanced understanding of the technology motivated management to consider a project. After implementation, although all systems could demonstrate expert performance in their domains, only those systems that made immediate, tangible benefits to the business (i.e., CARE and LMS) were embraced by management and the organization.

4.1.4. Management commitment and organizational support

Most ES are developed under the patronage of a 'champion': the person who wants the system


developed and operationalized. Typically the champion provides the financial resources directly or takes the lead in attaining them. Having a top manager as a champion gives the project high visibility, stimulates organizational support and fosters operational use.

CARE and LMS had the direct support of top managers who provided the requisite financial support. The managers became champions and gave the projects high visibility. (Neither project, however, experienced significant user rejection, which would have tested the degree of management commitment.) CLASS also had the support of top managers, but other business activities were more important than the implementation. Bank management committed no direct funds to the project, and may not have been so concerned with its ultimate success. CXPRO had the support of middle management, but when the development team noticed that only the users involved in CXPRO's development were using it, management was unable to affect wider operational use. This suggests that implementation sponsors were too low in the organization to give the project high visibility. The lack of immediate, demonstrable benefits in the CLASS and CXPRO projects failed to generate management commitment and stimulate operational use.

A network of supporters and champions reflect organizational support (Leonard-Barton, 1987), and without a supportive network, technically successful systems may fall by the wayside. As a system is introduced into an organization, lower-level 'gatekeepers' may emerge to facilitate or impede organizational acceptance. In the CARE and LMS projects, the network of supporters and champions grew: management support and the benefits derived from the success of early data base applications contributed to network growth, and thus there was considerable organizational support in both projects. Only the development team supported CXPRO, and the project never achieved a critical mass of supporters and champions. During CLASS's implementation, organizational support and operational use declined because data entry bottlenecks at the financial analyst level prevented lending officers from using the system on a regular basis. Ques- tionable benefits, soft management support and low implementation priority partially explain the minimal attention given to data entry.

4.2. Technical validity

4. 2.1. Experts Experts must be cooperative, willing to partici-

pate fully as members of the project team (Irgon et al., 1990), and those that are instructors have their knowledge organized, enabling efficient knowledge transfer. When experts come from the user community, they can provide insight into user issues and become advocates for the system. These factors enhance a development team's capabilities and stimulate operational use. The CARE and CLASS projects had experts who understood the data requirements and expectations of their user communities. The expertise and user perspectives for LMS came from multiple sources and emerged during development and implementation. The development team's experts created CXPRO; they were also from the user community. The source and nature of the systems expertise appears to have had no discernable impact on the project outcome in all four cases.

4.2.2. Project team and project planning The project or development team consists of

experts, developers and often (but not always) users, and it is generally recognized that the development team should meet regularly and have a full-time project leader. The LMS and CXPRO projects had experts, developers and users on the development team, while the CLASS project only included users during implementation as part of the implementation team. The CARE project did not have users as a formal art of the team, but use of prototyping afforded users considerable opportunities to influence development. Lack of user involvement from an early stage is certainly a reason for the failure of CLASS, and the involvement of users (albeit managed differently in the two projects) contributed to the success of LMS and CARE.

Developers that are knowledgeable about ES languages, shells and development tools, as well as knowledge acquisition methods, provide a sound technical basis for a development team. They are responsible for choosing the most appropriate hardware and software for the problem. Before starting the ES development effort, they require training in the application domain to learn pertinent vocabulary and concepts that help them interface with experts (Irgon et al., 1990). All the


projects considered here had capable development teams: the developers possessed knowledge of ES languages and tools, and had previous development experience with conventional applications, modeling applications, Decision Support Systems and/or ES. Lack of technical skills did not appear to be a problem in any of the four projects.

With regard to project planning, a proposal that contains detailed cost-benefit data is more likely to get the support of management than one based upon qualitative information (Currie, 1989). A formal requirements specification that is developed either with or without a 'test' system provides valuable information about the ES's desired problem-solving behavior and performance objectives (Irgon et al., 1990). Additionally, to successfully manage the implementation, developers need a project plan that contains milestones and tasks, and provides realistic time for ES development (Leonard-Barton, 1987).

Interestingly, only the CARE project used formal project planning, but not at the level of detail typically associated with engineering projects. A detailed requirements analysis, intermediate reporting and strict budgeting are notably absent from all four projects. Despite this, the projects exhibited some short-term planning for subsequent stages, typically three- to six-month hori- zons. Thus, the formal project planning had a negligible impact on project success.

4.2.3. Problem structure The nature of the problem largely determines

the software development method, the hardware delivery platform and the validation method. If the problem is ill-structured and its boundaries are unclear, the project will take longer, consume more resources, and require more sophisticated hardware/software and development and validation methods. Poor ES problems are ones where the cost of a bad decision is high, solutions are based upon unpredictable or poorly understood factors and the task requires common sense (Coats, 1988).

From a technical standpoint, each of the four problems were suitable for available development technologies, and all the systems achieved expert performance. The CLASS project indicated that commercial lending may be a poor domain: recommendations could not be immediately verified

for correctness because loan default occurs well after the decision is made (in some cases, several years after a loan is granted), and actions taken (or not taken) have a direct impact on a company's performance. Additionally, the risks a loan offi- cer faces - loss of business and loan default - are significantly high. Similarly, CXPRO's problem prevented immediate measurement, with im- provements in customer service only gradually occurring over time.

These were not issues in the LMS and CARE projects. LMS users can implement the system's recommendations and view the impact on the line; a new building superintendent who insisted on LMS being 'unplugged' in his building saw line performance deteriorate rapidly. CARE also provides immediate feedback: it returns a design or model in real time. Moreover, the risk associated with not finding or designing a design is negligible: the user simply has to resort to previous manual methods.

4.2.4. Technical management Technical management encompasses hardware

and software, development methodology, and validation issues. Hardware and software constitute the delivery system, which in turn determines the system's user interface, performance, response time, ability to interface with other hardware and software and knowledge explanation facilities. In- adequacies in any one of these areas can stifle operational use and prevent the system from attaining its technical objectives (Leonard-Barton, 1987). A development environment that supports multiple problem-solving paradigms and knowledge representation schemes can facilitate rapid prototyping and the development of a robust ES that satisfies the technical requirements of the project (Irgon et al., 1990).

All the projects considered here successfully overcame some technical hurdles. If anything, LMS and CARE represented greater technical challenges than CLASS and CXPRO: LMS is a large real-time system using various data sources, and CARE combines ES and data base methods with an intelligent interface. Both required a mixture of programming languages. CLASS was firmly based upon GENX, but input facilities had to be developed outside the shell. Although the developers report that ESE was not always easy to use, writing external routines allowed the job


to be done. Regarding hardware, no problems are noticeable across any of the four projects (this is probably as would be expected in technology- driven companies, especially IBM).

Generally, developers use prototyping and iterative development methods to build ES because the problem's boundaries and the system's capabilities are unknown at the start of the project (Leonard-Barton, 1987). Prototyping also fosters direct user participation during development by allowing users to view, evaluate and modify the system. Users become codevelopers and gain a sense of ownership and hence timing is considered vital: developers need to have quick turnaround to keep user expectations from dete- riorating and to avoid losing enthusiasm. Any approach that involves users promotes operational use.

All projects used iterative evolutionary development, and in all some prototyping was done. With CARE and LMS, prototyping played a large role in defining the systems functionality and user requirements, with developers allowing users to recommend changes and enhancements to the

interface and reporting features. Thus, users became codevelopers of the systems and part own- ers. Moreover, LMS developers were able to quickly respond to user requests; sometimes in less than 24 hours. The short turn-around time on requests and user codevelopment partially contributed to successful use of the system during implementation.

In the CXPRO project, although users could provide feedback during implementation via menu options, low user involvement on the development team and during software development may have contributed to poor operational use. CLASS had no one from the commercial lending office participate in software development because the system was created outside of the organization. During implementation, although users were part of the implementation team and recommended system modifications that were implemented gradually, a sense of ownership never material- ized. The implementation became more of a technology 'hand-ofF than a joint implementation project.

System validation requires the developers to

Table 3 Contrasting success and failure factors for all four projects

Successes Failures

LMS CARE CXPRO CLASS

Management commitment and organizational support." Adequate develop- Upfront provision Regular review Limited initial ment resources of funds and and injection commitment

people of funds Commitment to Driven by observed Driven by manage- Died as project

implement benefits ment and user progressed support

Benefits: Demonstrable Reduced cycle time,

benefits greater production Reduced base costs,

engineering errors, and cycle time

Growth in underlying application

Technical management: Rapid turn-around of users' requests

Very strong

Rapid response to suggested changes

Strong

Rapid response to suggested changes

Understanding and control of data

Undertaken before development of ES

Undertaken before development of ES

No measurable improvement in customer service

Discontinued

Interest did not spread beyond users who aided development

Sold as an ES

Limited initial commitment

Priority reduced

No foreseen improvement in loan defaults, etc.

Slow

Most of system built before system implemented

Sold as an ES, subsequent data entry problems


test the system against the domain expert, other experts a n d / o r test cases to determine correctness and credibility of solutions (O'Keefe and O'l_~ary, 1992). Additionally, it involves testing the system against performance and user-speci- fied requirements, including the interface and knowledge explanation facilities. All systems underwent some formal validation. CLASS received more attention in this aspect than the other systems (relative to total development effort), probably due to the nature of the problem. With LMS and CARE, validation was subsumed into the extensive evolutionary development, such that when the system performed badly this was quickly noticed and changes were made.

4.3. Discussion

It is apparent that there is no one factor that distinguishes successful from less successful ES implementation efforts. In the cases considered here, six factors emerge which are related to success. These are summarized in Table 3.

The four cases confirm the importance of two management commitment and organizational support factors. As the LMS and CARE projects demonstrated, top management commitment requires a willingness to provide adequate development resources (e.g., people, time and funding). As evidenced by the CLASS implementation, it also requires that top management give the implementation a high priority: making it at least as important as other business activities, especially when implementation problems begin to emerge. This can be termed commitment to implement. High project priority prevents obstacles which can derail the project, such as the CLASS data entry bottleneck, from emerging. These factors reflect management commitment and generate organizational support.

With regard to benefits, two factors are evi- dent in the LMS and CARE projects but are noticeably absent in the CLASS and CXPRO projects. First, an ES must have demonstrable benefits. These may not necessarily be financial, or even quantifiable, but they have to be immediately obvious. The benefits of CARE and LMS were immediately recognized; those of CLASS and CXPRO were not. With CARE and LMS, this strengthened management commitment and organizational support. Moreover, problem struc-

ture affects immediacy of benefits. Developers should pursue implementations in domains that lead to immediate benefits, or provide education that keeps expectations reasonable and promotes patience among managers and users.

Second, ES are successful where there is growth in the underlying application. The Burling- ton plant came under considerable pressure to produce just as LMS was coming on stream; similarly, GE Motors' business underwent considerable growth. This contrasts with the CXPRO experience where the intention was to wind down customer support for a discontinued computer line. With the CLASS project, commercial loan analysis work was increasing, but not at rates which forced management to evaluate or radically alter existing practices. Growth in the underlying application creates perceived need for the system among management and users.

With regard to technical management, all four projects included user participation through the use of an iterative evolutionary development method. However, as the system matured, both the CARE and LMS projects continued to provide rapid turn-around of users' requests for alter- ation and extension. The CXPRO project involved only a portion of the user community, with the result that only users who provided input during system development subsequently used the system. CLASS's developers made efforts to gather user feedback and modify the system ac- cordingly, but the slow turn-around stifled a sense of real participation.

Finally, both the LMS and CARE projects highlighted the importance of understanding and control of data. Each system was initially a data- oriented application, and ES modules were only added later to expand the functionality of the systems. To users, the ability to access and use data was new and useful - the ES facilities can be viewed as 'icing on the cake'. What is more, the ES capabilities were hidden from the users, such that users consider LMS and CARE as useful systems, not necessarily useful ES. This contrasts markedly with CLASS and CXPRO which both started out as ES projects and were sold as such. Lack of early attention to data characteristics and entry procedures turned out to be disastrous for the CLASS project. Although this factor has not appeared in previous MIS literature on implementation, organizing and ob-


taining necessary data has been suggested as a problem with ES (Peterson and Reitman, 1992), and the LMS development team stated that their initial focus on 'timely reliable data sources' was derived from Gene Woolsey's principles of M S / O R practice (Sullivan and Fordyce, 1990).

5. Conclusions

It is striking that implementation factors often discussed in the ES literature as contributing to success (particularly knowledge acquisition, knowledge representation and system validation) fail to differentiate the successful from less successful implementations here. Because all systems achieved expert performance, it appears that the development teams made correct technical decisions. Our study confirms the importance of management commitment, organizational support and user participation, and highlights the importance of demonstrable benefits, growth in the underlying application and understanding and control of data.

To be committed, management needs to provide adequate resources and make the implementation a high priority. Without sufficient management commitment organizational support may wane, as apparent in the CLASS and CXPRO projects. Through iterative evolutionary development and prototyping, developers create an environment that gives users a sense of real participation which increases the chance of success. As the system matures, rapid turn-around of users' requests must be continued, as evidenced in the LMS and CARE projects.

Applications that can produce immediate, visi- ble benefits secure management commitment and organizational support. However, it is easier for developers to produce application benefits in some domains than in others, as the cases demonstrate. Since ES tend to be highly linked to specific tasks, typically more so than other information systems, the importance of and growth in the need to perform the task are critical. Addi- tionally, understanding and control of data emerges as an important success determinant, especially in data-oriented applications. It allows developers to focus on critical issues related to the organization and availability of data, and to

provide benefits accruing from data management at an early stage of the implementation.

Acknowledgments

We thank the managers, project leaders and developers who shared their knowledge and expe- riences with us at IBM, GE C R & D , Eastman Kodak, Gensym Corporation, and Coopers and Lybrand.

References

Andrews, B. (1989), Successful Expert Systems, Financial Times Report, London, UK.

Coats, P.K. (1988), "Why expert systems fail", Financial Man- agement 17/3, 77-86.

Cox, J.F., Zmud, R.W., and Clark, S.J. (1981), "Auditing an MRP system", Academy of Management Journal 24/2, 386-402.

Currie, W.L. (1989), "The art of justifying new technology to top management", OMEGA 17/5, 409-418.

Duchessi, P., Schaninger, C.M., Hobbs, D.R., and Pentak, L. (1988), "Determinants of success in implementing material requirements planning (MRP)", Journal of Manufacturing and Operations Management 1/3, 263-304.

Duchessi, P., Schaninger, C.M., and Hobbs, D.R. (1989), "Implementing a manufacturing planning and control information system", California Management Review 31/3, 75-90.

Duchessi, P., Shawky, H., and Seagle, J.P. (1988), "A knowledge-engineered system for commercial loan decisions", Financial Management 17/3, 57-65.

Feigenbaum, E., McCorduck, P., and Nii, P. (1988), The Rise of the Expert Company, Times Books, New York.

Ginzberg, M.J. (1981), "Key recurrent issues in the MIS implementation process," MIS Quarterly 5/2, 47-59.

Greiner, L.E., and Barnes, L.B. (1976), "Orgnaizational change and development," in: P.R. Lawrence, L.B. Barnes and J.W. Lorch (eds.), Organizational Behavior and Admin- istration, Irwin, Homewood, IL, 625-636.

Irgon, A., Zolnowski, J., Murray, K.J. and Gersho, M. (1990), "Expert system development: A retrospective view of five systems", IEEE Expert 5/3, 25-39.

Leavitt, H.J. (1965), "Applied organizational changes in industry: Structural, technological, and heuristic approaches", in: J.G. March, (ed.), Handbook of Organiza- tions, Rand McNally, New York, 1144-1170.

Leonard-Barton, D. (1987), "The case for integrative innova- tion: An expert system at Digital", Sloan Management Review 29/1, 7-18.

Leonard-Barton, D., and Sviokla, J. (1988), "Putting expert systems to work", Harvard Business Review 66/2, 91-98.

Kwon, T.H., and Zmud, RW. (1987), "Unifying the frag- mented models of information systems implementation", in: R.J. Boland and R.A. Hirschheim (eds.), Critical Issues


in Information Systems Research, John Wiley, New York, 135-156.

Meyer, M.H., and Curley, K.F. (1991), "Putting expert systems technology to work", Sloan Management Review 32/2, 21-31.

O'Keefe, R.M., and O'Leary, D.E. (1992), "Performing and managing expert system validation", in: M. Grabowski and W.A. Wallace, (eds.), Advances in Expert Systems and Artificial Intelligence or Management, JAI Press, Green- wich, CT, forthcoming.

Peterson, W., and Reitman, W. (1992), "Some issues in the development and implementation of large-scale commercial expert systems", in: M. Grabowski and W.A. Wallace,

(eds.), Advances in Expert Systems and Artificial Intelli- gence for Management, JAI Press, Greenwich, CT, forthcoming.

Prerau, D.S. (1990), Developing and Managing Expert Systems, Addison-Wesley, Reading, MA.

Schultz, R.L. and Ginzberg, M.J. (eds.) (1984), Management Science Implementation, JAI Press, Greenwich, CT.

Sullivan, G., and Fordyce, K. (1990), "IBM Burlington's Lo- gistics Management Systems", Interfaces, 20/1, 43-64.

Waterman, D.A. (1986), A Guide to Expert Systems, Addison-Wesley, Reading, MA.

Woodward, J. (1965), Industrial Organizations: Theory and Parts, Oxford University Press, London, UK.

Documents

Contrasting successful and unsuccessful expert systems