Assessing the impact of expert systems: The experiences of a small firm

Pergamon

Expert Systems With Applications, Vol. 7, No. 2, pp. 249-257, 1994 Copyright © 1994 Elsevier Science Ltd Printed in the USA. All rights r-~erved

0957-4174/94 $6.00 + .00

Assessing the Impact of Expert Systems: The Experiences of a Small Firm

RITU AGARWAL

Department of MIS and Decision Sciences, University of Dayton. Dayton, OH

SUSAN BROWN

Carlson School of Management, University of Minnesota, Minneapolis, MN

M O H A N TANNIRU

School of Management, Syracuse University, Syracuse, NY

A b s t r a c t - - T h e impacts of information processing technologies have been the subject of considerable debate for some time now. Several alternate views to assessing impacts have been proposed, predominantly in the context of management information and decision support systems. In the case of expert systems technology, the few systems described in the literature have primarily been deployed in large firms with significant resource outlays. This article details the significant impact of expert systems technology on a small f irm--a regional credit union. Employing an assessment framework that examines the effects of information technology at the level of both product and process, impact is assessed along two orthogonal dimensions of breadth (local and global effects) and depth (direct and induced effects). Furthermore, the framework suggests how a longitudinal evaluation that allows for a con- sideration of both system impact and potential obsolescence may be conducted. Although the framework is used here to assess the impact of a specific application, it has broader applicability to other types of systems as well.

1. INTRODUCTION

WITH INFORMATION TECHNOLOGY (IT) becoming a key determinant in a firm's ability to survive and com- pete, there has been an increasing emphasis in business on searching for new and improved technologies. However, although it is important for an organization to seek out new technologies that can help retain its competitive posture, it is just as important for it to continually reevaluate its technology portfolio in order to assess its suitability in a changing environment. This type of assessment is particularly critical for smaller organizations that typically cannot afford to waste their precious resources on either outdated or state-of-the- art technologies without careful planning.

Requests for reprints should be sent to Mohan Tanniru, School of Management, Syracuse University, Syracuse, NY 13244-2130.

The objective of this article is to study the impact of a particular IT---expert systems (ES)---on a small business. While there is recognition that ES technology can be a valuable strategic weapon for a firm, not much has been written about the impacts of such systems, with the exception of a few celebrated cases (Sviokla, 1989-90, 1990). The relative infancy of the technology and the lack of any established framework to guide such an assessment are plausible explanations for this lack of interest, and whatever impact that was reported is for systems that are predominantly deployed in large firms with significant resource outlays. Thus, a related objective of this article is to present and operationalize a conceptual framework for assessing impact that dis- tinguishes between product-related and process-related measures. The framework provides a parsimonious ye t

powerful categorization of impact that spans many levels of analysis. It may be used to perform a static examination of system impact as well as to conduct a

249

250 R. Agarwal et aL

longitudinal analysis of the IT application. Although the framework is used here to assess the impact of a specific expert systems application, it has broader relevance to other types of IT applications also.

The organization of this article is as follows. The next section reviews prior work in impact assessment and presents some preliminary concepts underlying the product/process framework. The following section de- scribes the small business investigated in this study-- a credit union (CU)--and details the construction and implementation of an expert system. The product versus process framework is then used to illustrate the impact of the system on the firm. The fifth section provides an approach to perform a longitudinal analysis of the system's impact on the CU operations. The last section provides some concluding comments on the assessment strategy used in this study.

2. ASSESSING IT IMPACT

The impact that a new information processing technology can have on a business has been the subject of debate for several years. Prior research has proposed several alternate metrics to assess this impact. In the early years of computing, March and Simon (1958) suggested that individual and organizational performance programs (significant changes, attributable to IT, in the way certain tasks are performed) are appropriate measures of IT impact. Galbraith (1977) advo- cated the use of the information processing capacity of a firm (which manifests itself in the number of inputs used, outputs generated, and changes in the observed level of task performance) as a measure. Since the early writers, several others have also presented procedures and metrics for impact assessment; these works are reviewed below.

In a recent article, McKeen and Smith (199 l) presented compelling arguments in favor of a more broad- based and holistic approach to assessing the value of IT. Drawing upon previous studies and critiques, they identified four primary shortcomings with existing ap- proaches to value assessment: the lack of universal measures that link organizational performance to IT investment, the level of analysis (individual projects versus the entire portfolio of IT applications), the analysis techniques utilized (industrial engineering ap- proaches that tend to be largely deterministic), and the lack of good causal models. Arguing for a resource an- alytic approach to value assessment, they further identify three intervening factors that could play a me- diating role in resource analysis: the level of computerization of a company, the revenue pattern of a company, and the type of industry involved.

Huber (1990) sets forth several propositions that hypothesize the effects of advanced information technology on organizational design, intelligence, and decision making. Specifically, he develops propositions

to monitor decisions at the subunit and organizational level, along with those that deal with organizational memory and performance. The propositions at the subunit level suggest that technology provides more information sources to make a decision, reduces the number/variety of individuals involved in the decision process, and reduces the overall time spent in decision- related meetings. The propositions at the organizational level state that technology changes the nature of decision making (partially decentralizes centralized decision making and centralizes decentralized decision making) depending on certain organizational characteristics, reduces the number of levels involved in the decision process, and reduces the number of human nodes involved in supplying the needed information. Two additional propositions address how IT can improve organizational memory on data and knowledge of its past activities. The last set of propositions relates to the ability of the technology to enhance organizational performance by allowing it to identify problems/ opportunities rapidly; facilitating the use of accurate, timely, and comprehensive information in decision making; improving the quality of decisions; reducing the time needed to make decisions; and reducing the time needed to implement organizational action following a decision. ~is article synthesizes many decision oriented criteria that have been the subject of much MIS and DSS evaluation research.

The impact of IT on "decision oriented criteria," as they influence the allocation of "organizational resources" seems to offer a greater challenge when applied tosmall firms. The basic tenets of McKeen and Smith's assessment framework (i.e., a broad-based, longitudinal approach where the appropriate level of analysis is the organization and the resource measures are in main- frame hours expended) appear not applicable in the case of small firms, where IT investment is character- ized by extremely low levels of computerization (often in the form of personal computers and workstations) and the critical resource is often the decision t ime of individual decision makers. Also, the relationship between decisions and individuals (who make these decisions) is overlapping and less formal in many small firms, thus making it difficult to measure the impact of IT on several of the decision variables identified by Huber.

Although researchers have addressed the subject of impact assessment, a review of the literature highlights a definite need for more robust and broadly applicable frameworks. The key to successful impact assessment is to recognize that IT impact manifests itself in several ways. Measures proposed by both these studies and those reported elsewhere can be conceptually classified into two categories: product measures and process measures. A holistic assessment of technology impact must examine the role of technology both on the product (outputs/decisions) as well as the process (inputs

Assessing the Impact of Expert Systems 251

used, transformation, and the performance level of this transformation). In addition, impact must be assessed both locally (where the technology is applied) and globally (where its impact may be felt). Furthermore, impact may be direct (anticipated) or induced (caused by an unexpected shift in resources).

As illustrated in the following section, this type of analysis allows one to systematically move from the unit, where the technology is initiated, outward (pos- sibly along the value chain) to establish the value of the IT on many different dimensions. Whether fortui- tous or planned, the impacts of IT often can extend far beyond the immediate application it was intended to automate. Previous use of this product/process framework to analyse the impact of DSS and ES was reported elsewhere (Agarwal & Tanniru, 1992).

3. CONSTRUCTION AND IMPLEMENTATION OF AN

ES FOR CREDIT UNION

The one industry whose fortunes rise and fall with the vagaries of the economy is the banking and credit union industry. Smaller firms that do not possess large asset bases are particularly vulnerable to environmental un- certainties. Haunted by the specter of recent debacles in the savings and loan industry and high mortality rates, credit unions are reassessing the way they conduct business and manage their assets and trying to repo- sition themselves strategically for the upcoming decade. Information technology can be one enabling factor in this repositioning.

3.1. The Credit Union and Its Business Processes

The Mutual of New York Federal Credit Union (FCU), located in Syracuse, New York, serves the employees of the 14th largest mutual insurance company along with a few other organizations in the United States. Employing seven full time and five part time employees, the FCU manages the interests of about 7,000 shareholders in the United States and abroad. With assets of over $12 million, a loan portfolio of $9 million, and net income in 1991 of $770,000, it can be classified as relatively small (Table 1). It does, however, recognize the fact that in order to survive in an uncertain economy it must adopt both a conservative and aggressive

TABLE 1 The Organization

The Mutual of New York Federal Credit Union

Employees 7 full time, 5 part time Shareholders 7,000 Assets $12,000,000 Loan portfolio $9,000,000 Net income (1991) $770,000

posture: conservative in its management of risk, and aggressive in order to grow and improve its profita- bility. These two sometimes conflicting goals are an integral component of the FCU's marketing strategy for the 90s.

The business processes of credit unions in general and the FCU in particular are relatively straightforward. They perform many of the activities a bank performs including loan services, checking and savings account processing, payroll deductions, and other miscellaneous banking services. With an aggressive promotional campaign, the FCU is hoping to increase its membership to 8,000 by attracting members of various other organizations. Consumer loans are a major source of income for a credit union, and the effective management of these loans (i.e., balancing loans against liabilities--deposits) is very critical for its survival. In order to reach the growth goals through the generation of additional business, it is imperative that loan processing take as little time as possible. The organization must also simultaneously protect itself from accepting loans that will default in the future.

3.2. The Loan Evaluation Process

Preliminary investigation of a loan application is performed by credit union staff, based on some general guidelines. If they are unable to accept an application it is turned over to a credit committee. This committee includes several professionals who donate their time to the credit union and make loan evaluation decisions so that these remain consistent with the policies set by the credit union board. The authority to deny a loan rests solely with the credit committee; the credit union staff approves only loans that meet all the favorable criteria. In 1989, the credit committee met 104 times to evaluate about 875 loans. In general, the loan evaluation process was considered the biggest bottleneck in the FCU's business operations--its ability to grow was seriously impeded by its inability to process loans in a timely manner. Besides a lack of timeliness, loan approval decisions were not made in a consistent manner, and there was no capture of statsfics or past data to help management evaluate performance.

3.2.1. Problem Definition. The FCU decided to deploy ES technology to evaluate loan applications and achieve consistency in their processing. This exploration was considered critical as FCU is looking for technologies to help it reach some of its stated growth goals. During the investigation it was determined that although the initial process of accepting a loan is relatively straightforward (based on the existence of certain policy-dictated information), the process used by the credit committee to deny or accept an application is considerably more subjective. Indeed, given the large number of committee members involved in the deci-

252 R. Agarwal et al.

sion process at various points in time, it was conceiv- able that individual biases could (and they did) play a role in the evaluation: Such outcomes were not acceptable to the credit union board.

3.2.2. Knowledge Acquisition. In order to develop a system that reflected the FCU's philosophy accurately, the board selected one senior administrator and two credit union staff members as the primary sources of knowledge for the system: The former would provide information on credit union policies based on his ex- perience, while the latter would clarify the day-to-day logistics of the loan approval process. Knowledge acquisition attempted to extract both surface knowledge (procedures followed to accept the application), and a deeper understanding of the issues that guided the credit committee's decision. Traditional knowledge acquisition techniques, such as interviews and observation, were used to determine the roles of different variables and the relationships between them in the loan decision. Because the composition of the credit committee varies at different points in time, it became necessary to talk to multiple experts and resolve certain conflicts and inconsistencies.

Several contingencies and complications arise in the acquisition of knowledge from more than one expert. Differences in vocabulary, terminology, heuristics, as- sumptions, etc. must be resolved by the knowledge en- gineer (Boose, 1984; Reboh, 1983). These differences can be resolved prior to knowledge-base construction-- a "consensus seeking" approach, in which the objective of knowledge acquisition is to construct a single line of reasoning to which all experts have consented. The "consensus seeking" approach to resolve conflicts was used here because the credit board members possess varied but "related expertise" (Cung & Ng, 1989) and were willing to discuss their differences in order to develop a "consistent," a priori agreed-upon policy.

During the reconciliation process, when there was a disagreement, the group did hear the rationale behind the alternatives proposed, and adopted, in some instances, the minority opinion. One of the reasons for accepting minority opinion was the recognition of an individual's specific expertise in a given case. For example, it was pointed out that certain credit-reporting agencies assign a high-risk score to an individual based on the delays in payments, but often the delays are caused by stores who issue credit but delay the sending of an invoice to the customer (thus causing payment delays). This information was useful because it made the committee monitor not only the number of pay- merits that were delayed, but also the type of loan on which payments were delayed before making any assessment on the credit-worthiness of the applicant. In about 15% of the cases in which differences were observed, minority opinion prevailed.

3.2.3. Knowledge Representation and Implementation. The system is designed to classify cases into four categories: accept; committee accept (recommendation for the committee to accept); committee reject; and review (i.e., the system cannot provide an unequivocal recommendation, and the case must be reviewed by the credit committee). Decision logic formalized as rules appeared to adequately capture domain knowledge and experts' reasoning processes, and hence, a rule-based expert system shell was used for development. A PC- based shell was used to build initial prototypes, with rules being automatically generated using a set of sample cases and revised based on committee feedback. Because MONY-FCU is a part of the Mutual of New York insurance organization, and the parent firm cur- rently has expert systems developed using AICORP's KBMS, it was considered essential that any system developed at FCU fully take advantage of the technology resident at MONY. With the hardware platform acceptable to FCU being constrained to a PC environment, the product FirstClass (a PC-shell by AICORP) appeared to be the logical platform for the development.

3.2.4. Testing. After cycling through many revisions and validations using test cases, the prototype was considered acceptable for transfer to production. Valida- tion here primarily focused on ensuring that the system was able to mimic the "accept" decision of the credit union staffand provide appropriate recommendations to the credit committee in other cases. Over 40 cases were tested against the system to ensure that it was classifying instances correctly. Only a couple of changes were made to the system during this testing process. To ensure further validity, the system was run in par- allel with the current nonautomated system for over 2 months, and the results were compared. Again, no major changes were made to the decision logic during this period except for some format and interface adjustments.

3.2.5. Transfer to Production. The system was implemented in a procedural environment with a data base interface. This was done for several reasons. There were many problems with the input/output formats associated with the expert system environment, and some of the information initially gathered to make a loan decision had to be altered as the scope of the system was broadened to include statistical analysis of loans processed. In addition, it was considered necessary to reduce the paperwork associated with loan processing by better consolidation with other systems. While this type of transfer (from a rule-based to a procedural environment) is not always feasible and advisable, given the small rule base size and resource constraints, this translation allowed the CU to bring the system into


production quickly and to integrate it with its current technology environment. Thus, each technology cho- sen for implementation was appropriate given the specific needs of the application at that time: in the early stages, in which the focus was on eliciting and for- realizing decision logic, the shell facilitated prototyping, whereas at project completion, the platform for implementation needed to address efficiency and integra- tion issues.

The prototype was developed over a 2-month period (after four knowledge acquisition sessions) and was revised for about a month before it was given for internal testing. The system was tested for about a month, and testing continued while it was transferred to production. In total, the system reached its usable state within 6 months of its inception to the point where the manager and the committee could use its recommendations on a day-to-day basis. The system has been in place for about a year and is considered by the FCU staff and the committee to be an unqualified success. The following section provides a more formal analysis of the impacts associated with the system.

4. IMPACT ASSESSMENT OF ES TECHNOLOGY ON THE CREDIT UNION

Whereas it is relatively straightforward to assess a system's impact in terms of cost reductions and productivity improvements, it is much more difficult to isolate effects on other equally important dimensions, such as the organization structure, decision processes, and the elusive "strategic" benefits (Kriebel, 1989). Technology can induce change in the process and the product and, furthermore, can have a ripple effect both globally (the way the business system interfaces with other systems) and locally (the way decisions are made over time).

The product dimension examines the role of technology on the quality of the product generated, whereas the process dimension measures the impact of technology intervention on the resources used to generate the product. The impact of improvement in product quality may be felt globally (among tasks that use this product as their input) and locally (by changing the way in which the process is performed in the future). Furthermore, the impact of IT can be classified as direct or induced depending on the relationship between the desired (or expected) outcome of IT utilization and the observed impact. If technology is intended to influence product quality, then any local impact on the process that was not anticipated due to the introduction of the technology is referred to as "induced." Similarly, if technology is intended to influence the process without impacting the product, then unanticipated impacts on the product will be referred to as induced. While these two induced effects are local to the process that is influenced by technology, such unanticipated effects at

the local level may, in turn, affect processes outside the local process, creating an "induced global effect."

In this specific case, the objective of the loan evaluation system was to reduce the time expended by the credit committee to evaluate loan applicants (better resource utilization) by delegating some of the evaluation task to an expert system. There was no direct attempt to alter the quality of the product (i.e., the loan decision). In other words, the (a priori) intended use of technology was to impact process without impacting the product. However, this explicit intention proved to be a naive and incomplete assessment, as the impact has been felt in many unpredictable ways, as mentioned below.

4.1. Direct/Local Impact

Even though the number of applications reviewed by the committee has not declined, the time it takes for them to review these has been cut in half. Part of this can be attributed to the streamlining of all the information that was used to make the preliminary decision by the system. In addition, for cases the system is unable to accept outright, its ability to provide a rationale for its recommendation made the committee take less time than before to review and finalize the loan decision. Thus the system really operates in two modes: provid- ing expertise for the initial screening and decision support for the committee's deliberations. In the words of one of the committee members, "it enables us to target trouble areas of an application quicker." The time saved is used to analyze review appfications and suggest their approval when followed with a collateral, co- signer, or other relevant information.

4.2. Induced/Local Impact

Given that multiple individuals are involved in the loan evaluation decision process, the policies that are followed and the variables considered critical to the decision process seem to vary somewhat. The knowledge acquisition techniques used to formalize this decision logic forced differences to the surface for immediate clarification. The attempt to focus on improving the process made the committee take a closer look at the product that was being delivered: the acceptability or denial of the loans themselves. This broadening of the project scope, induced by the technology, resulted in the committee's examining and constructing policies that placed dollar limits on auto and unsecured loans and differentiated car and debt consolidation loans based on implied collateral. Also, certain criteria were given greater importance (such as good credit), whereas the criticality of others was reduced (such as the extent to which an applicant has overextended himself or her- self). These issues were resolved by the committee, and

254

TABLE 2 A Summary of Loan Statistics Provided by the System

Number of loan applications approved by loan type and month

Number of loan applications declined by loan type and month

Number of counter offers Credit score of loans approved and declined by month Number of loans processed by loan processors and

credit committee Number of loans recommended for acceptance, decline,

and review

it is anticipated that these changes will impact the product-- the final outcome of the loan decision itself. In fact, according to one of the board members, Wendy Faulkner, "The significant benefit of the system is that it made the board take a careful look at the loan philosophy and agree on a common set of guidelines that everyone understands."

4.3. Direct/Global Impact

One of the benefits of the system is its ability to capture past loan evaluation decisions so that these data can be analyzed and used for policy evaluation. From the point of view of the credit committee and the credit union manager, this information is very valuable. Analysis of prior loans provides critical information that will keep loan acceptance policies adaptive to changing economic conditions and also help monitor the causes of any patterns in delinquencies. This is one of the by-products of the ES, even though it was not the primary motivating force behind the initial construction of the system. See Table 2 for an illustration of some of the analyses the system provides.

Because the time to process regular loans was cut in half, the committee started to examine "debit cards" (similar to other credit cards) that were approved pre- viously with only a cursory evaluation. Prior to the introduction of system, the committee used to evaluate about 100 loans/month; after system installation, it processed about 85 loans and about 70 debit card loans that it did not have time to process earlier. Because over 60% of these debit card loans were rejected, the implication, according to the General Manager, was

R. Agarwal et al.

that the system forced them to take a closer look at "card loans" with a high potential for future default.

4.4. Induced/Global Impact

The streamlining of loan evaluations made the overall decisions on loan acceptances/rejections consistent over time. It is anticipated that this will impact the overall product quality and result in more timely and market-oriented loan decisions, while reflecting the credit union's philosophy consistently. However, one has to monitor the loan performance over time to ensure that the system is staying consistent with the CU objectives; the next section examines this issue in greater detail. Table 3 summarizes the impacts discussed thus far.

4.5. Discussion

The impact observed is consistent with several propositions identified by Huber (1990). IT has reduced the decision time but has not altered the decision meeting frequency, has not altered the composition of the group making the decisions, and has no impact on the information used to make decisions. Structurally, some decisions that were made solely by the credit committee prior to the introduction of technology (centralized) have been transferred to loan processors, thus adding another level to the decision hierarchy. Because the application's objective is to capture expertise, it was able to store decision logic associated with loan evaluation and data on past loans, thus allowing the organization to standardize and formalize its loan decisions. The general consensus of the credit committee is that the quality of the decisions has improved sig- nificantly, decisions are less sensitive to the decision group composition, the decision time of a loan application has been reduced by 50%, and problem areas are spotted quickly for resolution. Notice that much of this impact was a result of"decision time" (resource) saved by the committee.

The impact of ES technology, independent of which way it is analyzed, has been significant for the CU operations, the greatest impact being on the loan decision process. In order to assess this impact fully, one needs to continuously observe the performance of the system

TABLE 3 The Impacts of the Expert Systems Application

Local Global

Direct Cut loan decision time in half Analyze loan data for policy evaluation Streamline loan application information Examine other loan types, such as debit cards Target trouble areas quickly Change in loan evaluation policies Consistency in loan decisions

Induced Improve loan quality over time Result in more market-oriented decisions


against the decisions made by the commit tee--not only to improve the system performance by revising the embedded logic, but also to evaluate the relevance of the logic in changing market conditions. The next section presents a method developed to analyze the system performance over time relative to the decisions made, so that the impact of the system can be monitored longitudinally. This type ofanaiysis is particularly crit- icai if the impact of IT on the decision is time-depen- dent, as is the case at the CU.

5. IMPACT OF SYSTEM PERFORMANCE ON T H E LOAN DECISION PROCESS

In order to analyze the impact of the ES on loan decisions over time, one can monitor the way loans are categorized by the system and relate them to the final decision taken by the committee. This contrast in the decisions is shown in Table 4.

In Table 4, rows correspond to the system recommendation and columns correspond to the action taken by the committee based on the system recommendation. For example, when the system says "accept," the committee always accepts those loans (i.e., these are situations that are relatively straightforward). If, however, the system simply recommends acceptance, then these are looked at by the committee in a cursory manner to see if any anomalies may call for overriding the system. The other cases: " recommend to reject" and "review" are thoroughly examined by the committee. This type of data can prove invaluable in monitoring system performance over time; the following discussion illustrates how an analysis of the cases that fail in these cells can be used to assess the impact of the system on the loan decision process.

In a sample of 100 processed loans, the system accepted outright 15 (C 1) and denied 0 (C2) due to policy limitations. The system sent 60 to the committee with "accept" as a recommendation, and the committee accepted 59 of these (C3) and rejected 1 (C4). The system sent 24 to the committee with "deny" as a recommendation, and 20 were denied (C6) and 4 were accepted (C5). In this sample, only 1 of the applications came to the committee for "review" (C7 and C8), and it was accepted. These data may be utilized both to under-

stand the global impact of the system and to monitor system performance and potential obsolescence over time.

5.1. Cell 1 and Cell 2: System Accepts or Rejects Applications Outright

In this case, the system accepts loan applications outright only ifail the needed information is available and acceptable. No loans will fail in cell 2 because the system cannot reject loans, as a policy.

5.2. Cell 3 and Cell 6: Committee Accepts System's Recommendation

These cells represent cases in which the recommendation provided by the system is accepted by the committee. Although these are still reviewed by the committee, the time they take for such a review is much less. As more cases fail into Cell 3 over time, one needs to explore ways to embed them into the system logic, that is, delegate such cases to the system (move to cell 1) and improve the committee's productivity. The number of cases that fail in Cell 6, however, have to be reviewed personally by the committee.

5=3. Cell 5: System Recommends Denial and Committee Accepts

This cell provides information on those cases in which the committee does not accept the recommendation of the system and "accepts" the loan application. The fewer the cases that fail in this cell the better the system is in reflecting company policy. In the early stages of system implementation, because of the induced effect, it is possible that the recommendations made by the system are in fact in line with the FCU board's changed policies, and the committee may be unaware of some of these changes. So these cases have to be analyzed carefully to ascertain whether these differences can be attributed to changed policies or to other qualitative factors the system has no knowledge of. If reconciled and streamlined over time, such knowledge can be used to alter the system's logic, that is, to transfer such cases to Cells 1 and 3.

TABLE 4 A Longitudinal Analysis of System Effectiveness

Committee Decision System

Recommendation Accept Deny Total

Accept 15 0 15 Recommend to accept 59 1 60 Recommend to deny 4 20 24 Review 1 0 1 Total 79 21 100

5.4. Cell 4: System Recommends Acceptance and Committee Denies

It is difficult to assess how well these would have turned out, had the committee accepted the system recommendation. Some follow-up of these cases, when appropriate, might provide insights into the credit policies of the organization. One has to be concerned if too many cases fall in this cell as it may reflect either an obsolete system or inadequate knowledge acquisition and representation in constructing the expert system.

256 R. Agarwal et al.

5.5. Cell 7 and Cell 8: System Sends the Cases for Committee to Review

Here the system does not have sufficient logic to provide any kind of recommendation. If many cases fall into this area, these should be reviewed by the board for possible policy reevaluation and for embedding some of this reasoning into the system. The fewer the. proportion of the cases that fall into these cells, the greater the gain in productivity for the committee and the organization. On the other hand, if many fall into these cells, one may have to question the viability of the system as a support tool.

5.6. Further Discussion

As system monitoring progresses over time, our objective is to minimize the number of cases that fall in cells 3 and 7 (by shifting the burden to the system), cell 5 (by keeping the system logic up-to-date with the changing conditions of the market place), and cell 4 (by altering system logic so as to minimize any undue system influence on the credit committee's decisions). With respect to cells 6 and 8, our objective is to move as many of them to cell 6 as possible from cell 8.

Some of these shifts can be accomplished by changing the system's reasoning process so that it mirrors the committee's philosophy as closely as possible. This can make the system complex by increasing the number of criteria and/or their values that are used in making the decisions. This, in turn, can impact both the knowledge acquisition (during the refinement of the system) and the consultation (by seeking information on many more variables). "Robustness," which is often associated with simpler, more generalizable, and easier to understand reasoning processes is sacrificed somewhat ifeach special case is automatically used to revise the system. The added rules and parameters may impact the clarity of the reasoning used for explanation and training. Alternate methods available for managing such a shift would include changing the current knowledge representation and/or seeking the use of a combination of representations to represent decision com- plexity (such as hybrid representations) (Agarwal, Brown, & Tanniru, 1991).

6. CONCLUSIONS

The case presented here described the significant impact a new information technology had on a small organization. Whereas one can attribute the success to a variety of factors, such as a judicious use of resource for technology exploration and deployment, the selec- tion of a problem that had strategic value, and a strong commitment to do whatever it takes to make the project a success, no one anticipated the types of changes the expert system resulted in. The credit union has

started to move in three major directions to meet its growth needs, and the system designed, either directly or indirectly, has an impact on all these directions.

Given the perils many banking and savings and loan institutions are facing these days, it is critical that a bank take stock of its asset and liability situation and balance them effectively. Because the assets of a bank are tied up in the "loans" it underwrites, its ability to monitor the loan portfolio effectively is crucial for survival and growth. As the credit union undertakes a critical assessment of its assets against liabilities (the deposits it holds) and tries to accomplish the goal of balance, the expert system is proving valuable through its ability to deliver statistics on the types of loans processed and their characteristics, such as risk, volume, and amount.

The credit union is trying to grow by expanding its membership base and introducing new loan opportunities, such as credit cards. With this new thrust it is critical that the committee that has to oversee the loan evaluation process not be overburdened and ineffective. The decision support role of the expert system is crucial here because it allows the committee to process more loans without affecting their productivity. According to the General Manager, the expert system used to process individual loans can easily be tailored to process any other types of loans introduced as a part of their product portfolio. Whereas many of these initiatives are being undertaken by new management and board of directors who have a proactive attitude toward the use of technology, some of these initiatives are in part influenced by the critical role the expert system has played. For example, one of the initiatives being developed to reward tellers and service personnel for loan referrals was implemented quickly and cheaply by modifying the current loan expert system. One of the ultimate compliments to the expert system was made by the General Manager, Ross Irvin, when he said, "The real source of its success is the fine logic which was developed to analyze the credit. Without that, none of the other initiatives could have been accomplished".

6.1. State of the ES Today at MONY-FCU

Since the successful assimilation of ES technology for loan processing, the CU has expanded the system to include credit card evaluation. According to Ross Irvin, much of the logic remained the same except for some minor adjustments to tailor it specifically to credit card applications. Since this extension has been added, over 237 applications were processed in a single month (over a threefold increase), and half of these were applications for credit cards. Such an increase in the processing ca- pability was considered infeasible prior to the introduction of the system. Beyond this extension, no other major applications have been initiated in the ES area. As discussed above, maintaining a balance between


assets and liabilities is considered a top priority, and organizat ion has shifted its at tention to this task at this time. It is, however, impor tan t to note that the expe- rience with the introduct ion o f ES technology (using prototyping for feasibility assessment) has made the organization use similar strategies for investigating the use o f a "decision model" for asset/liability balancing using a spreadsheet language.

One final side effect o f the process o f ES introduction and diffusion at F C U is that now the system is viewed simply as a compute r program that provides useful informat ion for loan decisions; the users' perceptions are that it is no different f rom the other informat ion processing technologies that existed before. Hence, while there was excitement when the system initially replaced the tedious task o f loan evaluation, it has now become an integral c o m p o n e n t o f the user's task. Such an acceptance, coupled with a full appreciat ion o f the value the system provides, is an indicator o f the success with which the system was developed and integrated into the task env i ronment o f the user.

R E F E R E N C E S

Agarwal, R., & Tanniru, M. (1992). Assessing the organizational impacts of information technology. International Journal of Tech- nology Management, 7, 626-643.

Agarwai, R., Brown, S., & Tanniru, M. ( 1991). Alternate acquisition and representation formalisms for expert systems: A case study. Indian Computing Congress, Hyderabad, India, pp. 193-202, and Working Paper, Syracuse University, Syracuse, NY, 13244.

Boose, J.H. (1984). Rapid acquisition and combination of knowledge from multiple experts in the same domain. Proceedings oflEEE Conference on .41 Applications, pp. 461--466.

Cung, L.D., & Ng, T.S. (1989). DESPLATE: A diagnostic expert system for faulty plan view shapes on steel plates, in J.R. Quinlan (Ed.),Applications of expert systems(Voi. 2, pp. 156-169). Read- ing, MA: Addison Wesley.

Galbraith, J. (1977). Organization design. Reading, MA: Addison- Wesley.

Huber, G. (1990). A theory of the effects of advanced information technologies on organizational design, intelligence, and decision making. Academy of Managemem Review, 15,(1), 47-71.

Kdebel, C.H. (1989). Understanding the strategic investment in information technology. In K.C. Laudon & J. Turner (Eds.), In- formation technology and management strategy (pp. 106-118). Englewood Cliffs, N J: Prentice-Hall.

March, J.G., & Simon, H.A. (1958). Organizations, New York: Wiley. McKeen, J.D., & Smith, H.A. (1991). The value of information tech-

nology: A resource view, Proceedings of the 12th International Conference on Information Systems, New York. pp. 41-52.

Reboh, R. (1983). Extracting useful advice from conflicting expertise. Proceedings of the International Joint Conference on Artificial Intelligence. Karlsruhe, West Germany. pp. 145-50.

Svioideh J.J. (1989-90). Expert systems and their impact on the firm: The effects of planpower use on the information processing ca- pability of the financial collaborative. Journal of MIS, 6(3), 65- 84.

Sviokla, J.J. (1990). An examination of the impact of expert systems on the firm: The case of XCON. MIS Quarterly, 14, 127-140.

Documents

Assessing the impact of expert systems: The experiences of a small firm