Pitfalls in Analyzing Systems in Organizations · PDF filePitfalls in Analyzing Systems in ... Pitfalls in Analyzing Systems in Organizations ... the major types of pitfalls demonstrated

Pitfalls in Analyzing Systems in Organizations

Steven Alter School of Business and Management

University of San Francisco San Francisco, CA 94117

[email protected] 415-422-6383

© Steven Alter, 2003 1

mailto:[email protected]

Pitfalls in Analyzing Systems in Organizations

Abstract

Despite the availability of elaborate methods for defining data and business processes, huge amounts of time and effort

are wasted on system projects that produce disappointing results. An important contributing factor is the significant

difficulty business and IT professionals experience when they try to describe, evaluate, and/or analyze systems in

organizations even at a cursory level. Between 1997 and 2003, the author's information system courses for evening

MBAs and EMBAs have required students to write two group papers that present a business-oriented analysis of a real

world system in an organization and propose preliminary recommendations for improvements. If these working students

are representative of the types of business professionals who are involved in systems in organizations, it is plausible that

the major types of pitfalls demonstrated by their papers are representative of the pitfalls that contribute to disappointing

results with systems.

An examination of 202 group papers submitted by evening MBA and EMBA students between 1997 and 2003 revealed

pitfalls in the following categories: difficulty defining systems in organizations, difficulty identifying the information in

information systems, aversion to using measures of performance, reluctance to mention organizational and personal

issues, susceptibility to techno-hype and jargon, inadequate critical thinking, and difficulty applying abstractions and

formal methods. This paper illustrates these problems through examples from 25 of the student papers. Assuming that

typical business professionals encounter the same types of pitfalls, both MBA programs and analysis and design methods

should provide concepts and techniques that help in identifying and minimizing the related problems.


1. Introduction

Why is the success rate of system-related projects so abysmal? Every other year the Standish Group publishes a new

study showing that fewer than a third of IT-projects are completely successful and many are complete failures. (e.g. [1])

One frequently encounters claims that a substantial percentage of CRM projects or ERP projects or outsourcing projects

are partial or total failures. New technologies encounter surprisingly long assimilation gaps. [2] Perhaps IS, instead of

economics, should be termed the dismal science.

One of many reasons for these problems is that business professionals are often ineffective in communicating with IT

professionals and in their roles related to identifying system-related problems, determining system requirements, and

implementing systems in their organizations. An important goal of generalist MBA courses in information systems is to

impart knowledge that will help business professionals perform these roles more effectively, but the courses often

become an exercise in learning and using jargon rather than an effective way to learn how to analyze systems from a

business viewpoint. Despite the availability of elaborate methods that IT professionals can use for defining data and

business processes, little attention has been devoted to developing organized methods for business professionals.

Starting around 1992 I decided that my introductory IS courses should address these problems directly by focusing on

how business professionals can think about systems for themselves and by requiring students to write two group papers.

The first paper develops and justifies tentative recommendations for system improvements based on a preliminary

analysis of a real world system in an organization. The second paper focuses on processes through which real world

systems change or are supported. Starting in 1997 I began requesting that students submit electronic versions of their

major papers. In addition to providing a form of feedback about teaching effectiveness, this helped in the continuing

development of the work system method for understanding and analyzing systems from a business viewpoint. [3, 4, 5]

Examples from 25 of the 202 papers will be cited to illustrate different types of systems analysis pitfalls encountered by

these early-career business professionals, most of whom are not IT professionals.


2. Source of Data and Method for Data Analysis

To date, 202 student papers have been collected, although an unknown but relatively small number were never received

or have been lost due to a combination of syllabus variations, lack of follow-though when students failed to submit

electronic copies, and the use of five different computers at home and at work between 1997 and 2003. Over 90% of

these group papers concern a real world system in an organization to which at least one team member had access. In

other words, the vast majority of the papers were not attempts to assemble material from the Web or to analyze an

existing case study. The fact that these were group papers had a number of effects. On the one hand, students working

together sometimes clarify each other’s ideas and generate better analyses than would appear in individual papers. On the

other hand, the management of the writing and reviewing process is often haphazard in student teams. In many cases the

final papers do not hang together very well due to a combination of poor writing skills, inadequate review, and

interpersonal conflicts within the teams.

The goal of examining these papers was to identify the major types of problems encountered by business professionals

when they attempt to analyze systems from a business viewpoint after some preliminary training in thinking about

systems in organizations. Although no precise information is available about the amount of business experience of the

authors of specific papers, at least half of the students who submitted these papers had five or more years of business

experience based on their admission to USF’s PMBA or EMBA programs. Less than half of the regular MBA students

had five years of business experience.

The analysis process was informal. Based on experience teaching IS courses for over 15 years I identified what I

believed were the major categories of problems. Many of these problems were apparent in project proposals that the

student teams were required to submit before doing their research. In many cases my detailed feedback about these

proposals helped the teams recognize and avoid likely pitfalls. Although I did not keep records of my responses to the

original proposals, I believe that most of the teams would agree that my initial feedback helped them write more

successful papers.


I numbered the papers and then looked at each paper to identify representative examples and to find other types of

problems. Many of the 202 papers were very good and illustrated no obvious pitfalls; some of the others illustrated little

more than inattention and poor writing. The 25 papers that provide the 30 illustrative examples cited here vary in quality,

and were chosen mainly because the examples of pitfalls were understandable and possible to describe briefly. It is likely

that some of the problems would have been more severe had I not provided feedback on proposals to write the papers.

Although students knew I would keep an electronic copy and had been instructed to avoid topics that they were not

willing to discuss in class presentations on these papers, confidentiality concerns dictated that no one else could inspect

or code these papers. The examples cited here are disguised to avoid identifying specific companies. In several cases

system-related terminology was changed to assure that the company cannot be identified.

Because the source data (the papers) was a byproduct of topics and assignments that that were revised continually over

the years, and because examples would provide the best insights about specific pitfalls, no attempt was made to code

subjective observations about the papers or use quantitative procedures that might have been relevant if the source data

had been collected under more controlled circumstances.

3. Major Categories of Pitfalls

This section is the heart of this paper. It presents numerous examples illustrating pitfalls in the following categories:

difficulty defining systems in organizations, difficulty identifying the information in information systems, aversion to

using measures of performance, reluctance to mention organizational and personal issues, susceptibility to techno-hype

and jargon, inadequate critical thinking, and difficulty applying abstractions and formal methods.

It is surely possible that someone else examining the 202 papers would have responded to specific examples differently

and would have found other categories of problems. Nonetheless, the categories and examples should contribute to the IS

field’s understanding of difficulties encountered in analyzing systems, confusions and pitfalls encountered by business

professionals, and expectations about what can or should be learned from MBA courses in information systems. Most

important, this paper should contribute to the discussion of what can be done to help business professionals understand


systems in organizations more fully, thereby reducing the impacts of one of the many factors contributing to system

disappointments and failure.

Difficulty Defining Systems in Organizations

Every systems analysis method starts with defining the problem or opportunity and also the system that is being

examined. Although students can easily memorize the steps for analyzing a system and discuss those steps in the

abstract, it is much more challenging for them to use those steps in an open-ended situation that has not been pre-

digested in a case study. My information system courses address this challenge directly because I believe this is the most

important thing MBA and EMBA students who use IT at work every day can learn from an introductory IS course.

These courses emphasize the work system method for understanding systems in organizations. (see [3, 4, 5]). That

method’s underlying premise is that business professionals need a clear understanding of a specific work system (e.g., a

firm’s system for hiring people, finding sales prospects, designing products, manufacturing products, or developing an

annual plan) in order to contribute to the development or improvement of an information system that supports the work

system. Thus, the work system is not the software or the information system. It is the system whose performance is to be

improved as a result of the analysis. In general, the work system should be the smallest work system that exhibits the

problem or opportunity. If it is much larger, the analysis will be unnecessarily complicated. If it is smaller, the analysis

may ignore important issues that must be addressed. The work system method is organized around nine elements that

can be used to describe any work system. The first four, the components of the work system, include the business

process, participants, information, and technology within the system. Five additional elements that round out the basic

understanding of any system in an organization include the products and services produced, the customers, environment,

infrastructure, and strategy. [4]

In many cases, defining the work system is the most difficult and contentious part of the analysis, especially for student

teams with extensive business experience. A number of more experienced student teams have reported spending five or

more hours arguing about the definition of the system before finally settling on a definition, and some of those teams had

to change the definition after gathering more information during the analysis. Common difficulties in defining a work


system fall into a number of categories including failure to define the system clearly, viewing technology as the system,

and confusion between the system in operation and the system being built.

Failure to Define the System. Although the work system method specifically requires users to define the system being

analyzed, student teams sometimes bounce back and forth between several overlapping systems without clarifying which

one they believe the analysis is focusing on.

Example - Minimizing unnecessary expenses related to desktop printers: A large financial institution attempting to

control IT-related costs created a system for analyzing printer usage and deciding whether individual desktop printers are

justified. The student paper analyzing this situation failed to establish and maintain a consistent view of what work

system it was considering. At some points, it seemed to be discussing a work system of collecting information about

printers in use. At other points, it seemed to be discussing a system of “rightsizing”, i.e., the system of gathering

information, negotiating with managers, and helping make informed decisions about how many printers are needed. The

lack of clarity about which system was being analyzed led to confusion because these two systems had different scope,

different measures of performance, and in this situation encountered different types of problems.

Example - Tracking pharmaceutical sales representatives: A large pharmaceutical company developed an information

system for tracking physician visits by sales representatives. The student paper was sometimes confusing because it

switched back and forth between discussing the work system of providing information and service to physicians and the

work system of record keeping related to those visits. As with the printer analysis in the previous paragraph, the two

systems had different scope, different customers, different measures of performance, and in this situation encountered

different types of problems. If the work system is about tracking the visits, the measures of performance are about

whether information is gathered efficiently and completely. Whether or not the sales work is done well is beyond the

scope of the analysis. The customer in this case is the sales rep (who will use the information) and sales management. If

the work system is about serving physicians, the measures of performance are about whether the sales work is done

efficiently and successfully, and secondarily whether the information is gathered. The customers in this case are the

physicians, with managers and sales reps as secondary customers for the information that is generated.


Example - Sales process for expensive capital equipment: A major manufacturer began to implement what it called the

“sales coordination process” (SCP), which created selling teams consisting of everyone who had contact with certain

large customers. The selling teams were supposed to develop and track coordinated action plans. The company’s IT

group developed the “SCP network,” which was to be used to store all of the information relevant to a selling team’s

activities. The student paper had difficulty separating the effectiveness of SCP from the effectiveness of the SCP

network. Although the SCP effort had made some progress in getting teams to share information, less than a third of the

teams had produced action plans and only half of those were current. This problem was attributed to technical

shortcomings of the SCP network, although it certainly seemed possible to develop the plans outside of the network.

Example - Role of technology in an MBA program: A student paper viewed an MBA program as “a complex work

system that has in recent years become more reliant on technology as a tool to support information and aid in the

teaching practice.” The analysis looked at three representative separate courses and how technology was used in each.

The courses were unrelated and the use of technology in the courses was unrelated. The result was a set of separate

stories but no real system view of the situation. In contrast, it might have been possible to establish a work system view

of a typical student’s MBA program and then show how consistent, interrelated use of technology within various courses

could contribute to an overall experience. Alternatively, it might have been possible to establish a work system view of

the entire teaching effort.

Thinking of the Technology as the System. An important reason for using the work system approach is to recognize

that from a business viewpoint the headline is the system of doing the work rather than the technology that is being used.

In too many cases, systems that use IT are called IT systems. (e.g., “The IT System that Couldn’t Deliver,” an HBR case

study [6] discussed in [7].)

Example - Data warehouse in a finance department: A student paper focused on a data warehouse acquired to provide a

finance department and other departments convenient access to operational and accounting information captured in an

ERP system and several other information systems. The paper said the steps in the business process included extracting

data from the ERP system, filtering and aggregating the data, copying data from other information systems, and using the


information for financial analysis. The paper described how the technology was acquired and configured but only rarely

used in any significant way. The analysis would have been much more effective if it had started from specific finance

work systems such as closing the books each month and performing specific types of financial analysis. Starting from the

goal of improving those work systems would have led directly to identification of specific shortcomings of the data

warehouse that could have been fixed by changing its configuration. Focusing on the data warehouse as the system left

the link between finance work and operational problems in the data warehouse unclear; it was difficult to decide whether

the recommendation would solve anything, or whether the result would be a more efficient data warehouse that still

would not be used very much.

Example - Lead generation system in a software firm: A student paper attempted to analyze how a software firm’s sales

department could improve a lead generation process that used CRM software. The paper implied that the CRM software

was the system being studied. The analysis cited statistics and projections showing that the current lead generation effort

was insufficient to support the company’s sales goals. It spoke of burnout among telemarketers and argued that the lead

generation team needed better tools and information and that they lacked training. Because the work system was never

defined, it was never clear which of the following possible issues were within the scope of the system: the group

answering incoming inquiries doesn’t know how to do inbound sales; the telemarketers don’t know how to do outbound

sales; nobody knows how to use the CRM software properly; account execs (who receive the leads) don’t know how to

evaluate whether they are properly qualified or don’t know how to sell. A more effective analysis would have identified

steps in the business process and would have discussed separate performance issues for the various parts of the work

system. Based on responses to questions in the class presentation it seemed likely that some of the problems were related

to the CRM software while other parts were related to business process, personnel, and training issues.

Example - Manufacturing system at a food processing company. A food processing company uses an outdated software

product called manufacturing control system (MCS). The group paper analyzing various aspects of the company’s

manufacturing summarized the business process as follows: generating the sales forecast, recording raw material usage,

recording manufacturing yields, recording packaging completions, recording inventory data in MCS, ordering raw

material, and recording raw material receipts. In reference to raw material management it noted that the process is time

consuming, the system is inflexible (because it provides results only once a week), the system is costly (because greater


use of automated manufacturing would be cheaper), and the system is not reliable (because manual inventory counting is

error-prone). Unfortunately, the analysis uses the word “system” to denote both the MCS software and the system of

doing manufacturing. The analysis is ambiguous because it is sometimes unclear whether the topic is the manufacturing

software or the manufacturing.

Example - Brokerage firm’s network for financial consultants: Financial consultants working for an international

brokerage firm accessed client account information, sales material, research information, and email through customized

work stations. As the consultants became more mobile, the need to make the workstation mobile increased. The student

paper viewed the work station as the work system being studied, as revealed by describing the consultants as customers

and the business process as logging on, accessing information, and logging off. With such a limited view of the work

system, the recommendations said that technical glitches in the workstation software should be fixed but couldn’t offer

other ideas. A broader view of the way increasingly mobile financial consultants provide services for customers might

have led to a much more valuable recommendations about improving consultant efficiency and/or effectiveness.

Confusion between the System in Operation and the System Being Built. Over the years, a number of proposed

papers contained confusions about the difference between a system in operation and a system being built. This type of

confusion is revealed through business processes that include some steps about building and/or maintaining a technical

system and other steps related to using the technical system to accomplish business tasks. Both are important, but when

these separate activities are treated as a single work system the measures of performance become confused and the

recommendations may include anything from improvements in processes for doing technical system maintenance work

through improvements in performing customer-facing processes. In most cases, feedback about project proposals

clarified this issue for students, but in one paper the issue remained:

Example - Providing Web access to existing industry magazines: The analysis concerned a strategy decision related to

how a magazine publisher could use the Web to serve the customers that have previously been served using paper

periodicals. The analysis did not use system ideas effectively because it combined two systems with totally different

operational issues, measures of performance, and participants. The confusion was apparent from the way the business


process and participants were summarized. The business process included two parts. First, the magazine’s editorial and

marketing staff collected information, wrote product reviews, and generated revenue through sponsorships, advertising,

and subscriptions. Second, Web site users logged on, found information related to products and other aspects of their

industries, and in some instances made purchases. Perhaps because the paper combined two systems into one, it did not

discuss measures of performance and focused primarily on goals that the publisher might pursue. A more effective

analysis might have viewed the customer’s use of the Web site as a work system, and might have been more specific

about how the customer’s efforts to obtain value could be evaluated and improved. For example, that analysis might have

asked whether the customers actually want the types of information provided in magazines and whether some form of

genuine computer interaction, as opposed to data retrieval, might provide more value for some of the products reviewed

in the magazines.

Difficulty Identifying the Information in an Information System

It might seem surprising to specialists in information systems, but many of the MBA and EMBA students who wrote

these papers seemed to have relatively little propensity or ability to identify the information used or produced by the

systems they analyzed. Despite having seen demos of relational databases and despite having done several in-class

exercises involving the use of ERDs to help in specifying data requirements, many of the students seemed satisfied with

vague statements such as the information is outdated, historic data is required, and the data in the system is

manufacturing data.

Example - Financial analysis at a clothing manufacturer: The finance department of a clothing manufacturer

consolidated sales forecasts from various regions to produce a combined sales forecast for the firm. The student analysis

attempted to suggest ways to do this work more efficiently. However, from the analysis it was unclear why the forecasts

were being produced and whether the forecasts were at the SKU level or at some aggregated level. My comments to the

student team included the following: “Many of the confusions in reading the paper might have been cleared up if it had

said something like the following: ….. The forecast is by product line and extends five months into the future by month.

There are 27 product lines and the each product line is sold in 14 regions. The job Finance does is to consolidate

forecasts from each of the regions in order to create revenue forecasts for the next five months. These forecasts have


nothing to do with determining exactly what will be produced by SKU in the next five months because that was

determined in the annual production planning cycle. ……. The previous four or five sentences may have been

completely wrong, but something of that general form would have been a starting point toward providing enough clarity

about what the system is trying to do and why your recommendation would really create an improvement.”

Example - Production planning at a biotech manufacturing firm: A biotech manufacturing firm attempting to use an

MRP system encountered a variety of problems related to capacity planning, cost-effective purchasing, shop floor

control, and timely delivery. Among other things, the analysis called for the use of historic machine hours, scrap levels,

material consumption, cycle times and efficiencies for each product. Compared to recommendations about information

requirements in other papers, this recommendation was more specific about the types of information required. However,

with the large number of products the company produces, steep learning curves, and lumpy demand, it is not clear

whether this recommendation would truly solve the company’s problems, especially since another part of the paper says

that production orders are often generated months before they are needed. The paper did not explain whether the

recommended information currently existed in any form and how much effort, and by whom, would be needed to capture

and store it. Furthermore, the heart of the problem may have been the lack of information for several high volume

products rather than for the thousands of products the company occasionally produced.

Aversion to Using Measures of Performance

MBA and EMBA programs include extensive coverage of managerial and financial accounting, the need for

performance measurement in process improvement, and the use of balanced scorecards. Nonetheless, over the years most

student papers seemed to reflect a strong aversion to mentioning measures of performance. Even when the instructions

invited students to estimate performance indicators that were not readily available or to note that seemingly proprietary

performance data had been modified for purposes of writing a paper, students often omitted that information and

attempted to justify recommendations based totally on qualitative concerns and vague generalizations. I eventually began

to require that student papers include several tables listing estimated or actual values of important performance measures

and estimating the extent to which their recommendations would affect these values.


Example - Tracking of demonstration inventory for electronic equipment: A manufacturer of electronic equipment

provided demonstration models to customers but didn’t track the demonstration models effectively. The demonstration

models were often shipped through standard shipping channels without a designation that these were on loan and were

not being sold. This generated inaccurate sales, inventory, and financial data used by other information systems.

Although the analysis provided an estimate of the amount of incorrectly classified inventory, the analysis did not present

measures of performance describing other aspects of the problem, such as the amount of time and effort that is required

for reconciliations, the positive or negative impact of providing so much demonstration equipment, and any impacts on

real revenues and commissions. More attention to measures of performance in other areas might have led to a powerful

recommendation, such as a system that would charge the sales organization for the demonstration equipment. Instead,

the student team’s recommendation seemed rather toothless. It proposed creating an Equipment Request Form (ERF), a

written procedure for filling out the ERF, in-service training on the formal procedure, and education about the need for

the process.

Example - Product development at a food manufacturer: A student paper analyzed the product development process at a

food manufacturer that creates many new products each year. The paper produced the bland recommendation that a

committee should look at the product development process more closely. My feedback to the team included, “The paper

described a current product development system, but did not provide a clear statement of a problem or opportunity that

was to be analyzed. It also seemed to pull its punches on how well the product development system is operating. For

example, it might have said that developing a new product currently costs $X on average and requires Y hours of work

on average, and that these numbers could be improved to the following degree.... if we made these specific changes

…..”

Example - Reconciling local and national reservation data: Five years ago a major hotel chain had a national reservation

system and local reservation systems controlled by each of its properties. The data from the national reservation system

was downloaded periodically to the local reservation systems, which were characterized as electronic replicas of the

paper-based reservation systems that the local properties used previously. Not surprisingly, inconsistencies were

common because reservations could also be created or modified at the local properties, especially when guests left early

or extended their visits. The student team analyzed the situation and recommended that an integrated system be


developed (which was happening at the time). The paper mentioned that discrepancies as large as 20% sometimes

occurred, requiring as many as 300 reconciliations in a day between the local and national systems. Although the 20%

and 300 reconciliations per day represented measures of performance, the analysis never gave a clear idea of the

magnitude of the problem or of the number of people who are really involved in correcting the inconsistencies. Other

measures of performance that might have been relevant to the analysis and the justification of recommendations included

the estimated number of guests who were inconvenienced when their reservations could not be filled, the number of

rooms that went unused unnecessarily, and the amount of clerical time required to do the reconciliations.

Example - Production planning at a biotech manufacturing firm: A biotech manufacturing firm encountered a variety of

problems related to capacity planning, cost-effective purchasing, and timely delivery (mentioned earlier). Although the

paper mentioned many problems, it made no effort to quantify the extent of the problems. For example, capacity

planning was a problem, but it did not mention the extent of delays or lost orders that might have been reduced with

better capacity planning. Incorrect MRP parameters were cited as a problem, but there was no estimate of how

inaccurate the parameters were. Production planning was frustrating and time consuming due to incorrect MRP

parameters and other problems, but no measures of performance for the efficiency or quality of planning were cited.

Example - Lead generation system in a software firm: A previously mentioned student paper attempted to analyze how a

software firm’s sales department could improve a lead generation process that used CRM software. Although the paper

discussed the company’s sales goals and estimated number of leads that would be required to meet those goals, it was

quite imprecise about performance measures for the lead generation process. For example, it said that inbound phone

calls represent the bulk of the leads entering the system, not 60% or 95%, but “the bulk” of the leads. Similarly, it said

that the leads were “often inaccurate” but did not estimate what percentage of the leads were totally non-qualified and

what percentage over-estimated or under-estimated the number of seats that might be purchased. Greater specificity

about performance measures such as these would have helped in evaluating the validity of the recommendations.


Reluctance to Mention Organizational and Personal Issues

Although work systems rely on the human participants who do the work, MBA and EMBA students analyzing systems

tend to emphasize technology and business process problems when trying to explain performance issues and recommend

directions for performance improvement. As illustrated by the following examples, they often seem reluctant to mention

organizational and personal issues.

Example - Travel reservations at a large consulting company: A student team analyzed an ongoing change in the travel

reservations system used by a large consulting company. The goal was to provide a Web-like interface that would

encourage consultants to make reservations themselves, thereby eliminating contractors who served as an in-house travel

agency. Three years after implementation, new online capabilities were being used extensively by around 50% of

younger consultants and less than 10% of senior consultants. The analysis mentioned the relative age and billing rates of

different groups of consultants, and it called for additional encouragement and training. It did not observe that these low

adoption rates after three years indicate that the consultants did not want to use the new system and would continue

avoiding its use until it became compulsory.

Example - Tracking pharmaceutical sales representatives: A large pharmaceutical company developed an information

system for tracking physician visits by sales representatives. The student paper mentioned earlier discussed the need to

do sales work, to keep managers informed, and to do record keeping so that successor sales people would start with

usable customer information. Not mentioned in the paper was the fact that many of the sales representatives resented

spending time entering the data because they believed they were sales representatives, not data entry clerks. The

discussion following the presentation also revealed that the information entered about particular physician visits was

often fudged. To avoid annoying a physician struggling through an overloaded day, a representative might do little more

than say hello, but would still record the visit so that the information transmitted to management would look good.

Example - Time reporting system at a professional services firm: A group paper observed that the firm’s employees “are

more creatively than financially oriented. They … do not naturally understand or appreciate the importance of time

reporting nor are they held accountable for accurate time reporting. The challenge is to implement a change


management process that educates and creates incentives for these employees so that they gain an appreciation and

understanding of the financial importance to the business of accurate data.” Other possible interpretations are that the

company’s management is not aware of the problem or that it views accurate time reporting as far less important than

keeping customers and valued staff happy.

Example - Supporting inexperienced insurance agents: A group paper analyzing systems that support inexperienced

insurance agents noted that many new agents fail because they run out of money before producing enough new business.

The paper’s recommendations included increasing the base salary for the first few years, providing better technology,

and providing mentoring on selling and on the general challenges faced by insurance sales people. Not mentioned in the

paper, but discussed following the presentation, was the possibility that the company’s strategy was to put the new

agents on the street, see whether they could sell, and devote more resources to them only after they proved themselves.

Susceptibility to Techno-hype and Jargon

Even after the Internet bubble burst, the IS field remains rife with techno-hype and jargon. Terms such as CRM, ERP,

EAI, and data mining all set up expectations that may be unrealistic. In too many situations, textbooks do little to

puncture inflated hopes. To the contrary, they sometimes promote images that barely fit current reality. Despite my

repeated attempts to warn students about the exaggerated expectations that techno-hype and jargon sometimes engender,

a number of papers encountered problems in this area.

Example - Use of “extensible knowledge blocks” in a documentation department: A large electronic manufacturer has a

documentation department that manages contractors who write and revise technical manuals. To minimize costs,

department managers adopted the strategy of using “extensible knowledge blocks” (EKBs), electronic blocks of images

and text that could be reused and extended from one product revision to the next. Contractors were rewriting the manuals

from scratch and the EKB system was barely used. The student paper recommended that the company should set aside

additional funds for contractors to encourage them to find the relevant EKBs in a proprietary company database that had

not been made available to them. The funds for searching were necessary because the EKB database contained no

indexing or cross-reference information, making it necessary to scan through the database manually to find the EKBs


that were relevant to any particular new manual. My impression was that the oversold EKB concept had seduced the

student team. The real problem seemed to be the lack of a convenient way for the contractors to obtain the electronic

versions of the existing manuals. EKBs may have been an elegant approach at one time, but a feasible implementation of

that idea had not been created. Instead of focusing on the use of EKBs, the students would have done much better to

simply ask how the work system of producing those manuals might be improved.

Example - Mobile wireless panacea for medical care: A student paper argued that mobile wireless technology and PDAs

would streamline patient care in hospitals, would drastically reduce the amount of time nurses must devote to record

keeping, would reduce error rates, and would improve communication. Although some version of these benefits will

surely occur to some extent, the students seemed to be swept away with hype about a wireless future. Their paper

mentioned “being able to electronically access data anytime, anywhere.” It claimed, “there is little end-user training

required to use this technology.” It stated that the use of “standard operating systems like Microsoft’s Windows 2000” on

laptops “reduces and, in most cases, eliminates any custom application or integration development needed to access

existing network applications.” The paper ignored many difficult issues that bedevil American medical care, including

lack of nurses, difficulties with the details of different insurance schemes, ambiguities in coding of information about

medical conditions and treatments, difficulties integrating information and processes across different departments,

unwillingness of doctors and others to change traditional work practices, difficulties with privacy and security, and

difficulties dealing with downtime and bugs if all patient information is computerized.

Example - Six Sigma: A student paper tried to explain how a six sigma process including six sigma green belt and black

belt certification for employees could be used for building information systems and applied elsewhere. Although some

six sigma ideas are useful in managing projects, one might wonder whether the students understood its applicability at a

deeper level than that of slogans. The paper says, “Six Sigma is a highly disciplined process that helps businesses focus

on developing and delivering near-perfect products and services. Sigma is a statistical term that measures how far a

given process deviates from perfection. A Six Sigma process translates into approximately 3.4 defects per 1 million

opportunities.” I doubt that any IT groups, even those at capability mature model (CMM) level 5, have attained an error

rate of 3.4 per million lines, and if they have attained it, at what cost. CMM was discussed in the class, but somehow the

promise of six sigma seemed to overwhelm the discussion of CMM costs.


Inadequate Critical Thinking

Although some student papers make cogent arguments that use concepts and information effectively, over the years

many of the shortcomings of other student papers seemed as much due to inadequate critical thinking as to inadequate

understanding of IS concepts. This is a difficult issue at the MBA and EMBA level because the students are college

graduates who presumably have learned about critical thinking and careful writing in their high school and

undergraduate work. Unfortunately, wishful thinking doesn’t solve this widespread shortcoming in educational

background. Table 1 lists common types of pitfalls related to various aspects of critical thinking. The examples following

Table 1 illustrate how these pitfalls appeared in several papers.

Table 1: Common problems related to critical thinking

Aspect of critical thinking

Common pitfalls

Defining the problem, issue, or question

Failure to clarify exactly what problem or opportunity is being pursued. Example: “We will analyze system X” [but won’t start with a problem or opportunity that guides our analysis.]

Identifying assumptions, values, and point of view

Arguing based on values and opinions without being aware that information is lacking. Example: “People need to have upbeat personalities in order to contribute.” [but some people with pessimistic personalities may also contribute]

Gathering information and evidence

Failure to gather or consider important information. Example: “The sales training was a great success” [but sales results remained inadequate]

Evaluating the quality of the evidence

Repeating quotations from individuals or published material without evaluating the source or the underlying incentives. Example: “The company produces the highest quality products at the lowest costs.” [The CEO may have said so, but that doesn’t make it true.]

Using concepts, and frameworks to shape the information and evidence

Recitation of facts without shaping them in a way that helps in interpretation. Example: “The process has 47 steps. Step 1 is …. “ [without listing every step one can say that the process is over-structured, participants have no autonomy, and management is complacent]

Drawing inferences and interpretations

Failure to explain the significance of information presented. Example: “Company X has a tradition of excellent service.” [but information presented seems to contradict that tradition]

Searching for alternatives and possibilities

Arguing for one option without ever mentioning other possibilities. Example: “We recommend that company X buy an ERP system.” [but we seem not to have considered less expensive measures that might be more practical]

Drawing conclusions or recommendations

Substituting opinions for carefully supported recommendations Example: “We believe the manager should be replaced.’ [but we haven’t explained exactly why the facts justify that conclusion]

Presenting a complete argument

Ignoring factors that most readers might wonder about. Example: “The error rate is 22%.” [and it is also noteworthy that the system of doing


the work is highly error-prone.] Avoiding internal contradictions

Saying X in one location and saying something that contradicts X somewhere else. Example: The organization’s greatest resource is its people, but the resistance to the new information system sometimes resembles sabotage. [if the people are so great, perhaps they could have negotiated instead of sabotaged]

Reducing a software firm’s travel and entertainment budget: A student team noted that a software firm’s T&E

expenditures were too high and suggested that the firm move from one travel service provider to another, thereby

reducing those expenses 35% to 50%. Although the paper mentioned some shortcomings of the current travel services,

there was no analysis of specific T&E expenses and no justification that these expenses would drop by any particular

amount. For example, if the root cause was that salespeople were taking too many trips or that they were choosing

expensive travel options, merely moving to a new travel service provider without creating new internal controls might

make little difference.

Time reporting system at a professional services firm: A student paper presented internally inconsistent views of whether

that system was successful. My comments about the paper questioned its logic, saying, “On the one hand, the

information gathered is very important for billing and for long-term revenue growth. Senior management likes the

system, thinks the information is important, uses the information, etc. On the other hand, people who enter the

information fake it at least to some extent. The degree of the faking is not estimated. What is the basis for [the paper’s

statements that] the system operates at “an impressive level?” What is so impressive? What are the implications about

the degree to which senior management really cares about this information? Without direct knowledge of the situation, it

sounds to me as though they care very little about it and are quite willing to live with faked information as long as the

faking is within reason and within budget, and as long as the company is able to produce high quality results and

maintain client relationships.”

Minimizing unnecessary expenses related to desktop printers: A large financial institution attempting to control its IT-

related costs created a system for analyzing printer usage and deciding which type of printer is appropriate in any

particular situation. The student group recommended taking printers away from non-managers but the only data

presented was that the company had too many printers in relation to the number of pages it printed. A convincing

justification would have shown that savings from reducing the number of printers would have been greater than the


additional cost of doing work less efficiently due to the need to share printers. Furthermore, in many cases the non-

managers probably needed the printers more than managers. Allocating printers based on status and rank might have

been politically expedient, but the paper did not explain why that would attain the best business results.

Producing non-standard reports for state insurance commissions: An otherwise well argued paper about a national

insurance company’s internal inefficiencies in generating certain reports stated that “agent listings are consistently

defective with 22% to 38% wrong information in each listing.” Given that this was a regulatory compliance situation

with the data being cross-checked by the states, such a high error rate seemed implausible. More explanation to describe

the nature of the errors and what is done to fix the errors before the data is submitted to the states would have clarified

what was meant and would have eliminated the suspicion that something was wrong with the analysis.

Web-based clothing retailer: The retailer had roughly 100 orders per month and a total of $50K of sales during the

previous year, but the student paper described the retailer’s market as millions of people who use the Web. It also said

that the retailer offered thousands of semi-customized product options, more than any other retailer in the same product

category within clothing. Much of the analysis seemed disconnected from reality. Almost none of the millions of Web

users knew the Web site existed, and the fact that the retailer’s competitors did not offer as many options might be

related to the impracticality of providing quick service without maintaining an excessively large inventory of clothing in

a pre-customized state. Overall, the paper provided no convincing arguments that this company had a sustainable,

profitable business model.

Suggestion that consumer-oriented agencies offer Web-based purchasing: A paper from 1997 recommended that travel

agents provide better service to their customers by offering Web-based, ATM-like kiosks that the customers could use to

cut the time to make a reservation from 53 minutes using a conventional travel agent to 3 minutes using an ATM-like

device. My response questioned how they students arrived at the 3-minute estimate and asked why the travel agents

don’t use this type of device if it could provide such a dramatic time saving. And if this type of tool were available

through the Web, why would travelers need the travel agents?


Discussion of CRM systems: Instead of analyzing a system in a particular organization, a student group wanted to write

a paper about CRM systems. At various points the paper stated that:

-- CRM is “an information technology that allows companies to manage every customer request / need in one integrated

platform.”

-- CRM is “a way to consolidate all isolated IT systems, so that they can communicate with each other.”

-- CRM is “about bringing together different areas of an organization to gain insight into customer behavior and value.”

-- CRM is “an ongoing dialogue between a company and its customers.”

-- CRM is “any customer-centric automated process that holds the customer as the cornerstone of the firm.”

Although the paper cited several long cases and many sources from the Web, some of which apparently supplied the

above statements, it did not present a good analysis of CRM because it never defined what it meant by CRM.

Difficulty Applying Abstractions and Formal Methods

Many of the topics covered in business courses related to systems are abstractions based on or related to the concepts of

system, information, and process, which themselves are abstractions. Formal methods such as data flow diagrams and

entity relationship diagrams involve the creation of abstractions using particular symbols and rules. Although abstraction

and formal methods are two of the fundamental ideas of computer science, teaching abstractions and formal methods to

non-technical MBAs and EMBAs often takes a surprising amount of time and effort. The many possible reasons for

these difficulties include lack of appreciation or interest in formal methods and a propensity to prefer concrete examples

rather than abstractions (based on the Myers-Briggs psychological type of the majority of MBA students).

An example of a comparatively simple abstraction is the work system framework that forms the basis of the work system

method, various versions of which were used in the student papers. The first step in using the work system method is to

summarize the situation in terms of the nine elements. The substantial challenges of defining a system were mentioned

earlier, but even applications of the individual terms in the model are sometimes problematic. In a previously mentioned

student paper about pharmaceutical sales, the students identified physicians as the customers of the work system but

didn’t mention the sales managers as customers even though it was sales managers who benefited most directly from a

somewhat controversial information system for collecting data about physician visits. Similarly, an analysis of how a


small manufacturer granted credit to its customers identified commercial credit reports as the product of the system.

Those commercial credit reports were the products of organizations in the business of producing those reports, but for

the work system of granting credit, the credit reports should be treated as information used by the business process.

Formal methods that use precise notations present a deeper form of abstraction because the user is expected to apply

particular concepts to create an abstraction and then document the abstraction using a particular “grammar.” [8] A

number of researchers have studied the difficulties related to using ERDs and other conceptual modeling methods. [8].

In introductory IS courses most MBAs and EMBAs quickly grasp the way an ERD describes the logical structure of a

relational database, but without substantial practice they often have difficulty creating meaningful ERDs. The earlier

group papers in the sample contained at least five examples of DFDs and ERDs that demonstrated a poor understanding

of what the tools were attempting to accomplish. Around four years ago I decided to cover ERDs in database examples

and classroom discussions but to stop asking the students to produce ERDs in their group papers because attaining good

results would have required three or four times the amount of class time that I believed the topic deserved.

4. Discussion and Conclusions

This paper has cited 30 examples that illustrate various types of pitfalls encountered by teams of evening MBA and

EMBA students attempting to analyze real world systems in the organizations they work in. At least half the students

involved in these papers had five or more years of business experience and are therefore reasonably representative of the

types of individuals who may need to make decisions about systems in organizations or who may serve as user

representatives on committees devoted to building or maintaining systems in organizations. As a representation of the

categories and extent of the difficulties that typical business professionals encounter the examples presented here are

actually just the tip of the iceberg. All of these examples come from students who are ambitious enough to study for an

MBA, who are part way through a course in information systems, and who have received some form of feedback about

likely problems related to a proposed topic for a group paper. Typical business professionals attempting to understand a

system by themselves or attempting to collaborate with IT professionals do not have the same advantages, but will still

have to deal with the same pitfalls and perhaps others.


The examination of 202 papers began with a set of categories for pitfalls, and examples in all of the categories were

found. The specific categories included difficulty defining systems in organizations, difficulty identifying the

information in an information system, difficulty using measures of performance, reluctance to mention organizational

and personal issues, susceptibility to techno-hype and jargon, inadequate critical thinking, and difficulty applying

abstractions and formal methods. From the outset I was certain that I had encountered all of these difficulties in the

original proposals for topics, but I was not sure whether my feedback to the teams about likely problems would filter out

certain categories of pitfalls that therefore would not appear in the final papers. Pitfalls in each category did appear, but

my impression is that the frequency and severity of the pitfalls was somewhat reduced. Notice how only some of the

pitfalls are addressed either in typical information system courses or in systems analysis and design texts.

Difficulty defining systems in organizations: Systems analysis authors almost always say that it is necessary to define a

system before analyzing it, but this is easier said than done. It is interesting that the most experienced student teams

tended to report longer and more contentious debates about the identity and boundaries of the system they should analyze

in order to produce recommendations related to a specific real world problem or opportunity. The less experienced

student teams sometimes seemed not to realize why this was an important issue. In both cases, as the student teams

progressed with the analysis they often found that the original definition of the system was inadequate. In some cases

they even redefined the problem they were trying to solve.

Difficulty identifying the information in an information system: Information system courses and systems analysis

methods also place substantial emphasis on databases and data definitions. The findings from this study indicate that

MBA and EMBA students who can have seen demonstrations of relational databases may still tend to be vague about

data requirements and may have difficulty using ERDs to describe the data requirements of a new situation.

Difficulty using measures of performance, reluctance to mention organizational and personal issues, and susceptibility

to techno-hype and jargon: Information system courses and systems analysis methods often do not put much emphasis

on measures of performance, full engagement with human and organizational issues, and vulnerability to techno-hype

and jargon. Formal analysis methods for IS provide rigorous techniques for defining data but provide sketchy guidance

or none at all for dealing with human and organizational issues. Information systems courses and systems analysis


methods usually promote the use of measures of performance, but guidance about which measures of performance to

consider is often sketchy at best. (In contrast, the work system method used by the students who wrote these papers

encourages them to consider at least several measures of performance for each work system element.) Information

system courses may or may not warn students about techno-hype and the excesses of jargon; in some cases the courses

themselves seem to promote techno-hype by implying that the newest and greatest technology is important for success or

that the ability to use fancy jargon is impressive.

Inadequate critical thinking and difficulty using abstractions: Issues about critical thinking and difficulty in using

abstractions and formal methods are especially troublesome because they fall outside of the purview of the IS field, yet

are a major determinant of whether students produce logical papers with well-supported recommendations. There are

times when I wonder whether half or more of the quality of most papers is the result of good critical thinking rather than

of mastery of specific system-related topics. It is easy for professors to avoid this issue by grading through tests or other

exercises that have a demonstrably correct answer and therefore do not exercise critical thinking to the same extent. I

take my chances with relatively open-ended assignments because I believe these assignments present students with a

more realistic replica of the challenges they will face in their jobs.

Implications for Action

Assuming that the categories of pitfalls illustrated here are common, what should be done about them? The first thing to

do is to make sure that typical MBA courses about information systems address these issues. Table 2 identifies some of

the possible approaches that can be used in relation to each area of difficulty. Some of these suggestions may seem a bit

basic, and may not seem as exciting as talking about the strategic uses of information technology, the impacts of

technology on society, or the new hot technologies that may become commonplace soon. On the other hand, focusing on

basic topics such these may help the students acquire or extend what is probably most valuable for them, an ability to

participate fully in the analysis, design, implementation, and operation of systems in organizations.

Table 2: How typical MBA courses in information systems might address common pitfalls

Category of pitfall How an MBA course might address this pitfall Difficulty defining systems in organizations

• Provide exercises in defining systems. For example, give the students a situation and a problem or opportunity. Ask them to identify two possible views of what the


relevant system is, and to explain why one view might be preferred. • Have the students write real world case studies as projects.

Difficulty identifying the information in an information system

• Provide examples of relational databases within Microsoft Access or another DBMS. View the ERD. • Allow the students to perform transactions to modify the data on pre-existing database examples; perform queries and ask the students about the role of the database’s structure

Aversion to using measures of performance

• Provide examples of situations in which many different measures of performance might apply. Ask the students to identify several plausible measures of performance for different elements of a system, and then compare the answers. • Ask the students to identify three or more important measures of performance for work systems they have encountered in their lives.

Reluctance to mention organizational and personal issues

• Provide examples that illustrate why systems in organizations are more than just technology, information, and business process. • Provide numerous examples in which system participants might have personal incentives contrary to the goals of management • Ask the students to identify systems they encountered in which system participants might have personal incentives contrary to the goals of management

Susceptibility to techno-hype and jargon

• Provide examples showing that the same jargon term can mean many different things to different people at different times. • Provide examples that make fun of overblown jargon. One I use is called “Hyper-competitive global empowerment-speak.”

Inadequate critical thinking

• Present examples such as those in Table 1 and encourage students to think about those examples as they write their papers • Provide a refresher about critical thinking at the beginning of an MBA program or in a communication class.

Difficulty applying abstractions and formal methods

• Demonstrate abstractions and formal methods using realistic examples • Ask students to criticize examples of DFDs or ERDs that may contain incorrect descriptions (e.g., is the 1:1 relationship between professors and offices correct?) • Consider the possibility that most business professionals will not find highly abstract methods very useful in their own work.

The individual suggestions in Table 2 address particular issues but do not directly address the larger question of how

business professionals should be able to avoid these pitfalls and should be able to understand systems more completely,

perform a preliminary analysis of a system by themselves, and communicate more effectively with IT professionals.

Each of these topics should be the subject of future research because the stakes are so high in terms the unacceptable rate

of system-related disappointments and system failure.

The underlying question might be viewed as a methods and quality control issue. Assume that business professionals do

not have the types of training that IT professionals have. Assume that they are not uniformly skilled in critical thinking.

Assume that they have many unrealistic beliefs about technology. Even with all of these assumptions it might be possible

to help business professionals by providing organized methods that guide them toward a better understanding and better

analysis. Such methods might incorporate paper or computer-based tools for producing preliminary definitions of


systems. They might incorporate system principles in a way that would help in identifying issues and in recognizing that

changes in one part of a system could affect other parts of the system. They might incorporate system development and

system implementation principles that would help in identify likely pitfalls in the process of creating or improving a

system. Whether business professionals (or MBA or EMBA students) would seek such knowledge or methods is

debatable. However, given the importance of systems in organizations, the unacceptable rates of disappointment and

failure, and the pitfalls that business professionals currently encounter, progress in these directions is of great

importance.

References

[1] Johnson, J. K.D. Boucher, K. Connors, and J. Robinson (2001) “Collaborating on Project Success,” Software Magazine, Sponsored Supplement, February 2001, http://www.softwaremag.com/L.cfm?Doc=archive/2001feb/CollaborativeMgt.html viewed on May 6, 2003 [2] Fichman, R.G. and C. F. Kemerer (1999) “The Illusory Diffusion of Innovation: An Examination of Assimilation Gaps,” Information Systems Research, (10:3), Sept. 1999, 255-275. [3] Alter, S. (2002a) Information Systems: Foundation of E-Business, 4th ed., Upper Saddle River, NJ: Prentice-Hall, 2002. [4] Alter, S. (2002b) “The Work System Method for Understanding Information Systems and Information Systems Research”, Communications of the AIS (9:6), pp. 90-104 [5] Alter, S. (2002c) “The Collaboration Triangle”, CIO Insight, pp. 21-26. [6] Reimus, B. (1997) “The IT System that Couldn’t Deliver,” Harvard Business Review, May-June 1997 [7] Alter, S. (2003). “Pervasive Real-Time IT as a Disruptive Technology for the IS Field,” Proceedings of HICSS-36, The 36th Annual Hawaii International Conference on System Sciences. [8] Wand, Y. and R. Weber (2002) “Research Commentary: Information Systems and Conceptual Modeling – A Research Agenda,” Information Systems Research, (13:4), December 2002, pp. 363-376.


http://www.softwaremag.com/L.cfm?Doc=archive/2001feb/CollaborativeMgt.html

Documents

Pitfalls in Analyzing Systems in Organizations · PDF filePitfalls in Analyzing Systems in ... Pitfalls in Analyzing Systems in Organizations ... the major types of pitfalls demonstrated