An Integrated

8/2/2019 An Integrated

http://slidepdf.com/reader/full/an-integrated 1/14

An integrated evaluationsystem based on thecontinuous improvement

model of IS performanceChoon Seong Leem and

Injoo Kim

1. Introduction

Recently, the importance of Information

Systems (IS) is being rapidly increased as a key

strategic mean promoting the efficiency of

enterprise activity. According to dramatic

progresses of info-technology, the typical usersof IS are expected to use various applications in

dynamic enterprise environments.

Furthermore, most enterprises pursue the

renovation of business process and strategies

through IS.

In order to adequately response these trends,

enterprises have to establish comprehensive

concepts and goals based on evolutionary

characteristics of IS and to identify their

objectives from the continuous evaluation of

current IS conditions by a scientific and

systemic methodology.

In spite of these needs, previous researches

have not been very successful for identifying

accurate problems based on the evaluation of

performance maturity of IS. Additionally, as

they dealt with conceptual evaluation

frameworks and focused on fragmentary

evaluation areas, it is difficult to figure out the

detailed evaluation results of IS for the entire

scope of enterprise.

This paper examines the evaluation issues of

enterprise IS performance dealing with:

(1) suggestions for the performance

improvement model based on the

evolutionary characteristics of IS;

(2) development of an integrated evaluation

system based on the improvement model;

and

(3) verification of efficiency and applicability of

the evaluation system through a large-scale

case study.

2. Previous research

This work focuses on improvement of IS

performance by systematic evaluation

methodology. Previous researches can be

classified into two types regarding improvement

models and evaluation models of IS

performance. Also, the researches related to the

evaluation models concern three kinds of topic

which are evaluation model, evaluation fields,

and evaluation items of IS performance.

The authors

Choon Seong Leem is a Professor at the School of

Computer and Industrial Engineering, Yonsei University,

Seoul, Korea.Injoo Kim is a Research Fellow at the Naval Forces

Development Command, Kyeunggi, Korea

Keywords

Continuous improvement, Performance management,

Modelling, Information systems

Democratic People's Republic of Korea

Abstract

In order to achieve competitive business goals, every

enterprise needs to evaluate the current level of information

systems performance and their utilization. The evaluation

measures the technical capacity and operational capability of

enterprise information systems and diagnoses their

effectiveness in business goals and efficiency in resources.

An integrated evaluation system is developed based on the

continuous improvement model of information systems

performance. The system has been applied to performance

measurement of information systems with a huge set of data

from Korean industries, and proven reliable and practical

Electronic accessThe Emerald Research Register for this journal is available at

www.emeraldinsight.com/researchregister

The current issue and full text archive of this journal is

available at

www.emeraldinsight.com/0263-5577.htm

115

Industrial Management & Data Systems

Volume 104 . Number 2 . 2004 . pp. 115-128

# Emerald Group Publishing Limited . ISSN 0263-5577

DOI 10.1108/02635570410522080



2.1 Previous researches on the

improvement model of IS performance

There are two types of researches related to

improvement of IS performance. The one is on

improvement processes and the other is on

improvement stages of IS performance.

Plan-Do-Check-Act (PDCA), Initiating-Diagnosing-Establishing-Acting-Learning

(IDEAL), and Quality Improvement Paradigm

(QIP) are typical researches on improvement

process. The PDCA initialized by Shewhart

(1931) and generalized by Deming (1986) after

the Second World War is the improvement

process of product quality based on feedback

cycle that can optimize unit production process.

The QIP by NASA Software Engineering

Laboratory is the improvement process of

software quality based on the meta-lifecycle

model to improve long term quality. This

process has several functions; packing,

assessing, and increasing comprehension of

development experience for software. The

IDEAL by Software Engineering Institute (SEI)

in the Carnegie Mellon University is the process

improvement model focused on project

management. This model is composed of five

steps that are continuously and recursively

performed (McFeeley, 1996). The kaizen

model to improve the process performance has

been applied to the ESPRIT project. The basic

concept of this model is called `àdoption curve''

to take up new technology which is proposed by

Conner and Patterson. Table I briefly

summarize these researches (see Renaissance

Consortium, 1997)

Also, there are several researches on

improvement stages of IS performance. Nolan

and Wetherbe (1980) suggested six maturity

stages of IS focused on data, and Venkatraman

(1997) also proposed a five stage model focused

on structure innovation of organization by IS.

Vernadt (1996) presented a three stage modelof systems integration according to expansion of

the Computer Integrated Manufacturing

(CIM) integration range. The Capability

Maturity Model (CMM) by SEI is composed of

five stages derived from the degree of process

maturity (Bate et al.,1995). The Information

Systems Management (ISM) model by Tan

(1999) is based on balance between

organizational structure and IT components.

This model is originated from MIT90s

framework that is composed of the levels of

IT-enabled business reconfiguration by

Venkatraman. In this model, IS fields are

divided into three parts which are external

environments, organization environments, and

IS environments. Table II shows the researches

related to improvement stages of IS

performance.

2.2 Previous researches on the evaluation

model of IS performance

The evaluation diagnoses the current condition,

and utilizes its results for future plans, so that

the organization could get the better

performance. For instance, the Japanese

Deming prize, USAs Malcolm Baldrige Award

called ``criteria for performance excellence'',

and European's ``Business Excellence model''

are known to significantly contribute to quality

improvement of products and process. Also, theTable I Researches on important processes of IS performance

Title Improvement processes Focus

PDCA Plan ± do ± check ± act Product quality improvement

QIP Characterize the environment ± set

goals ± choose and tailor a processmodel ± execute the process ±

analyze the collected data ± learn

and feedback

S/W quality improvement

IDEAL Initiating ± diagnosing ±

establishing ± acting ± leveraging or

learning

Process improvement

Kaizen Contact ± awareness ±

understanding ± evaluation ± trial

use ± adoption ± institutionalization

New technology adoption

Table II Researches on improvement stages of IS performance

Title Improvement stages Focus

Nolan Initiation ± contagion ± control ± integration ± dataadministration ± maturity

Data

CIM Physical system integration ± application integration ±

business integration

System

ISM Functional integration ± cross-functional integration ±

process integration ± business process redesign ±

business redesign or business scope redefinition

Business

CMM Performed informally (initial) ± planned and tracked ±

well-defined ± quantitatively controlled ± continuously

improving

Process

116

An integrated evaluation system

Choon Seong Leem and Injoo Kim


Volume 104 . Number 2 . 2004 . 115-128



USA, the UK, Japan and OECD are

continuously working out the national IS

indices, so as to gradually increase the level of

IS performance (Jeong, 1996).

In the research related to the evaluation

model of IS, the DeLone and McLean's (1992)

IS success model based on the works byShannon and Weaver (1949) and Mason

(1978) is well known. This model is examined

and improved by Seddon and Kiew (1994)

which suggests the measures of six fields and

proves their appropriateness. Since the IS

model did not cover the appropriate measures

coincided with the characteristics of

organization, Saunders and Jones (1992)

developed the `ÌS function performance

evaluation model'' which encompasses a

selection method of appropriate measurescorresponding to organization features. Myers

et al. (1997) worked out the ``Comprehensive IS

assessment model'' that expanded the six

evaluation fields of DeLone and McLean's

model into eight fields and combined these

fields into organizational and external

environments. Also, Goodhue and Thompson

(1995) and Goodhue(1998) proposed the

Task-to-Performance Chain (TPC) model

based on a fitting technique into individual

performance. The focus of the model is to apply

the technique to individual tasks to calculate

their positive impact on individual

performance. Additionally, there are several

researches related to IS framework. Tan (1999)

suggested the ``Consistency model'' composed

of seven components that expanded the

MIT90s model and SEI also proposed a

framework composed of seven evaluation fields

(Bergey et al., 1997).

As researches related to identification of the

evaluation items, the Goal-Question-

(indicator)-Measures (GQM) methodology wasintroduced by Basili and Rombach (1988),

refined by AMI (1992), Pulford et al. (1996) in

ESPRIT project, and was applied to the

goal-driven software evaluation by Park (1996)

in SEI. Especially, Mendonca (1997, 1998)

converted the GQM to another

Goal-Question-Metric for improvement of

evaluation processes. Sometimes researches

related to the evaluation model of IS

performance (see Table III).

2.3 Limits of previous works

Although previous researches addressed IS

framework, performance improvement,

evaluation processes and evaluation items, but

their theoretical models may not be easily

applied to real situations. Their inherent

limitations are as followed.First, while the improvement of IS

performance has to be continuously pursued

from IS strategy planning to construction and

maintenance, previous researches have a

tendency to workout only for limited parts of

the whole.

Second, although the evaluation model of IS

performance put emphasis on identification of

the evaluation items and investigation of their

relationships, but they are not successful in

embodying detailed and protocol evaluation

procedures.

Third, usual IS frameworks do not address

the relationship between IS evaluations and

their business implications. As a result, it is

difficult to use the evaluation results directly to

any related business decision-makings.

In order to overcome these limits of previous

works, this paper proposes an improvement

model of IS performance and develops

corresponding evaluation system which can

estimate the current IS status and feedback the

results into IS construction process. The

improvement model is described in the

following section 3 and the integrated

evaluation system is explained in section 4. A

case study with real data from 219 Korean

leading companies in section 5 proves the

practical values of the model and system.

3. The improvement model of ISperformance

This paper defines the IS as integrated systemsthat collect data, analyze that, generate the new

useful information, transmit it and use

information related with business activities in

organizations, typically business process in

enterprises.

``The IS performance'' that is usually divided

into several stages is defined as the degree of

effectiveness and efficiency in business goal

accomplishment by IS. The `Ìmprovement of

IS performance'' implies that the IS

117




Volume 104 . Number 2 . 2004 . 115-128



performance is improved to become flexibly

commensurate with changes in internal and

external environments and various

requirements of users, so that the IS

performance can be optimized with activities in

organization.

``The improvement model of ISperformance'' is a representation of their

relationships. This paper presents the

improvement model of IS performance, which

consists of improvement stages and cycles.

3.1 Improvement stage of IS performance

The improvement stage of IS performance

plays major role in overall evaluation of IS

performance. The improvement stage is

suggested to consist of five stages and Figure 1

shows them.

As shown in Figure 1, the five improvement

stage of IS performance in this research are

function integration, process integration,

business integration, industry integration, and

role-model generation. The level of the stage

can be determined by the six comprehensive

fields of IS performance which are vision,

organization and institution, infrastructure,

supporting, application, and usage of IS. The

``function integration'' represents to

computerize the individual tasks within isolated

systems The ``process integration'' combines

the individual processes and functions into

corresponding working group via IS. The

``business integration'' is defined to integrate

the working groups into the level of entire

organization, and the `ìndustry integration''should be cover up to partner companies and,

individual customers, outside the organization.

In the ``role model generation'' stage, the

organization can flexibly accommodate to new

external environment by itself and naturally

create new business models by accumulated

information and updated IS.

The improvement stage of IS performance

has important meanings that can quantitatively

represent the current IS status and target IS

status in future. Seeing that the IS

environments have many diverse qualitative

factors and these factors are tangled with each

other, it is very difficult for organization to

decide level of the stages for current IS status

or target IS status. Therefore, in order to

decide the stage correctly, these stages should

be characterized and explained by various

factors. This paper suggests these decision

factors based on the IS framework that are

divided into six fields; IS vision, IS organization

Table III Researches on the evaluation model of IS performance

Category Researcher Key elements

Evaluation model Shannon and Weaver Technical level ± semantic level ± effectiveness or influence

Mason Production ± product ± receipt ± influence on recipient ±

influence on system

DeLone and McLean System quality ± information quality ± use ± user satisfaction ±individual impact ± organization impact

Barry L. Myers Service quality ± system quality ± information quality ± use ±

user satisfaction ± individual impact ± workgroup impact ±

organization impact

Goodhue and Thompson Task characteristics ± technology characteristics ± individual

characteristics ± task-technology fits ± utilization ± performance

impact

IS framework MIT90S Strategy ± structure ± technology ± people ± management

Consistency model IS strategy ± information systems ± users ± data and technical

infrastructure ± IS organization ± IT specialists ± IS management

SEI framework Organization ± project ± legacy systems ± system engineering ±

software engineering ± technology ± target systems

Identifying process of

evaluation factors

GQM Goal ± question ± indicator ± measures

Mondonca's GQM Capture data user's goals ± identify relevant entities ± identify

relevant attributes ± map attributes to existing metrics

118




Volume 104 . Number 2 . 2004 . 115-128



and institution, Infrastructure, supporting,

application, and usage. Table IV summarizes

these factors.

3.2 Improvement cycle of IS performanceThe improvement model of IS performance in

this paper consists of three components:

improvement stages, integrated evaluation

system, and construction process, and should

be applied by five continuous and circular

cycles; initiation, goal establishment, diagnosis

and evaluation of IS performance, constructionprocess, and leveraging and learning. Figure 2

shows the cycle.

Figure 1 Five improvement stages of IS performance

Table IV Decision factors for improvement stages of IS performance

Classification Function integration Process integration Business integration Industry integration Role-model generatio

IS vision No or low level ISP

Low level project control

Informally performed

process

Basic ISP

Short-term construction

plan

Planned and tracked

process

High level ISP

Mid/long-term

construction plan

Fragmentary product

management

Well-defined process

Circulatory ISP

management

Circulatory project

management

Enterprise product

management

Quantitatively controlledprocess

Circulatory ISP

management

Circulatory project

management

Enterprise product

management

Continuous creation of optimized process

IS organization and

institution

Basic computerization

organization

Basic IS organization

Basic IS rules, institutions, and directions

Enhancement of IS education and expansion of

IS minds

Specialized and detailed IS organization

Settlement of IS rules, institutions, and directions

Infrastructure Stand-only system

Platform for each

function (individual tasks)

Network for units

Platform for working

groups

Standardization

Network for enterprise

Platform for enterprise

Integrated DB

Global network for

companies and customers

Platform for enterprise

Integrated DB

Supporting Development of simple

application

Establishment of supporting systems

Usage of CASE toolsEnhancement of user services

Maintenance of management supporting systems

Automated reflection of user requirements

Application Computerization of

individual tasks

Office automation

IS of working groups

Data sharing

Supporting of decision

making

ERP

Information sharing

MIS

EC, CALS,

Knowledge sharing

CRM, SCM

New knowledge creatio

New business strategy

creation

Usage Intermittent usage

(individual usage)

Uncontrolled participation

Partially indispensable

usage (group)

Controlled participation

Mostly indispensable

usage (enterprise)

Controlled/self-sustained

participation

Entirely indispensable usage

Fully self-sustained participation

119




Volume 104 . Number 2 . 2004 . 115-128



As shown in Figure 2, the improvement of IS

performance can be achieved by five processes.

First, the motive to improve IS performance is

triggered by stimulus originated from changes

in internal and external environment. Second,

the organization should establish the goal (IS

vision) that can flexibly cope with the trends of

IS environment. Third, the organization should

evaluate the current IS status, identify future

objectives, and analyze the gap through the

comparison between goal states and current

states. Fourth, detailed problems in current

states should be considered in planning and

construction of IS projects. Finally, information

and knowledge acquired from previous

processes should be utilized with recursive

iterations of the cycle, the IS environments can

be continuously reconciled with management

environments of the organization.

Figure 3 shows the detailed processes of

improving IS performance in parallel with the

IS construction steps; planning (ISP), analysis,

design, construction, and maintenance.

4. The integrated evaluation system of ISperformance

The integrated evaluation system of IS

performance is designed to diagnose the current

IS status, and identify the deficiencies of

current status for target systems by gap analysis.

This system consists of three parts; evaluation

procedures, evaluation fields, and evaluation

methods. The evaluation procedures can be

decomposed into five steps; preparation,

measurement, analysis, interpretation, and

feedback. The evaluation fields which are

originated from IS framework can be

decomposed into three parts; measurement

factors, influence factors, and evaluation

factors. The measurement factors mean the

static standpoint of IS framework, the influence

factors mean the dynamic standpoint that

represents the relationship between subject and

object in IS framework, and the evaluation

factors are considered to supply useful

information to decision-makers. These factors

are measured, analyzed, and interpreted by

various evaluation methods. Figure 4 shows a

schematic diagram of the integrated evaluation

system of IS performance.

4.1 Evaluation procedures

The evaluation of IS performance is to measure

current IS performance, to analyze the

measured data, to interpret the analysis result,

and to return these outcomes to the

improvement model and construction process,

so that the current IS performance can beevolved into target stage. The evaluation

procedure is defined as a series of sequential

actions that should continuously perform these

evaluating activities efficiently. This paper

presents five practical steps to evaluate the IS

performance; preparation, measurement,

analysis, interpretation, and feedback which are

originated from human problem solving

processes. Also the order of evaluation

procedures is made up by the relationship

Figure 2 Improvement cycles of IS performance

120




Volume 104 . Number 2 . 2004 . 115-128



between input and output of each steps.

Although these steps could be decomposed by

deeper, this paper introduces second sub-level

of the steps. Figure 5 shows the evaluation

procedures for IS performance.

4.2 Evaluation framework and perspectives

In order to achieve business goals using IS, IS

ought to be considered as a part of entire

organization, and the evaluation framework of

IS performance should be accurately defined

with respect to the organization.

This paper presents an integrated evaluation

framework, as a part of improvement model of

IS performance, that can comprehensively

represents entire IS environment as a part of

real world. Figure 6 shows this framework.

Figure 6 illustrates the integrated evaluation

framework embodying all the measures of IS

performance. This framework contains static

and dynamic perspectives. The static perspective

of IS environments consists of external

environment, enterprise environment and IS

environment while the dynamic perspective of

improvement process implies the relationships

between activities being continued from legacy

IS (level n) to target IS (level n + 1) by various

projects. In addition to the two perspectives, this

paper considers the evaluation perspective of

manager executing the project. Now the

framework is extended with three perspectives

that are measurement (static perspective),

interpretation (dynamic perspective), and

evaluation (evaluation perspective).

``Measurement perspective'' of IS framework

views the components of IS framework as static

ones points, so that organization can measure

the level of IS assets and the degree of IS

performance. In order to measure IS assets of

organization, the IS framework in this

standpoint should be gradually decomposed

into measurement items. Figure 7 shows the IS

framework with measurement perspective

which is composed of three-level subject areas.

Figure 3 Detailed processes of improving IS performance

Figure 4 Integrated evaluation system of IS performance

121




Volume 104 . Number 2 . 2004 . 115-128



`Ìnterpretation perspective'' of IS framework

means the dynamic perspective regarding how

the IS environment efficiently contributes to

business activities of organization. The

evaluation factors in this standpoint are

relationships between subjects and objects in IS

framework. Figure 8 shows the interpretation

perspective of IS framework that is composed of

capacity, capability, and performance. The

capacity of IS means physical characteristics of

components in the IS area such as speed of

CPU and expertise of provider.

The capability means the ability to influence

the business area with capacity. The

performance denotes the realization of

improvement of enterprise competitiveness via

the IS environment. As shown in Figure 8,

measurement process has to be orderly

progressed from capacity to performance, and

analysis process of these measurements has to

be reversibly progressed.

Figure 8 Interpretation perspective of IS

framework `èvaluation perspective'' of IS

framework means the classification of

Figure 5 Evaluation procedures of IS performance

Figure 6 Integrated evaluation framework of IS performance

122




Volume 104 . Number 2 . 2004 . 115-128



evaluation items and factors to manager's

interests, so as to help decision making. This

perspective is utilized as a criterion to determine

an improvement stage, and is decomposed into

six fields; IS vision, IS organization and

institution, IS infrastructure, IS supporting, IS

application, and IS usage. Consequently, the

manager or decision-maker can get the practical

information using the evaluation perspective.

Table V shows the descriptions of six fields of

IS performance in this perspective.

4.3 Evaluation fields, factors, and itemsFor characterizing of IS framework, its

construction structure that converts the

conceptual IS framework to measurable items

should be defined holistically. Although the

GQM method can be considered, it is difficult

to identify input, output, and relationships

between components. Thus, it is not easy for an

evaluator to practically apply this method to

real world. This paper presents the

characterization structure of IS framework with

three hierarchical components; field, factor, and

item. In the characterization structure, the

evaluation fields are decomposed into

evaluation factors with respect to the dimension

subjects (who), objects (what), and

relationships between three perspectives.

Table VI summarizes the derived evaluation

factors for each evaluation field.

The evaluation items can be identified from

the characteristics of evaluation factors, and

these factors may be measured by measurable

items to represent them. According to the

characterization structure, this work identifies

345 measurable items which are consist of 46

items in IS vision, 80 items in IS infrastructure,

88 items in IS organization and institution, 22

items in IS supporting, 61 items in IS

application, and 48 items in IS usage.

4.4 Integrated evaluation system

An integrated evaluation system should have

various functions that are data collection,

measurement, analysis, interpretation, and

feedback. Table VII represents these functions

of the evaluation system.

In order to pursue automation of these

evaluation functions, the conceptual framework

of integrated evaluation system as is designed

by Figure 9.

5. Case study

In order to verify adoptability and practicality of

the proposed integrated evaluation system, a

case study has been performed with the support

from Korean Ministry of Information and

Communication. The system has been applied

to the 219 leading companies in Korea using

the 345 measurable items in six fields of IS

performance for two years from 1999 to 2000.According to the study, the enterprises could

get many useful evaluation results of IS

performance. Also, a company could find its

relative status of IS performance and assets

comparing with in the same industrial sector.

With this information of evaluation, these

enterprises have been excitedly motivated to

improve their IS performance. The results of

the case study in concert with the goals of this

paper are briefed as followed.

Figure 7 Measurement perspective of IS framework

Figure 8 Interpretation perspective of IS framework

123




Volume 104 . Number 2 . 2004 . 115-128



Table V Descriptions of six fields of IS performance in evaluation perspective

Fields Definition and meaning

IS vision Degree of IS vision's contents, scope, and fidelity in concert with business goal

Proportion schemes in keeping with management objectives of organization

IS infrastructure Function, performance, and retention status of IS infrastructure (facilities, H/W, S/W, DB,

network, and CASE tools for system development, etc.)Optimum level of IS infrastructure in concert with application

IS organization and

institution

Optimum level of organization, rules, institutions, and direction for IS usage (IS planning/

development/maintenance team, CIO, training system, information resource management

system, maintenance system, and so on)

Degree of adoption and usage of them

IS supporting Degree of retention and specialty for personnel supporting IS development, maintenance,

management

Degree of activities for these organizations and personnel

IS application Optimum level of application in concert with business

Degree of adoption for new technology

Degree of effectively cooperating with management strategy and business processIS usage Specialty level of general users

Degree of usage for IS application in keeping with their task

Table VI Evaluation fields and factors of IS performance

Level Subject (who) Object (what) Relationship (how)

IS vision ISP

IS project plan

Business goal

Business process

External environment (legal, technical)

Available resources

Available time

Comprehensibility

Suitability

Availability

IS infrastructure H/W

Database

Network

Supporting tools

Application

User

IS provider

Sufficiency

Suitability

Usability

Satisfactory

IS organization and

institution

IS organization

IS institution/rule/direction

IS mind

Application

Infrastructure (H/W, S/W, N/W)

User

IS provider

Adoptability

Comprehensibility

Supportability

IS supporting IS supporting team Infrastructure (H/W, S/W, N/W)Supporting tool

Application

User

SupportabilityUsability

IS application Application

Information for service

Business process

User

Comprehensibility

Degree of integration

Suitability

Satisfactory

IS usage User Application

IS institution/rule/direction

Usability

Satisfactory

124




Volume 104 . Number 2 . 2004 . 115-128



As shown in Figure 10, the evaluation results in

year 2000 show that the most companies in

Korea are staying at the ``business integration''

stages. While 14 percent stays at the ``process

integration'' stage, only 4 percent barely enter

into the ``role model generation'' stage. The

comparison between years 1999 and 2000shows that not a few companies in 1999 have

experienced improvement to stand at higher

stages in 2000. That is, the companies of

``function integration'', ``process integration'',

or ``business integration'' stage in 1999 have

been respectively improved into ``business

integration'' or `ìndustry integration'' stages in

2000. Also, the distribution of IS level in 2000

has been compressed against 1999, which it

means that the gap between advanced

companies and underdeveloped companies is

gradually reduced. However, only 1 percent in

the `ìndustry integration'' stages in 1999 has

been evolved into ``role model generation'' in

2000, it implies that it is more difficult for the

higher level companies to improve their

performance.As shown in Figure 11, IS performance in

most evaluation fields are reached about 60

percent, while the IS application and

supporting fields stands about 50 percent in

2000. That is, although the increased interests

in IS lead these companies to design their IS

vision, to use existing IS, and to review their

organization and institutions, they do not

practically accomplish applying IS technology

as well as developing the business application.

Table VII Function for integrated evaluation system

Function Sub-function

Data collection Questionnaire generation, measurable item identification and storage

Interview material generation, interview item identification and storage

Existing document survey and storage

Measurement Scale (nominal, ordinal interval, ratio) generation and matching it to questionnaireYardstick generation for each scale

Item measurement, scoring and data storage

Analysis Weight generation

Score accumulation for each field and level and ordering

Calculations for various useful information

Interpretation Comparison, gap analysis

Feedback Flowing back-gap analysis results into continuous improvement process

Figure 9 A conceptual design of the integrated evaluation system

125




Volume 104 . Number 2 . 2004 . 115-128



Also, they do not keep efficient maintenance for

their IS resources.

In Figure 12, the results of comparison

between years 1999 and 2000 show that the

increasing trend of IS performance is the

highest in manufacturing and construction

companies, while the lowest in upper 30

companies in performance. This implies that

the companies with higher IS performance

levels has rather slower speed of improvement.

The above phenomenon appears that the

improvement in small and medium-sized

enterprises, often called SMEs, are more

notable them in large-scale companies.

Figure 13 represents a list of focused IS

indices and their results in year 2000. Not a few

interesting IS trends in Korea can be found. For

example, while internet and intranet are actively

used, corresponding business applications are

not activated relatively, and especially, IT

outsourcing barely activated in 2000, Korea.

Figure 14 shows the evaluation results of IS

performance level and improvement stage for a

logistics service company K. The K company's

IS level in 2000 is considerably increased. The

performance levels of IS support and IS usage

fields are significantly improved, although the

levels of IS infrastructure and application

approach to a standstill. The company can

utilize the evaluation results to improve its IS

performance in balance and to reach the

role-model generation stage.

By this case study involving the 219 Korean

companies during two years from 1999 to 2000,

the proposed the integrated evaluation system

of IS performance has been proven adoptable

and practical. Not only company K, every

company participated in the case study received

its own evaluation results of the 345

measurement items with detailed explanations.

As the integrated evaluation system gave the

quantitative results of company's IS

performance, the managers or decision-makers

are expected to use them for right alternatives

and correct directions on there is strategy.

Figure 10 Level of IS performance based on the improvement stage Figure 12 Comparison of IS level between years 1999 and 2000

Figure 11 IS performance in evaluation fields

126




Volume 104 . Number 2 . 2004 . 115-128



6. Conclusion

In order to improve IS performance, this paper

develops an integrated evaluation system that

can diagnose the performance level, identify the

deficiencies of current status, and return this

information into improvement processes with a

corresponding model. The improvement model

and the integrated evaluation system of IS

performance are tested with a large-scale case

study to prove their adoptability and efficiency.

The work offers several benefits; some of

which are as follows. First, it defines the

concepts of IS performance improvement.

Second, it expands the IS framework

comprehensively via systematic translation of

the conceptual framework into measurable

items. Third, it supports practical evaluation

tools. In spite of these benefits, as every

organization has its own business scope,

characteristics, and cultures, it is recommended

to apply this system with reasonable

customizations. Therefore, the proposed system

ought to be continuously revised and updated,

so as to fit into various organization types and

manager's needs.

References

Basili, V.R. and Rombach, H.D. (1988), ``The TAME project:

toward improvement-oriented software environment'',

IEEE Transactions on Software Engineering , Vol. 14

No. 6, pp. 758-73.Bate, R., Kuhn, D., Wells, C., Armitage, J., Clark, G. and

Cusick, K. (1995), A Systems Engineering Capability

Maturity Model, Version 1.1, CMU/SEI-95-MM-003,

Carnegie Mellon University, Pittsburgh, PA.Beckett, A.J., Wainwright, E.R. and Bance, D. (2000),

`Ìmplementing an industrial continuous improvement

system: a knowledge management case study'',

Industrial Management & Data Systems , Vol. 100

No. 7, pp. 330-8.Bergey, J.K., Northrop, L.M. and Smith, D.B. (1997),

Enterprise Framework for the Disciplined Evolution of

Legacy Systems , CMU/SEI-97-TR-007, Carnegie Mellon

University, Pittsburgh, PA.DeLone, W.H. and McLean, E.R. (1992), `Ìnformation

systems success: the quest for the dependent

variable'', Information Systems Research , Vol. 3 No. 1,

pp. 60-95.Deming, W.E. (1986), Out of the Crisis , MIT Center for

Advanced Engineering Study, MIT Press,

Cambridge MA.Goodhue, D.L. (1998), ``Development and measurement

validity of a task-technology fit instrument for user

evaluations of information systems'', Decision

Sciences, Vol. 29 No. 1, pp. 105-38.

Figure 13 Focused IS indices in year 2000

Figure 14 Level of IS performance in company K

127




Volume 104 . Number 2 . 2004 . 115-128



Goodhue, D.L. and Thompson, R.L. (1995), ``Task-technology

fit and individual performance'', MIS Quarterly ,

pp. 213-36.Jeong, G.-H. (1996), Weights for Informatization Index , NCA,

pp. 8-44.McFeeley, B. (1996), IDEAL: A User's Guide for Software

Process Improvement , CMU/SEI-96- HB-001, Carnegie

Mellon University, Pittsburgh, PA.Mason, R.O. (1978), ``Measuring information output:

a communications system'', Information &

Management , Vol. 1 No. 5, pp. 219-34.Mendonca, M.G. (1997), `Àn approach to improving existing

measurement frameworks in software development

organizations'', doctoral dissertation, Department of

Computer Science, University of Maryland, College

Park, MD, pp. 10-16.Mendonca, M.G., Basili, V.R., Bhandari, I.S. and Dawson, J.

(1998), `Àn approach to improving existing

measurement framework'', IBM Systems Journal ,

Vol. 37 No. 4, pp. 484-501.

Myers, B.L., Kappelman, L.A. and Prybutok, V.R. (1997), `Àcomprehensive model for assessing the quality and

productivity of the information systems function:

toward a theory for information systems assessment'',

Information Resources Management Journal , Vol. 10

No. 1, pp. 6-25.Nolan, R.L. and Wetherbe, J.B. (1980), ``Toward a

comprehensive framework for MIS research'',

MIS Quarterly , pp. 1-19.Park, R.E., Goethert, W.B. and Florac, W.A. (1996),

Goal-Driven Software Measurement ± A Guidebook ,

CMU/SEI-96-HB -002, Carnegie Mellon University,

Pittsburgh, PA.Pulford, K., Kuntzmann-Cambelles, A. and Shrilow, S.

(1996), A Quantitative Approach to Software Management: The AMI Handbook , Addison-Wesley,

Wokingham.Renaissance Consortium (1997), ``Renaissance Framework ±

method and tools support for the evolution and

reengineering of legacy systems'', Renaissance

Consortium.Saunders, C.S. and Jones, W. (1992), ``Measuring

performance of the information systems function'',

Journal of Management Information Systems , Vol. 8

No. 4, pp. 63-82.Seddon, P.B.and Kiew, M.Y. (1994), `À partial test and

development of the DeLone and McLean model of IS

success'', Proceedings of the International Conference on Information Systems, Vancouver, Canada ,

pp. 99-110.Shannon, C.E. and Weaver, W. (1949), The Mathematical

Theory of Communication , University of Illinois Press,

Urbana, IL.

Shewhart, W.A. (1931), Economic Control of Quality of Manufactured Product , D. Van Nostrand Company,New York, NY.

Tan, D.S. (1999), ``Stages in information systemsmanagement'', Handbook of IS Management ,CRC Press LLC, Boca Raton, FL, pp. 51-75.

Venkatraman, N. (1997), ``Beyond outsourcing: managing IT

resources as a value center'', Sloan Management Review , Spring, pp. 51-64.

Further reading

AMI Consortium (1992), The AMI Handbook: A Quantitative Approach to Software Management , South BankPolytechnic, London.

Callon, J.D. (1996), Competitive Advantage through Information Technology , McGraw-Hill, Maidenhead,pp. 5-12.

Chandra, C. and Kumar, S. (2001), `Ènterprise architectural

framework for supply-chain integration'', Industrial Management & Data Systems , Vol. 101 No. 6,pp. 290-303.

Kim, I.-J. (1999), ``Development of an integrated evaluationsystem based on the continuous improvement modelof information system performance'', PhD dissertationfor Yongsei University, Seoul.

Kim, I.-J. and Leem, C.-S. (1999), `Àn integrated evaluationframework for continuous capability maturity of enterprise information systems'', Proceedings of APIEMS'99 , pp. 69-72.

Leem, C.-S. (1999), 1999 Annual Reports for Evaluation of IS Performance , IT Research and Consulting.

Leem, C.-S. (2000), 2000 Annual Reports for Evaluation of IS

Performance , IT Research and Consulting.Loughman, T.P., Fleck, R.A. and Snipes, R. (2000),

`À cross-disciplinary model for improved informationsystems analysis'', Industrial Management & Data Systems , Vol. 100 No. 8, pp. 359-69.

Saad, G.H. (2001), ``Strategic performance evaluation:descriptive and prescriptive analysis'', Industrial Management & Data Systems , Vol. 101 No. 8,pp. 390-9.

Sakaguchi, T. and Dibrell, C.C. (1998), ``Measurement of theintensity of global information technology usage:quantitizing the value of a firm's informationtechnology'', Industrial Management & Data Systems ,Vol. 98 No. 8, pp. 380-94.

Seddon, P.B. (1997), `À respecification and extension of theDeLone and McLean model of IS success'', Information Systems Research , Vol. 8 No. 3, pp. 240-53.

Vernadat, F.B. (1996), Enterprise Modeling and Integration: Principles and Applications , Chapman & Hall, London,pp. 14-16, 317-34.

128




Volume 104 . Number 2 . 2004 . 115-128

Documents

An Integrated