_Chen___Hirschheim_2004_

8/2/2019 _Chen___Hirschheim_2004_

1/39

Info Systems J(2004) 14, 197235

2004 Blackwell Publishing Ltd

197

Blackwell Science, LtdOxford, UKISJInformation Systems Journal1350-1917Blackwell Publishing Ltd, 200414

3197235

Original Article

Examination of IS researchW Chen & R Hirschheim

A paradigmatic and methodologicalexamination of information systems researchfrom 1991 to 2001

WenShin Chen* & Rudy Hirschheim

*Bauer College of Business, University of Houston, Houston, TX 77204-6282, USA, and

Ourso College of Business Administration, Louisiana State University, Baton Rouge, LA

70803, USA, email: [email protected]

Abstract. The field of information systems (IS) has evolved for more than threedecades. Although many schools of thought have emerged and even become well

established, few historical analyses of research paradigms and methodologies

have been undertaken. One of the rare exceptions is Orlikowski & Baroudi (1991).

Yet, the IS research community has evolved substantially since 1991 in many

aspects. A variety of journal outlets have emerged and become well established.

More attention has been paid to paradigmatic and methodological issues. Political

and professional contexts have also changed noticeably. Therefore, it should be an

opportune time for the field to ask: What changes are manifested in journal pub-

lications? Is the field making progress regarding pluralism in IS research? How

will the fields publications practices change in the future? The purpose of thispaper is to investigate these questions and, in turn, reflect on the paradigmatic and

methodological progress made since 1991. We examined 1893 articles published

in eight major IS publication outlets between 1991 and 2001. Our findings suggest

that the long-term endeavours of interpretivist researchers might need to continue

because the paradigmatic progress appears somewhat inconsequential; positivist

research still dominates 81% of published empirical research. In particular, US

journals, as opposed to European journals, tend to be more positivist, quantitative,

cross-sectional and survey oriented. With respect to research design, survey

research is still the most widely used method (41%), although case studies have

gained substantial recognition (36%). Further, the increase of qualitative research

(30%), empirical studies (61%) and longitudinal cases (33%) at the expense of lab-

oratory experiments (18%) might suggest that IS researchers have become more

interested in obtaining scientific knowledge in real world settings. In summary, we

suggest that the field has been dominated by the positivist paradigm, despite calls

to the contrary. Indeed, if the field was to truly embrace pluralism, it would have to

find ways to fundamentally change the publication practices of the journal system,

including the current tenure and promotion system, which pose considerable

obstacles for the acceptance of alternative paradigms.


2/39

W Chen & R Hirschheim

2004 Blackwell Publishing Ltd, Information Systems Journal14

, 197235

198

Keywords:

IS research, paradigms, research methodology, positivism,

interpretivism

INTRODUCTION

There appears a growing interest as to how the information systems (IS) research community

evolves with respect to philosophical and methodological assumptions. Considerable attention

has been paid to the issues of research diversity and methodological pluralism during the past

two decades (e.g. Hirschheim & Klein, 1989; 1992; Galliers, 1991; Klein et al., 1991; Olaisen,

1991; Walsham, 1995a; Hirschheim et al., 1996; Wicks & Freeman, 1998; Cook & Brown,

1999). Specifically, alternative paradigms such as interpretivism have emerged in the field

(Walsham, 1995b) and have become more widely accepted even in traditionally positivistori-

ented journals such as MIS Quarterly

(

MISQ

) (Trauth & Jessup, 2000 ). Qualitative researchmethods, as opposed to the dominant quantitative methods, have also become more popular

in the field (Lee, 1989; Walsham, 1995a; Silverman, 1998).

Yet, the belief that alternative paradigmatic research exists within the IS field remains largely

anecdotal and conceptual (Hassard, 1988; Hirschheim & Klein, 1989; Klein et al

., 1991;

Olaisen, 1991). Little effor t has been made to empirically examine the philosophical and meth-

odological assumptions underlying IS journal publications. This is not to say that there are no

visible examples of alternative paradigmatic publications adopting Burrell & Morgans (1979)

four paradigms (functionalism, interpretivism, radical structuralism and radical humanism);

their differences with respect to ontological and epistemological assumptions have been widely

discussed in IS literature (cf. Hirschheim & Klein, 1989; Olaisen, 1991; Hirschheim et al

., 1995;

Walsham, 1995a).

While conceptual discussion brings attention to the issues surrounding alternative para-

digms, an empirical examination would help to reveal how such attention is reflected in journal

publications. After all, to discuss is one thing, to reflect on real

research practice is another. As

such, we extend the analysis of Orlikowski & Baroudi (1991) which is one of the rare empirical

examinations of IS researchs paradigmatic and methodological base. Orlikowski and Baroudi

examined 155 articles published between 1985 and 1989 in MISQ

, Communications of the

ACM

(

CACM

), Management Science

and Proceedings of the International Conferences on

Information Systems

(

ICIS

). Their findings indicated that the positivist paradigm overwhelm-

ingly dominated the IS research community (96.8%) whereas little attention was paid to the

interpretive paradigm (3.2%) and no empirical research work was done using a critical para-

digm (0%). With such an extreme positivist dominance, it was not surprising that methods such

as survey research (49.1%) and laboratory experiments (27.1%) that aim for deductive, hypo-

thetical reasoning and generalization, were the dominant research designs.

Such paradigmatic and methodological skewing in a research community might not be

healthy. Orlikowski & Baroudi (1991), thus, ar ticulated the essence of each paradigm and spec-

ified the appropriateness of employing other research methodologies to better investigate

research questions of interest. While the positivist paradigm could aim for replicability and gen-


3/39

Examination of IS research


, 197235

199

eralizability, the interpretivist paradigm could enhance the in-depth understanding of the phe-

nomenon examined. Each paradigm provides unique attributes for different purposes of

scientific inquiry; the same argument could also be applied to each methodology. The issue,

thus, should not be simply about paradigm wars (Tashakkori & Teddlie, 1998) but about theresearch questions investigated. Alternative paradigms or methodologies such as interpretiv-

ism and qualitative methods should be welcomed and encouraged because they provide dif-

ferent dimensions for research investigation that the positivist paradigm and survey methods

would not be able to accomplish.

Although Orlikowski & Baroudi (1991) brought attention to the paradigmatic and method-

ological issues of IS research to the community, their examination was conducted more than

a decade ago. Much has changed in the IS research community since 1990. First, many new

journals have been established to provide more research outlets and different perspectives.

For example, Information Systems Research

(

ISR

) launched its debut issue in 1990 and

became one of the top journals in the field.Accounting, Management, and Information Tech-nology

(

AMIT

) (

Information and Organization

after 2001), Information Systems Journal

(

ISJ

)

and European Journal of Information Systems

(

EJIS

) all started in 1991 and became some of

the major research outlets, especially in Europe. The recently established electronic journals,

Communications of the AIS

and Journal of the AIS

, are only a few years old and just finding

their respective niches.

Second, more attention has been paid to paradigmatic and methodological pluralism. Landry

& Banville (1992), for example, highlighted the issue of the IS field being more of a fragmented

adhocracy than a unified discipline. Hirschheim & Klein (2003) took this viewpoint one step fur-

ther to explain why the fragmented adhocracy occurred and how the IS community could pos-

sibly build a body of knowledge which might help establish the identity of IS as a discipline. In

addition, alternative research paradigms have been considered more acceptable in the field.

For example, Wicks & Freeman (1998) challenged the traditional dominant positivist position;

Walsham (1995a; 1995b) indicated the emergence of interpretivism; Klein et al

. (1991) advo-

cated the necessity of pluralism for any serious IS research agenda. Other discussions in rela-

tion to paradigmatic or methodological issues in recent years can also be found in Cook &

Brown (1999), Farhoomand & Drury (1999), Goles & Hirschheim (2000), Hirschheim & Klein

(1992), Iivari & Hirschheim (1996), Iivari et al

. (1998), Silverman (1998) and Trauth & Jessup

(2000).

Furthermore, the political and professional context has also changed significantly since

1990. For example, the mainstream journal, MISQ

, publicly announced its acceptance of alter-

native research approaches in 1993. Allen Lee, a qualitative researcher and pluralist, was later

appointed as its editor-in-chief. His effort to change the political context and advocate research

pluralism should not go unnoticed. The debate between Benbasat & Weber (1996) and Robey

(1996) indicates the variety of interests in the field, even among some of the most well known

researchers. While Benbasat and Weber argue for a unified discipline, Robey advocated the

necessity of research diversity and suggested specific ways to achieve it. Their argument

should also provoke deeper discussion in the community as to how the field should be estab-

lished. Moreover, a special issue of MISQ

in March 1999 gathered a group of well-known


4/39



, 197235

200

researchers (Applegate & King, 1999; Benbasat & Zmud, 1999; Davenport & Markus, 1999;

Lee, 1999; Lyytinen, 1999; Markus & Lee, 1999) to debate the issue of rigour vs. relevance.

Communications of the AIS also dedicated an entire issue to a similar discussion in March

2001.All of this suggests that it has been an exciting and progressive period for the IS research

community since 1990. The communitys interest in paradigmatic and methodological issues

has widened considerably. With an expanded variety of recently established journal outlets,

alternative voices should be expected to be heard. As such, it would be reasonable to expect

that these changes of paradigmatic and methodological understanding would be reflected in

journal publications. Our research purpose, thus, is to uncover whether the journal publications

do indeed reflect the changes occurring since 1990. To this end, we undertook an empirical

analysis of 1893 articles published in eight major IS journals from 1991 to 2001. The selection

of journals four European and four US and the period of investigation should help us under-

stand the progress that has been made in recent years, including:

the contrast between the traditionally dominant paradigm, positivism (Orlikowski & Baroudi,

1991), and an emerging paradigm, interpretivism (Walsham, 1995b), in each journal;

a cross-continental comparison between European and North American journals, as con-

ventional wisdom has long perceived that European journals are more receptive to interpretive

approaches while North American journals tend to be positivist; and

four areas of comparison of methodology: empirical vs. non-empirical, qualitative vs. quan-

titative, cross-sectional vs. longitudinal, and the comparison of five research designs (survey,

laboratory experiment, field experiment, case study and action research).

Such an empirical examination would not just continue the research tradition of Orlikowski &

Baroudi (1991) but also extend the empirical understanding of the current state of the IS dis-

cipline. Conceptual arguments, regardless of how convincing they might appear, would not

reflect the state of concurrent research practice in the journal publications. The empirical result

of the study could, thus, help us to understand better how the IS research community has

evolved with respect to its paradigmatic and methodological assumptions.

In order to achieve this goal, the paper is organized as follows. First, we articulate our clas-

sification of paradigms and methodologies. Then, our research procedure is described espe-

cially as it relates to the journals chosen for study and the paradigms and methodologies used

in the analysis. The section that follows outlines the findings of the empirical examination of 1893

articles. The discussion of the results and the implication of the analysis are then presented in

relation to our research questions. Finally, we present some concluding remarks, which reflect

upon our history, and we also offer some suggestions for future research endeavours.

PARADIGMATIC

CLASSIFICATIONS

Considerable variation exists when it comes to classifying paradigms. For instance, Landry &

Banville (1992) categorize researchers into three groups: (1) mainstream navigators

who


5/39



, 197235

201

inherit a logical positivism tradition; (2) unity advocates

who favour methodological monism;

and (3) knights of change

who advocate methodological pluralism. This categorization broadly

differentiates those who endorse one single paradigm (mainstream navigators and unity advo-

cates) and those who advocate methodological pluralism (knights of change). Similar to Burrelland Morgans typology, Landry and Banvilles grouping differentiates functionalists (main-

stream navigators and unity advocates) from non-functionalists (interpretivist, radical humanist

and radical structuralist). According to Hirschheim & Klein (1992), neohumanism and radical

structuralism have not been well developed in the IS research community.

1

Orlikowski &

Baroudi (1991) and Goles & Hirschheim (2000) indicate that positivism dominates IS research

while other paradigms are relatively small in number. In fact, the only real alternative paradigm

observable in any numbers in IS research is interpretivism (Walsham, 1995a; 1995b; Nandha-

kumar & Jones, 1997; Trauth & Jessup, 2000). Thus, the paradigmatic comparison in our anal-

ysis will simply focus on positivism vs. interpretivism.

The major differences between positivism and interpretivism concerning research arethreefold. Ontologically

, positivists believe that reality exists objectively and independently

from human experiences while interpretivists emphasize the subjective meaning of the real-

ity that is constructed and reconstructed through a human and social interaction process

(Burrell & Morgan, 1979). Epistemologically

, positivists are concerned with the hypothetic-

deductive testability of theories. Scientific knowledge should allow verification or falsification

and seek generalizable results. As such, a causal relationship is usually presented and a

tight coupling among explanation, prediction and control is expected (Orlikowski & Baroudi,

1991). Interpretivists, by contrast, assume that scientific knowledge should be obtained not

through hypothetic-deductive reasoning but through the understanding of human and social

interaction by which the subjective meaning of the reality is constructed (Walsham, 1995a).

Methodologically

, positivists contend that, to test hypothetic-deductive theory, research

should take a value-free position and employ objective measurement to collect research evi-

dence. A quantitative method such as the survey is a typical positivist instrument. Interpre-

tivists, on the other hand, argue that to understand the meaning embedded in human and

social interaction, researchers need to engage in the social setting investigated and learn

how the interaction takes place from the participants perspective. Field studies that engage

researchers in the real social setting would be more appropriate for generating interpretive

knowledge (Orlikowski & Baroudi, 1991).

In brief, positivist research is mostly depicted through: (1) the formulation of hypotheses,

models, or causal relationships among constructs; (2) the use of quantitative methods,

although not always necessary, that test theories or hypotheses; and (3) researchers objec-

tive, value-free interpretation. A classic example of positivist research listed in Hirschheim &

Klein (1992) is the Dickson et al

. (1977) discussion of the Minnesota experiments. Interpretivist

studies, in contrast, could be observed through: (1) evidence from a non-deterministic (free

1

Alternative paradigmatic classification could be found in Chua (1986), in which the three basic paradigms are positivism,

interpretivism and critical theory (Hirschheim & Klein, 2003). Critical theory, however, is not yet well established in the IS

field (Orlikowski & Baroudi, 1991).


6/39



, 197235

202

will) perspective; (2) researchers engagement in the specific social and cultural setting inves-

tigated; and (3) an analysis based on participants viewpoints (Orlikowski & Baroudi, 1991;

Walsham, 1995a). Examples of interpretive studies that include these attributes are Orlikowski

(1993) and Orlikowski & Robey (1991).

METHODOLOGICAL

CLASSIFICATIONS

Various schemes have been applied among researchers to classify research methodology. For

example, Cash & Nunamaker (1989; 1990; 1991) compare qualitative research methods to

experimental and survey research methods. They discovered that only 17% of the papers

included in their analysis employed qualitative research methods. Alavi et al

. (1989) analyzed

empirical and non-empirical articles and found that empirical articles constitute only 46.5% of

publications in the period between 1968 and 1988. They further divide empirical studies intoeight categorizes: laboratory experiment, field experiment, field study, case study, survey,

development of MIS instruments, ex post descriptions and others. Orlikowski & Baroudi (1991)

surveyed 155 articles in four major North American journals. The major categories in their find-

ing are surveys (49.1%), laboratory experiments (27.1%), case studies (13.5%), mixed meth-

ods (3.2%), instrument development (2.6%), protocol analysis (1.3%) and action research

(0.6%). Galliers & Land (1987) and Galliers (1991) created a taxonomy for IS research

approaches within which 10 modes were discussed: theorem proof, laboratory experiment,

field experiment, case study, survey, forecasting and future research, simulation and game/role

playing, subjective/argumentative, descriptive/interpretive and action research. One of the

more recent examinations is by Farhoomand & Drury (1999) in which the methodology of the

studies they analyzed were categorized as non-empirical (39%), survey (32%), case study

(17%), laboratory experiment (10%) and field experiment (2%).

Among these different taxonomies the most consistent comparisons are: empirical vs. non-

empirical (Alavi et al

., 1989) and quantitative vs. qualitative (Cash & Nunamaker, 1989).

Detailed examinations of research designs appear diverse and ambiguous. For example,

although Galliers (1991) included more research methodologies than others, forecasting/future

research, simulation and game/role playing, and theorem/proof, never gained much attention

in the IS community. Additionally, although Alavi et al

. (1989) distinguish between case study

and field study, this distinction is not well articulated. Most case studies are conducted in the

field; hence, the attempt to separate field study from case study appears unnecessary.

Our methodological classification will primarily follow the classification of Orlikowski &

Baroudi (1991) for two reasons. First, Orlikowski & Baroudi (1991) are concerned not just with

methodology but also with paradigmatic trend. Thus, with a similar purpose, both studies could

provide a better comparison if the same classification is employed. Second, their sample was

collected between 1985 and 1989 from four major US research outlets, while ours was col-

lected between 1991 and 2001 from eight major journals. A better reflection, as to how the field

has progressed, could be made if the same classification is used. However, three categories

have been modified for the following reasons:


7/39



, 197235

203

1

Mixed method is not a clear category on their list. We compare mixed method with quanti-

tative and qualitative methods. Articles that use more than one research design (e.g. laboratory

experiment, case study) will be counted in each category and will also be considered as mixed

method.

2

Protocol analysis and instrument development are more practitioner oriented. As this is not

the main interest of our research, we place them in the others category.

3

With respect to research evolving over time, we distinguish between repeated measure

design and cross-sectional, multiple snapshots. While repeated measure design involves

multiple occasions, its focus is on understanding how phenomena change as time passes,

which makes it more dynamic than cross-sectional, multiple snapshots.

RESEARCH

PROCEDURE

Our research procedure involved two stages. The first stage was conducted from fall 2000 to

spring 2001. We collected and analysed articles published in MISQ andAMITto capture and

compare overall trends, if any, in US and European journals. Based on the results of the initial

analysis, we extended article collection to an additional six research outlets and modified our

categorization approach in the second stage, which was from fall 2001 to spring 2002. The

comparison of research duration was extended to four categories: cross-sectional, multiple

snapshots, longitudinal and repeated measure designs. We believe that this comparison pre-

sents a better picture of the different research endeavours involved. Repeated measure design

is of particular interest because it emphasizes the time factor and, thus, can provide a richer

understanding of the researched phenomenon.

Journal representations

For the purpose of illustrating the research trend in IS and comparing the traditionally domi-

nant positivist approach (Orlikowski & Baroudi, 1991) and the emerging interpretive

approach (Walsham, 1995a; 1995b), we decided to focus on mainstream journals which, we

thought, would reflect changes or progress in the field. As such, we examined four main-

stream US publications MISQ

, ISR

, Journal of Management Information Systems

(

JMIS

)

and ICIS

and four main European journals

2

AMIT

,

3

ISJ

,

4

Journal of Information Technol-

ogy

(

JIT

) and EJIS.

The rationale behind these choices has been well explained byWalsham (1995a) and Orlikowski & Baroudi (1991). In brief, these US journals

5

represent

the largest forums of IS publications and arguably, the most influential IS researchers. The

2

This distinction between European and US is simply based on where the journal originates, i.e. the location of the pub-

lisher. To a large extent, it also reflects the composition of the editorial board (Galliers & Meadow, 2003).

3

In 2001,AMITchanged its name to Information and Organization.4ISJwas called Journal of Information Systems until 1997.5We are, of course, aware that ICIS is not a journal but its acceptance rate (traditionally around 10%) is much like the top

IS journals. It also has a pre-eminent position with respect to other conferences in the scholarly community.


8/39


2004 Blackwell Publishing Ltd, Information Systems Journal14, 197235

204

four selected European journals, on the other hand, embrace a more non-traditional, non-

positivist nature and attract IS researchers with non-traditional orientations. The contrast

and comparison between these two groups should reflect IS research trends and, thus,

illustrate any differences within the IS research community.

6

The only difference betweenour list and Walshams (1995a) is that we have replaced CACMwith JMIS. Our reasoning is

that CACM has changed so as to appeal more to general readers and members of the

Association for Computing Machinery (ACM), thereby reducing the scholarly nature of its

publication. Thus, the journals desire to become practical to meet the needs of the larger

practitioner community makes it inappropriate for this study.

Paradigmatic representations

As Goles & Hirschheim (2000) indicate, Although the proportion of positivist to non-

positivist articles varies somewhat from study to study, the indisputable consensus is thatpositivism dominates information systems research (p. 254). Hirschheim & Klein (1992)

further explain that examples of IS research in radical structuralism and neohumanism

(radical humanism) are rarely found. What might gain increasing attention in IS research,

according to Walsham (1995a), is interpretive approach. As such, comparing and contrast-

ing the trend of positivist and interpretive paradigmatic approaches might better reflect IS

research.

Positivist

The criteria for categorizing positivist articles are the indications of hypotheses, propositions,model formation, quantifiable measures of variables and the inferences drawn from samples to

populations (Orlikowski & Baroudi, 1991), all of which demonstrate the belief that objective

data could be collected to predict the relationship among factors and to test hypotheses or the-

ories (Walsham, 1995a).

Interpretive

The criteria for selecting interpretive articles are threefold. First and foremost, articles

should not involve any positivist indicators as described above, that is, no deterministic

perspectives imposed by the researchers. Second, participants perspectives are taken as

the primary sources of understanding and investigating the phenomena. Third, the phe-

nomena are examined with respect to cultural or contextual circumstances (Walsham,

1995a).

6Alavi et al. (1989) also include journals that are not in our list such as Data Base, Decision Science, Harvard Business

Review, Management Science and Sloan Management Review. They are either practitioner oriented (Harvard Business

Reviewand Sloan Management Review), not IS focus (Management Science and Decision Science) or according to Hard-

grave & Walstrom (1997), not considered part of the of the top-ranking journal list ( Data Base).


9/39



205

Methodological representations

Our analysis of research methodology focuses on the following areas: empirical vs. non-

empirical; quantitative vs. qualitative; cross-sectional vs. longitudinal; and the comparisons

among five research designs (survey, laboratory experiment, field experiment, case study and

action research).

Empirical vs. non-empirical

While empirical studies rely on observations and data, non-empirical studies emphasize ideas

and concepts (Alavi et al., 1989). Each of which makes different contributions to the research

community. While non-empirical studies help to develop concepts and build theory, empirical

studies provide concrete evidence for testing theories. As a discipline becomes more mature,

one might expect theory-testing to outweigh theory-building efforts. Therefore, a comparison

between empirical and non-empirical studies serves as a meaningful indication of the progressmade in the field.

Our criterion for categorizing studies as empirical or non-empirical is determined by whether

the articles obtain real data or observations, which could be gathered through quantitative,

qualitative, or a mixed approach, including archival data.7

Quantitative vs. qualitative

Although qualitative research methods such as case studies have gained increasing attention

(Benbasat et al., 1987; Lee, 1989; Klein et al., 1991; Trauth & OConnor, 1991; Visala, 1991;

Walsham, 1995b; Darke et al., 1998; Silverman, 1998; Klein & Myers, 1999; Markus & Lee,1999), the comparison between quantitative and qualitative research methods in IS research

has been treated somewhat inadequately. While the former typically uses numerical analysis

to illustrate the relationship among factors in the phenomenon studied, the latter emphasizes

the description and understanding of the situation behind the factors. To make the comparison

meaningful, both methods should be categorized as empirical studies. As non-empirical studies

typically involve describing or arguing for theories and frameworks, they could be categorized

as a qualitative research method but this would muddy the distinction of quantitative methods.

As such, our criterion for categorizing research methods as quantitative or qualitative is based

on whether studies use a statistical or numerical approach to collect and analyse data. There

is a possibility that research could use both quantitative and qualitative methods in differentstages of the study. On such occasions, we categorize them as a mixed research method.

Cross-sectional vs. longitudinal

As described previously, in contrast to Orlikowski & Baroudi (1991), our categorization of the

duration of a research project distinguishes between repeated measure design and multiple

7There are occasions when we locate articles with data/observations in the non-empirical category. In such cases, articles

were merely descriptive without scholarly research questions, practitioner oriented for systems development, or mathe-

matic simulation and analysis.


10/39



206

snapshots. Four categories are designated: longitudinal, repeated measure design, multiple

snapshot and cross-sectional each requiring different research efforts and time commitment.

We define longitudinal as research that evolves over an uninterrupted period of time and

focuses on process. Cross-sectional, on the other hand, is research that collects data throughone snapshot at a particular point of time (Orlikowski & Baroudi, 1991). Multiple snapshots are

also cross-sectional in nature but involve more than one single data-collection period or setting,

which could mean different experiments, treatments or subjects and so on. Repeated measure

designs are those that conduct studies at various time periods to examine how a phenomenon

evolves at various periods of time.

Six major research designs

Six research designs were identified, as per Orlikowski & Baroudi (1991):

Survey: Studies that fall into this category gather data by means of questionnaires. Case study: Studies that are involved with a single site or a few sites over a certain period oftime are located in this category. The case study inquiry usually relies on multiple sources of

evidences (Yin, 1994, p. 13).

Laboratory experiment: These include studies that take place within a designed, controlledenvironment and usually involve special treatments of different groups to contrast the precise

relationships among variables (Galliers, 1991).

Field experiment: As with laboratory experiments, special treatments are used to study twoor more controlled groups. Field experiments differ in that they are conducted in a real-world

setting. Action research: Often embodied in a case study, action research distinguishes itself in thatthe researchers are an integral part of the phenomenon under study. The researchers input

often influences the outcomes of the phenomenon and his/her role could change from

researcher to subject (Galliers, 1991).

Others: this category includes articles that are practitioner oriented (systems or tools devel-opment), non-empirical pieces, or descriptive/argumentative as noted in Gallierss (1991) clas-

sification. Research with secondary data such as public records or existing datasets is also

located in this category.

The following section presents our empirical findings. Our analysis discusses overall trends,

which combines the results of all eight journals from 1991 to 2001 and a total of 1893 articles.

The detailed categorization of each article is listed in the Appendix wherein readers can see

how each article was located with respect to the classification criteria: (1) positivist vs. inter-

pretive; (2) empirical vs. non-empirical; (3) quantitative vs. qualitative vs. mixed; (4) longitudinal

vs. repeated measure vs. multiple snapshots vs. cross-sectional; and (5) survey vs. lab exper-

iment vs. field experiment vs. case study vs. action research vs. others. 8 These detailed cat-

8Includes subcategories of conceptual, existing (archival) data, documents/reports, meta-analysis, systems design, news

data, simulation/mathematic calculation, Delphi techniques, observation and tutorials.


11/39



207

egorizations are listed by journals, and then summarized in terms of the number of articles by

year and categories. More details for each journal are listed in the Appendix under the title

Summary of Categorization of Journal Articles.

FINDINGS

Whilst a detailed analysis of 1893 articles might appear overwhelming, in this section we focus

our analysis on three major areas that better fit our research interests: (1) overall trend; (2) the

cross-continental comparison; and (3) the trend of individual journals. Table 1 displays a sum-

mary of all articles as they relate to our classification criteria.

Overall trend: all journals

Positivist vs. interpretive

Figure 1 shows that positivist research maintains a clearly dominant position while the number

of interpretive studies has increased slightly over the years. As can be seen from Table 1, the

overall percentages of positivist and interpretive studies are 81% (917) and 19% (214) respec-

tively. What is perhaps somewhat surprising is the fact that the overall trend has remained

remarkably constant over the past 10 years.

Empirical vs. non-empirical

Figure 2 illustrates the trend of empirical vs. non-empirical studies. In the early 1990s, non-

empirical studies were in the majority in IS research. This is consistent with the finding of Alavi

et al. (1989). Since 1993, however, the number of empirical-research studies has exceeded

non-empirical studies and the gap has widened over the years. Moreover, among the non-

empirical studies there are many articles that contain empirical data but we locate them in the

non-empirical category as they are typically practitioner-oriented studies. As such, the real

number of empirical-research studies, i.e. those which contain empirical data, is more than

what Figure 2 illustrates. In Table 1, the overall percentages of empirical and non-empirical

studies are 60% (1131) and 40% (762) respectively. This exactly mirrors the findings of Far-

hoomand & Drury (1999) who examined 2098 articles published between 1985 and 1996 and

noted that the breakdown between empirical and non-empirical studies was 61% and 39%

respectively. Even though the journal selections are significantly different in the two studies, the

results are the same. So the dominant trend of empirical research in IS is demonstrably visible.

Quantitative vs. qualitative

Figure 3 shows that quantitative research still dominates the IS discipline but the number of

qualitative research papers has been increasing over the years. The gap between quantitative


12/39



208

and qualitative research in 1997 and 1998 was very narrow. Subsequently, the gap has wid-

ened. Mixed method research, on the other hand, has remained steady but not substantial from

1991 to 2001. As noted in Table 1, the overall percentages of quantitative, qualitative and

mixed methods are 60%, 30% and 10% respectively.

Longitudinal vs. cross-sectional

Figure 4 shows that there has been a steady, continuous popular trend of cross-sectional stud-

ies although there has been a considerable growth in longitudinal studies as well. The growth

of longitudinal research corresponds well with the trend toward qualitative methods, as shown

in Figure 3. This should be expected because the qualitative research method usually involves

case studies, which requires a long-term research process. Repeated measure design and

multiple snapshots, on other hand, continue to be flat and few in number, indicating a very lim-

ited growth over the decade. Figure 5 displays that the overall percentage of longitudinal,

repeated measure design, multiple snapshots and cross-sectional as 33%, 4%, 4% and 59%

respectively.

Major research designs

Figure 6 illustrates that survey research and case studies are, by far, the most popular research

designs while action research and field experiments are rather insignificant. Laboratory exper-

iments, on the other hand, maintain a somewhat steady position throughout the years, albeit

with occasional fluctuations. While the survey method remains a steady research force, case

studies demonstrate a noticeable growth throughout the years. In the years 1997, 1998, 2000,

the number of case studies actually exceeded that of survey research. This could be an indi-

cation that IS researchers are increasingly interested in gathering rich data through the use of

the case study method.

Table 1. Summary of journals in all categories

Year Positivist Interpretive Empirical Non-empirical Quantitative Qualitative Mixed Cross-sectional

1991 63 3 66 82 50 6 10 53

1992 60 4 64 72 49 7 8 45

1993 73 12 85 64 49 22 14 55

1994 92 10 102 79 73 23 6 70

1995 89 10 99 76 63 29 7 66

1996 97 20 117 70 65 36 16 65

1997 79 32 111 66 54 47 10 50

1998 89 31 120 48 59 52 9 63

1999 100 24 124 72 75 34 15 65

2000 92 42 134 60 70 54 10 69

2001 83 26 109 73 73 31 5 66

Total 917 214 1131 762 680 341 110 667


13/39



209

Figure 7 offers the overall percentages of survey, case study, laboratory experiment, action

research and field experiment as 41%, 36%, 18%, 3% and 2% respectively. This result differs

somewhat from that of Orlikowski & Baroudi (1991) where the survey method embodied 49.1%

of IS research production followed by laboratory experiments (27.1%) and case studies

(13.5%). It is, however, more in line with Farhoomand & Drury (1999) where the survey, case

study and laboratory experiment share was 52%, 28% and 16% respectively. This indicates

that while surveys maintain their steady popular position, case studies have gained consider-

able interest over the decade. It must be noted, however, that research categorized as others

makes up 46% of the entire research production. In another words, the IS research community

still relies heavily on non-empirical studies such as conceptual/framework development.

longitudinal

Multiple

snapshots

Repeated

measures Survey

Case

study

Lab

experiment

Action

research

Field

experiment Others

11 0 2 30 15 20 0 3 90

12 5 2 38 16 15 0 1 81

23 2 5 35 30 23 0 0 75

22 4 6 52 25 15 3 2 94

30 3 0 51 30 15 1 1 87

40 6 6 44 42 28 7 2 87

51 7 3 42 52 16 6 2 74

52 3 2 41 56 11 1 2 64

41 12 6 47 47 27 3 3 88

56 5 4 35 53 21 8 2 85

36 1 6 39 36 13 3 1 96

374 48 42 454 402 204 32 19 921

Figure 1. Overall paradigmatic trend.

0

20

40

60

80

100

120

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

Nu

mberofArticles

Positivist Interpretive


14/39



210

Cross-continental comparisons

US vs. European journals: positivist vs. interpretive

The focus of our second analysis is on the cross-continental comparison between US and

European journals. Conventional wisdom suggests that US journals are more positivist ori-

ented and use more quantitative methods while European journals tend to be more inter-

pretive with more qualitative methods. Here, we empirically examine whether this

assumption holds true. Figure 8 illustrates that US positivist research clearly dominates with

58% while US interpretive articles account for only 7%. This clearly indicates that US

research is overwhelmingly dominated by the positivist paradigm. On the other hand, the

Figure 2. Overall trend of empirical vs. non-empirical studies.

0

20

40

60

80

100

120

140

160

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles

Empirical Non-Empirical

Figure 3. Overall trend of quantitative vs. qualitative studies.

0

10

20

30

40

50

60

70

80

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles

Quantitative Qualitative Mixed


15/39



211

difference between European positivist (23%) and European interpretive studies (12%) is

considerably less significant. Moreover, as Figure 9 shows US positivist research not only

remains a clearly dominating force but it has also maintained its predominant position over

the years. The same, although to a somewhat lesser extent, can be seen in Europe where

approximately two-thirds of the papers are positivist. Interpretive studies in both continents,

while showing a modest increase, are still quite small in number. Interpretive researchers,

expecting a more equitable publication distribution may or may not be surprised by this

result.

US vs. European journals: quantitative vs. qualitative

As seen in Figure 10, US quantitative research is vastly dominant (46%), but the second most

popular type of publication is qualitative research in European journals (17%). It slightly edges

Figure 4. Overall trend of cross-sectional vs. longitudinal studies.

0

10

20

30

40

50

60

70

80

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles

Cross-sect ional Longitudinal Multiple Snapshots Repeated Measures

Figure 5. Percentage of cross-sectional vs. longitudinal studies.

Cross-Sectional

59%

Longitudinal33%

MultipleSnapshots

4%

RepeatedMeasures

4%


16/39



212

out qualitative research in the USA (13%) and quantitative research in European journals

(14%). Mixed methods in both US and European journals are still relatively small, comprising

only 6% and 4% respectively.

From Figure 11, one notes that as far as the overall trend is concerned, the only method

that clearly stands out throughout the years is the quantitative method in the USA. It has

remained consistent over the past 11 years and even increased slightly, in recent years. The

gradual increase of European qualitative research is also noteworthy. No other trend is

discernible.

Figure 7. Percentage of five research designs.

Survey

41%

Casestudy

36%

Labexperiment

18%

Actionresearch

3%

Field

experiment2%

Figure 6. Overall trend of five major research designs.

0

10

20

30

40

50

60

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001Year

NumberofArticles

Survey Case Study Lab Experiment Action Research Field Experiment


17/39



213

Trends of individual journals

To simplify our analysis of the individual journals, we restrict our examination to paradigmatic

trend and research design. For each journal, we will present an overall paradigmatic trend and

a pie chart indicating the percentage of the five major research designs employed in the jour-

nal. The sequence of presentation is MISQ,AMIT/IO, EJIS, JIT, ISJ, ISR, JMIS and ICIS, with

no particular priority.

MISQ trend

The total number ofMISQ articles included in our analysis is 232, broken down into 166 empir-

ical and 66 non-empirical studies. The paradigmatic trend of MISQ shown in Figure 12 very

Figure 8. Paradigmatic comparison: US vs. European journals.

Positivist

(US)

58%

Interpretive

(European)

12%Interpretive

(US)

7%

Positivist

(European)

23%

Figure 9. Paradigmatic trend: US vs. European journals.

0

10

20

30

40

50

60

70

80

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles

US Positivist European Positivist

US Interpretive European Interpretive


18/39



214

closely corresponds to the overall paradigmatic trend with positivism dominating interpretivism

throughout the years. Never in any given year has interpretive studies produced more than five

articles in MISQ. Occasionally (e.g. 1991, 1992 and 1995), no interpretive research appears in

the journal. With respect to research design (Figure 13), survey research is still the most pop-

ular data collection method with 46% while case studies make up 34% of the research output.

The other three methods appear much less significant. Although MISQ has publicly announced

Figure 10. Quantitative and qualitative percentages: US vs. European journals.

Mixed

(US)

6%

Quantitative

(European)

14%

Qualitative

(US)

13%

Quantitative

(US)46%

Mixed

(European)

4%

Qualitative(European)

17%

Figure 11. Quantitative vs. qualitative trend: US vs. European journals.

0

10

20

30

40

50

60

70

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles

US Quantitative European Quantitative US Qualitative

European Qualitative US Mixed European Mixed


19/39



215

its interest in accepting alternative research, overall, MISQ remains a positivist (91%), quan-

titative (64%), cross-sectional (60%) and survey research (46%) dominated journal.

AMIT/IO trend

We examined 122 articles in AMIT/IO, of which 65 were empirical and 57 non-empirical.

Figure 14 depicts an interesting paradigmatic trend in AMIT/IO. The two paradigmatic lines

are intertwined particularly in the mid-1990s, until interpretive studies became the main-

stream in 1997. In 2001, IO published zero positivist research papers thus establishing

Figure 12. Paradigmatic trend of MISQ.

0

2

4

6

8

10

12

14

16

18

1991 1993 1995 1997 1999 2001

Year

NumberofArticles

Positivist

Interpretive

Figure 13. Percentage of research designs in MISQ.

Survey46%

Labexperiment

15%

Fieldexperiment

4%

Casestudy34%

Actionresearch

1%


20/39



216

interpretive research as the paradigm of choice. The overall percentage of interpretive and

positivist studies in AMIT/IO are 63% and 37% respectively. Among the journals examined,AMIT/IO is the only one in which interpretive research is the mainstream research

production.

Figure 15 demonstrates that the most popular method inAMIT/IO is the case study making

up 64% of the publications. Positivist research methods such as surveys, laboratory experi-

ments and field experiments have relatively marginal representation with 14%, 15% and 2%

respectively. Action research appears to have gained increasing attention in this outlet with a

modest total of 5% production. This should be encouraging for action researchers because

most journals pay little attention to this type of research (e.g. EJIS). Clearly, this journal is

receptive to alternative research approaches.

Figure 14. Paradigmatic trend ofAMIT/IO.

0

1

2

3

4

5

6

7

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles


Figure 15. Percentage of research designs in AMIT/IO.

Survey

14%

Case

study64%

Actionresearch

5%

Field

experiment

2%

Lab

experiment15%


21/39



217

EJIS trend

We examined 233 EJIS articles with a breakdown of 118 empirical and 115 non-empirical stud-

ies. Figure 16 indicates that EJISs paradigmatic trend looks similar to that of MISQ a clear

indication of the dominance of positivist research, the past 2 years notwithstanding. In 1991,

1992, 1995 and 1996, there were no interpretive articles published in EJIS. The percentage of

positivist studies in EJIS is 88%, which closely resembles that ofMISQ (91%). If conventional

wisdom is that European journals are more interpretive and qualitatively oriented, EJIS does

not follow this trend. In terms of research design, Figure 17 shows that EJIS is clearly domi-

nated by survey methods (51%) while action research (2%) and field experiment (1%) are

insignificant. Case studies also gain wide acceptance in EJIS although looking at its paradig-

matic trend (Figure 16), it is clear that these are likely to be positivist case studies.

JIT trend

We examined 198 articles in JITwith 120 empirical and 78 non-empirical studies. As Figure 18

indicates, our study only involves articles published from 1994 onwards. This was unfortunately

due to our inability to obtain the earlier issues. We note that the gap between interpretive and

positivist studies has narrowed noticeably since 1996. In fact, if one excludes 1994 and 1995,

the difference between the number of positivist and interpretive studies is, at best, marginal.

The percentages of positivist and interpretive research in JITare 60% and 40% respectively.

With respect to research design, Figure 19 reveals that JITis receptive to qualitative research

as case studies and action research share relatively high percentages of 52% and 7% respec-

tively. Only 39% of the articles used survey methods, a somewhat low percentage relative toother outlets.

Figure 16. Paradigmatic trend of EJIS.

0

2

4

6

8

10

12

14

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

Numbe

rofArticles



22/39



218

ISJ trend

We examined 172 articles in ISJ, of which 90 were empirical and 82 non-empirical.Figure 20 demonstrates an interesting paradigmatic pattern of ISJ. Both lines of the para-

digmatic trends punctuate around each other, whereas interpretive studies occasionally

exceed positivist ones (e.g. 1997 and 2000). The number of interpretive studies has also

shown, by and large, an increasing trend through the years. The overall percentages of pos-

itivist and interpretive studies in ISJ are 60% and 40% respectively. The percentage of

research designs in ISJalso appears noticeably different. While case studies dominate the

journal, a relatively high number of action research papers (12%) indicates its traditional

European qualitative orientation (Figure 21). Survey research, comprising only 23%, is the

lowest of any examined outlet.

Figure 18. Paradigmatic trend of JIT.

0

2

4

6

8

10

12

14

16

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles

Posi tivist Interpretive

Figure 17. Percentage of research designs in EJIS.

Survey51%

Casestudy40%

Actionresearch

2%

Labexperiment

6%

Fieldexperiment

1%


23/39



219

ISR trend

Out of the 213 ISR articles examined, 108 were empirical and 105 were non-empirical.

Figure 22 indicates that interpretive studies in ISR are virtually non-existent. In 6 out of

11 years, ISR published no interpretive studies. The gap between positivist and interpretive

research has, if anything, increased over the years. The total percentages of positivist and

interpretive studies in ISR are 94% and 6% respectively. In the 11-year period, ISR produced

only seven empirical, interpretive studies. Quantitative studies make up 87% while cross-

sectional studies are similarly high at 74%. Like MISQ, ISR is positivist, quantitative and cross-

sectional oriented. Figure 23 further indicates that ISR is a traditional positivist outlet where

Figure 19. Percentage of research designs in JIT.

Survey

39%

Case

study

52%

Action

research

7%

Field

experiment

0%

Lab

experiment

2%

Figure 20. Paradigmatic trend of ISJ.

0

1

2

3

4

5

6

7

8

9

10

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

N

umberofArticles

Positiv ist Interpretive


24/39



220

laboratory experiments (45%) and survey research methods (38%) dominate more than 80%

of the published research.

JMIS trend

We examined 370 JMIS articles comprising 234 empirical and 136 non-empirical studies.

Figure 24 illustrates that the paradigmatic trend ofJMIS also shows the extremely dominating

position of positivist research throughout the years. As in the case of ISR, the overall percent-

ages of positivist and interpretive studies in JMIS are 94% and 6% respectively. With respect

to research design, Figure 25 demonstrates that surveys are the preferred instrument (50%)

while qualitative approaches comprise less than one fourth of research publications (22%). If

Figure 21. Percentage of research designs in ISJ.

Survey

23%

Labexperiment

14%Field

experiment1%

Casestudy50%

Actionresearch

12%

Figure 22. Paradigmatic trend of ISR.

0

2

4

6

8

10

12

14

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles



25/39



221

traditional wisdom considers US journals to be more positivist and quantitative, then MISQ, ISR

and JMIS certainly bear this out.

ICIS trend

We examined 351 ICIS articles comprising 230 empirical and 121 non-empirical studies. Sim-

ilar to the other US outlets, ICIS also strongly tends to be positivist (83%). As illustrated by

Figure 26, positivist research clearly dominates interpretive studies with the number of its pub-

lications remaining high. Indeed, the gap was largest in 2001. With respect to research design,

ICIS papers appear to be more equally divided among surveys (39%), case studies (36%) and

laboratory experiments (22%) (Figure 27).

Figure 23. Percentage of research designs in ISR.

Survey

38%

Labexperiment

45%

Fieldexperiment

3%

Casestudy14%

Actionresearch

0%

Figure 24. Paradigmatic trend of JMIS.

0

5

10

15

20

25

30

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

NumberofArticles



26/39



222

DISCUSSION

The empirical evidence gathered from 1893 articles in eight leading IS publication outlets

between 1991 and 2001 demonstrates that even after years of advocacy of paradigmatic plu-

ralism, not much has really changed. The gap between positivist and interpretive research is

still significant. The increase of interpretive studies is still marginal and a considerable dis-

crepancy can be easily found in MISQ, ISR, JMIS, ICIS and EJIS. In other words, five out of

eight journals are dominated by the positivist approach. This is even more apparent in ISR and

JMIS where positivist research comprises 94% of the publications. The only journal in which

Figure 25. Percentage of research designs in JMIS.

Survey

50%

Lab

experiment

25%

Casestudy

22%

Field

experiment

3%

Action

research

0%

Figure 26. Paradigmatic trend of ICIS.

0

5

10

15

20

25

30

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

Numb

erofArticles

Posi tivist Interpretive


27/39



223

interpretive research exceeds positivist research isAMIT/IO. This exception might be due to

the editorial policy advocated in its debut issue:

The journal is open to all forms of scholarship and analysis concerning information

systems . . . . No reference discipline, research method or theoretical perspective will be pre-

ferred over another. The journal is open to established topics and approaches . . . as well as

to more openly interpretive and critical analysis. (Boland & OLeary, 1991, p. 1)

With respect to the cross-continental comparison, US journals tend to be dominated by the

positivist approach. Despite calls for research methodological pluralism and the appointment

of a sympathetic editor-in-chief ofMISQ, no significant change is reflected in the mainstream

research publications. A recent paper by Galliers & Meadow (2003) offers a possible expla-

nation. While examining articles published in two mainstream North American (ISR and MISQ)

and two European journals (ISJand JSIS) from 1994 to 2000, Galliers and Meadow discovered

that the ISR and MISQ editorial board members were primarily North American researchers

(between 75% and 87.5% forISR and between 51.2% and 86.8% forMISQ). These journals

also tended to publish articles written by North American authors (74% for ISR and 83% for

MISQ). A similar situation was found in European journals where the editorial board members

were largely European as were the papers published. Galliers and Meadow concluded that theIS field maintained a strong home grown perspective.

While these results may be of little surprise to some, the research community might wish

to assess what progress, if any, has been made over the last decade towards pluralism.

The analysis of Orlikowski & Baroudi (1991) noted the overwhelming dominance of positiv-

ism (96.8%). Our analysis shows that the positivist paradigm still accounts for 81% of all

empirical publications, and almost 86% of the US journal empirical publications (661 out of

738 articles). Clearly, paradigmatic progress over the past 11 years has been marginal. If

diversity is to be encouraged to avoid an unnecessarily restrictive and insular perspective

for studying IS phenomena (Orlikowski & Baroudi, 1991; Robey, 1996; Hirschheim & Klein,

Figure 27. Percentage of research designs in ICIS.

Survey

39%

Lab

experiment

22%

Case

study36%

Action

research

3%

Field

experiment

0%


28/39



224

2002a), IS researchers need to ask: What can the IS research community do to accept

alternative research perspectives?

Consistent results could also be observed in the comparison between quantitative and qual-

itative research and between longitudinal and cross-sectional studies. It is quite noticeable thatquantitative and cross-sectional studies enjoyed more popular positions throughout the

decade. This might be due to the publish or perish research publication games (Walsham,

1995a). Quantitative research usually employs survey questionnaires or laboratory experi-

ments. Researchers time commitment and engagement in the research process are, relatively

speaking, minimal. As pressures converge for tenure evaluation, researchers will predictably

choose an easier approach to ensure that their research works are published in time for the

promotion. Quantitative and/or cross-sectional studies, therefore, appear to be a better solution

for that purpose.

Another interesting observation is illustrated by the comparison between empirical and

non-empirical studies. Our analysis demonstrates that IS researchers have increased theiruse of empirical data collection. Descriptive, conceptual and analytical types of research are

still popular but are in decline (Alavi et al., 1989). Such a result is consistent with Far-

hoomand & Drury (1999) in which 61% of research is empirical in nature. This might be

due to long-term advocacy and discussion of rigour and relevance. For example, Communi-

cations of AIS devoted an entire issue to the subject in March 2001 while MISQ did some-

thing similar in March 1999. As a field matures, theoretical and conceptual developments

become less appealing. Empirical studies, in contrast, become more popular because of the

need for theory testing and practical relevance. The growing number of empirical studies

might be an indication that the IS field has become more mature and is continuing to

evolve.

With respect to research design, our findings show some change. Orlikowski & Baroudi

(1991) noted that the most frequently adopted methods were surveys (49.1%), laboratory

experiments (27.1%) and case studies (13.5%). Our analysis, in contrast, demonstrates that

while survey (41%) is still the most widely used research design, the number of case studies

(36%) has increased substantially. Farhoomand & Drury (1999) found a similar situation with

case studies comprising 28% and surveys 52% of all empirical publications. Such results might

indicate that researchers are not just engaging in more empirical studies but rather paying

more attention to how the phenomenon evolves. [This is based on the belief that the case

study, normally conducted through an uninterrupted period, might better reveal the detailed

process of the phenomenon investigated and/or even be incorporated into positivist, theory-

testing research (Lee, 1991; Sarker & Lee, 1998).]

Implications

Our analysis indicates that whilst there has been some paradigmatic change in the IS research

community since 1990, this change has not significantly manifested itself in journal publica-

tions. In most research journals, positivism maintains its prevailing, dominant position. This

could have implications with respect to pluralists and general researchers.


29/39



225

First, if the field were to become a pluralist discipline (Klein et al., 1991; Robey, 1996), the

long-term advocacy for pluralism would need continuous and stronger endeavours. It is rea-

sonable to claim, a single research perspective for studying information systems phenomena

is unnecessarily restrictive (Orlikowski & Baroudi, 1991, p. 1). It could also be true that thereare signs that interpretivism is gaining ground (Walsham, 1995b, p. 376) and There has been

an increase in recent years in the number of in-depth case studies which focus on human

actions and interpretations surrounding the development and use of computer-based informa-

tion systems (IS) (Walsham, 1995a, p. 74). The empirical evidence shown in this study, how-

ever, suggests that while the emergence of interpretivism in IS research and the wider use of

case studies might be true, their influences are still rather insignificant, particularly given that

case studies have been widely used in a positivist fashion (e.g. Swanson, 1992; Jones, 1994;

Kirsch & Cummings, 1996; Nelson et al., 1997; Reeves-Ellington & Anderson, 1997). As Lan-

dry & Banville (1992) claimed: it is relatively easy to get in, but getting in the field is quite afar

from exerting a meaningful and lasting influence (p. 86). Although plentiful activities haveevolved in the IS community since 1990, to assure a long lasting influence, pluralist research-

ers will need a stronger commitment and continuous effort.

Particular attention could be paid to the publication system and the tenure and promotion

system in the field. Traditional wisdom has long recognized that positivist research is more eas-

ily accepted because its research tradition has been more successfully established (Hir-

schheim & Klein, 2002a). Positivist research is generally considered more rigorous, scientific,

generalizable and reliable, and therefore it is encouraged and widely accepted (Tornatzky &

Klein, 1982). As such, it is safer for authors to stick to positivist orthodoxy (Walsham, 1995b,

p. 391). Walsham (1995b) further indicates:

[P]ositivist research is typically less time consuming than interpretive methods such as eth-

nography, certainly in elapsed time terms, and thus authors are likely to choose the former in

response to our publish or perish age. (p. 391)

While under the pressure of tenure and promotion, researchers would tend to choose a less

time consuming approach positivism to avoid perishing. This might help explain why inter-

pretive influences are still marginal and likely to continue to be this way. As such, for a pluralist

research tradition to be established, both the publication and tenure and promotion systems

might need some modifications. The reason that most researchers employ a less time-

consuming approach is because for tenure and promotion, the quantity of publications counts.

The less time consuming the approach, the more productions are possible; the more produc-

tions, the better the chance for tenure and promotion. While the tenure and promotion system

values the quality of publications as well, it is difficult to get tenure at a good school without

suitable numbers of publications. In the existing system, researchers are more concerned

about how to get a research project published rather than asking significant research ques-

tions. Many research papers fail to address meaningful issues as they are too difficult and take

too long to publish.

On the publication system side, pluralists need to develop ways of educating journal editors

so that they would be more receptive to alternative research approaches. First, editors should


30/39



226

move beyond conservatism and parochialism (Galliers & Meadow, 2003, p. 6) and be more

receptive to authors from other continents. Failing to do so could indicate that the development

of the discipline itself could well be retarded by our failure to appreciate the work of colleagues

elsewhere in the world . . . journal editors may well be the unwitting mouthpieces of unwittingresearchers, publishing results of partial, culturally biased accounts (Galliers & Meadow, 2003,

p. 6). In addition, editors traditionally play the role of gatekeeper rather than facilitator to

ensure the rigour of research publications. However, such a view of rigour typically means

applying a positivist approach and thus excludes other scholarly research (Hirschheim & Klein,

2002a). As Hirschheim and Klein argue, there are many scholarly vehicles for knowledge cre-

ation and they need to be recognized as rigorous as long as they employ sound forms of rea-

soning and evidence giving (p. 13). While positivist research might serve the purpose of

generalizability, interpretivist studies aim for an in-depth understanding of meaning embedded

in the phenomenon investigated (Orlikowski & Baroudi, 1991). They provide an alternative

dimension for scientific inquiries and should supplement rather than exclude each other.In other words, the editors evaluation of a research article should not be based upon the

approach employed, but on the reasoning and evidence provided. Editors should not reject

interpretive research simply because there is no objective criterion for the evaluation. Following

Habermass notion of the force of the better argument, Hirschheim & Klein (2002a) suggest

three aspects for the objective evaluation of research: intelligibility, noveltyand believability.

Intelligibilityrelates to the question how well the research approach and results are compre-

hensible, i.e. how closely others can follow them with similar qualification. Novelty can be

judged in at least three ways: (1) by the amount of new insight added; (2) by the significance

of the research reported in terms of the implications it has for seeing important matters in a

new light and/or provide a new way of thinking about the phenomenon under study; (3) by the

completeness and coherence of the research report(s) . . . Believability, on the other hand,

relates to how well the research arguments make sense in light of our total knowledge. The key

question for believability is how well the research in method and results fits with other ideas and

arguments that are taken for granted within the current stage of knowledge (p. 13). These cri-

teria could then help journal editors to evaluate research employing alternative approaches,

more reasonably.

If the tenure and promotion system would better incorporate the quality of articles into its

evaluation criteria, it would encourage researchers to be more committed to conduct research

projects that require in-depth, long-term endeavour. If the publication system could be more

receptive to alternative research approaches and evaluate the research based on reasoning

and argument instead of the traditional form of rigourgeneralizability, it could encourage

researchers to ask different research questions requiring alternative research approaches. For

a pluralist research tradition to be established, the collaboration of these two systems would

then be essential.

Third, our analysis could help general researchers better allocate their research endeavours

in terms of their publication outcomes. For quantitative researchers, better publication outlets

appear to be MISQ, ISR, EJIS, ICIS and JMIS. The survey method is prevalent in almost every

journal but particularly dominates the research in these five outlets. For qualitative researchers,


31/39



227

on the other hand, the best publication opportunities are found in Information and Organiza-

tion, JITand ISJalthough MISQ, ICIS and EJIS appear increasingly receptive to alternative

methods such as the case study.

There are three implications of these findings. First, for authors, recognizing the political andprofessional preferences of each journal could help them allocate their resources with respect

to the reviewing and publication process. In responding to the publish or perish game, authors

could then better ensure their survival chances. Second, for members of tenure and promotion

committees, a variety of journal outlets such as Information and Organization, JITand ISJ,

among others that publish high quality research using alternative approaches, should be rec-

ognized and taken into consideration in tenure and promotion evaluations. Third, for publish-

ers, an attempt to enhance the availability and visibility of alternative journals is critical. For

example, Information and Organization is not yet electronically available which reduces its

exposure to the general research community.

It is, therefore, not surprising that Information and Organization, JITand ISJdo not appearon the journal-ranking list of ISWorld. High quality research that involves a long-term commit-

ment and is published in these journals appears not to be seriously recognized and appreci-

ated. This may lead researchers to shun alternative research approaches because such

research is not recognized in the tenure and promotion system. Clearly, this serves to dampen

any drive toward pluralism in the IS field.

Limitations

While this paper provides a reflection of IS research, some limitations need to be noted to avoid

possible misinterpretation. First, our goal has been reflection rather than generalization. For

researchers who are interested in generalizing results, the scope of data collection would need

to be broadened. Management Science and Organization Science, for example, are not

included in our survey due to their non-IS orientation. Their inclusion might, however, have

some effect on the field. The most cited articles between 1986 and 1995, according to Wal-

strom & Leonard (2000), include many influential works that are published in those journals:

DeSanctis & Gallupe (1987), Daft & Lengel (1986), Swanson (1974) in Management Science;

and DeSanctis & Poole (1994) in Organization Science. Therefore, for the purpose of gener-

alization, a broader survey that includes a greater variety of journals should be encouraged.

Second, while categorizing the paradigmatic position of articles, we adopted the approach

of Walsham (1995a) and Orlikowski & Baroudi (1991). As Walsham (1995a) indicated, even

within the interpretive paradigm, there is no homogeneous school of thought with respect to

knowledge claims. Different interpretive researchers might apply different categorizing

schemes with different scaling justifications, resulting in different outcomes. Given this poten-

tial subtle bias, we can only claim that our findings serve as a reflection of IS research based

on the Walsham (1995a) and Orlikowski & Baroudi (1991) interpretive school of thought.

Finally, our analysis is based on quantity evidence. One may argue that the quantity of arti-

cles does not accurately reflect their influence. Alternative approaches, such as citation anal-

ysis, that emphasize quality should also be examined. If so, we would then conduct an analysis


32/39



228

based on the results of Walstrom & Leonard (2000). However, none of the most cited articles

on their list was published between 1991 and 1995, which is the overlapping period of their and

our data collection. Therefore, it would not appropriately reflect the influence of articles pub-

lished between 1991 and 2001. A parallel analysis of classic articles published between 1991and 2001 might be needed to better illustrate the influence of research, with respect to quality.

CONCLUSION

Although many schools of thought have emerged and even become well established, reflective

studies that trace such historical trends appears inadequate. Hirschheim & Klein (2002b) once

stated:

We are not sure whether the field considers itself too young to need such a reflection or

whether there simply are not enough old timers around who could provide such a view.

Whatever the case, we believe this to be a serious shortcoming of the IS discipline.

This paper, therefore, aims to reduce such a deficiency. We have first stated the needs of

conducting such a historical reflective piece, described some prior works related to paradig-

matic and methodological issues, and explained our rationale of classifying paradigms and

methodology. Our intent is to continue the research tradition of Orlikowski & Baroudi (1991) and

reflect on the progress made in the IS discipline since 1990. We have argued and illustrated

that many activities have evolved in the IS research community since the paper of Orlikowski

& Baroudi (1991) was published. We then ask, Are those changes reflected in journal publi-

cations? Our empirical analysis of 1893 articles reveals a result similar to that of Orlikowski &

Baroudi (1991), particularly in relation to the paradigmatic trend. The influence of long-term

advocacy of paradigmatic pluralism, thus, appears negligible. In other words, changes that

have occurred and efforts that have been devoted since 1990 still have not manifested them-

selves in journal publications. This could be good news for mainstream navigators (traditional

positivists) and unity advocates (Landry & Banville, 1992) because a more unified, positivist

tradition may have led to a better-established IS field thus far.

Such a result might be due to the existing tenure and promotion and publication systems.

The tenure system overly emphasizes the number of publications in promoting a faculty mem-

ber and, thus, encourages researchers to engage in less time-consuming projects. Positivist

and quantitative research, as opposed to interpretive or qualitative research, generally suits

this purpose better, particularly when researchers are under tremendous publish or perish

pressure. In addition, on the publication system side, journal editors and reviewers tend to be

less receptive to interpretive and qualitative research because the evaluation criteria are less

objective (Hirschheim & Klein, 2002a). Consequently, researchers may be more reluctant to

conduct time-consuming research projects particularly in the domain of interpretivism where

the reward of publication is less likely.

As such, if the IS discipline believes that a monistic, Kuhnian view of IS research is inad-

equate (Hirschheim et al., 1996; p. 4), then a continuous commitment to paradigmatic plural-


33/39



229

ism is critical. This would likely involve the need to modify the current tenure and publication

systems. While the former could consider incorporating the quality of publications into the eval-

uation criteria, the latter could develop a more receptive attitude toward alternative research

approaches. In doing so, IS researchers would not just conceptually embrace the possibility ofalternative assumptions and interests but also empirically allow the notions to be manifested in

journal publications. Only when alternative approaches are empirically reflected in the journal

publications would the IS discipline establish itself in a pluralist research tradition that allows

alternative approaches to help build its body of knowledge.

REFERENCES

Alavi, M., Carlson, P. & Brooke, G. (1989) The ecology of

MIS research: a twenty year status review. In: The Pro-

ceedings of the Tenth International Conference on Infor-

mation Systems, DeGross J.I., Hendersonm, J.C. &

Konsynski, B.R. (eds), pp. 363375. Boston, MA.

Applegate, L.M. & King, J.L. (1999) Rigor and relevance:

careers on the line. MIS Quarterly, 23, 1718.

Benbasat, I., Goldstein, D.K. & Mead, M. (1987) The case

research strategy in studies of information systems. MIS

Quarterly, September, 369386.

Benbasat, I. & Weber, R. (1996) Research commentary:

rethinking diversity in information systems research.

Information Systems Research, 7, 389399.

Benbasat, I. & Zmud, R.W. (1999) Empirical research ininformation systems: the practice of relevance. MIS

Quarterly, 23, 316.

Boland, R.J. Jr & OLeary, T. (1991) Editorial Technolo-

gies of inscribing and organizing. emerging research

agendas. Accounting, Management and Information

Technology, 1, 17.

Burrell, G. & Morgan, G. (1979) Sociological Paradigms

and Organizational Analysis. Heinemann, London, UK.

Cash, J. & Nunamaker, J (1989) The Information Systems

Research Challenge, Vol. I: Qualitative Research Meth-

ods. Harvard Business School Press, Boston, MA, USA.

Cash, J. & Nunamaker, J (1990) The Information Systems

Research Challenge, Vol. II: Experimental Research

Method. Harvard Business School Press, Boston, MA,

USA.

Cash, J. & Nunamaker, J. (1991) The Information Systems

Research Challenge, Vol. III: Survey Research Method.

Harvard Business School Press, Boston, MA, USA.

Chua, W.F. (1986) Radical developments in accounting

thought. The Accounting Review, 61, 601632.

Cook, S.D.N. & Brown, J.S. (1999) Bridging epistemolo-

gies: the generative dance between organizational

knowledge and organizational knowing. Organization

Science, 10, 381400.

Daft, R.L. & Lengel, R.H. (1986) Organizational informa-

tion requirements media richness and structural design.

Management Science, 20, 178188.

Darke, P., Shanks, G. & Broadbent, M. (1998) Success-

fully completing case study research: combining rigour,

relevance and pragmatism. Information Systems Jour-

nal, 8, 273289.

Davenport, T.H. & Markus, M.L. (1999) Rigor vs. relevance

revisited: response to Benbasat and Zmud. MIS Quar-

terly, 23, 1923.

DeSanctis, G. &

Documents

_Chen___Hirschheim_2004_