GIScience Journals Ranking and Evaluation: An International Delphi

Transactions in GIS

, 2008, 12(3): 293–321

© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd

Blackwell Publishing LtdOxford, UKTGISTransactions in GIS1361-16821467-9671© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing LtdXXX Original Articles

GIScience Journals Ranking and EvaluationC Caron, S Roche, D Goyer and A Jaton

Research Article

GIScience Journals Ranking and Evaluation: An International Delphi Study

Claude Caron

GeoBusiness GroupUniversité de Sherbrooke

Daniel Goyer

CGI – Groupe Affaires électroniquesMontréal (Québec), Canada

Stéphane Roche

Center for Research in GeomaticsUniversité Laval

Annick Jaton

Department of Geomatics SciencesUniversité Laval

Keywords

Abstract

Researchers’ fame in most scientific fields is closely linked to their publishingcapacity, both in terms of quantity and quality. In GIScience, as in other fields, thissituation demands that the researcher evaluate and to be very familiar with thescientific journals in which they could publish. Some specialized journals (e.g. Journalof Citation Reports or JCR) are devoted to ranking these reviews according to variousmethods and criteria. Compared to other scientific communities, GIScience isrelatively new and constantly evolving. Therefore, the journals of this field do notbenefit from any real formal ranking yet. The objective of this paper is to presentthe process and results of a study aimed at addressing this gap. More specifically,the challenge is to elaborate an importance ranking of the scientific journals in thefield of GIScience. To do so, both a qualitative (Delphi study carried out with 40international experts) and a quantitative (JCR impact factor) approach has beenimplemented. This triangulation method leads to an early global ranking of thejournals of this field.

1 Introduction

Researchers’ reputation in most scientific fields is closely linked to their publishingcapacity (Hamilton and Ives 1980, Latour 1987, Hardgrave and Walstrom 1997). Their

Address for correspondence:

Claude Caron, GeoBusiness Group, Faculté d’administration, Universitéde Sherbrooke, Sherbrooke (Québec), Canada, J1K 2R1. E-mail: [email protected]

294

C Caron, S Roche, D Goyer and A Jaton


Transactions in GIS

, 2008, 12(3)

production is an important evaluation criterion during the “tenure track” period, aswell as for potential promotions. This production is evaluated not only on the quantityof their published works (for instance, the number of papers published per year), butalso on their quality (awards for the best papers, personal invitation to publish, etc.), aswell as on the way these publications impact their area of expertise (particularly thefrequency with which articles are quoted) (Dubois and Reeb 2000). This

modus operandi

also implies an evaluation of the scientific journals in which researchers publish theirresults. Consequently, in both applied science and humanities, these journals are beingclassified according to various criteria, based on various methods. Specialized journalsare dedicated to this task, such as the Journal of Citation Reports (JCR), which is oneof the most respected by the international scientific community.

Numerous research studies have tackled these issues of evaluating and rankingscientific journals (Garfield 1972, Hamilton and Ives 1980, Tijssen and Van Raan 1990,Gillensen and Stutz 1991, Hardgrave and Walstrom 1997, Moed et al. 1998, Duboisand Reeb 2000, Rousseau 2002). These works have facilitated the evaluation of thespecific contribution of each journal regarding the dissemination of research resultswithin the scientific community, as well as the structuring of communities of practice.

It is now clear that the issue of evaluating and ranking scientific journals is animportant one. Yet, in some relatively new fields, this type of ranking is still quiteinformal. This is particularly the case in GIScience. As far as it is known, journalsrelated to GIScience have not been subject to any formal evaluation. No overall evaluationexists enabling prioritization of the scientific journals in which researchers from ourcommunity publish their works. These journals, however, do not have the same impacton the dynamic of this specific field, and on the transfer of knowledge between GIScientists.Some studies have been carried out in the related field of geography, the most notableof which are Gatrell and Smith (1984) and Lee and Evans (1984, 1985). These studiesare interesting as some of the journals already ranked in the 1980s are still respected byresearchers in GIScience (

Annals of the Association of American Geographers

,

Environmentand Planning A

,

Environment and Planning B

, for instance).Without venturing into any epistemological analysis of GIScience, we assume that

there is

de facto

a community of specialists and scientists who do care about the issuesrelated to the formalization, analysis and use of geographical concepts and data (e.g.University Consortium for Geographic Information Science (UCGIS), Association ofGeographic Information Laboratories for Europe (AGILE)). The GIScience communityhas been developing for more than a decade (Goodchild 1992, Hendriks 1998, Ottens1999, Caron 2000, Fryrear et al. 2001, Murayama 2001, Gatrell 2003). According toBruno Latour’s (1987) analysis, GIScience as described in Goodchild et al. (1999) ismore and more considered to be a special field. Therefore, on the basis of this veryactive community, we wish to identify, then evaluate and rank the journals consideredas being related to GIScience by the practitioners of the field.

Evaluating the specialized journals in GIScience is highly relevant because it willgive a clearer picture of how papers contribute to the advancement of knowledge(Gatrell and Smith 1984). This evaluation will also identify trends that lead to thedevelopment of research policies (Wilson 1985), and that characterize the scientificcommunity (identification of the sub-domains, possible differentiation of the researchers’behaviors according to their type and age, etc.) (Lanegran 1992). The improvement ofthe comprehensive knowledge about the GIScience literature also offers to juniorresearchers, early in their careers, the possibility to identify which journals are the most

GIScience Journals Ranking and Evaluation

295


Transactions in GIS

, 2008, 12(3)

relevant to publish in. Furthermore, depending on their publishing strategy, this knowledgeallows them to target selective and prestigious journals or ones that are not too demanding(Hamilton and Ives 1980, Dubois and Reeb 2000).

It is with a view to improving the knowledge of the GIScience-specialized literaturethat we have carried out this study. The objective is to offer a preliminary formalranking of the scientific journals in GIScience. To do so, we chose a methodologicalapproach focusing on both quality and quantity. First of all, a Delphi study

1

, qualitativeand consensual by nature, was carried out on nearly 40 international experts of thefield. Then, the results obtained were consolidated, using triangulation and comparison,with quantitative approaches (particularly the JRC impact factor).

First we develop a contextual framework by providing an update on the majorscientific journal ranking schemes. Then, we explain the methodological approach, andmore particularly the implementation parameters of the Delphi study. Finally, wepresent the results and the consequent ranking. We conclude with the limits andperspectives of this research.

2 Research Conceptual Framework

2.1 GIScience in Action

Before building up a ranking system of the GIScience journals, it is necessary to identifyall the journals in this area. Therefore, a major precondition is to define the currentboundaries of GIScience. Yet, this is a very difficult task since, reciprocally, the decisionof considering whether or not each journal is a part of GIScience and contributes to thevery definition of the field. Bruno Latour’s works offer in this context an interestingframework (Latour 1987). He believes that publications and journals fully participatein the actors’ network, on which scientific fields develop. Consequently, whatever thedifficulty, proposing a reading and understanding grid of this actors’ network meanscontributing to a better understanding and definition of GIScience.

What does GIS mean? GIS tools, systems or science? : This is the type of questionson which the scientific community, involved in geographic information research, hasbeen focusing for over a decade (Chrisman 1999, Pickles 1997, Wright et al. 1997, Mark2000, Schuurman 2000). Our purpose is not to continue this epistemological debate, butrather to emphasize how much this community is alive, how much this new “discipline”or “science” is going through a very rich development period. Whether this dynamismconfers on this discipline the status of science or not is hard to say, but it surely highlightsthe researchers’ strong will to formalize its basic concepts and to define its boundaries.

The last decade has been particularly rich with discussions that fostered theprogressive emergence of a (more or less) shared vision of GIScience. Numerous formsof research, and numerous positions have been proposed since the Spatial DataHandling conference keynotes presented by Michael Goodchild in Zurich, July 1990,and then in Brussels, April 1991, during the EGIS – European GIS conference (Schuurman1999). In 1991, Jean-Paul Cheylan also developed the GIScience concept in an editorialof the

Revue internationale de géomatique

(previously

Sciences de l’informationgéographique et de l’analyse spatiale

). In Canada, the 1980s and the 1990s marked theconceptualization of the field of geomatics (Paradis 1981), and many other publicationsfollowed (e.g. Gagnon and Coleman 1990, Gagnon and Bédard 1994). In 1993, the

Office de la langue française du Québec

(Canada) defined geomatics as the:

296



Transactions in GIS

, 2008, 12(3)

Discipline dedicated to the management of spatially referenced data, and thusrelies on the scientific concepts and technologies implied in the acquisition,storage, analysis, and distribution of the data (Bergeron 1993).

The first attempts to formalize GIScience plead for their recognition as a fullscientific discipline. This reasoning emerged out of a strong distinction between GIScienceand GIS software, and went even beyond the debate “tools versus system” thatanimated the community at the beginning of the 1990s. Michael Goodchild (1992) gaveno formal definition of GIScience, but he clarified the scope of this field by identifyingits main characteristics. One of the first definitions was proposed by Frank (1992):GIScience addresses the spatial representation of the data allocated to represent a partof the Earth’s surface.

The 1990s provided a wealth of thoughts and debates. The book coordinated byJohn Pickles (1995) is, in this regard, a significant contribution to the reflections onwhich GIScience developed. A few more papers on that subject were published duringthis period (Pickles 1997, Wright et al. 1997). The “critical GIS” stream arose aroundthe same time, and was widely supported by the publications (articles and researchreports) produced through the National Center for Geographic Information Analysis’s(NCGIA’s) Research Initiative 19. The Varenius project, which followed the NCGIA’sinitiatives, began with the publication of a series of papers in the

International Journalof Geographic Information Science

(the “

S

” was then no more for

system

but for

sciences

, as was the case for other journals that were renamed as the GIScience conceptemerged). In the introduction of the Varenius project, Goodchild et al. (1999) devoteda large part of their article to marking the boundaries of GIScience, to define, but alsoto justify their relevance to scientific, societal and technological influences.

GIScience deals with the fundamental issues resulting from the creation, processing,storing, representation, distribution and use of geographical information. That is to say,and as Goodchild et al. (1999) also specified it, the terms geomatics, geoinformatics,spatial information sciences can be considered as synonymous. A recent definitionposted on the Geographic Information Science Center at the University of California-Berkeley website (see http://www.gisc.berkeley.edu/about/whatisgis.html for additionaldetails) supports this statement:

GIScience has recently emerged as the field where problems of data capture,encoding, storage, analysis, retrieval, synthesis, and dissemination of geographicinformation are studied. These problems have increased in significance as newcomputer-based technologies such as geographic information systems (GIS),digital remote sensing (RS) and Global Positioning Systems (GPS) evolve.GIScience is the foundation upon which these GI technologies are built.

Inspired by the above, we marked the boundaries of GIScience as they have developedfor the last two decades, and as they are considered today. We decided to adopt thisformal and operational definition of GIScience through all the current study. On thisbasis, we constituted the initial corpus of journals for the Delphi study (section 3).

The previous timeline briefly emphasizes the dynamics of GIScience developmentand underlines the significant role of publications and journals in the formalization ofthis scientific discipline. They are crucial in communicating ideas within communities ofscientists, because of the concepts and ideas they explicitly formalize through theirnames (which often change as the discipline evolves), and finally because of the way they

http://www.gisc.berkeley.edu/about/whatisgis.html


297


Transactions in GIS

, 2008, 12(3)

are recognized. GIScience emerged and developed mainly in and through communicationsbetween scientists, more formally through publications and journals.

Apart from any formal ranking, all GIScience journals do not count equally. Thereare differences, especially between peer reviewed journals (such as

Transaction in GIS

)and non-reviewed journals (such as

GIM Magazine

). Among peer reviewed journals,some score significantly higher than other ones. This is the reason why the GISciencejournal evaluation and ranking methodology chosen here is so important, not only forthe development of this still relatively new field, but also as an active component ofGIScience in action.

2.2 Journal Ranking Methodologies

The ranking approach elaborated here required the development of a conceptual frame-work, built on the basis of an analysis carried out to identify the main priority-rankingmethods for scientific journals. The criteria being used are variable: some are quantitative,some are qualitative; some rely on statistical or bibliometric methods, or on qualitativeevaluation processes. Singleton (1976), in his work dealing with the ranking of physicalsciences journals, proposed bringing all of these methods together into three majorapproaches: the analysis of journal article citations; scientists’ assessment of journals intheir field of study; and the evaluation of the size or productivity of the journals.

The first approach adopts the analytical techniques dedicated to the citationspublished in journal articles. The “impact factor” of the Journal of Citations Report(JCR) is surely the most common quantitative evaluation process for citations (Rousseau2002). This approach made clear the fact that the quality of scientific journals is linkedto the impact factor, a term created and used for the first time in 1963 by EugeneGarfield, founder of the

Science Citation Index

at the beginning of the 1960s. Amin andMabe (2000) underlined that the value of the impact factor is directly affected by theresearch field at stake, the type and size of the journal, and may fluctuate over time forsome journals (Gillenson and Stutz 1991). Amin and Mabe (2000) also mentioned thatthe “impact factor” is not only a means to describe the impact of journals on the scientificliterature of a specific field, but it is a strong measurement process for the ranking ofscientific journals, yet this type of indicator should only be used independently for eachdiscipline. Some secondary journals in one specific field may be primary journals inanother field. This type of approach was developed by Gatrell and Smith (1984) toassess the relationships between 22 geography journals. In this study, the authorscalculated how many times journal articles were cited by other journals.

The second methodological approach relies on the researchers’ judgement andappraisal (often qualitative) of journals in their own field. It may have to do with the“freshness” or mean elapsed time between the submission of an article and its publication,or with the calibre and recognition of the members appointed to the peer reviewers oran editorial review board. It may also be related to the review process itself (anonymousor not), or with the calibre of the journal. Hamilton and Ives (1980), Gillenson andStutz (1991) and Walstrom et al. (1995) focused on assigning a rating to various journalswith the help of a numerical scale. For instance, Hamilton and Ives (1980) askedexperts in information system management to systematically score the relative importanceof journals in related fields. Their results revealed that the experts’ perception isvery important. Katerattanakul et al. (2003) went even further, asking academics togive their opinion about the relative value of computer science journals via a series of

298



Transactions in GIS

, 2008, 12(3)

questionnaires. Formal techniques have also been developed to assess researchers’perceptions towards journals in their field of study, for instance: Koong and Weistroffer(1989) in management information systems (MIS) and Dubois and Reeb (2000) in inter-national business. Following this approach, Sutter and Kocher (2001) measured, over10 years, the fluctuating recognition of economics journals so as to establish an overallranking through time.

The third methodological approach relies on the quantitative evaluation of the sizeor productivity of journals. Many indicators are used: number of readers, number ofarticles submitted, number of articles accepted, number of issues sold (or delivered),diversity of the subject fields covered, etc. For instance, Todorov and Glaenzel (1990)developed a journal ranking method on the basis of computerized bibliometrictechniques. The journals were organized by speciality areas, relying on the measures ofdispersion (or of concentration) of the subjects treated. Closer to our own field, Lee andEvans (1984, 1985) used this approach to rank 34 geography journals according to thenumber of issues delivered. In this regard, Lanegran (1992) mentioned that

“The growthin the number and circulation of geographical journals has led to speculation about theirprestige and impact

”.So, in many disciplines, studies have been carried out to establish scientific journal

operational rankings. In disciplines closely related to GIScience, issues relating to journalranking have also been tackled. For instance, Brancheau et al. (1996) determined thecritical issues with regard to information systems. Sabrin (2002) developed a classificationof the academic institutions offering information systems training in the United States.Walstrom et al. (1995) dealt with the ranking of information systems conferences. Yet,nothing of the kind formally exists in GIScience.

Granting agencies, or research institutions themselves have a true interest in possessingsuch a ranking system, since publications still constitute today one of the major criteriato evaluate researchers (obtaining grants, tenure and promotion, etc.). For instance, theFrench

Centre National de la Recherche Scientifique

(CNRS) launched in 2004 anaction to assess the place and influence of French journals dedicated to human andsocial sciences at the international level. An inquiry based on the impact factor principlewas then started. Finally, 18 disciplines were chosen, and the journals were analysedbetween 1992 and 2001. This analysis ranked 125 journals in three categories: (A) veryhighly rated international journals, (B) highly rated international journals and (C)national audience journals. This ranking is now a basis for the evaluation of researchersand French laboratories.

3 Description of Methodological Approaches

We believe that an intersectional approach could strengthen the ultimate outcome;consequently we have opted for a methodology integrating the first and second approach.The first step is mainly based on the Delphi method (Loo 2002). The rationale forchoosing this qualitative and consensual method was two-fold: the appropriateness ofbeing directly in contact with the researchers in a still young and developing field;and the lack of quantitative data relating to GIScience journals (with particularly fewjournals mentioned in the JCR). This step allowed us to generate a consensus as well asan initial ranking of GIScience journals, so as to better interpret the results obtainedwith the Delphi.


299


Transactions in GIS

, 2008, 12(3)

3.1 The Delphi Method

The Delphi method has been the subject of many studies in various fields: health(Demeyrick 2003), information systems (Brancheau et al. 1996, Schmidt et al. 2001),banking (Bradley and Stewart 2003), and marketing (Mitchell and McGoldrick 1994).Yet, in GIScience, this method of investigation has not been used extensively (Roche etal. 2003, Péribois et al. 2005). Consequently, before going further, it is useful to reviewsome of the elements relating to this approach. There are several variants (Linstone andTuroff 1975, Mitchell and McGoldrick 1994, Brancheau et al. 1996). We have chosenLoo’s method (2002), which appears to be the more straight forward one. Loo (2002)identifies four major steps: (1) define the problem; (2) select the participants; (3) determinethe size of the sample; and (4) conduct the iterative surveys of the Delphi method.

Kaynak and Macaulay (1984) underscored the fact that the Delphi method is usedin response to a situation or a phenomenon rather than to test hypotheses. That is tosay, the Delphi method is efficient at explaining a problem or defining a situation, ratherthan assessing the relationships between hypotheses. Therefore, we consider that it iswell adapted to develop a GIScience journal ranking methodology.

General approach.

Successive questionnaires are sent to a targeted group of expertson a given theme, until a consensus on the questions posed is reached. This method isused to establish a consensus of opinion on a more or less quantitative specific subject.It allows the researchers to predict, develop and explore the attitudes of a group, as wellas their needs and priorities (Jairath and Weinstein 1994). Iterations allow for feedbackfrom one survey to the other. Usually, three iterative surveys are enough to complete thestudy (Duffield 1988).

Jury.

The Delphi study is directed by a small group of jurors. The members of thejury are not chosen at random but selected according to their subject matter knowledge(Deitz 1987).

Expert.

The definition, the choice and the maintenance of anonymity of the expertsare the key elements of a Delphi study. They prevent personal opinions from leadingthe study. Moreover, anonymity stimulates ideas, as well as observations without anypressure of external factors or from pairs (Goodman 1987).

Group of experts.

The constitution of the group of experts is critical to the rigor ofthe Delphi study (Linstone and Turoff 1975). The chosen respondents must be repre-sentative of the group and be familiar with the discipline or issue. The constitution ofthe group of experts in the present study is mainly based on Martino’s (1983) theories.In some specific fields, experts are too few in number to be chosen randomly. There isno formal rule regarding the size of a group of experts (Linstone and Turoff 1975, Loo2002). Martino (1983) proposes a rule of thumb: a group of 15 to 30 experts is necessaryfor a heterogeneous population, where a group of 5 to 10 experts is enough for ahomogeneous population. To test a medicine, a heterogeneous population will consistof physicians in an academic health-science centre, whereas a homogeneous populationwill consist of specialist physicians in cardiology, for instance.

Limitations/Sources of error.

The Delphi method does not escape criticism. Sackman(1975), and then Woudenberg (1991), raised a series of questions about the scientificbases of the Delphi method, whereas ample scientific evidence clearly demonstrates itsusefulness. Linstone and Turoff (1975), Martino (1983), and Loo (1997, 2002) confirmedthe good performance of the Delphi technique, but also specified the precise criteria thatare required to ensure efficiency and exactness. Constituting the group of experts and

300



Transactions in GIS

, 2008, 12(3)

determining the sample size are particularly important criteria. Choosing experts formdifferent geographical locations, or confirming the experts’ reputation through informaldiscussions is also essential. Based on successful research outcomes, Mitchell andMcGoldrick (1994), Blow and Sprenkle (2001) and Bradley and Stewart (2003) allagree. In order to mitigate the limits of the Delphi method, Dootson (1995) suggestedadopting a triangulation approach, using other complementary methods as well, whichis adopted in the present study.

The sampling of experts.

The sampling process of experts is non-probabilistic innature, since the individuals are not chosen randomly. Sacks (2000) mentioned that themore qualitative studies typically imply small samples, and enhance the sample repre-sentativeness rather than its size. We used Martino’s (1983) guidelines to gather ahomogeneous group.

In this study, the experts were chosen from a thorough study of the peer-reviewjournals in the fields of geomatics, geography, geographical information systems, andother related disciplines. We listed all the Internet sites of these scientific journals so asto identify the members of their respective editorial boards. This identification tookplace over a period of several weeks. Based on an initial population of 1,110 expertsreported in the editorial review boards, a first sample of 174 experts was drawn. Thisfilter was carried out according to the following criteria: experts serving on at least twoeditorial review boards and with an expertise in GIScience. The group includes selectedexperts from all over the globe.

3.2 Constitution of the Initial Corpus of GIScience Journals

First, an initial listing of GIScience journals was compiled. The sampling process, usedto generate this basic corpus from the Delphi study, is non-probabilistic since the jour-nals were not chosen at random. More specifically, we have mainly used the followinginformation sources:

• Journal of Citation Reports 2003 – Social Sciences Edition, Institute for ScientificInformation (ISI), Thomson Scientific, Philadelphia, microfiches.

• A selection of journals from the lists of the following sources : http://www.crg.ulaval.ca/revues/revues.asp (June 2002), http://www.geomatique.georezo.net (June 2002), andhttp://www.forum-pggq.com/liens/index.asp?section=10 (June 2002).

• Research guide 2002–2003, Business Administration Department, Université deSherbrooke.

• Taylor and Francis Journals: http://www.tandf.co.uk/journals/sublist.asp• Blackwell Journals: http://www.blackwellpublishing.com/listofj.asp• Elsevier Journals: http://www.elsevier.com/wps/find/journal_browse.cws_home• Biblio-GIS – Journals: http://perso.wanadoo.fr/dupuy.allignet/bibliosig/revues.htm

An initial sorting, produced by the jury

2

, reduced the corpus of 121 journals identifiedat the beginning in the sources mentioned above, to 84 journals. After the first iteration ofthe Delphi study, scientific journals that were region-based or unknown to most expertswere removed, as well as those published in languages unfamiliar to the experts, or thosewhose content was not relevant to the definition of GIScience adopted in this study.

After the experts’ suggestions and the jury deliberation, the list of scientific journals,used from the second iteration of the Delphi study with regard to the classification,consisted of 54 titles

3

. Among these 54 journals, some are Francophone, but the vast

http://www.crg.ulaval.ca/revues/revues.asp

http://www.crg.ulaval.ca/revues/revues.asp

http://www.geomatique.georezo.net

http://www.forum-pggq.com/liens/index.asp?section=10

http://www.tandf.co.uk/journals/sublist.asp

http://www.blackwellpublishing.com/listofj.asp

http://www.elsevier.com/wps/find/journal_browse.cws_home

http://perso.wanadoo.fr/dupuy.allignet/bibliosig/revues.htm


301


Transactions in GIS

, 2008, 12(3)

majority is Anglophone, i.e. most of them come from North America and WesternEurope.

3.3 Comparison with Other Measurement Approaches

To validate the classification resulting from the Delphi, we have compared it with theJCR impact factor. Only some of the journals listed in the Delphi appear in the JCR,which limits the comparison to only some of the classified journals. The ISI

(

Institutefor Scientific Information) JCR multidisciplinary database offers statistical data used toassess how significant publications are in various fields. Two versions are available:

•

JCR Web Science Edition

: 5,000 leading international scientific journals•

JCR Web Social Sciences Edition

: 1,600 key international journals

The impact factor is a quantitative measure of the frequency with which the articlesin a particular journal of a given field have been cited, compared to the other journalsof the same field. The impact factor is defined as follows:

The average numbers of times recent articles in a specific journal were cited inthe JCR cover year. For JCR impact factors recent articles and those publishedin the two years preceding the JCR cover year. There are other ways ofcalculating journal impact (Garfield 1972).

The impact factor is calculated by dividing the total number of citations publishedduring the previous two years by the total number of articles published in those twoyears. For example, the impact factor of the

American Political Science Review

is 2.448,resulting from the following calculation:

– Number of citations in 2002: 123– Number of citations in 2001: 152– Number of articles published in 2001: 50– Number of articles published in 2000: 46– (123 + 152)/(50 + 46) = 235/96 = 2.448

The comparative approach used here consists of determining whether the sortingin descending order of importance resulting from the Delphi also corresponds to adescending impact factor.

4 Methodology Used in This Study

4.1 Questionnaires

A questionnaire was developed for each round of the Delphi (Appendices 1–4), and wasthen distributed iteratively to the experts. The questionnaire was produced in English

4

.To ensure a good understanding of the questionnaire, pre-tests were conducted beforeeach iteration on the jury of the Delphi and researchers from the GeoBusiness Group(Université de Sherbrooke) and the Center for Research in Geomatics (Laval University).Furthermore, a definition of “GIScience” was included with the questionnaire, in orderto make clear the boundaries of this particular field. To facilitate the experts’ work, thequestionnaires were developed using a four-level classification of journals (Facultéd’administration 2002):

302



Transactions in GIS

, 2008, 12(3)

– Level 1: journals ranking in the top 15% of their field– Level 2: journals ranking between 16% and 50% of their field– Level 3: journals ranking between 51% and 100% of their field– Level 4: journals that are not part of the first three categories

5

When an expert had difficulties classifying some of the journals in one of the fourcategories, this person could indicate “unknown” rather than a number from 1 to 4(Hufnagel and Conca 1994).

An identical analytical method was used between each survey. The jury studied theresults and made certain decisions, such as discarding some redundant journals, andthen developed a questionnaire for the next survey. Each questionnaire listed the jour-nals in alphabetical order to prevent the experts from being influenced by a presentationof the journals in descending order (which may have led them to concentrate on theevaluation of each journal individually).

4.2 First and Second Rounds

In August 2003, we sent an e-mail to the 174 experts of the original group. Thesee-mails included a customized introductory letter, as well as the first questionnaire(Appendix 1). A first reminder was sent three weeks after the initial message, but onlyto 140 experts, for some e-mail addresses were invalid and certain experts had alreadyaccepted or refused to participate in the study. Most of the refusals were on the groundof lack of time, or because some of those contacted were deemed insufficiently qualifiedwith regards to GIScience. Thirty-seven experts (21%) answered the first round.

With the first questionnaire, the experts’ mandate was two-fold: to comment on therelevance of each of the 84 journals appearing on the list, and to suggest adding missingjournals (Appendix 1). The following choices were offered: to decide whether a journalshould remain or be removed from the list for the second round, or to make no com-ment if the expert believed they did not have the necessary knowledge to judge thejournals or to include additional journals. At the end of the first round, the experts’suggestions and comments enabled the jury to correct and enrich the original list.

Twenty-eight additional journals were proposed by the experts during the firstround (Table 1). Once the analysis and screening (number of experts who suggested ajournal, relevance of the journal regarding GIScience and breadth of the journal), thesix following journals were added by the jury:

Cybergeo

,

Directions Magazine

,

GIMInternational

,

ISPRS Highlights

,

Marine Geodesy

and

The Professional Geographer

.Moreover, updates were proposed, because some journals had changed their names

in recent years. It is the case for the

Journal of the North American CartographicInformation Society

, known today as

Cartographic Perspectives

, but also

MappingAwareness

that is now

GI News,

or even the

Revue de géographie de Lyon

which todayis named the

Géocarrefour

. Some experts also emphasized certain redundancies –the same journal was listed twice, but under two different names:

Cartography andGeographic Information Systems

(same as

Cartography and Geographic InformationScience

),

Geographical Systems

(same as

Journal of Geographical Systems

).The second round took place from November 2003 through January 2004. Its

process was the same as the first one, both in terms of questionnaire and outcomemeasures. It included two reminders over a period of a month. Out of the 37 expertscontacted during the second round, only 28 answered the second questionnaire


303


Transactions in GIS

, 2008, 12(3)

(Appendix 2). This questionnaire required the experts to proceed in the same way as forthe first round, but then, they also had to propose a first classification for each journal.To conclude this second round, the jury had once more, and using the same method, todecide about the new journals added, as well as about the journals that the experts couldnot make up their mind, whether they should remain on the list or not.

4.3 Third and Fourth Rounds

The third and fourth rounds took place, respectively from February 2004 to March2004, and from April 2004 to May 2004. Both rounds followed the same patterns andincluded two reminders over a period of a month. The questionnaires distributed to theexperts during both rounds are very similar (see the questionnaire of the third round,Appendix 3). Both iterations have the same goal, namely to request each expert to

Table 1 List of additional journals proposed by the experts (round 1)

Journals proposed Number of suggestions

Administrative Sciences Quarterly 1Arcuser 1Statistics in Medecine 1Catena 1Computer and Operations Research 1Cybergeo 3Directions Magazine 1Environmental Modeling 1Geoderma 1Geo-spatial Information Science (Wuhan) 1GIM International 1Hydrological Processes 1Imaging Notes 1ISPRS Highlights 1Journal Mathematical Geology 2Journal of Geographical Information Science 1Journal of Hydrology 1Journal of the Association for Computing Machinery 1Marine Geodesy 1Mondo GIS 1Network and Spatial Economics 1Ocean and Coastal Management 1Policy Analysis 1Professional Geographer 4Public Administration Review 1Sistema Terra 1Journal of Geography in Higher Education 1XYZ 1

304 C Caron, S Roche, D Goyer and A Jaton

© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing LtdTransactions in GIS, 2008, 12(3)

classify each journal according to the categories ranging from 1 to 4. Unlike the firsttwo rounds, these last two were no longer offering the possibility to add or removejournals from the list. Each expert had two pieces of key information at their disposalto proceed to a classification, that is to say: (1) their own evaluation of each journalduring the previous round and (2) the group of experts’ average rating of each journal.The fact that each expert could have access to the average rating given by the grouprepresented an opportunity to be influenced. The expert could then maintain his/herevaluation for each journal, or follow the overall opinion, so that a group consensuscould be achieved progressively and anonymously.

In fact, 27 experts answered the third iteration, whereas at the end of the fourthand last iteration they were down to 26, which corresponds to a net response rate of14.9% (26/174). This rate is very good and in line with expectations, since the literatureforecasts a 15% response rate for Delphi studies (Linstone and Turoff 1975). Theerosion of the group of experts after each round is a consistent characteristic of Delphistudies, the average desertion rate usually reaching 10% to 40% (32% in our study).The distribution of experts by country at the beginning and end of the study issummarized in Table 2. The large majority came from Northern America (50%) andWestern Europe (27%).

The various iterations of a Delphi are complete only when a final consensus isreached. Therefore, for the third and fourth rounds, the jury had to decide whether theiterations should stop or continue. With regard to our study, four rounds were carriedout, which is in line with the development standards achieved by a Delphi study, whichusually consists of three to six rounds (Bradley and Stewart 2003, Linstone and Turoff1975).

In order to determine accurately the achievement of a consensus, the jury measuredthe stability factor (Scheibe et al. 1975) during the third and fourth rounds. For a giveniteration, the stability factor is the ratio between the total of units changed (in this case the

Table 2 Distribution of experts by countries

Countries Original number of experts Final number of experts

United-States 11 10France 8 3England 5 3Austria 3 2Canada 3 3Germany 1 1Australia 1 0Italy 1 1Netherlands 1 0Switzerland 1 1Tunisia 1 1Japan 1 1Total 37 26

GIScience Journals Ranking and Evaluation 305


differences between the answers given for each journal in two successive questionnaires),and the number of participants (the total number of experts in a given iteration). Thiscalculation gives a percentage change, which should remain below 15% to have theconsensus achieved.

Table 3 shows the classification obtained at the end of the fourth iteration of theDelphi. This ranking reveals a consensus among the experts contacted (for comprehensiveresults, see Appendix 4).

5 Analysis and Triangulation of Results

Once the final results of the Delphi were obtained, we proceeded with the second partof the study, namely the comparison with the JCR impact factor. To do so, we produceda table comparing all the journals of the Delphi, ranking them from 1 to 3 that werementioned in the JCR (Table 4).

Table 4 indicates the JCR Rank in its main field for each journal analysed, whichenables the calculation of the ranking percentage for each of those journals. Forinstance, the Applied Geography journal ranks 25th among the 35 journals listed in thefield “Geography”, that is to say 71st out of 100. Within a categorization ranking from1 to 4, a ranking of 71% based on the JCR data corresponds to a level 3. Out the 44journals ultimately classified from the Delphi, the same journal ranks 34th, which alsocorresponds to a level 3. It is, thus, easy to make comparisons between the Delphi andJCR approaches.

We have been able to compare nearly 50% of the journals from the Delphi (21/44).Based on the analysis of these 21 journals, we can say that for:

57%: both classifications coincide43%: both classifications do not coincide, but out of these 43%:20%: the ranking resulting from the Delphi is just above the one calculated from

the JCR;23%: the ranking resulting from the Delphi is just below the one calculated from

the JCR.

Consequently we can conclude that there is a match between the classificationresulting from the Delphi and the one compiled from the JCR (57%). When there is no match(43%), the ranking given by the Delphi is equally just above or below the one resultingfrom the JCR. On that account, the triangulation of the Delphi results and the JCR dataconfirms the classification obtained. Therefore, we believe that, in regard to the 44 journalsthe international experts consensually classified, the Delphi constitutes an innovativeand sound methodology to determine the importance of the journals in GIScience.

The final ranking, built on the basis of the Delphi and confirmed through a trian-gulation with the JCR, is presented in Table 5.

6 Conclusions

The classification, developed in a rigorous qualitative manner (Delphi method), has beencorroborated by a quantitative approach (JCR). Consequently, the resulting overallranking helps to correct a deficiency in GIScience.



Table 3 Final ranking obtained from the experts who participated to the Delphi

Journal Delphi

International Journal of Geographic Information Science 1.12International Journal of Remote Sensing 1.27Photogrammetric Engineering & Remote Sensing (PE&RS) 1.45Computers and Geosciences 1.52Transactions in GIS 1.58GeoInformatica 1.60Geomatica 1.60Cartography and Geographic Information Sciences 1.64Environment and Planning B 1.64IEEE Transactions Geoscience and Remote Sensing 1.65Remote Sensing of Environment 1.67Computers, Environments and Urban Systems 1.70Annals of the Association of American Geographers 1.72URISA Journal 1.77Environment and Planning A 1.82Landscape Ecology 1.92Cartographica 1.95Journal of Geographic Information and Decision Analysis 2.12Marine Geodesy 2.15Revue Internationale de Géomatique 2.15Geographical System 2.20Professional Geographer 2.20Transactions of the Institute of British Geographers 2.20Earth Surface Processus and Landforms 2.25Progress in Human Geography 2.28Spatial Cognition and Computation 2.29Mapping Sciences and Remote Sensing 2.38Canadian Journal of Remote Sensing 2.39Cartographic Journal 2.39Canadian Geographer 2.45Geoscience Canada 2.47Cybergeo 2.50International Journal of Urban and Regional Research 2.50Applied Geography 2.53Surveying and Land Information Science 2.53Earth Observation and Remote Sensing 2.67Cartographic Perspectives (Journal North American Cartographic Information

Society)2.71

Acta Cartographica 2.73Remote Sensing Reviews 2.80Espace Géographique 2.90Geofocus International Review of GI Science and Technology 3.00Mappemonde 3.07Geomatique Suisse 3.11Geographical Analysis 3.20Géospatial Solutions 3.43Geocarto International 3.71

GIScience Journals R

anking and Evaluation307

© 2008 The A

uthors. Journal compilation ©

2008 Blackw

ell Publishing LtdTransactions in G

IS, 2008, 12(3)

Table 4 Ranks obtained using Delphi and JCR approaches

Whole TitleJCR Rank Field (JCR)

% Field

JCR Calculated Rank

DELPHI Rank

Applied Geography 25/35 Geography 71% 3 34/44Canadian Geographer 21/35 Geography 60% 3 30/44Cartographic Journal 34/35 Geography 97% 3 29/44Environment and Planning A 8/35 Geography 23% 2 15/44Environment and Planning B 9/50 Environmental Studies 18% 2 9/44Geographical Analysis 19/35 Geography 54% 3 43/44Geoscience Canada 110/128 Geosciences, Interdisciplinary Computer Science 86% 3 31/44International Journal of Geographic Information Science

31/78 Computer Science, Information Systems 40% 2 1/44

International Journal of Remote Sensing 3/13 Imaging Science & Photographic Technology 23% 2 2/44International Journal of Urban and Regional Research 8/28 Urban Studies 29% 2 34/44Photogrammetric Engineering & Remote Sensing 21/35 Geography 60% 3 3/44Remote Sensing of Environment 1/11 Remote sensing 9% 1 11/44Professional Geographer 10/35 Geography 29% 2 22/44Progress in Human Geography 1/35 Geography 3% 1 25/44Landscape Ecology 50/128 Geosciences, Multidiciplinary 39% 2 16/44IEEE Transactions Geoscience and Remote Sensing 2/11 Remote Sensing 18% 2 10/44GeoInformatica 22/31 Geography, Physical 71% 3 6/44Earth Surface Processus and Landforms 42/128 Geosciences, Multidisciplinary 33% 2 24/44Earth Observation and Remote Sensing 119/128 Geosciences, Multidisciplinary 93% 3 36/44Canadian Journal of Remote Sensing 7/11 Remote Sensing 64% 3 28/44Computers and Geosciences 77/128 Geosciences, Multidisciplinary 60% 3 4/44



Table 5 Final ranking of GIScience journals

Journal Rank

International Journal of Geographic Information Science 1International Journal of Remote Sensing 1Photogrammetric Engineering & Remote Sensing (PE&RS) 1Computers and Geosciences 1Transactions in GIS 1GeoInformatica 1Geomatica 1Cartography and Geographic Information Sciences 2Environment and Planning B 2IEEE Transactions Geoscience and Remote Sensing 2Remote Sensing of Environment 2Computers, Environments and Urban Systems 2Annals of the Association of American Geographers 2URISA Journal 2Environment and Planning A 2Landscape Ecology 2Cartographica 2Journal of Geographic Information and Decision Analysis 2Marine Geodesy 2Revue Internationale de Géomatique 2Geographical System 2Professional Geographer 2Transactions of the Institute of British Geographers 3Earth Surface Processus and Landforms 3Progress in Human Geography 3Spatial Cognition and Computation 3Mapping Sciences and Remote Sensing 3Canadian Journal of Remote Sensing 3Cartographic Journal 3Canadian Geographer 3Geoscience Canada 3Cybergeo 3International Journal of Urban and Regional Research 3Applied Geography 3Surveying and Land Information Science 3Earth Observation and Remote Sensing 3Cartographic Perspectives (Journal North American Cartographic Information

Society)3

Acta Cartographica 3Remote Sensing Reviews 3Espace Géographique 3Geofocus International Review of GI Science and Technology 3Mappemonde 3Geographical Analysis 3Geocarto International 3Geomatique Suisse 4Géospatial Solutions 4



Indeed, there are limits to these results. First of all, the group of experts contactedis over-represented by residents from North America and Western Europe. Generalizationto the whole international community is, thus, precarious. For instance, how relevantthese results would be to Asia, especially China? Moreover, we did not carry out ananalysis aimed at determining in detail the experts’ specialized fields of expertise.GIScience is characterized by a large variety of disciplinary backgrounds and expertise.These specializations could, if the case arises, challenge the representativeness of thegroup. For example, what could be the impact on the classification of certain journalsthat are more specialized in photogrammetry or remote sensing? It is difficult to say.

The scale proposed here to carry out the Delphi (ranks from 1 to 4) entaileddifficulties in the experts’ decision-making process regarding the evaluation of somescientific journals. For instance, some of them gave half scores, such as 2.5 or 3.5. Then,we had to contact the experts to ask them to make a choice. Other experts informed usthat they had difficulties to carry out the exercise, mainly due to the very limited ratingscale. The rating exercise may have been easier if the experts had a wider choice thanthe initial ranking from 1 to 4. For example, how might a five or seven-point Likertscale have impacted the results?

Finally, the list of scientific journals covered by the Delphi does not pretend to beexhaustive, even though it was compiled on the basis of various sources of information,as well as on suggestions made by the participating experts. It is clear that the methodused to develop this ranking will have to evolve with time and according to the experts’commentaries and further inquiries.

Sometimes though, this classification will necessitate updates. Journals evolve andtheir rankings change over time as well, GIScience will evolve too, just like disciplinescontinually do. The constant redefinition and the steady integration of new conceptsand new information technologies (IT) imply the need to maintain and improve ourglobal knowledge in this field. In order to make guarantees about how current, andthus, useful this classification is, it is important to keep on improving and enhancingthis first version.

Generally speaking, it would be interesting to transpose this type of exercise toconference rankings in GIScience (e.g. COSIT, GIScience conference), with respect tothematic content and prestige, for example. Furthermore, we are convinced that themethodology of this work could be generalized to research fields other than GIScience.The Delphi method appears to be an innovative and efficient way to achieve this purpose.

We consider the innovative approach used in the present study as complementaryto the JCR approach. It efficiently fills some of the gaps of the JCR, such as the lack ofconsideration for some key journals, and because few other rankings of journals fromthe emerging field of GIScience have been attempted. This complementarity might implysome changes relating to the criteria used to constitute the JCR, i.e. to complete thesecriteria with others based on consensus building, such as the Delphi method.

Moreover, since the approach used in the present study aims at reaching consensusamong subject-field experts, we consider it as a sensible means for publishers, people incharge of tenure-track and promotion decisions, and granting agencies to evaluate thecontribution and reputation of journals. We do hope that the present research will leadto the development of a new journal ranking approach in the field of GIScience, andpossibly in other research fields. In this regard, we are ready to get involved in anyinitiative aiming at structuring some kind of process to periodically update the resultsof the present study.



7 Acknowledgements

The authors thank all the anonymous experts who participated to this research. Thequality of the results is directly connected to their generosity. Geoffrey Edwards, David Bachy,and Mokhtar Saada should also be thanked for their review, as well as the NaturalSciences and Engineering Research Council of Canada (NSERC) for their financial support.

Notes

1 The method for structuring a group communication process so that the process is effective inallowing a group of individuals, as a whole, to deal with a complex problem confidentially(Linstone and Turoff 1975).

2 The jury of the Delphi study consists of the authors of this paper.3 One mistake was made during the Delphi study. The title: “Geographical Information Systems

for Urban and Regional Planning” was part of the final list compiling the 54 journals, eventhough it is not a journal but a book. One of the experts had suggested adding this title duringthe first iteration of the Delphi. But, neither the experts nor the jury of the Delphi ever detectedthis mistake afterwards. Yet, it hardly affected the final ranking. We ultimately removed thistitle. This mistake does not invalidate the remainder of the classification obtained.

4 A French version of the questionnaire was available upon request.5 The fourth level corresponds to journals without editorial boards, that is to say professional

or transfer journals.

References

Amin M and Mabe M 2000 Impact factors: Use and abuse. Perspectives in Publishing 1: 1–6Bergeron M 1993 Vocabulaire de la Géomatique. Québec, Cahiers de l’Office de la langue

française, Publications du QuébecBlow A J and Sprenkle D H 2001 Common factors across theories of marriage and family therapy:

A modified Delphi study. Journal of Marital and Family Therapy 27: 385–402Bradley L and Stewart K 2003 A Delphi of Internet banking. Marketing Intelligence and Planning

21: 272–81Brancheau J C, Janz B D, and Wetherbe J C 1996 Key issues in information systems management:

1994–95 SIM Delphi results. MIS Quarterly 20: 225–42Caron C 2000 Les projets de géomatisation: contexte social et organisationnel. Revue Internationale

de Géomatique 10: 1–26Chrisman N R 1999 What does “GIS” mean? Transactions in GIS 3: 175–86Deitz T 1987 Methods for analysing data from Delphi panels. Technological Forecasting and

Social Change 31: 79–85Demeyrick J 2003 The Delphi method and health research. Health Education 103: 7–16Dootson S 1995 An in-depth study of triangulation. Journal of Advanced Nursing 22: 183–7Dubois F L and Reeb D 2000 Ranking the international business journals. Journal of International

Business Studies 31: 689–704Duffield C 1988 The Delphi technique. Australian Journal of Advanced Nursing 6(2): 41–5Faculté d’administration 2002 Guide de la recherche 2002–2003. Sherbrooke, Université de

Sherbrooke Faculté d’administrationFrank A 1992 Spatial concept geometric data models, and geometric data structures. Computer

and Geosciences 18: 409–17Fryrear R, Prill E, and Worzala M 2001 The use of Geographic Information Systems by corporate

real estate executives. Journal of Real Estate Research 22: 153–64Gagnon P, and Bédard Y 1994 From surveying to geomatics: Evolution of education needs to

adapt to a new paradigm (a Canadian perspective). In Proceedings of the Third InternationalThink Tank of the Atlantic Institute, Vienna, Austria



Gagnon P and Coleman D J 1990 La géomatique: une approche systémique intégrée pour répondreaux besoins d’information sur le territoire. CISM Journal 44: 383–9

Garfield E 1972 Citation analysis as a tool in journal evaluation. Science 1: 527–44Gatrell T 2003 GIS and public health. Economic Geography 79: 341–2Gatrell T and Smith A 1984 Networks of relations among a set of geographical journals. Profes-

sional Geographer 36: 300–7Gillenson M L and Stutz J D 1991 Academic issues in MIS: Journals and books. MIS Quarterly

15: 447–52Goodchild M F 1992 Geographical Information Science. International Journal of Geographical

Information Systems 6: 31–45Goodchild M F, Egenhofer M J, Kemp K K, Mark D M, and Sheppard E 1999 Introduction to

the Varenius project. International Journal of Geographical Information Science 13: 731–45Goodman C M 1987 The Delphi technique: A critique. Journal of Advanced Nursing 12: 729–34Hamilton S and Ives B 1980 Communication of MIS research: An analysis of journal stratification.

In Proceedings oh the International Conference on Information Systems: 220–32Hardgrave B C and Walstrom K A 1997 Forums for MIS scholars. Communication of the ACM

40(11): 119–24Hendriks P H J 1998 Information strategies for Geographical Information Systems. International

Journal of Geographical Information Science 12: 621–39Hufnagel E M and Conca C 1994 User response data: The potential for errors and biases.

Information Systems Research 5: 48–73Jairath N and Weinstein J 1994 The Delphi methodology: Administrative applications. Canadian

Journal of Nursing Administration 7(4): 29–42Katerattanakul P, Han B, and Hong S 2003 Objective quality ranking of computing journals.

Communications of the ACM 46(10): 111–8Kaynak K and Macaulay J A 1984 The Delphi technique in the measurement of tourism market

potential. Tourism Management: 87–101Koong K S and Weistroffer H R 1989 Faculty usage of management information systems journals:

A survey. Journal of Computer Info Systems 30(1): 1–4Lanegran 1992 Communication. In Abler R F, Marcus M G, and Olson J M (eds) Geography’s

Inner Worlds. New Brunswick, NJ, Rutgers University Press: 187–211Latour B 1987 Science in Action: How to Follow Scientists and Engineers through Society.

Cambridge, MA, Harvard University PressLee D and Evans A 1984 American geographers’ rankings of American geography journals.

Professional Geographer 36: 292–300Lee D and Evans A 1985 Geographers’ rankings of foreign geography and non-geography journals.

Professional Geographer 37: 396–402Linstone L and Turoff M 1975 The Delphi Method: Techniques and Applications. Reading, MA,

Addison-WesleyLoo R 1997 Managing workplace stress in Canadian healthcare organizations. Journal of Man-

agement Development 16: 680–9Loo R 2002 The Delphi method: A powerful tool for strategic management. International Journal

of Police Strategies and Management 25: 762–9Mark D M 2000 Geographic Information Science: Critical issues in an emerging cross-disciplinary

research domain. Journal of Urban and Regional Information Systems Association 12(1): 45–54

Martino J P 1983 An Introduction to Technological Forecasting. London, Gordon and BreachMitchell V W and McGoldrick P J 1994 The role of geodemographics in segmenting and targeting

consumer markets: A Delphi study. European Journal of Marketing 28(5): 54–75Moed H F, Van Leeuwen T N, and Reedijk J 1998 A new classification system to describe the

aging of scientific journals and their impact factors. Journal of Documentation 54: 387–419Murayama Y 2001 Geography with GIS. GeoJournal 52: 165–71Ottens H 1999 Geographic Information Systems: An introduction. GeoJournal 48: 341–4Paradis M 1981 De l’arpentage à la géomatique. The Canadian Surveyor 35: 262–8Péribois K, Roche S, and Caron C 2005 Etudes des variables conditionnant l’utilisation de

l’information géographique pour la participation publique locale. In Actes du Colloque Inter-national de Geomatique et d’Analyse Spatiale – SAGEO, Avignon, France



Pickles J (ed) 1995 Ground Truth: The Social Implications of Geographic Information Systems.New York, Guilford Press

Pickles J 1997 Tool or science? GIS, technoscience and the theoretical turn. Annals of the Associationof American Geographers 87: 363–72

Roche S, Sureau K, and Caron C 2003 How to improve the social-utility value of geographicinformation technologies for the French local governments? A Delphi study. Environmentand Planning B 30: 429–47

Rousseau R 2002 Journal evaluation: Technical and practical issues. Library Trends 50: 418–39Sabrin M 2002 A ranking of the most productive business ethics scholars: A five-year study.

Journal of Business Ethics 36: 355–80Sackman H 1975 Delphi Critique. London, Lexington BooksSacks A M 2000 Research Measurement, and Evaluation of Human Resources. Scarborough,

Ontario, Nelson Thomson LearningScheibe M, Skutsch M, and Schofer J 1975 Experiments in Delphi methodology. In Linstone L and

Turoff M (eds) The Delphi Method: Techniques and Applications. Reading, MA, Addison-Wesley 1–12

Schmidt R, Lyytinen K, Keil M, and Cule P 2001 Identifying software project risks: Aninternational Delphi study. Journal of Management Information 17: 5–36

Schuurman N 1999 Speaking with the enemy? An interview with Michael Goodchild. Environmentand Planning D 17: 1–15

Schuurman N 2000 Trouble in the heartland: GIS and its critics in the 1990s. Progress in HumanGeography 24: 569–90

Singleton A 1976 Journal ranking and selection: A review in physics. Journal of Documentation32: 258–89

Sutter M and Kocher M G 2001 Tools for evaluating research output: Are citation-based rankingsof economics journals stable? Evaluation Review 25: 555–66

Tijssen R J W and Van Raan A E J 1990 A net citation balance: A measure of influence betweenscientific journals. Journal of the American Society for Information Science 41: 298–304

Todorov R and Glaenzel W 1990 Compureur bibliometrics for journal classification. InformationProcessing, and Management 26: 673–80

Walstrom K A, Hardgrave B C, and Wilson R L 1995 Forums for management informationsystems scholars. Communications of the ACM 38(3): 93–107

Wilson A G 1985 Raising the levels of ambition in research: Some lessons from the journal.Environment and Planning A 17: 465–70

Woudenberg F 1991 An evaluation of Delphi. Technological Forecasting and Social Change 40:131–50

Wright D J, Goodchild M F, and Proctor J D 1997 Demystifying the persistent ambiguity of GISas “tool” versus “science”. Annals of the Association of American Geographers 87: 346–62

Appendix 1: Questionnaire #1

Please, answer the following 3 questions that represent your opinion.

1) Look at the scientific and professional journals above and indicate those which,according to your expertise, do not deserve to be evaluated because they do notbelong in the field GIS. In the second column entitled “RELEVANT”, pleaseanswer by Yes or No.

2) Always according to the field of GIS, indicate the scientific or professional journalsthat we omitted and those which deserve to be considered in the reserved area:“Omitted journals”.

3) Please reply to us as soon as possible with the questionnaire completed at:

• Email adress: [email protected]• Fax number: (819) 821-7934. (Directed to the attention of Claude Caron)



Journal Relevant? (Yes/No)

Acta CartographicaAnnales de GéographieAnnals of the Association of American GeographersApplied GeographyAustralian GeographerCanadian GeographerCanadian Journal of Remote SensingCartographic JournalCartographicaCartographie MondialeCartography and Geographic Information SciencesCartography and Geographic Information SocietyCartography and Geographic Information SystemsCasual CartographerComputers and GeosciencesComputers, Environments and Urban SystemsContourEarth Observation and Remote SensingEarth Observation MagazineEarth Surface Processes and LandformsEconomic GeographyEnvironment and Planning AEnvironment and Planning BEnvironment and Planning CEnvironment and Planning DEurasian Geography and EconomicsEuropean Journal of GeographieEuropean Urban and Regional StudiesGeo InfoGeo Info systemsGeocarto InternationalGeofocus International Review of Geographical Information Science

and TechnologyGeoforumGeographical AnalysisGeographical and Environmental ModelingGeographical Information Systems for Urban and Regional PlanningGeographical JournalGeographical ReviewGeographical SystemGéographie Physique et QuaternaireGeographyGeoInformaticaGeoinfosystemsGeomatica



Omitted journals

Geoscience CanadaGéospatial SolutionsGéoworldGIS EuropeGIS WorldGIS World Reports IEEE Transactions Geoscience and Remote SensingInternational Journal of Geographical Information ScienceInternational Journal of Remote SensingInternational Journal of Urban and Regional ResearchJournal of Geographic Information and Decision AnalysisJournal of Geographical SystemsJournal of GeographyJournal of the North American Cartographic Information SocietyJournal of Urban Planning and Development – ASCEJournal of Urban TechnologyKarlrusher Geoinformatik ReportLandscape EcologyLes Cahiers de Géographies du QuébecL’espace GéographiqueMap World MagazineMappemondeMapping (Spanish)Mapping AwarenessMapping Sciences and Remote SensingMensuration, Photogrammétrie et Génie RuralMeridianPhotogrammetric Engineering & Remote Sensing (PE&RS)Progress in Human GeographyRemote Sensing of EnvironmentRemote Sensing ReviewsRevue Belge de GéographieRevue Géographique de LyonRevue Internationale de GéomatiqueSpatial Cognition and ComputationSurveying and Land Information SystemsTransactions in GISTransactions of the Institute of British GeographersUrban GeographyURISA Journal




Dear Sir/Madam,

We are pleased to note that you are among the 38 experts that answered our first surveyconcerning our research project for classifying the scientific journals in the field ofGeographic Information Science (GIS). For a second time, we request your professionnalskills to complete Survey # 2.

Following your suggestions from the first round, we eliminated nearly 20 journals.However, we need your opinion, a second time, to evaluate some other journals fromthe first round. Also, some new suggestions of journals provided by some of yourcolleagues who answered the first survey need to be evaluated.

To do so, we provide you a definition of Geographic information science:

“GIScience has recently emerged as the field where problems of data capture,encoding, storage, analysis, retrieval, synthesis, and dissemination of geo-graphic information are studied. These problems have increased in significanceas new computer based technologies such as geographic information systems(GIS), digital remote sensing (RS) and Global Positioning Systems (GPS) evolve.GI Science is the foundation upon which these GI technologies are built.”

(Source: http://www.gisc.berkeley.edu/about/whatisgis.html)

Please, answer the following two questions that represent your opinion.

1) Look at the scientific journals above and indicate those which, according to yourexpertise, do not deserve to be evaluated because they do not belong in the field ofGeographic Information Science. Please answer the question in the column entitled“Yes, No or ?” question. (If you are unfamiliar with the journal, put a questionmark (?))

2) For each journal in both lists, identify its ranking according to the scale below(If you are unfamiliar with the journal, put a question mark (?))

– Level 1: Peer journal classified among the first 15% of their field– Level 2: Peer journal classified between 16% and 50% of their field– Level 3: Peer journal classified between 51% and 100% of their field– Level 4: Journal not part of the first three categories

(For example: A journal that does’t have a referee process.)

Please reply by December 12th, 2003 with the survey completed at:

• Email address: [email protected]• Fax number: (819) 821-7934. (Directed to the attention of Claude Caron)

Journal (Yes, No or ?) Level

Canadian GeographerCartographie MondialeCasual CartographerCybergeoDirections magazine




Earth Surface Processes and LandformsEspace GéographiqueGeo InfoGeoscience CanadaGI News (Mapping Awareness)GIM internationalInternational Journal of Urban and Regional ResearchISPRS highlightsJournal of Urban TechnologyLandscape EcologyMarine GeodesyMensuration, Photogrammétrie et Génie RuralProfessional GeographerProgress in Human GeographyTransactions of the Institute of British Geographers

Part 2 of the survey Level

Acta CartographicaAnnals of the Association of American GeographersApplied GeographyCanadian Journal of Remote SensingCartographic JournalCartographicaCartography and Geographic Information SciencesComputers and GeosciencesComputers, Environments and Urban SystemsEarth Observation and Remote SensingEarth Observation MagazineEnvironment and Planning AEnvironment and Planning B

Journals Level

Geocarto InternationalGeofocus International Review of Geographical Information Science and

TechnologyGeographical AnalysisGeographical Information Systems for Urban and Regional PlanningGeographical SystemGeoInformaticaGeomaticaGéospatial Solutions (GeoInfo systems)Géoworld (GIS World)



We warmly thank you for your participation in this research.


Dear Sir/Madam,

We are pleased to note that you are among the experts that answered our second DelphiSurvey concerning our research project for classifying the scientific journals in the fieldof Geographic Information Science (GIS). We are providing you with the results ofSurvey #2 and we require your professionnal skills to complete Survey # 3.

Referring to the definition of Geographic Information Science *, please answer thetwo questions below:

*“GIScience has recently emerged as the field where problems of data capture,encoding, storage, analysis, retrieval, synthesis, and dissemination ofgeographic information are studied. These problems have increased insignificance as new computer based technologies such as geographicinformation systems (GIS), digital remote sensing (RS) and Global PositioningSystems (GPS) evolve. GI Science is the foundation upon which these GItechnologies are built.”

Source: http://www.gisc.berkeley.edu/about/whatisgis.html

IEEE Transactions Geoscience and Remote SensingInternational Journal of Geographical Information ScienceInternational Journal of Remote SensingJournal of Geographic Information and Decision AnalysisCartographic Perspectives (Journal North American Cartographic Information Society)MappemondeMapping Sciences and Remote SensingPhotogrammetric Engineering & Remote Sensing (PE&RS)Remote Sensing of EnvironmentRemote Sensing ReviewsRevue Internationale de GéomatiqueSpatial Cognition and ComputationSurveying and Land Information SystemsTransactions in GISURISA Journal

The research team:

Claude CARON, research responsible Annick JATON, research adviserUniversity of Sherbrooke (Québec, Canada) Laval University (Québec, Canada)

Stéphane ROCHE, researcher Daniel GOYER, research assistantLaval University (Québec, Canada) University of Sherbrooke (Québec,

Canada)And University of Angers (France)




1) Specify in which domain you consider yourself as an expert. (Cartography,Geography, Geodesy, . . .)

2a) In the listing below, take notice of your first evaluation in the column “You” andthen compare it with the column “Group” (mean value of the group).

2b) For each journal that you evaluated, you have two options:

A. You can propose a new ranking* in the column “Now”.B. You can keep your old ranking by leaving the column “Now” empty.

* (Don’t forget that you must rank the journals with an integer)

2c) Also, if you wish to rank a journal that you have not evaluated in the first survey,you are welcome to do so.

Please reply by March 5th 2004 with the survey completed at:

• Email address: [email protected]• Fax number: (819) 821-7934. (Directed to the attention of Claude Caron)

(If you are unfamiliar with the journal, put a question mark (?) or leave it empty)

– Level 1: Peer journal classified among the first 15% of their field– Level 2: Peer journal classified between 16% and 50% of their field– Level 3: Peer journal classified between 51% and 100% of their field– Level 4: Journal not part of the first three categories

Journal You Group Now

Acta Cartographica 2.7Annals of the Association of American Geographers 1.8Applied Geography 2.3Canadian Geographer 2.4Canadian Journal of Remote Sensing 2.3Cartographic Journal 2.2Cartographic Perspectives (Journal North American Cartographic

Information Society)2.6

Cartographica 1.9Cartography and Geographic Information Sciences 1.6Computers and Geosciences 1.5Computers, Environments and Urban Systems 1.8Cybergeo 2.7Directions magazine 3.8Earth Observation and Remote Sensing 2.6Earth Observation Magazine 3.5Earth Surface Processus and Landforms 2.4Environment and Planning A 1.8Environment and Planning B 1.6Espace Géographique 2.4Géo Europe (GI News – Mapping Awareness) 3.7Geo Info 3.7



We warmly thank you for your participation in this research.

Geocarto International 2.8Geofocus International Review of Geographical Information Science

and Technology3.1

Geographical Analysis 1.5Geographical Information Systems for Urban and Regional Planning 2.1Geographical System 1.8GeoInformatica 1.7Geomatica 2.3Geoscience Canada 3.0Géospatial Solutions 3.6Géoworld 3.7GIM international 3.6IEEE Transactions Geoscience and Remote Sensing 1.6International Journal of Geographical Information Science 1.1International Journal of Remote Sensing 1.4International Journal of Urban and Regional Research 2.3ISPRS highlights 3.3Journal of Geographic Information and Decision Analysis 2.1Landscape Ecology 1.8Mappemonde 2.9Mapping Sciences and Remote Sensing 2.3Marine Geodesy 2.5Mensuration, Photogrammétrie et Génie Rural 3.0Photogrammetric Engineering & Remote Sensing (PE&RS) 1.5Professional Geographer 2.1Progress in Human Geography 1.9Remote Sensing of Environment 1.7Remote Sensing Reviews 2.7Revue Internationale de Géomatique 2.2Spatial Cognition and Computation 2.4Surveying and Land Information Systems 2.5Transactions in GIS 1.7Transactions of the Institute of British Geographers 1.9URISA Journal 1.8

The research team:

Claude CARON, research responsible Annick JATON, research adviserUniversity of Sherbrooke (Québec, Canada) Laval University (Québec, Canada)

Stéphane ROCHE, researcher Daniel GOYER, research assistantLaval University (Québec, Canada) University of Sherbrooke (Québec,

Canada)And University of Angers (France)

320C

Caron, S R

oche, D G

oyer and A Jaton

© 2008 The A


2008 Blackw


IS, 2008, 12(3)

Appendix 4: Delphi Final Results

# Journal

Average (Rank) Std. Deviation Stability

Rnd. #2 Rnd. #4 Rnd. #2 Rnd. #4 Rnd. #3 Rnd. #4

33 International Journal of Geographical Information Science 1.08 1.12 0.27 0.33 0.12 0.0434 International Journal of Remote Sensing 1.38 1.27 0.59 0.46 0.13 0.0043 Photogrammetric Engineering & Remote Sensing (PE&RS) 1.48 1.45 0.59 0.67 0.09 0.0010 Computers and Geosciences 1.52 1.52 0.58 0.51 0.13 0.0451 Transactions in GIS 1.71 1.58 0.84 0.65 0.12 0.1325 GeoInformatica 1.74 1.60 0.91 0.68 0.11 0.1526 Geomatica 2.31 1.60 0.86 0.60 0.07 0.059 Cartography and Geographic Information Sciences 1.59 1.64 0.71 0.66 0.09 0.0918 Environment and Planning B 1.59 1.64 0.72 0.73 0.13 0.0932 IEEE Transactions Geoscience and Remote Sensing 1.63 1.65 0.49 0.49 0.06 0.0546 Remote Sensing of Environment 1.65 1.67 0.75 0.59 0.11 0.1111 Computers, Environments and Urban Systems 1.78 1.70 0.51 0.47 0.17 0.002 Annals of the Association of American Geographers 1.78 1.72 0.93 0.74 0.08 0.0853 URISA Journal 1.77 1.77 0.74 0.61 0.09 0.1417 Environment and Planning A 1.77 1.82 0.80 0.80 0.23 0.0538 Landscape Ecology 1.80 1.92 0.79 0.86 0.17 0.158 Cartographica 1.91 1.95 0.61 0.59 0.10 0.0537 Journal of Geographic Information and Decision Analysis 2.12 2.12 0.62 0.49 0.17 0.1241 Marine Geodesy 2.50 2.15 1.00 0.55 0.25 0.0048 Revue Internationale de Géomatique 2.15 2.15 0.80 0.80 0.29 0.1524 Geographical System 1.78 2.20 0.73 0.62 0.06 0.0544 Professional Geographer 2.13 2.20 0.75 0.83 0.06 0.0052 Transactions of the Institute of British Geographers 1.92 2.20 0.86 0.77 0.17 0.0016 Earth Surface Processes and Landforms 2.43 2.25 0.79 0.71 0.00 0.2545 Progress in Human Geography 1.92 2.28 0.92 0.83 0.08 0.1149 Spatial Cognition and Computation 2.36 2.29 0.65 0.61 0.08 0.00

GIScience Journals R

anking and Evaluation321

© 2008 The A


2008 Blackw


IS, 2008, 12(3)

40 Mapping Sciences and Remote Sensing 2.33 2.38 0.75 0.51 0.15 0.005 Canadian Journal of Remote Sensing 2.31 2.39 0.59 0.50 0.06 0.066 Cartographic Journal 2.22 2.39 0.65 0.50 0.06 0.064 Canadian Geographer 2.38 2.45 0.50 0.52 0.00 0.0927 Geoscience Canada 3.00 2.47 0.71 0.64 0.13 0.1312 Cybergeo 2.71 2.50 1.20 0.97 0.25 0.2035 International Journal of Urban and Regional Research 2.25 2.50 1.04 0.71 0.38 0.103 Applied Geography 2.33 2.53 0.80 0.70 0.20 0.0550 Surveying and Land Information Systems 2.53 2.53 0.74 0.64 0.13 0.0014 Earth Observation and Remote Sensing 2.57 2.67 0.74 0.62 0.14 0.007 Cartographic Perspectives (Journal North American Cart. Inf. Society) 2.62 2.71 0.74 0.61 0.07 0.141 Acta Cartographica 2.69 2.73 0.63 0.59 0.00 0.0747 Remote Sensing Reviews 2.67 2.80 0.48 0.42 0.10 0.1019 Espace Géographique 2.38 2.90 0.97 0.74 0.22 0.3022 Geofocus International Review of GI Science and Technology 3.11 3.00 1.10 0.60 0.09 0.0839 Mappemonde 2.92 3.07 0.66 0.47 0.14 0.2142 Mensuration, Photogrammétrie et Génie Rural 3.00 3.11 0.76 0.33 0.25 0.2223 Geographical Analysis 1.54 3.20 0.87 0.79 0.20 0.3028 Géospatial Solutions 3.58 3.43 1.08 0.53 0.20 0.1429 Geoworld 3.73 3.50 0.45 0.76 0.25 0.1536 ISPRS highlights 3.25 3.63 1.04 0.52 0.29 0.2515 Earth Observation Magazine 3.50 3.64 0.74 0.50 0.14 0.1420 Geo Info 3.71 3.65 0.48 0.49 0.06 0.0631 GIM international 3.64 3.69 0.65 0.48 0.09 0.0013 Directions magazine 3.83 3.71 0.41 0.49 0.00 0.0021 Geocarto International 2.80 3.71 0.70 0.47 0.24 0.0030 GI News (Mapping Awareness) 3.69 3.71 0.61 0.46 0.00 0.08

Average 2.36 2.42 0.73 0.61 0.132 0.091

# Journal

Average (Rank) Std. Deviation Stability

Rnd. #2 Rnd. #4 Rnd. #2 Rnd. #4 Rnd. #3 Rnd. #4

Appendix 4: Continued

Documents

GIScience Journals Ranking and Evaluation: An International Delphi