C O M M E N T A R Y

Trouble with the Gray Literature

Richard T. Corlett1

Department of Biological Sciences, National University of Singapore, 14 Science Drive 4, Singapore 117543

ABSTRACT

In much of the tropics, the gray literature—published by nongovernmental organizations, governments, intergovernmental organizations, consultancies, private com-panies, and individuals—has a greater volume than the peer-reviewed scientific literature in ecology and conservation. I discuss why this is a problem in terms ofquality, discovery, access, and archiving. Unpublished dissertations and theses are another vast untapped source of information in tropical biology. Internet searchengines can potentially integrate access to all information sources, but only if the producers of gray literature and theses greatly improve electronic access. Digitalrepositories can provide both this accessibility and permanent archiving.

Key words: archives; data bases; digital repository; dissertations; gray literature; NGOs; peer review; theses.

IF YOU ARE READING THIS, IT IS BECAUSE IT PASSED A FIRST SCREENING BY

THE EDITOR OF BIOTROPICA, was sent out to referees, modified in the

light of their comments, and a final version approved by all con-

cerned. Unless everyone was asleep on the job, this process will have

ensured that you are not totally wasting your time. Obvious errorsand omissions will have been corrected and the conclusions will at

least be defendable, if not necessarily correct. Moreover, you need

not be reading this in 2011, but could be accessing it at any time in

the future. In short, this article is now part of the scientific litera-

ture: an accumulating body of formally published science. A min-

imum quality standard is ensured by formal peer review, while

accessibility follows from the desire of publishers to maximize their

profits and authors to maximize their citations. Archiving in insti-tutional libraries is as permanent as anything is. Electronic journals

and open access have modified this traditional model, but peer

review is usually retained, publication is mostly by professionals,

accessibility is increased, and the need for archiving is recognized

(e.g., http://www.portico.org).

In the tropics, however, there is an at least equal volume of

material related to ecology and conservation in the so-called ‘gray

literature’. There is no simple definition of which literature is ‘gray’,but the key feature is that it is not published and disseminated by

commercial publishers, but by organizations where publishing is

not the primary activity (http://www.greynet.org). Journals and

monographs published by universities and other academic institu-

tions are also usually considered ‘nongray’, but conference proceed-

ings are definitely gray (e.g., Lacanilao 1997), unless published as

part of a continuing series. In tropical biology, most gray literature

is produced by nongovernmental organizations (NGOs, includingcharities, nonprofits, and various other civil society organizations),

governments at various levels, intergovernmental organizations,

such as the United Nations and World Bank, environmental con-

sultancies, private companies involved in businesses other than

publishing, and, increasingly, freelance individuals. This literature

is not formally peer reviewed—often not reviewed at all—and the

quality varies hugely. Most is initially made available on the Inter-

net, if you know where to look, but much also is not, for a variety of

reasons, including interagency rivalries, commercial sensitivity,

government secrecy, fear of plagiarism, and lack of fundingor technical skills. The gray literature appears to be most volumi-

nous in Southeast Asia, which is underrepresented in the global

scientific literature (Sodhi & Liow 2000), but it is a pantropical

phenomenon.

The most common complaints against the gray literature—

low quality and low accessibility—might appear to cancel each

other out, but quality and accessibility are not correlated. Most re-

ports contain some useful information and the best are excellent.Most often it is the data that is of lasting scientific value, not the

analysis and conclusions. The big international NGOs (BINGOs)

have more money than most local researchers in the tropics, so their

reports may have useful information derived from remote sensing,

camera-traps, and other resource-intensive techniques, while gov-

ernment departments have access to data that are not available to

other researchers. Moreover, gray literature commissioned by

governments may have large impacts on policy irrespective of itsquality.

In the tropics, the gray literature is often the only source of

information on particular sites or projects, or on particular species

(Mrosovsky & Godirey 2008). Distribution and status studies that

ignore the gray literature may well be misleading. Meta-analyses

that exclude the gray literature risk over-representing statistically

significant results and large effect sizes for the simple reason that

they are more likely to be published (Jennions & M�ller 2002,Conn et al. 2003). Even the worst of the gray literature is at least a

record of how governments and NGOs have spent other peoples’

money. The effectiveness of conservation efforts will only increase

when it is possible to evaluate the results and see what worked and

what did not (Ferraro & Pattanayak 2006, Linkie et al. 2008), yet

this is rarely possible at present since past conservation efforts in the

tropics—especially those that failed—are generally not accessibly

documented. Even in the developed world, practical conservationReceived 19 May 2010; revision accepted 15 July 2010.1Corresponding author; e-mail: corlett@nus.edu.sg

BIOTROPICA 43(1): 3–5 2011 10.1111/j.1744-7429.2010.00714.x

r 2010 The Author(s) 3

Journal compilation r 2010 by The Association for Tropical Biology and Conservation

work, such as the management of biological invasions, tends to be

documented only in the gray literature (Genovesi 2005, Simberloff

2009).

Proponents of the gray literature use two major arguments:first, it is in addition to the scientific literature not instead of it and,

second, it is actually more accessible to practitioners in the field than

is scientific literature, in part because it is usually free. The first

argument is at least partly valid, since it is undoubtedly easier to

produce a gray literature report than to get the same information

into the scientific literature and many reports would not survive

peer review in their present form. The most valuable data in many

reports could, however, be published in local or regional naturalhistory journals. These journals vary in quality and accessibility, but

in tropical Asia several are decades old and permanently archived in

libraries around the world. The oldest journals are mostly in

English, but there are now on-line biological journals in all the

world’s major tropical languages. These journals need our support

and using them to archive material from the gray literature is one

way to ensure their continued survival.

The second argument simply does not match with my ownexperience in the tropics or that of anyone else I have asked. Yes, a

well-publicized report available on a well-designed web site should,

in theory, be more easily accessible to most people than an article in

a scientific journal, but this potential is achieved only by a few

sources or institutions (such as the Center for International Forestry

Research, CIFOR). More often you need to know the right people

and/or knock on the right doors, and neither of these are guarantees

of success. Even in the developed world, where government agen-cies are often required by law to provide public access to docu-

ments, standards of accessibility still vary widely. In contrast, GoogleScholar and an email address now provide easy discovery and free

access to much of the recent scientific literature. The contents pages

and abstracts of most scientific journals are easily located and freely

accessible on the Internet, while authors, who are increasingly

assessed on citations, are usually delighted to respond to requests

for a pdf or reprint. Accessing journal archives, however, is usuallymore difficult and expensive.

The gray literature is growing exponentially, so we need solu-

tions that recognize its potential. Of the four principal concerns—

quality, discovery, access, and archiving—scientific quality is likely

to remain low, since much of the work that is reported is done by

inexperienced and isolated nonspecialists. Moreover, the aims are

rarely primarily scientific, although the methods often at least try to

be. Large improvements could be made fairly easily if reports werereleased first in draft form for comments and corrections. A soft

pre-release may weaken the public relations impact of the final

report, but it would payoff in quality. Pre-submission circulation of

manuscripts for comment is standard practice in science—I did it

for this article—because it works so well.

Regarding discovery and access, reports produced for private

companies can legitimately be restricted, although requests for

access are often treated favorably, suggesting that these companiesmay simply need to be made aware of the value of these reports to

academics and others. Governments are a more complex case, since

there are grounds for secrecy in regards to security and economic

affairs, but it is hard to argue that the release of ecological or con-

servation information is a threat to any legitimate government

right. More than 60 countries worldwide currently have legislation

that protects the right of citizens to access information held by gov-ernment bodies, but such legislation is rare in the tropics (http://

www.transparency.org). NGOs have no legitimate reason for lack

of transparency, although with many local NGOs limited technical

capacity and lack of continuity in staffing are real barriers to both

access and archiving. There are no such excuses for the BINGOs,

which have the resources, but making their data available to re-

searchers, managers, and conservationists in the tropics appears to

be a low priority for most. That it does not have to be like this isshown by the exceptions, such as a recent study of wildlife moni-

toring in the Congo by the Wildlife Conservation Society, for

which the results and methods were published in a peer-reviewed

open-access journal and all the data have been uploaded to public-

ally accessible data bases (Stokes et al. 2010).

It should also be added, in fairness, that problems with data

access and archiving are not confined to the gray literature, but are a

pervasive problem in ecology, with most scientific journals puttingthe onus on individual authors to archive data, although this may be

changing (Bruna 2010). A central repository for gray literature

reports relevant to tropical ecology and conservation would be the

ideal solution, but this would require a greater degree of collabora-

tion between organizations than is currently displayed in the tropics.

A distant second best would be for each organization to improve ac-

cess and archiving of its own output by setting up a digital repository,

such as those operated by many academic institutions.Unpublished dissertations and theses are not strictly gray liter-

ature, because they are produced primarily for examination rather

than dissemination, but they are another vast untapped—and cur-

rently ‘untappable’—source of information on tropical biology.

This is particularly true in Southeast Asia, where hundreds of uni-

versities offer higher degrees by research, but very few make disser-

tations available to outsiders. Many universities also require

undergraduates to do research dissertations in order to graduate,with ecology a very popular subject for these studies. There is no

doubt that the annual volume of research deposited in these various

research reports exceeds that in the scientific literature and gray lit-

erature combined. In terms of quality, they generally lie between

the two. Publication biases are likely to be a particular problem with

theses, because supervisors typically cherry-pick from their stu-

dents’ work, resulting in the publication only of studies with ‘sta-

tistically significant’ results (the ‘file drawer’ problem; Rosenthal1979). On-line access should be automatic for graduate theses, as it

is in many nontropical universities, and at least the abstracts of

undergraduate dissertations should be accessible, as pointers to

promising research areas. It is up to individual academics to press

for these changes at their own institutions.

Currently, most of the big academic data bases, such as Scopusand the ISI Web of Knowledge, cover only the peer-reviewed litera-

ture. Google Scholar claims to find relevant work from ‘articles, the-ses, books, abstracts and court opinions, from academic publishers,

professional societies, online repositories, universities and other

web sites’, but the published scientific literature overwhelmingly

4 Corlett

dominates search results. To a large extent this reflects the invisibil-

ity of most gray literature to the automated software—web

crawlers—that Google Scholar uses to locate documents for inclu-

sion in search results. Even when gray documents are located, theyare often poorly indexed because the software is unable to identify

bibliographic data. Scientific publishers routinely boost the visibil-

ity of their products by paying close attention to the inclusion

guidelines issued for each data base (e.g., for Google Scholar: http://

scholar.google.com.sg/intl/en/scholar/inclusion.html). Producers

of gray literature need to do the same, which may require listing

reports on a separate page that is structured in a way that makes

them easy to ‘crawl’. Larger documents (4 5 MB) in any languagecan be uploaded to Google Books by the copyright holder.

This paper was prompted by my recent experiences while writ-

ing a book on the ecology of tropical East Asia (Corlett 2009), for

which I attempted to access all relevant gray and nongray literature

produced over the previous decade, as well as recent dissertations.

The main focus was on material produced in English and, for the

north of the region, in Chinese, but I also searched for additional

literature produced only in Thai and Malay. Overall, the volume ofthe gray literature found (or identified but not seen) greatly

exceeded that of the scientific literature. An iceberg analogy is

unavoidable, if geographically inappropriate: nine tenths of what has

been done in this region is hidden from sight, in the gray literature

and dissertations. We need to make this content visible. This will

only happen if tropical biologists pay as much attention to the means

of access and archiving of their output as they do to its quality. This

requires easy discovery with a free search engine and either a freelydownloadable pdf or an email address for reprint requests.

ACKNOWLEDGMENTS

I have incorporated suggestions from John Fellowes, David Dud-

geon, Keith Hamer, Navjot Sodhi, Nina Ingle, Hugh Tan, Peter

Todd, and William Laurance, but this does not mean that they

agree with my conclusions.

LITERATURE CITED

BRUNA, E. M. 2010. Scientific journals can advance tropical biology and con-servation by requiring data archiving. Biotropica 42: 399–401.

CONN, V. S., J. C. VALENTINE, H. M. COOPER, AND M. J. RANTZ. 2003. Greyliterature in meta-analyses. Nurs. Res. 52: 256–261.

CORLETT, R. T. 2009. The ecology of tropical East Asia. Oxford UniversityPress, Oxford, U.K.

FERRARO, P. J., AND S. K. PATTANAYAK. 2006. Money for nothing? A call forempirical evaluation of biodiversity conservation investments. PLoSBiol. 4: e105.

GENOVESI, P. 2005. Eradications of invasive alien species in Europe: A review.Biol. Invasions 7: 127–133.

JENNIONS, M. D., AND A. P. M�LLER. 2002. Publication bias in ecology and evo-lution: An empirical assessment using the ‘trim and fill’ method. Biol.Rev. Camb. Philos. Soc. 77: 211–222.

LACANILAO, F. 1997. Continuing problems with gray literature. Environ. Biol.Fishes 49: 1–5.

LINKIE, M., R. J. SMITH, Y. ZHU, D. J. MARTYR, B. SUEDMEYER, J. PRAMONO, AND

N. LEADER-WILLIAMS. 2008. Evaluating biodiversity conservationaround a large Sumatran protected area. Conserv. Biol. 22: 683–690.

MROSOVSKY, N., AND M. H. GODIREY. 2008. The path from grey literature toRed Lists. Endang. Species Res. 6: 185–191.

ROSENTHAL, R. 1979. The file drawer problem and tolerance for null results.Psychol. Bull. 86: 638–641.

SIMBERLOFF, D. 2009. We can eliminate invasions or live with them. Successfulmanagement projects. Biol. Invasions 11: 149–157.

SODHI, N. S., AND L. H. LIOW. 2000. Improving conservation biology research inSoutheast Asia. Conserv. Biol. 14: 1211–1212.

STOKES, E. J., S. STRINDBERG, P. C. BAKABANA, P. W. ELKAN, F. C. IYENGUET,B. MADZOKE, G. AIME, F. MALANDA, B. S. MOWAWA, C. MOUKOUMBOU,F. K. OUKABADIO, AND H. J. RAINEY. 2010. Monitoring great ape andelephant abundance at large spatial scales: Measuring effectiveness of aconservation landscape. PLoS ONE 5: e10294.

Trouble with the Gray Literature 5