The Computer-Assisted Literature Review

http://ssc.sagepub.com/Social Science Computer Review

http://ssc.sagepub.com/content/2/3/137The online version of this article can be found at:

DOI: 10.1177/089443938600200303

1984 2: 137Social Science Computer ReviewEdward E. Brent, JR

The Computer-Assisted Literature Review

Published by:

http://www.sagepublications.com

can be found at:Social Science Computer ReviewAdditional services and information for

http://ssc.sagepub.com/cgi/alertsEmail Alerts:

http://ssc.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://ssc.sagepub.com/content/2/3/137.refs.htmlCitations:

What is This?

- Jul 1, 1984Version of Record >>

at SUNY BINGHAMTON on November 18, 2014ssc.sagepub.comDownloaded from at SUNY BINGHAMTON on November 18, 2014ssc.sagepub.comDownloaded from

http://ssc.sagepub.com/

http://ssc.sagepub.com/content/2/3/137

http://www.sagepublications.com

http://ssc.sagepub.com/cgi/alerts

http://ssc.sagepub.com/subscriptions

http://www.sagepub.com/journalsReprints.nav

http://www.sagepub.com/journalsPermissions.nav

http://ssc.sagepub.com/content/2/3/137.refs.html

http://ssc.sagepub.com/content/2/3/137.full.pdf

http://online.sagepub.com/site/sphelp/vorhelp.xhtml



137

The Computer-AssistedLiterature Review

Edward E. Brent, Jr.

Computers and the Social Sciences 2 (1986)<9Paradigm Press, Inc.

Abstract: This paper describes several tech-

niques for using computers to assist with litera-ture reviews. It identifies various literature review

tasks, then discusses how existing programs canbe used to address them. Programs discussed in-clude word processors, qualitative analysis pro-grams, content analysis programs, file manage-ment systems, database management systems,spreadsheets, and statistical programs. The pa-per concludes that judicious use of existing pro-grams offers a powerful means to transform theliterature review from a burdensome task result-

ing in a quickly outdated product to a moremanageable task producing a product that is flex-ible, cumulative, and better able to cope with therapid growth of knowledge.

A computer-assisted literature review, that is, anyreview in which a computer helps with some as-pect of the review process, might use some of theprograms discussed below or others. Distin-

guished in both depth and scope from computer-ized bibliographic retrieval (see, e.g., Wynar,1980; Leibold, 1983) the computer-assisted liter-ature review relies on computerized referencesystems such as MEDLARS and ERIC. Bibliograph-ic retrieval is concerned simply with identifyingthe relevant literature and selecting references;the literature review is concerned with thedetailed analysis, interpretation, and organizationof thoughts derived from those bibliographic

sources. While bibliographic retrieval systems in-clude small amounts of information on many ar-

ticles, computer-assisted literature review in-cludes in-depth information about a smaller num-ber of articles. The literature review demandssufficient depth to identify relevant variables, de-scribe the findings, distinguish basic theoreticalperspectives, and report the methods used.Hence, computer-assisted literature reviewsshould be narrower in scope than bibliographicretrieval and should be conducted by profession-al researchers in the field to achieve the requireddepth of analysis.

The Literature ReviewThe literature review offers an opportunity to getan overview of a field not possible from singlestudies (Cooper, 1980). Rapid increases in scien-tific publications have increased the dependenceof the individual scientist on literature reviews byothers (Menzel, 1966). This dependence hasprompted a number of authors to view it as anempirical research project, much like primaryempirical research, its data being the informationavailable. The literature review, then, should besubject to the same standards as other empiricalresearch (Jackson, 1980; Cooper, 1980; Taveggia,1974) and have the same stages, including defi-nition of the problem, data collection, data qual-ity assessment, analysis and interpretation, andfinally reporting.One effort to improve literature reviews has

been the development of quantitative methods forreviewing research results and developing gener-alizations (Smith and Glass, 1977; Rosenthal andRubin, 1978; Gage, 1978; Hunter and Schmidt,1978; Viana, 1980). Here we consider another

Acknowledgments: The author would like to thank Ronald E. An-derson and anonymous reviewers for helpful comments on earli-er drafts of this paper.

Edward Brent is an associate professor of sociol-ogy and Family & Community Medicine at theUniversity of Missouri - Columbia.

at SUNY BINGHAMTON on November 18, 2014ssc.sagepub.comDownloaded from


138

method, the computer-assisted literature review,in which information describing each article is en-tered into a computerized database by the review-er and the computer is then used to assist in ex-amining, analyzing, interpreting, and presentingthe data.

Problems of Literature Reviews

Burdened with both logistical problems andthreats to their validity, comprehensive literaturereviews in even relatively specialized areas nowoften involve scores, hundreds, even thousands ofstudies. New ones are published faster than oldones become outdated. Relative to the rapidgrowth of the literature, the review itself proceedsat a snail’s pace. New reviews usually must repli-cate much of the earlier reviews along with therequirement to add new studies. Months are of-ten required to retrieve the relevant studies andthen read, analyze, and summarize them. Oncecompleted, the review may take months or evenyears to reach publication and becomes outdat-ed quickly as new studies become available, newissues are raised, and important control variablesare recognized which were not included in theoriginal review.Perhaps prompted by their greater dependence

on them, many researchers have reexamined themethods and standards commonly applied toliterature reviews and found serious threats totheir validity. Judgments of the relevance of ar-ticles to a literature review have been found to beaffected by many factors: the way the study isdocumented in a literature retrieval system (Res-nick, 1961), the time available for the reviewer todecide whether the study is relevant (Davidson,1977), and the reviewer’s expertise and open-mindedness (Cuadra and Katter, 1967). Studiesfinding relatively low correlations between qual-ity judgments made by editors and authors (Gott-fredson, 1978) suggest that those judgments inliterature reviews are suspect as well. Still otherstudies report reviewers find more credible thosestudies whose results confirm their own expec-tations (Mahoney, 1977; Lord, et al., 1979). Withno common standards to prevent such biases, theliterature review can degenerate to the pointwhere reviewers critique studies disagreeing withtheir own point of view and find acceptable onlystudies confirming their own views (Glass, 1977).

To avoid bias, the reviewer often seeks bothgreater depth and greater scope. But not payingsufficient attention to conceptual and methodo-logical details risks missing distinctions essentialfor understanding the results (Bracht and Glass,1968). When too broadly defined, concepts canhide significant differences in results betweenstudies (Presby, 1978). Differences in results dueto the use of different methods (sometimes called&dquo;method-generated variance&dquo;) may outweigh oth-er substantive differences or similarities in stud-ies (Cooper, 1980; Pillemer and Light, 1980).Hence, greater depth is needed to assure that im-portant distinctions are not missed and importantelements are recognized.Greater scope is needed to assure that review-

ers do not unwittingly introduce bias into theirreviews through systematically excluding studiesunfavorable to their perspectives. While the largenumber of studies in many reviews creates a

temptation to focus on selected studies, that strat-egy is particularly susceptible to three deficien-cies : the biases discussed above (Cooper, 1980),the difficulty of weighing the volume of supportfor various positions (Light and Smith, 1971), andthe impediments to assessing the strength of rela-tionships (Cohen, 1977).The reporting of literature reviews has also

come under criticism. Like other studies, thesemay present a distorted picture of actual occur-rences, downplaying the false starts and deadends and conveying an unearned appearance ofdeliberateness (Matheson, et al., 1978). Perhapsmore serious are failure to describe the review

process (Bracht and Glass, 1968; Jackson, 1980)or exclusion from the report of information latershown to be critical (Meadow, 1976).

The Literature Review Task-Varieties

Although literature reviews vary considerably,common types of analysis, neither mutually ex-clusive nor exhaustive, may be identified. Anyliterature review may employ one or more; eachmay constitute the entire thrust of a review or

only one component.1) Conceptualizing. Reviews that put foi-th some

conceptual framework for classifying the liter-ature in an area are likely to identify importantnew concepts and may provide an entirely newperspective. An example (if it were a literature



139

review) would be a book such as Kuhn’s (1962)The Structure of Scientific Revolution.

2) Organizing. Less grand than conceptualizingreviews, organizing reviews proposes no newstructure for analyzing an area, but do provideorganization. They group similar articles to-gether for discussion, and they summarize andcontrast them. An example might be oneamong the many articles appearing in the An-nual Reviews series in various disciplines.

3) Cataloging. Primarily concerned with provid-ing a reference document, cataloging reviewsare represented by annotated bibliographieswhich may order articles by author or topic.

4) Full-text analysis. These reviews examine fea-tures of the original text and present sum-maries of the results.

5) Qualitative comparisons. Discursive compar-isons noting similarities and dissimilaritiesamong works may also identify trends or rela-tionships among those aspects.

6) Quasi-statistics. A term sometimes used to de-scribe frequency tables based on qualitativecomparisons, &dquo;quasi-statistics,&dquo; describes ta-

bles presented in Boys in White (Becker et al.,1961). Quasi-statistics may be used in reviews.

7) Quantitative statistical analysis. Reviews us-ing quantitative statistical analyses may pro-vide extensive summary statistics and contin-gency tables, may use some form of scalinganalysis to identify clusters of articles, and soon. A good example of such reviews are thoseperformed in meta-analysis to provide a quan-titative summary of several studies each per-forming similar statistical tests (Glass, 1977;Hunter, et al., 1982).

The Interface Between Retrieval and ReviewFor computers to be used in the literature review,information must be made available in computer-readable form, either entered by the reviewer orstored in full-text computer files and retrievedthrough computerized retrieval procedures. Anumber of strategies can make bibliographic in-formation suitable for literature analysis. Figure1 provides a summary of the possible transforma-tions for bibliographic files to literature reviewfiles.

Figure One. The Interface Between Bibliographic Retrieval and Literature Analysis



140

Manual notes. At the far right of Figure 1 isa representation of the manual process of retrievaland review: the information collected in the bib-liographic retrieval (citations and references) istypically used to identify books and articles forretrieval from a library. Those are then read,notes taken, and the information organized intocategories useful to the reviewer and sensible forthe field. The advent of computerized biblio-graphic retrieval and computerized methods foranalyzing literature provide many additionaloptions.Across the bottom of Figure 1 are displayed a

variety of programs which can be used to analyzeliterature. The three on the right, which workprimarily with text data structures, can accept asinput either the full text of an article or summa-rized notes. Full text, which typically placesheavy demands on computer memory, may bequite inefficient. Hence, even these programsusually work with summarized notes rather thanthe original text of references. The remaining pro-grams use alternative data structures and cannotwork directly with text. For these, the text of anarticle must first be converted into a suitable datastructure.

User-directed note taking. The conversion oftext to notes is usually performed by methodsmuch like those for the manual review. Importantcategories are identified, relevant informationabout each article recorded, and the informationorganized for presentation. These steps requiredirect involvement, oversight, and a word proces-sor or other computer program, but the processis essentially unchanged from its manual

counterpart.Natural language understanding. This ap-

proach requires full-text files already in machine-readable form and language understanding pro-grams which presently are limited to narrow con-tent domains and small vocabularies. One day,however, programs may routinely read and under-stand the full text of an article and convert it intosummarized notes in a data file for submission to

analysis programs.Micro-to-mainframe links. Information for a

computerized literature review might also be ob-tained intact from large mainframe databases us-ing &dquo;micro-to-mainframe&dquo; communications pro-grams. In this manner, information already in

database form can be manipulated and analyzedfor the literature review, without further data en-try. Several database management and statisticalanalysis programs, such as SPSS, have this capa-bility. The limiting factor for this procedure is thelack of existing mainframe databases having datastructures like those required for literaturereviews.

Varieties of Computer-Assisted LiteratureReviews

A wide range of existing programs for microcom-puters are good candidates for use in literaturereviews. Including statistical programs, databaseand file management systems, computerizedspreadsheets, word processors, and even specialpurpose programs designed for qualitative anal-ysis, these programs are displayed along the bot-tom of Figure 1. Additional programs such asthose using artificial intelligence techniques(Brent, 1984) may one day play a useful role inliterature reviews but, because they are not yetoperational, are not considered here. Operation-al programs which may be used for literaturereviews offer different levels of functionality (thefunctions they permit the computer to perform)and flexibility (the extent to which they permitdiverse data structures to be created and used bythe programs).The following sections will discuss briefly how

each of these programs might be used to conducta computerized literature review. Prominent inthis discussion will be the main features of the

programs, the functions they can perform thatmay be useful for literature reviews, and the con-straints they impose upon the data they use whichmay limit their flexibility or create the need fordifficult and time-consuming data conversion andentry problems.

Qualitative Analysis ProgramsAn increasing number of programs for analyzingqualitative data include THE ETHNOGRAPH bySeidel and Clark (1984) and LISPQUAL by Drass(1980). Generally operating on a text file, theyhave some of the text-editing features of wordprocessors such as find and replace, but lackingother powerful word-processing features such asformatting commands, they may be best used in



141

conjunction with word processors. What distin-guishes qualitative analysis programs from wordprocessors is their ability to perform qualitativeanalysis of field notes. These include the capaci-ty to demarcate and code segments of text, tomodify or delete those codes later as required,and to produce a summary output file of sorted,cross-referenced, coded segments of original text(Seidel and Clark, 1984). Typically, these pro-grams are used to analyze various text files suchas diaries, field notes, or transcripts.

In a literature review such programs can beused to categorize a text file of notes or even afull-text file of references. They can extract fromthat file all references to a particular concept or

phenomenon and place them in another file forfurther examination. The reviewer can examinethat file to identify trends and summarize theliterature on the topic. Using similar proceduresto create a whole series of such extracted files, thereviewer can develop a complete qualitative anal-ysis of the literature.A sample analysis of a partial text taken from

Presentation of Self in Everyday Life (Goffman,1959:22-24) will illustrate these steps. In Table 1below is displayed a segment of text as it mightlook categorized by such a qualitative program.Here segments are demarcated and labeled withthe concept or concepts to which they refer.

TABLE 1EXAMPLE PAGE OF TEXT AS IT MIGHT LOOK

CLASSIFIED BY CATEGORIES



142

From categorized text files similar to this, THEETHNOGRAPH can create a summary file of text

segments related to various labeled categories.For this example, the &dquo;performance&dquo; file wouldinclude all four segments, the &dquo;front&dquo; file, the lastthree, and the &dquo;setting&dquo; and &dquo;personal front&dquo;files one segment each. The reviewer could ex-amine such summary files for each concept.Qualitative programs are likely to be useful inliterature reviews for both full-text analysis andqualitative comparisons. For full-text analysis,

their greatest limitation is the restricted availabil-ity of full-text files.

Content Analysis ProgramsContent analysis programs available for main-frames include the General Inquirer (Stone et al.,1966), TEXTPACK (see Weber, 1984) and the Min-nesota Contextual Content Analysis (MCCA) pro-gram (McTavish and Pirro, 1984). In addition,Tamai et al. (1985) describe existing programs forperforming content analysis on microcomputers.

TABLE 2KEY-WORD-IN-CONTEXT AND FREQUENCY DISPLAYS

FOR GOFFMAN EXAMPLE

KEY-WORD IN CONTEXT

WORD FREQUENCIES

Gerardine DeSanctis, &dquo;Computer Graphics as Decision Aids: Directions for Research,&dquo; Decision Sciences15:4 (1984), published by Decision Sciences Institute. Reprinted with permission.



143

Such programs typically provide a variety of in-formation about text files including word-frequency lists, concordances, key-word-in-context lists, classifications of text by content cat-egories, category counts, and retrievals of textbased on categories (Holsti, 1969; Krippendorff,1980). Many content analysis programs can alsogenerate quasi-statistics and tabulate occurrencesof specific kinds of words or phrases, which mayhelp the analyst examine relationships amongvariables. Retrievals of text based on categorieswould look much like the summary filesdescribed above for qualitative analysis programs.In Table 2 are presented key-word-in-context andword-frequency displays for the Goffman textexample.

Clearly, content analysis programs could beused in literature reviews in much the same waythe qualitative analysis programs discussed abovemight be used, namely to identify categoriescharacterizing the text, to extract passages fittinga specific category and place them in another file,to count occurrences of a specific category, to pro-duce qualitative comparisons and to generatequasi-statistics.

Word Processors

Powerful word processors on microcomputerssuch as Wordstar, Microsoft Word, and AppleWriter probably need no introduction. Generallymore versatile and more flexible than the special-purpose qualitative and content analysis pro-grams described above, they typically lack someof their special features. Because of their flexibil-ity, word processors can be used for writing vir-tually any type of literature review. However, theircontribution may be more to the writing process(e.g., see Daiute, 1985) than to the analysis. Herewe focus on the use of word processors for or-

ganizing and full-text analysis.Organizing. Word processors can help the

reviewer organize during the review. Since theypermit various sections to be cut and pasted, theycan quickly search text for relevant passages, andthey permit flexible formatting to emphasizedifferent sections. Hence, they can be versatileprograms for constructing review notes. Closelyrelated to word processors are outlining programs(sometimes called &dquo;idea processors&dquo;) such asThink Tank or PC-Outline. These programs may

be useful when an outline format is appropriate,but tend to force the user into an outline modeeven when it is inappropriate. Outlines are ofteninappropriate for many stages in the literature re-view. Hence, these programs should be used withcaution.

Full-text analysis. Another use of word proces-sors in the literature review is the analysis and or-ganization of the original source text. Here, textfiles of articles, books, or other documents maybe summarized, manipulated, and reorganized us-ing the word processor. This strategy, whichclosely parallels the methods described above forprograms specifically designed for qualitativeanalysis or content analysis, has been describedby several different authors (e.g., see Brent, 1984;Gassaway, et al., 1984). Word processors can beused by the reviewer to mark passages, selectsegments discussing specific contents for displayin a separate file, and in short, perform virtuallyall the tasks performed by qualitative or contentanalysis programs. These are made easier by us-ing keywords or symbols (e.g., words surround-ed by angle brackets or other infrequently usedcharacters) within the text to mark passages forretrieval.

In most word processors these tasks must be

performed under the direction of the user and aremuch slower and much less automated than the

qualitative and content analysis programs. How-ever, word processors with built-in programminglanguages, macro facilities, or form letter capa-bilities can often create sequences of commandsto be invoked automatically. Word processorslacking these capabilities can do much the samething with keyboard macro programs such asPROKEY and KEYSWAP.

File and Data Base Management SystemsFile management systems are programs that or-ganize information in much the way the card cata-log in libraries organizes books. Informationabout each book is recorded on a card record. Oneach record (card) several fields describe thebook’s important elements, such as author, title,date published, and so on. These records arestored in file drawers. In the computer, file

management systems create similar data struc-tures, consisting of information fields that makeup records which in turn are grouped into files.



144

Like card catalogs, the records in a file may besorted into some order on one or more of thefields (e.g., organized by last name of author). Inaddition, multiple indexes can be created whichrefer to the records and permit them to be ac-cessed by their order on different combinationsof variables (card catalogs themselves are index-es permitting books to be accessed ordered by au-thor, title, or subject). Commonly used file

management systems for microcomputers are PC-FILE, Notebook, and PFS. Hutton and Hutton(1981) describe a database literature retrieval sys-tem for use in the social sciences. Kirk (1981)describes the use of a database program in an-

thropological research. Using file managementprograms, Brent (1981) describes strategies thatmight be applied to literature reviews.These systems can assist in literature reviews

in several ways. They are probably ideal forcataloging the literature to produce a referencedocument such as an annotated bibliography withthe references described by various fields andperhaps organized by key categories. For exam-ple, in one project a file management system(Notebook) was used to create and print an anno-tated bibliography of over 150 references. Theprogram’s capabilities permitted new referencesto be quickly sorted into the appropriate se-quence. The data fields on the entry form provid-ed a helpful template for describing each article.And the program’s unusually strong formattingcapabilities produced camera-ready copy for pub-lication far more quickly than is possible by handor even with a word processor.Another potential use of file management sys-

tems is qualitative comparisons among the refer-ences. They can, for example, search the refer-ences on two or more fields and print out sepa-rate files possessing particular combinations ofvalues. Those may be examined visually and evencounted and combined into contingency tables toprovide quasi-statistics.

It is important to note that not all file manage-ment systems are suitable for literature reviews.To perform adequately in this role they must per-mit large fields, preferably of varying length, sothat relatively large quantities of text can be en-tered as needed, and they must be relatively easyto restructure so that additional fields can be add-ed as new ideas develop. In addition, it is desira-

ble that they have sophisticated query facilitiesand be able to search for complex boolean com-binations of variable values located anywhere ina field rather than only text at the beginning ofa field. This capability permits many kinds of in-formation to be more loosely included in largefields rather than requiring the reviewer to be for-ever revising the structure to add still other fields.Powerful formatting features such as those avail-able in Notebook and Dayflo can also save con-siderable time by eliminating the need for

reprocessing the report file with a word processor.During data collection, file management pro-

grams guide the researcher through a conceptu-al template. Being prompted for each field in thedata structure is a benefit as long as the research-er can specify a useful structure at that stage. Forrelatively structured fields, the database manage-ment program can check data as they are enteredto determine whether they are of the proper form,rejecting letters in a numeric field or seven-figurenumbers in a six-figure field, and so on. Manyprograms permit the user to specify additionalchecks for assuring data quality on entry. Alsosubject to rejection are data not matching aprescribed list of accepted values, numbers be-yond a reasonable range, and other obviousanomalies.

File management systems as described abovehave what is called a &dquo;flat file&dquo; structure in which

every record is classified among the same fields.

Many authors distinguish such file managementsystems from database management systems(DBMSs). In these more sophisticated extensionsof file management systems, relational files canlink diverse files containing different types of in-formation. For conducting literature reviews,DBMSs can perform as well as file managementsystems, but they are unlikely to take advantageof their more powerful features such as relation-al files. In practice, DBMSs are often not moreuseful for literature reviews than file managersbecause their sophisticated relational data struc-tures are too complex and unwieldy to use withrapidly evolving conceptions of literature reviews.In the future, greater flexibility and ease ofchange may make DBMSs more useful for litera-ture reviews. Because of severe restrictions onrecord and field sizes, some of the more power-ful DBMSs available for microcomputers, such as



145

TABLE 3ASPREADSHEET DISPLAY OF LITERATURE

Gerardine DeSanctis, &dquo;Computer Graphics as Decision Aids: Directions for Research,&dquo; Decision Sciences15:4 (1984) published by Decision Sciences Institute. Reprinted with permission.



146

dBase II or Metafile, may be less useful for liter-ature reviews than nonrelational file managementsystems such as Notebook.

SpreadsheetsSpreadsheet programs such as Visicalc and Lo-tus 1-2-3 have received considerable attention inbusiness because they can create tables of infor-mation in which numbers are calculated for

specific cells by the computer according to chang-ing values in other cells. Beyond numeric data,these can also be useful for verbal descriptive ta-bles. Their ability to move rows and columnsaround, to insert and delete rows, to sort on vari-ous fields, and to tabulate summary data makesthem flexible tools which can display and manipu-late descriptive information about studies in aliterature review.

Spreadsheets, having the same flat file datastructure as file management systems, could beused in many of the same ways. They can recordcharacteristics of references in different fields, sorton specific categories, and display them for com-parison. Typically, however, spreadsheets lack theextensive data entry checking available with manyfile management systems and because of the dif-ficulty of displaying large fields on the screen arelimited in field size. If variable-width columns areavailable to minimize this problem, spreadsheetsare excellent for displaying records and makingqualitative comparisons among different studies.Thus able to insert, delete, and reorder rows andcolumns, these programs can reorganize informa-tion as part of an organizing review.

As an example, consider a spreadsheet filewhere each row represents a separate article andeach column a variable characterizing articles.This is illustrated in Table 3A which reproducesa portion of a Table 2 from a review of studies ofthe effectiveness of computer graphics carried outby DeSanctis (1984). (That the different articlesin this table are represented by a variable numberof rows greatly enhances the display, but can causeconsiderable difficulty for using the spreadsheet’scapabilities.)To make qualitative comparisons of two specif-

ic articles the reviewer can reorder rows to put thetwo articles next to each other and then scanacross the columns to compare the two articles onall variables. This capability is illustrated in Table3B where an article by Lusk and Kersnick (1979)which found tabular presentations to be better isdisplayed just above an article by Feliciano et al.(1963) which found graphs to be better. We caneasily scan across the two rows to find that thesestudies utilize a different context/task: one is com-

paring bar graphs while the other is not specificin the graphics used, and the dependent variablefor one is interpretation whereas the other exa-mines performance and confidence. Thus, a num-ber of differences might account for their disparateresults.To examine the relationship between two varia-

bles across different studies, the columns of thespreadsheet can be reordered to place the twovariables in adjacent columns and then the rowscan be sorted on one of the variables. It is easy toscan down the two adjacent columns to examinerelationships between the two variables, as is illus-trated in Figure 3B, where the articles in

TABLE 38COMPARISON (lF T6-0 RETICLES



147

DeSanctis’ original table are first sorted by thecontext/task (after the labels are revised to makethe computer-based alphabetic sort meaningful);columns are then reordered to place context/task

next to the results. The segment displayed here,showing little or no consistent relationship amongthese variables, supports DeSanctis’ observationthat &dquo;the results still lack a meaningful pattern.&dquo;

TABLE 3C

RELATIONSHIPS BETWEEN TWO VARIABLES



148

TABLE 4G~U~-1S I -S&dquo;’’~T I S T T C~li... SUMMARY

OF RESEARCH RESULTS COMPARING GRAPHS AND TABLESFOR SELECTED DEPENDENT VARIABLES*

*Numbers indicate number of studies for a giveri caceg<=<ry



149

As the review progresses and categories be-come more formalized, additional categories canbe summarized in tables using the mathematicalfeatures of spreadsheets. From these categoriesmay be created quasi-statistical tables such asthose in Table 4 (also reproduced from DeSanc-tis’ study). Eventually, a number of quantitativestatistics may also be created. Most spreadsheetshave common summary statistics such as mean,median, mode, and standard deviation. Morecomplex statistics can often be specified by theknowledgeable user. More sophisticated statisticsare usually best produced by moving the analy-sis to a statistical package. Fortunately, manyspreadsheets and statistical packages are compat-ible and permit easy data transfer of numeric dataand relatively structured qualitative data. Sincethe more qualitative and more flexible fields withextensive text will probably not transfer well,analysis of those should be completed before thetransfer to a statistical package.

Statistical Analysis ProgramsAs the categories crystalize and the coding be-comes more explicit, subsets of the data can beeither transferred to statistical packages on themicrocomputer or uploaded to a mainframe com-puter for analysis. Among increasingly sophisti-cated statistical analysis programs on themicrocomputer (e.g., see reviews by Carpenter, etal., 1984; Brent and Campbell, 1986) are SPSS/PCand BMDPC which are essentially powerful main-frame packages downloaded with most of theircapabilities to microcomputers.These versatile statistical programs offer a

wide range of analyses. They can, for example,perform the meta-analysis summarizing the evi-dence favoring a particular hypothesis derivedfrom several similar studies in the literature

(Hunter et al., 1982). They are, however, inflexi-ble and can examine only variables coded duringthe review process. Those must often be coded

inflexibly and very specifically. A reviewer mayexpect to spend far more time preparing the datafor a statistical analysis than analyzing them.

TABLE 5REVIEW TASKS ADDRESSED BY AVAILABLE PROGRAMS



150

Summary and Conclusions

Table 5 identifies the literature review tasks thatcan be addressed by each of these programs.Even this depth and variety of review tasks sub-ject to computerization should be superseded bya wider range of tasks. Until a comprehensiveliterature review program is available, however,the integration among these various programswill continue to be a concern.

Current Limitations of Computer-AssistedLiterature Reviews

Although existing computer programs can assistwith many tasks of literature reviews, several im-portant limitations remain.Integration among programs. Currently,

transferring data from one of these programs toanother is time-consuming and often technicallydifficult. Most reviews are therefore likely to useonly one or a few of the available programs.Program selection. Programs for literature

reviews should be selected to fit the reviewer’s

knowledge level and preferred review style. Areviewer with considerable knowledge of an area,having established categories for classifying ar-ticles, might want a highly structured databasemanager or statistical program to summarize ar-ticles by those categories. A researcher new to afield, with only tentative notions and changeablecategories, might select a program with greaterflexibility. Word processors or qualitative analy-sis programs permit variable length fields and theeasy addition or deletion of key terms to existingdatabases.

Program limitations. The successful use ofcomputers in literature reviews requires the selec-tion of programs having appropriate functionalcapabilities and the careful application of strate-gies to take maximum advantage of those capa-bilities. Desirable functional capabilities may in-clude large data fields in database programs,sophisticated query procedures, and variablecolumn widths in spreadsheet displays. Amongeffective strategies are creating summary entriesfor fields to make the limited display more use-ful and choosing consistent terms organized sothat an alphabetic ordering will be meaningful.

Advantages of Computer-AssistedLiterature ReviewsWithin the limitations noted above, the computer-assisted literature review offers several advan-

tages over traditional ones. It can increase theirspeed, reducing to mere minutes what might takehours or days manually. It can automate tedioustasks such as note-taking, cross-referencing arti-cles, compiling lists, sorting references, and com-paring studies on one or more indices. This flex-ible tool may be used over an entire career to inte-

grate new studies, add new categories, and incor-porate new ideas. For students, it can be an ef-fective teaching tool to try out their own hypoth-eses as they examine the literature.The computer-assisted literature review’s abil-

ity to organize and reorganize the database withminimal additional effort permits more extensiveinvestigations, both involving more comparisonsand searches, and addressing a broader range ofquestions than previously possible. Computerizedliterature reviews can:~ examine the same studies for trends over time;~ compare articles by author or clusters ofauthors to identify Crane’s (1969) &dquo;invisible col-leges,&dquo; groups of researchers who take a simi-lar approach to research, communicate with oneanother frequently, and cite each other’s work;

~ compare articles by the methods used for evi-dence of methods-induced variance; and

~ control for a variable or combinations of varia-bles thought to affect the results.The computerized database created for the

literature review, disseminated in machine-readable form to other researchers along with thetraditional written review, will provide more com-prehensive information than written reviews.Giving subsequent reviewers the option of start-ing where the old reviews finished, and updating,modifying, and expanding the review with newliterature and variables of interest, would makeliterature reviews more cumulative. Recipientsof the database may also validate and assess the

adequacy of the original work.To summarize, computer-assistance promises to

fundamentally transform the literature review. Atpresent, this onerous and increasingly burden-some task must be started almost anew with each



151

effort, and quickly becomes outdated as new stud-ies surface and as the importance of new varia-bles not previously considered becomes evident.In contrast, the computer-assisted literature re-view provides more complete information in aflexible format, adaptable to changing circum-stances, and amenable to quick and reliable com-puter analysis. Where the traditional review isstatic and quickly outdated, the computer-assisted one is dynamic and flexible, as cumula-tive as the literature reviewed, and a potentiallyeffective way to cope with the rapid growth ofknowledge. Judicious choice among the varietyof computer programs currently offers a power-ful means to improve literature reviews. As thefunctionality and integration of these programsimproves and as new programs directly addressissues associated with the review task, the

computer-assisted literature review promises aneven brighter future.

REFERENCES

Becker, Howard S., Blanche Geer, Everett C. Hughes, and An-selm L. Strauss, Boys in White: Student Culture in MedicalSchool. Chicago: University of Chicago Pre;ss, 1961.

Bracht, G., and G. Glass, "The external validity of experiments,"American Educational Research Journal, 5, (1968), 437-474.

Brent, Edward E. and James Campbell, "Scaling and Classifica-tion with Microcomputers," Computers and the SocialSciences. 2:2 (1986).

Cohen, J., Statistical Power Analysis for the Behavioral SciencesNew York: Academic Press, 1977.

Cooper, Harris, "The Literature Review: Elevating its Status toScientific Inquiry," Technical Report #238. Columbia, MO:Center for Research in Social Behavior, 1980.

Crane, D., "Social Structure in a Group of Scientists: A Test ofthe ’Invisible College’ Hypothesis," American Sociological Re-view, 34 (1969), 335-352.

Cuadra, C. and R. Katter, "Opening the Black Box of Relevance,"Journal of Documentation, 23 (1967), 291-303.

Davidson, D., "The Effects of Individual Differences of Cogni-tive Style on Judgments of Document Relevance," Journal ofthe American Society for Information Science, 8 (1977),273-284.

DeSanctis, Gerardine, "Computer Graphics as Decision Aids:Directions for Research," Working Paper MISRC-WP-84-04Minneapolis: University of Minnesota Management Informa-tion Systems Research Center, 1984.

Drass, Kriss, "The Analysis of Qualitative Data: A Computer Pro-gram," Urban Life, 9 (1980), 322-353.

Feliciano, G.D., R.D. Powers, and E.K. Bryant, "The Presentationof Statistical Information," Audio Visual Communication Re-view, 11 (1963), 32-39.

Gage, N., The Scientific Basis of the Art of Teaching. New York:Teachers College Press, 1978.

Gassaway, Robert, William Elder, and James Campbell, "WordProcessors for Qualitative Sociologists: A Review Essay,"Qualitative Sociology, 7 (1984), 157-168.

Glass, G., "Integrating Findings: The Meta-Analysis of Research,"Review of Research in Education, 5. Itasca, IL: F.E. Peacock,1977

Goffman, Erving, The Presentation of Self in Everyday Life NewYork: Doubleday, 1959.

Gottfredson, S., "Evaluating Psychological Research Reports,"American Psychologist, 33 (1978), 920-934.

Holsti, Ole R., Content Analysis for the Social Sciences and Hu-manities. Reading, MA: Addison-Wesley, 1969.

Hunter, J. and F. Schmidt, "Differential and Single Group Valid-ity of Employment Tests by Race: A Critical Analysis of ThreeRecent Studies," Journal of Applied Psychology, 63 (1978),1-11.

Hunter, John E., Frank L. Schmidt and Gregg B. Jackson, Meta-Analysis: Cumulating Research Findings Across Studies.Beverly Hills: Sage, 1982.

Jackson, G., "Methods for Integrative Reviews," Review of Educa-tional Research, 50 (1980), 438-460.

Krippendorff, Klaus, Content Analysis: An Introduction to itsMethodology. Beverly Hills: Sage, 1980.

Kuhn, Thomas, The Structure of Scientific Revolutions. Chica-go : University of Chicago Press, 1962.

Lawrence, John Shelton, The Electronic Scholar A Guide to Aca-demic Microcomputing. Norwood, NJ: Ablex, 1984.

Leibold, Anne, "Bibliographic Data Bases in the Humanities: APerformance Study," Sixth International Conference on Com-puters and the Humanities 1983. Rockville, MD: ComputerScience Press, 1983, 368-382.

Light, R. and P. Smith, "Accumulating Evidence: Procedures forResolving Contradictions Among Different Research Studies,"Havard Educational Review, 41 (1971), 429-471.

Lord, C., L. Ross, and M. Lepper, "Biased Assimilation and At-titude Polarization: The Effects of Prior Theories on Subse-quently Considered Evidence," Journal of Personality and So-cial Psychology, 37 (1979), 2098-2109.

Lusk, E.J., and M. Kersnick, "Effects of Cognitive Style and Re-port Format on Task Performance: The MIS Design Conse-quences," Management Science, 25 (1979), 787-798.

Mahoney, M., "Publication Prejudices: An Experimental Studyof Confirmatory Bias in the Peer Review System," CognitiveTherapy and Research, 1 (1977), 161-175.

Matheson, D., R. Bruce, and K. Beauchamp, Experimental Psy-chology, 3rd Ed. New York: Holt, 1978.

Meadow, C., Applied Data Management. New York: Wiley, 1976.Menzel, H., "Scientific Communication: Five Themes from So-

ciology," American Psychologist, 21 (1966), 999-1004.Pillemer, D. and R. Light, "Benefiting from Variation in Study

Outcomes," New Directions for Methodology of Social andBehavioral Science, 5 (1980), 1-11.

Presby, S., "Overly Broad Categories Obscure Important Differ-ences between Therapies," American Psychologist, 33 (1978),504-515.

Resnick, A., "Relative Effectiveness of Document Titles and Ab-stracts for Determining Relevances of Documents," Science,134 (1961), 1004-1006.

Rosenthal, R. and D. Rubin, "Interpersonal Expectancy Effects:The First 345 Studies," The Behavioral and Brain Sciences,3 (1976), 374-415.

Seidel, John and J. Clark, "The ETHNOGRAPH: A ComputerProgram for the Analysis of Qualitative Data," Qualitative So-ciology, 7 (1984), 110-125.

Smith, M. and G. Glass, "Meta-Analysis of Psychotherapy Out-come Studies," American Psychologist, 32 (1977), 752-760.

Stewart, David W., Secondary Research: Information Sources andMethods. Beverly Hills: Sage, 1984.

Taveggia, T., "Resolving Research Controversy Through Empir-ical Cumulation," Sociological Methods and Research, 2 (1974),395-407.

Viana, M., "Statistical Methods for Summarizing IndependentCorrelational Results," Journal of Educational Statistics, 5(1980), 83-104.

Wynar, B.S., Introduction to Cataloging and Classification. 6thEd., Littleton, CO.: Libraries Unlimited, 1980.

