0018-9162/99/$10.00 © 1999 IEEE February 1999 45

Guest Editors’ Introduction

Digital Libraries:TechnologicalAdvances and Social Impacts

The World Wide Web has made access to theInternet part of the structure of everyday life.Millions of people all over the world searchthe Web every day. But the commercial tech-nology of searching large collections has

remained largely unchanged since the 1960s, when itwas developed in the course of US government-spon-sored research projects.1 This public awareness of theNet as a critical infrastructure in the 1990s hasspurred a new revolution in the technologies for infor-mation retrieval in digital libraries.

Many believe that we are approaching the start ofthe Net Millennium, a time when the Net forms thebasic infrastructure of everyday life. For this transfor-mation to actually occur, however, the functionality ofthe Net must be boosted beyond providing mere accessto one that supports truly effective searches. Collectionsof all kinds must be indexed effectively, from smallcommunities to large disciplines, from formal to infor-mal communications, from text to image and videorepositories, and eventually across languages and cul-tures. The Net needs fundamentally new technology tosupport this new search and indexing functionality.2

Digital libraries are a form of information technol-ogy in which social impact matters as much as techno-logical advancement. It is hard to evaluate newtechnology in the absence of real users and large col-lections. The best way to develop effective new tech-nology is by undertaking multiyear large-scale researchprojects that develop real-world electronic testbeds usedby actual users and by aiming at developing new, com-prehensive, and user-friendly technologies for digitallibraries. Typically, these testbed projects also examinethe broad social, economic, legal, ethical, and cross-cul-tural contexts and impacts of digital library research.

This special issue describes a wide range of researchprojects that investigate the development and usage ofnew information technology for substantial collections.

The technologies contained within are a representativesample of the Net of the early 21st century. Particularemphasis is placed on retrospective papers from mul-tiyear projects, which reflect actual experiences on anexperimental basis with the use of new technologies.The issue thus also contains initial hints of the userexperiences that will be common in the future Net.

RESEARCH INITIATIVESIn May 1996, a special issue of Computer focused

specifically on a major new US government initiative—the Digital Libraries Initiative (DLI)—funded by theNSF, DARPA, and NASA. The six major projects sup-ported by the DLI each had a survey paper at thishalfway point in the initiative.

This issue focuses on practical outcomes fromresearch projects—major research testbeds and fun-damental research technologies that show what thelarge-scale future infrastructure might become. Thepapers are split between DLI and non-DLI projects.Digital libraries have become far more importantnationally and internationally in 1999 than in 1996.This is largely due to the exponential growth of infor-mation in the World Wide Web, which Web searchersare increasingly failing to handle successfully. This isa special case of the increasing dependence of modernsociety on information technology and the increasingfailure of fundamental infrastructure due to theabsence of fundamental new technology.

The just-released PITAC report (President’sInformation Technology Advisory Committee) makesthis point clearly.3 In this report, the leaders of the USinformation technology research community con-cluded that “the current Federal program is inadequateto start necessary new centers and research pro-grams.…The end result is that critical problems aregoing unsolved and we are endangering the flow ofideas that have fueled the information economy.”

Public awareness of the Net as a critical infrastructure in the 1990s hasspurred a new revolution in the technologies for information retrieval indigital libraries.

Bruce SchatzUniversity ofIllinois atUrbana-Champaign

HsinchunChenUniversity ofArizona

Gues

t Edi

tors

’ In

trod

uctio

n

.

46 Computer

The committee went on to recommend that “theFederal budget for the year 2000 should include acommitment to sustained growth in IT research, alongwith a new management system designed to fosterinnovative research.”

Digital Libraries Initiative-Phase 2 (DLI-2) is anNSF-led initiative that builds on the successes of DLI-1 and presages the even bigger efforts recommendedin the PITAC report. DLI-2 has made the initialawards for multiyear projects that will support abroader range of activities than DLI-1, includingsmaller projects and topics in medicine and humani-

ties. There will be an even stronger emphasis on test-beds with real users and real collections.

Many federal agencies are contributing to this ini-tiative—namely NSF, DARPA, NASA, NationalLibrary of Medicine (NLM), Library of Congress, andthe National Endowment for the Humanities. The“Funding Agencies” sidebar includes a contributionfrom the NSF program officer discussing DLI-2, aswell as contributions from the lead agencies DARPAand NLM describing their agencies’ other efforts tosupport digital library research.

The importance of digital library research is spread-

Funding Agencies

Digital Libraries: The View from NSFStephen Griffin, National ScienceFoundation

The Internet and WWW have demon-strated that scholars, students of all ages,and the general public have a boundlessappetite for information of all types.Millions now regularly use the Web as aprimary source of information, and as aninventive medium for communicating andsharing knowledge, enabling new rela-tionships, collaborations, and intellectualcommunities.

The Digital Libraries Initiative (DLI),funded by NSF, DARPA, and NASA from1994 to 1998, supported pioneeringexploration into issues of organization,access, security, and use of distributedinformation resources. DLI demonstratedthat large amounts of heterogeneousinformation can be organized into coher-ent, interoperable collections in comput-ing laboratory settings, and that these canbe searched and manipulated in new waysto yield useful knowledge. The six DLIprojects addressed a broad range of fun-damental research: new document mod-els, video capture and indexing, geo-graphic data spaces, image retrieval, con-cept spaces, agent-based synthetic globaleconomies, and new tools for classroomeducation, to name a few. (See DLINational Synchronization at http://dli.grainger.uiuc.edu/national.htm.)

The Digital Libraries Initiative-Phase 2(DLI-2) supported by NSF, DARPA,

NLM, LoC, NEH, NASA, and otheragency partners will address a refinedtechnology research agenda, and look tosupport new areas in the digital librariesinformation life cycle, including contentcreation, access, use and usability, preser-vation, and archiving. (See DLI-2 atwww.dli2.nsf.gov) DLI-2 will look to cre-ate domain applications and operationalinfrastructure, and understand their useand usability in various organizational,economic, social, and international con-texts. In short, DLI-2 will investigate dig-ital libraries as human-centered systems.DLI-2 involvement will extend farbeyond computing and communicationsspecialty communities to engage scholars,practitioners, and learners in not only sci-ence and engineering but also arts andhumanities. DLI-2 recognizes that knowl-edge access is inherently international andwill actively promote activities andprocesses that bridge political and lan-guage boundaries, including sponsoringprojects through a new program inInternational Digital Libraries Collabora-tive Research.

Many of the most important researchquestions regarding systems and use arebound into the process of building andusing real-world operational systems. DLIwas characterized by a single projectmodel addressing a broad, technology-cen-tered research agenda and building tech-nology testbeds. Content was of secondaryconcern and acquired primarily throughdonations. DLI research illuminated thecomplexity and difficulty of fundamentalissues of functionality, scalability, interop-

erability, reliability, and usability.Investigations into these and related

technologically grounded questions willcontinue in DLI-2, but until large-scaledistributed systems are built, instru-mented, filled with content of value, andopen to use by large and diverse popula-tions, many important questions will gounanswered. The DLI projects reachedtheir most potent stage as research enter-prises toward the end of their funded termas the testbeds matured and became heav-ily used. To scientifically understand howlarge-scale distributed digital librariesbehave in a global information environ-ment, and how they might be used to thegood of society, we must first begin tobuild and use them.

The recent President’s InformationTechnology Advisory Committee InterimReport to the President (Aug. 1998,www.ccic.gov/ac/interim) notes that cur-rent agency practices and modes of sup-port for IT projects are ineffective inaddressing research areas that require alarge- or medium-sized team and afocused effort of more than a few years.

Digital library technologies are a nat-ural outcome of earlier federal funding ofhigh-end computing systems and high-performance networks. Digital librariesare among the first and most promisinggenerations of applications to exploit andvalidate the continuing development ofthese basic technologies and services.Digital libraries fit the bill for new federalplans to provide for the nation’s informa-tion needs in the 21st century through net-worked computing.

.

ing beyond the US. The “International Activities”sidebar includes contributions describing the devel-oping activities in Europe and Asia, based on resultsfrom recent technical workshops. The sidebar con-cludes with the past president of the InternationalFederation of Library Associations discussing politi-cal and economic difficulties of spreading researchtechnologies into practical systems for searchingacross languages and across cultures.

The articles in this issue are careful retrospectiveson multiyear digital library research projects, whichdiscuss large-scale testbeds for text documents and

fundamental technologies for semantic interoperabil-ity beyond text.

LARGE-SCALE TESTBEDS Building an experimental testbed is an accepted

methodology for evaluating networked information sys-tems. A testbed is a prototype system with real collec-tions and real users, but supported as a research ratherthan a commercial product. Many national policy com-mittee reports such as the NRC National Collabora-tories,4 the NSF DLI-2 Planning,5 and the PITAC3 haveemphasized the necessity of large-scale testbeds as the

February 1999 47

Digital Libraries: The View from DARPARonald Larsen, Defense AdvancedResearch Projects Agency

Speed and precision, central to DARPA’sinformation management objectives, wereaddressed through vertical integration ofinformation systems in the 1980s. But in the1990s, this model broke down. Explosivegrowth of the WWW made vertical inte-gration an inadequate response, and wasreplaced by an emphasis on digital libraries.DARPA’s Information Management pro-gram (www.darpa.mil/ito/ research/im)addresses core digital library issues requir-ing revolutionary research technology.

• Federated repositories. The organi-zation of distributed repositories intoa coherent virtual collection is funda-mental, as demonstrated by the Net-worked Computer Science TechnicalReference Library (NCSTRL).

• Scalability. Managing billions of dig-ital objects and millions of sourcesposes challenges in identifying, cate-gorizing, indexing, summarizing, andextracting content.

• Interoperability. Digital librariesrequire semantic interoperabilityamong heterogeneous repositoriesdistributed across the network.

• Collaboration. Analysts work in dis-tributed teams, building on each other’sknowledge, experience, and, resources.

• Testbed development. DARPA is estab-lishing four DLI sites and NCSTRL aspartners in providing stable, accessibletest collections to the research com-munity. (www.dlib.org)

• Communication. Timely dissemina-tion of research results is the focus ofD-Lib, now mirrored internationally.

Defense requirements challenge digitallibrary technology, both qualitatively andquantitatively. DLI illuminated the com-plexity and difficulty of scalability, inter-operability, and usability. DARPA’s infor-mation management research programreflects a continuing commitment to theseissues.

Digital Libraries: The View from NLMMilton Corn, National Library of Medicine

Health-related activities depend on vastseas of information. The physician’s infor-mation needs differ from those of the mole-cular biologist, the patient, and the publichealth scientist, but whoever the questioner,the information is likely to be in a variety oflocations and in multiple formats, includingprint, images, graphics, and video. Massivedatabases of clinical information are becom-ing ubiquitous, but use of digitized infor-mation is complicated by problems genericto digital libraries for any domain: behaviorand cognition issues, lack of standards,legacy systems, distributed data, the need tonetwork among heterogeneous systems,inefficient information retrieval, and, par-ticularly for patient data, privacy concerns.

The National Library of Medicine, acomponent of the National Institutes ofHealth, is the world’s prime repository ofbiomedical information. The Librarymaintains and distributes without chargeon the Web a number of widely used data-

bases, such as MEDLINE, the biblio-graphic reference of biomedical literature,and GenBank, a key resource for molecu-lar biologists. The Library has been a pio-neer in technology to manage informationand supports research and developmentwork in medical informatics.

As examples of large, complex data-bases in text and in images, NLM is offer-ing two of its resources to the DLI-2community:

• the Unified Medical Language System(www.nlm.nih.gov/research/umls/umlsmain.html), an ambitious project offundamental semantic importancefor digital libraries, which is mappingconcepts across the myriad thesauriand vocabularies of biomedicine; and

• the Visible Human Project (www.nlm.nih.gov/research/visible/visible_human.html), by far the most com-prehensive online image repositoryof the male and female human body.

Because digital libraries, utilization ofdistributed databases, and data-miningissues are of national importance tohealthcare delivery and to research, theNational Library of Medicine recognizesthat a joint agency initiative, such as DLI-2, attacks problems generic to digitallibraries, and can both profit from andbenefit biomedical application. NLM hasa particular interest in projects of value toconsumers, but welcomes work relevantto computerized patient records, images,computational biology, education, publichealth, or any other health-related area.

.

48 Computer

only method for determining which information systemfeatures are actually useful in practice.

New technologies in digital libraries emerge fromlarge-scale research testbeds. To obtain the requisitecollections and users, these projects have concentratedon text documents, particularly articles already avail-able in electronic form. Text dominates use of infor-mation in the scholarly world, where experimentscould potentially be run. Thus, these representativepapers on digital library testbeds concentrate on jour-nal articles served to scholarly populations.

The Illinois DLI project was a classic testbed pro-ject, developing new technology and deploying itwidely on an experimental basis. The Illinois projectchose as its research paradigm the complete manip-ulation of structured documents—namely, the searchand display of engineering journal articles encodedin Standard Generalized Markup Language (SGML).The project developed federated search of documentstructure across multiple repositories from multiplepublishers, which was deployed in a testbed aroundcampus.

The Illinois DLI project was a research project

developing and experimentally testing new technol-ogy for federated search, by deploying real collectionsto real users on a production basis. The JSTOR pro-ject, in contrast, was intended to become a commer-cial service, now used by many academic institutions.They chose the mature technology of digitizedbitmaps (page images) rather than the immature tech-nology of SGML markup.

Many of the current generation of digital libraryresearch testbeds are turning into production services.For example, the DARPA D-Lib Test Suite6 is provid-ing continuing support for several of the DLI andrelated testbeds, and is actively seeking users to exper-iment further with these testbeds. These experiencesgive the first indication of usage patterns for search inthe Net of the 21st century.

SEMANTIC INTEROPERABILITYThe challenge of digital libraries has remained

unchanged from the goals described in the introductionto the 1996 special issue.7 The DLI projects pursueddeep semantic interoperability, making heterogeneousitems in heterogeneous sources spread across the net-

International Activities

Digital Library Research in EuropeAlan Smeaton, Dublin City University,Ireland

European research into digital libraries(DLs) is funded by the European Union aswell as by national sources. Several coun-tries, such as the UK, have launched spe-cific DL research programs. At theEuropean level, the Fourth FrameworkProgramme of the Commission of theEuropean Union is now concluding with-out having had a specific research programin DLs, although DL projects have beensupported by the Information Engineeringin Europe (www.echo.lu/ie), LanguageEngineering (www.echo.lu/langeng/en/lehome.html), and Esprit (www.cordis.lu/esprit) programs.

DELOS (www.iei.pi.cnr.it/DELOS) is aDL working group and is part of theERCIM (European Research Consortiumfor Informatics and Mathematics; www.ercim.org) Digital Library Initiativefunded by Esprit within the FourthFramework Programme. Its objective isto simulate DL research and collabora-tion. DELOS achieves this through work-

shops, including the series of EuropeanConferences on Research and AdvancedTechnology for Digital Libraries.

One of the most exciting developmentsin European DL research is an NSF-EUcollaboration that has formed five work-ing groups in the key technical areas ofinteroperability, metadata, intellectualproperty rights, resource indexing and dis-covery, and multilingual informationaccess. Emerging from these workinggroups will be a white paper driven byresearchers and scientists in the DL area,which will recommend a research agendafor joint research in digital libraries. It ishoped that this will influence the contentof the EU Fifth Framework Programme,the next major wave of technology fund-ing in Europe.

Digital Library Research in AsiaHsinchun Chen, University of ArizonaJerome Yen and Chris Yang, Universityof Hong Kong

Since 1995 digital library research hasbecome a national grand challenge inseveral countries in Asia. Most projectscan be classified into the following cate-gories:

• Nationwide digital library initiativeand special-purpose digital librar-ies—for example, the Library 2000Project in Singapore (to link alllibrary resources in Singapore) andthe Financial Digital Library at theUniversity of Hong Kong (to servethe needs of the Hong Kong stockmarket and users).

• Digital museum and historical docu-ment digitization—for example, theDigital Museum Project of theNational Taiwan University and thedigitization of the art collection of thePalace Museum in Taipei by IBM.

• Local language and multilingualinformation retrieval—for example,the Net Compass Project of TsinghuaUniversity in China, Chinese Infor-mation Retrieval at the AcademiaSinica, Taiwan, and New Zealand’smultilingual project.

Local language processing and histori-cal cultural content could be the mostimmediate Asian contribution to the inter-national DL community. There is signifi-cant interest among Asian DL researchersin exchanging research ideas and collabo-

.

work appear to be a single uniform federated source. Federating the search at a semantic level is an area

of active research in the digital library community.Statistical approaches in particular are leading theway toward scalable semantics—indexing deeperthan text word search that is computable on largereal collections. For example, concept spaces, whichcapture contextual information, have been computedfor collections of millions of documents.8,9

Semantic interoperability beyond federated searchalso involves making multiple sources appear as a sin-gle source, or making single systems with multiplefunctions. The Carnegie Mellon DLI project searchedmultimedia, particularly video segments, by generat-ing text indexes using speech understanding. TheNew Zealand project searched multilingual docu-ments, as well as nontextual search by singing a musi-cal phrase into a folk-song database. The StanfordDLI project searched across different engines usingmultiprotocol gateways. These articles represent agood sample of current research technology. Othereven harder issues remain untouched, such as multi-cultural search across context and meaning.

THE NET OF THE 21ST CENTURYIn the Net of the 21st century, there will be a bil-

lion repositories distributed over the world, whereeach small community maintains a collection of theirown knowledge.1 Semantic indexes will be availablefor each repository, using scalable semantics to gen-erate search aids for the specialized terminology ofeach community. Concept switching across semanticindexes will enable members of one community toeasily search the specialized terminology of another.10

The Internet will have been transformed into theInterspace, where users navigate abstract spaces toperform correlation across sources.11 Informationanalysis will become a routine operation in the Net,performed on a daily basis worldwide.12 Such func-tionality will first be used by specialty professionalsand then by ordinary people, just as has occurred withtext search. Information infrastructure will becomethe essential part of the structure of everyday life, anddigital libraries will become the essential part of infor-mation infrastructure.

This issue of Computer gives retrospectives for a rep-resentative sample of the major research projects in dig-

February 1999 49

ration on projects. The University ofLibrary and Information Science in Japanhas sponsored a series of InternationalSymposia on Digital Libraries (www.DL.ulis.ac.jp). Commercial companies havesponsored DL workshops, such as SunMicrosystem’s 1997 workshop in Beijing,and IBM’s Asian workshops.

More recently, the First Asia DigitalLibrary Workshop was held in Hong Kongin August 1998. The workshop, whichfocused on Asian DL research projects,attracted more than 120 participants fromnine Asia/Pan-Pacific countries. It hasserved as the catalyst for Asian DL collab-orations. Several countries have expressedstrong interest in sponsoring a SecondAsian Digital Library Workshop. An AsiaDigital Library Consortium is fosteringlong-term collaboration and projects in DL-related topics in Asia (see www.cyberlib.net/adl).

Digital Libraries: A Global ConnectionRobert Wedgeworth, University of Illinois at Urbana-Champaign and Int’l Federation Library Associations

Digital libraries are emerging in manyparts of the world to give access to the

world’s scientific developments. Searchingacross these multidisciplinary repositorieswill be a daunting task. Some of theresearch in the Digital Libraries Initiativesis beginning to indicate that this can bedone. But what about access to the gener-ations of scientific knowledge that alreadyexist in the world’s libraries?

A global interconnected library networkof existing collections has begun to emerge.Its Web site (www.ifla.org) is maintainedby the International Federation of LibraryAssociations and Institutions (IFLA).Recognizing some years ago that theresources would never be available to repli-cate the collections, facilities, staffing, andtechnologies of modern libraries in allparts of the world, IFLA launched an elec-tronic network strategy in 1994 as anexperiment to connect its members to itsconference in Havana, Cuba. By 1995,libraries in more than 70 countries wereconnected. At the beginning of 1998,libraries in more than 100 countries wereconnected to this global library network,sharing information and expertise.

Although there are concerns that a tech-nology-based strategy for library develop-ment could exclude some parts of the

world, many libraries report that needingto connect to the international world hasassisted them in obtaining the resources toconnect to the Internet. Others have foundthat rapid communications with librariesin their own region has facilitated accessto scientific information as well as access toadvice and consultations that previouslywould have taken many weeks.

Connectivity and training continue tobe the principal barriers to integrating theglobal network of libraries with the emerg-ing digital libraries. However, the existenceof a global communications network facil-itates training as well as access to scientificinformation. The community of learnersand researchers desiring access to scientificinformation will require many librariansand information specialists to assist in nav-igating complex search and retrieval sys-tems across heterogeneous repositories.Further integration of existing repositoriesof scientific information with the emerg-ing digital information systems will be nec-essary for the scientific community toretain access to the full record of scientificprogress. Utilizing such a global networkwill render scientific communications com-pletely independent of space and time.

.

Trans. Pattern Analysis and Machine Intelligence, Vol.18, Aug. 1996, pp. 771-782.

9. J. Alper, “Taming MEDLINE with Concept Spaces,” Sci-ence, Vol. 281, Sept. 18, 1998, p. 1,785.

10. H. Chen et al., “A Concept Space Approach to Address-ing the Vocabulary Problem in Scientific InformationRetrieval: An Experiment on the Worm Community Sys-tem,” J. Am. Soc. Information Science, Vol. 48, Jan.1997, pp. 17-31.

11. B. Schatz, “High-Performance Digital Libraries: Build-ing the Interspace on the Grid,”Seventh IEEE Int’l Symp.High-Performance Distributed Computing, July 1998,pp. 224-234.

12. Interspace Prototype, www.canis.uiuc.edu.

Bruce Schatz is director of the Community Architec-tures for Network Information Systems (CANIS) Lab-oratory at University of Illinois at Urbana-Champaignand a professor in the Graduate School of Library andInformation Science.

Hsinchun Chen is a professor in the Department ofManagement Information systems at the University ofArizona and director of the Artificial Intelligence Lab.

Contact the authors at schatz@canis.uiuc.edu orhchen@bpa.arizona.edu.

ital libraries. The fundamental new technology surveyedhere stands a good chance of becoming a fundamentalpart of everyday life in the foreseeable future. ❖

References1. B. Schatz, “Information Retrieval in Digital Libraries:

Bringing Search to the Net,” Science, Vol. 275, Jan. 17,1997, pp. 327-334.

2. J. Alper, “Assembling the World’s Biggest Library onYour Desktop,” Science, Vol. 281, Sept. 18, 1998, pp.1,784-1,786.

3. K. Kennedy and W. Joy, chairs, President’s InformationTechnology Advisory Committee (PITAC), Interim Reportto the President, Aug. 1998, www.ccic.gov/ac/interim.

4. V. Cerf and W. Wulf, eds., National Collaboratories:Applying Information Technology for Scientific Research,National Academy Press, Washington, D.C., Mar. 1993.

5. D. Atkins, ed., “Digital Libraries: Report of the SantaFe Planning Workshop on Distributed Knowledge WorkEnvironments,” Mar. 1998, www.si.umich.edu/SantaFe.

6. W. Arms, ed., D-Lib Test Suite, Summer 1998, www.dlib.org/test-suite.

7. B. Schatz and H. Chen, “Building Large-Scale DigitalLibraries,”Computer, Vol. 29, May 1996, pp. 22-26.

8. H. Chen et al., “A Parallel Computing Approach to Cre-ating Engineering Concept Spaces for SemanticRetrieval: The Illinois Digital Library Project,” IEEE

.