Machine Learning for Information Architecture in a LargeGovernmental Website ∗

Miles Efron, Gary Marchionini, John Elsas and Julinang ZhangSchool of Information and Library Science

CB#3360, 100 Manning HallUniversity of North CarolinaChapel Hill, NC 27599-3360

{efrom, march}@ils.unc.edu, {jelsas, junliang}@email.unc.edu

ABSTRACTThis paper describes ongoing research into the application ofmachine learning techniques for improving access to govern-mental information in complex digital libraries. Under theauspices of the GovStat Project (http://www.ils.unc.edu/govstat),our goal is to identify a small number of semantically validconcepts that adequately spans the intellectual domain ofa collection. The goal of this discovery is twofold. Firstwe desire a principled aid to information architects. Sec-ond, automatically derived document-concept relationshipsare a necessary precondition for real-world deployment ofmany dynamic interfaces. The current study compares con-cept learning strategies based on three document represen-tations: keywords, titles, and full-text. In statistical anduser-based studies, human-created keywords provide signif-icant improvements in concept learning over both title-onlyand full-text representations.

1. INTRODUCTIONThe GovStat Project is a joint effort of the University ofNorth Carolina Interaction Design Lab and the Universityof Maryland Human-Computer Interaction Lab. Citing end-user difficulty in finding governmental information (espe-cially statistical data) online, the project seeks to create anintegrated model of user access to US government statisti-cal information that is rooted in realistic data models andinnovative user interfaces. To enable such models and in-terfaces, we propose a data-driven approach, based on datamining and machine learning techniques. In particular, ourwork analyzes a particular digital library—the website of theBureau of Labor Statistics (http://www.bls.gov)—in effortsto discover a small number of linguistically meaningful con-cepts, or ”bins,” that collectively summarize the semanticdomain of the site.

∗This research was supported by NSF EIA grant 0131824.

The project goal is to classify the site’s web content accord-ing to these inferred concepts as an initial step towards datafiltering via active user interfaces (cf. [8]). Many digital li-braries already make use of content classification, both ex-plicitly and implicitly; they divide their resources manuallyby topical relation; they organize content into hierarchicallyoriented file systems. The goal of the present research is todevelop another means of browsing the content of these col-lections. By analyzing the distribution of terms across doc-uments, our goal is to supplement the agencies’ pre-existinginformation structures. Statistical learning technologies areappealing in this context insofar as they stand to define adata-driven—as opposed to an agency-driven—navigationalstructure for a site.

Our approach combines supervised and unsupervised learn-ing techniques. A pure document clustering [7] approach tosuch a large, diverse collection as BLS led to poor resultsin early tests [1]. But strictly supervised techniques [2] areinappropriate, too. Although BLS designers have definedhigh-level subject headings for their collections, as we dis-cuss in Section 2, this scheme is less than optimal. Thus wehope to learn an additional set of concepts by letting thedata speak for themselves.

The remainder of this paper describes the details of our con-cept discovery efforts and subsequent evaluation. In Section2 we describe the previously existing, human-created con-ceptual structure of the BLS. This section also describes evi-dence that this structure leaves room for improvement. Nextwe turn to a description of the concepts derived via con-tent clustering under three document representations: key-word, title only, and full-text. Section 6 describes a two-partevaluation of the derived conceptual structures. Finally, weconclude in Section 7 by outlining upcoming work on theproject.

2. STRUCTURING ACCESS TO THE BLSWEBSITE

The Bureau of Labor Statistics (BLS) is a federal govern-ment agency charged with compiling and publishing statis-tics pertaining to labor and production in the US and abroad.Given this broad mandate, the BLS publishes a wide arrayof information, intended for diverse audiences. The agency’swebsite (http://www.bls.gov) acts as a clearinghouse for this

Figure 1: Relation Browser Prototype

process. With over 15,000 text/html documents (and manymore documents if spreadsheets and typeset reports are in-cluded), providing access to the collection provides a steepchallenge to information architects.

2.1 The Relation BrowserThe starting point of this work is the notion that accessto information in the BLS website could be improved bythe addition of a dynamic interface such as the relationbrowser described by Marchionini and Brunk [8]. The re-lation browser allows users to traverse complex data sets byiteratively slicing the data along several facets. In Figure1 we see a prototype instantiation of the relation browser,applied to the FedStats website (http://www.fedstats.gov).

The relation browser supports information seeking by allow-ing users to form queries in a stepwise fashion, slicing andre-slicing the data as their interests dictatate. Thus in Fig-ure 1, users mights limit their search set to those documentsabout ”energy.” Within this subset of the collection, theymight further eliminate documents published more than ayear ago. Finally, they might request to see only documentspublished in PDF format.

As Marchionini and Brunk discuss, capturing the publica-tion date and format of documents is trivial. But successfulimplementations of the relation browser also rely on topicalclassification. This presents two stumbling blocks for systemdesigners:

• Information architects must define the appropriate setof topics for their collection

• Site maintainers must classify each document into itsappropriate categories

Figure 2: The BLS Home Page

These tasks parallel common problems in the metadata com-munity: defining appropriate elements and marking up doc-uments to support metadata-aware information access. Givena collection of over 15,000 documents, these hurdles are es-pecially daunting, and automatic methods of approachingthem are highly desirable.

2.2 A Pre-Existing StructurePrior to our involvement with the project, designers at BLScreated a shallow classificatory structure for the most im-portant documents in their website. As seen in Figure 2,the BLS home page organizes 65 ”top-level” documents into15 categories. These include topics such as Employment andUnemployment, Productivity, and Inflation and Spending.

We hoped initially that these pre-defined categories could beused to train a 15-way document classifier, thus automatingthe process of populating the relation browser altogether.However, this approach proved unsatisfactory.

In personal meetings, BLS officials voiced dissatisfactionwith the existing topics. Their existence, it was argued,owed more to the institutional structure of BLS than it didto the inherent topology of the website’s information space.In other words, the topics reflected official divisions ratherthan semantic clusters.

This impression was borne out quantitatively. This is a veryshallow classificatory structure; each of the 15 top-level cat-egories is linked to a small number of related pages. Thusthere are 7 pages associated with Inflation. Altogether, thelink structure of this classificatory system contains 65 doc-uments; that is, excluding navigational links, there are 65documents linked from the BLS home page, where each hy-perlink connects a document to a topic (pages can be linked

m00M0M

0

1010M10M

10

2020M20M

20

3030M30M

30

4040M40M

40

5050M50M

50

6060M60M

60

m00M0M

0

22M2M

2

44M4M

4

66M6M

6

88M8M

8

1010M10M

10

1212M12M

12

1414M14M

14

Eigenvalue RankMEigenvalue RankM

Eigenvalue Rank

Eigenvlue MagnitudeMEigenvlue MagnitudeMEi

genv

lue

Mag

nitu

de

Figure 3: Scree Plot of BLS Categories

to multiple topics). Based on this hyperlink structure, wedefined M, a symmetric 65 × 65 matrix, where mij countsthe number of topics in which documents i and j are bothclassified on the BLS home page.

From an intuitive standpoint, one of the most problem-atic aspects of the pre-existing BLS topic structure was thefact that its classes reflect the structure of the organiza-tion. Since many offices in BLS are concerned with issuesof Business Costs, for example, the home page of the Pro-ducer Price Index is linked to three of the fifteen categories.To analyze the redudancy inherent in the pre-existing struc-ture, we derived the principal components of M (cf. [12]).Figure 3 shows the resultant scree plot.

Because all 65 documents belong to at least one BLS topic,the rank of M is guaranteed to be less than or equal to15 (hence, eigenvalues 16 . . . 65 = 0). What is surprisingabout Figure 3, however, is the precipitous decline in eigen-value magnitude as eigenvalue rank exceeds three. The fourlargest eigenvlaues of the data account for 62.2% of the totalvariance in the data. This fact, in conjunction with the ob-vious gap between eigenvalues three and four suggests a highdegree of redundancy among the topics. While some inter-class redundancy is necessary (to accomodate multi-topicdocuments), this distribution reinforces the the agency’s as-sertion that the pre-existing categories were less than opti-mal.

3. A HYBRID APPROACH TO TOPIC DIS-COVERY

To aid in the discovery of a new set of high-level topics forthe BLS website, we turned to unsupervised machine learn-ing methods. In efforts to let the data speak for themselves,

we desired a means of concept discovery that would be basednot on the structure of the agency, but on the content of thematerial. To begin this process, we crawled the BLS web-site, downloading all documents of MIME type text/html.This led to a corpus of 15,165 documents. Based on thiscorpus, we hoped to derive k ≈ 10 topical categories, suchthat each document di is assigned to one or more classes.

Document clustering (cf. [11]) provided an obvious, but onlypartial solution to the problem of automating this type ofhigh-level information architecture discovery. The problemswith standard clustering are threefold.

1. Mutually exclusive clusters are inappropriate for iden-tifying the topical content of documents, since docu-ments may be about many subjects.

2. Due to the heterogeneity of the data housed in theBLS collection (tables, lists, surveys, etc.), many doc-uments’ terms provide very noisy topical information.

3. For application to the relation browser, we require asmall number (k ≈ 10) of topics. Without significantdata reduction, term-based clustering tends to deliverclusters at too fine a level of granularity.

In light of these problems, we take a hybrid approach totopic discovery. First, we limit the clustering process to asample of the entire collection, described in Section 4. Work-ing on a focused subset of the data helps to overcome prob-lems two and three, listed above. To address the problemof mutual exclusivity of standard clusters, we combine un-supervised with supervised learning methods, as describedin Section 5

4. FOCUSING ON CONTENT-RICH DOC-UMENTS

To derive empirically evidenced topics we initially turned tocluster analysis. Let A be the n× p data matrix with n ob-servations in p variables. Thus aij shows the measurementfor the ith observation on the jth variable. As described in[7], the goal of cluster analysis is to assign each of the nobservations to one of a small number k mutually exclusivegroups, each of which is characterized by high intra-clustercorrelation and low inter-cluster correlation. Though the al-gorithms for accomplishing such an arrangement are legion,our analysis focuses on so-called k-means clustering1, duringwhich, each observation oi is assigned to the cluster ck whosecentroid is closest to it, in terms of Euclidean distance. Themodel is fitted by minimizing the sum of squared error (sse),given in Equation 1:

1We have focused on k-means as opposed to other cluster-ing algorithms for several reasons. Chief among these is thecomputational efficiency enjoyed by the k-means approach.Because we need only a ”flat” clustering there is little tobe gained by the more expensive hierarchical algorithms. Infuture work we may turn to model-based clustering [2], al-though k-means is identical to standard EM clustering sub-ject to certain assumptions.

sse =

KXk=1

|k|Xi=1

= ‖xik − xk‖2 (1)

where xik is the ith observation in the kth cluster, and xk isthe centroid of the kth cluster.

Medioid-based clustering by k-means is well-studied in thestatistical literature, and has shown good results for textanalysis (cf. [4, 11]). However, k-means clustering requiresthat the researcher specify k, the number of clusters to de-fine. When applying k-means to our 15,000 document collec-tion, indicators such as the gap statistic [13] and an analysisof the mean-squared distance across values of k, suggestedthat k ≈ 80 was optimal. This paramterization led to se-mantically intelligible clusters. However, 80 clusters are fartoo many for application to an interface such as the relationbrowser. Moreover, the granularity of these clusters was un-suitably fine. For instance, the 80-cluster solution deriveda cluster whose most highly associated words (in terms oflog-odds ratio) were drug, pharmacy, and chemist. Thesewords are certainly related, but they are related at a levelof specificity far below what we desired.

To remedy the high dimensionality of the data, we resolvedto limit the algorithm to a subset of the collection. In con-sultation with employees of the BLS, we continued our anal-ysis on documents that form a series titled From the Editor’sDesk (http://www.bls.gov/opub/ted). These are brief arti-cles, written by BLS employees. BLS employees suggestedthat we focus on the Editor’s Desk because it is intended tospan the intellectual domain of the agency. The column ispublished daily, and each entry describes an important cur-rent issue in the BLS domain. The Editor’s Desk columnhas been written daily (five times per week) since 1998. Assuch, we operated on a set of N = 1279 documents.

Limiting attention to these 1279 documents not only re-duced the dimensionality of the problem. It also guaran-teed that the clustering process learned on a relatively cleandata set. While the entire BLS collection contains a greatdeal of non-prose text (i.e. tables, lists, etc.), the Editor’sDesk documents are all written in clear, journalistic prose.Each document is highly topical, further aiding the discov-ery of term-topic relations. Finally, the Editor’s Desk col-umn provided an ideal learning environment because it iswell-supplied with topical metadata. Each of the 1279 docu-ments contains a list of one or more keywords. Additionally,a subset of the documents (1112) contained a subject head-ing. This metadata informed our learning and evaluation,as described in Section 6.1.

5. COMBINING SUPERVISED AND UNSU-PERVISED LEARNING FOR TOPIC DIS-COVERY

To derive suitably general topics for the application of adynamic interface to the BLS collection, we combined doc-ument clustering with text classification techniques. Specif-ically, using k-means, we clustered each of the 1279 docu-ments into one of k clusters, with the number of clusterschosen by analyzing the within-cluster mean squared dis-tance at different values of k (see Section 6.1). Construct-

ing mutually exclusive clusters violates our assumption thatdocuments may belong to multiple classes. However, theseclusters mark only the first step in a two-phase process oftopic identification. At the end of the process, document-cluster affinity is measured probabilistically.

Once the Editor’s Desk documents were assigned to clusters,we constructed a k-way classifier that calculates P (Ck|di),the probability that a new document di is a member of classCk. We tested three statistical classification techniques:probabilistic Rocchio (prind), naive Bayes, and support vec-tor machines (SVMs). All were implemented using McCal-lum’s BOW text classification library [9]. Prind is a prob-abilistic version of the Rocchio classification algorithm [5].Interested readers are referred to Joachims’ article for fur-ther details of the classification method. Like prind naiveBayes attempts to classify documents into the most proba-ble class. It is described in detail in [10]. Finally, supportvector machines were thoroughly explicated by Vapnik [14],and applied specifically to text in [6]. They define a decisionboundary by finding the maximally separating hyperplane ina high-dimensional vector space in which document classesbecome linearly separable.

Having clustered the documents and trained a suitable clas-sifier, the remaining 14,000 documents in the collection arelabeled by means of automatic classification. That is, foreach document di we derive a k-dimensional vector, quanti-fying the likelihood that di belongs to each class C1 . . . Ck.Deriving topic scores for the entire 15 thousand-documentcollection thus requires only several minutes of CPU time.The output of this process is a score for every document inthe collection on each of the automatically discovered top-ics. These scores may then be used to populate a relationbrowser interface, or they may be added to a traditional in-formation retrieval system. In future work evaluation usingboth of these approaches will be undertaken.

6. EVALUATION OF CONCEPT DISCOV-ERY

Prior to implementing a relation browser interface and un-dertaking the attendant user studies, it is of course impor-tant to evaluate the quality of the inferred concepts, andthe ability of the automatic classifier to assign documentsto the appropriate subjects. To evaluate the success of thetwo-stage approach described in Section 5, we undertooktwo experiments. During the first experiment we comparedthree methods of document representation for the cluster-ing task. The goal here was to compare the quality of doc-ument clusters derived by analysis of full-text documents,documents represented only by their titles, and documentsrepresented by human-created keyword metadata. Duringthe second experiment, we analyzed the ability of the statis-tical classifiers to discern the subject matter of documentsfrom portions of the database other the the Editor’s Desk.

6.1 Comparing Document RepresentationsDocuments from The Editor’s Desk column came suppliedwith human-generated keyword metadata. Additionally, Thetitles of the Editor’s Desk documents tend to be germane tothe topic of their respective articles. With such an array ofdistilled evidence of each document’s subject matter, we un-

dertook a comparison of document representations for topicdiscovery by clustering. We hypothesized that keyword-based clustering would probably provide useful clusters. Butwe hoped to see whether comparable performance could beattained by methods that did not require extensive humanindexing, such as the title- or full-text representations. Totest this hypothesis, we defined three modes of documentrepresentation—full-text, title-only, and keyword only—wegenerated three sets of topics, Tfull, Ttitle, and Tkw, respec-tively.

Topics based on full-text documents were derived by appli-cation of k-means clustering to the 1279 Editor’s Desk doc-uments, where each document was represented by a 1908-dimensional vector. These 1908 dimensions captured theTFIDF weights of each term ti in document dj , where allterms that occurred at least twice in the data were retained.To arrive at the appropriate number of clusters for thesedata, we inspected the within-cluster mean-squared distancefor each value of k = 1 . . . 20. We found evidence of a de-crease in error as k approached 10. To select a single integervalue, we calculated which value of k led to the least varia-tion in cluster size. This metric stemmed from a desire tosuppress the common result where one large cluster emergesfrom the k-means algorithm, accompanied by several ac-cordingly small clusters. Without reason to believe thatany single topic should have dramatically high prior odds ofdocument membership, this heuristic led to kfull = 10.

Clusters based on document titles were constructed simi-larly. However, in this case, each document was representedin the vector space spanned by the 397 terms that occur atleast twice. Using the same method of minimizing the vari-ance in cluster membership, ktitle, the number of clusters inthe title-based representation was also set to 10.

The dimensionality of the keyword-based clustering was verysimilar to that of the title-based approach. There were 299keywords in the data, all of which were retained.The me-dian number of keywords per document was also 7, wherea keyword is understood to be either a single word, or amulti-word term such as ”consumer price index.” Using thekeywords, the documents clustered into 10 classes.

To evaluate the clusters derived by each method of doc-ument representation, we used the subject headings thatwere included with 1112 of the Editor’s Desk documents.Each of these 1112 documents was assigned 1 or more sub-ject headings, which were withheld from all of the clusterapplications. Our analysis began with the assumption thatdocuments with the same subject headings should clustertogether. To facilitate this analysis, we took a conservativeapproach; we considered multi-subject classifications to beunique. Thus if document di was assigned to a single sub-ject prices, while document dj was assigned to two subjects,international comparisons, prices, documents di and dj arenot considered to come from the same class.

Table 6.1 shows all Editor’s Desk subject headings that wereassigned to at least 10 documents. As noted in the table,there were 19 such subject headings, which altogether cov-ered 609 (54%) of the documents with subjects assigned.These document-subject pairings formed the basis of our

Table 1: Top Editor’s Desk Subject HeadingsSubject Countprices 92unemployment 55occupational safety and health 53international comparisons, prices 48manufacturing, prices 45employment 44productivity 40consumer expenditures 36earnings and wages 27employment and unemployment 27compensation costs 25earnings & wages, metro. areas 18benefits, compensation costs 18earnings & wages, occupations 17employment, occupations 14benefits 14earnings & wage, regions 13work stoppages 12earnings & wages, industries 11Total 609

Table 2: Contingecy Table for Three DocumentRepresentations

Representation Right Wrong AccuracyFull-text 392 217 0.64Title 441 168 0.72Keyword 601 8 0.98

analysis. Limiting analysis to subjects with N > 10 keptthe resultant χ2 tests suitably robust.

The clustering derived by each document representation wastested by its ability to collocate documents with the samesubjects. Thus for each of the 19 subject headings in Table6.1, Si, we calculated the proportion of documents assignedto Si that each clustering co-classified. Further, we assumedthat whichever cluster captured the majority of documentsfor a given class constituted the ”right answer” for that class.For instance, There were 92 documents whose subject head-ing was prices. Taking the BLS authors’ classifications asground truth, all 92 of these documents should have endedup in the same cluster. Under the full-text representation52 of these documents were clustered into category 5, while35 were in category 3, and 5 were in category 6. Taking themajority cluster as the putative ”right” home for these doc-uments, we consider the accuracy of this clustering on thissubject to be 52/92 = 0.56. Repeating this process for eachtopic across all three representations led to the contingencytable shown in Table 6.1.

The obvious superiority of the keyword-based clustering ev-idenced by Table 6.1 was borne out by a χ2 test on theaccuracy proportions. Comparing the proportion right andwrong achieved by keyword and title-based clustering led top ≈ 0. Due to this result, in the remainder of this paper,we focus our attention on the clusters derived by analysis ofthe Editor’s Desk keywords. The ten keyword-based clusters

Table 3: Keyword-Based Clustersbenefits costs international jobsplans compensation import employmentbenefits costs prices jobsemployess benefits petroleum youth

occupations prices productivity safetyearnings prices productivity safetywages index output healthpay inflation nonfarm occupational

are shown in Table 6.1, represented by the five terms mosthighly associated with each cluster, in terms of the log-oddsratio. Additionally, each cluster has been given a label bythe researchers.

Evaluating the results of clustering is notoriously difficult (cf[13]). In order to lend our analysis suitable rigor and utility,we made several simplifying assumptions. Most problematicis the fact that we have assumed that each document belongsin only a single category. This assumption is certainly false.However, by taking an extremely rigid view of what con-stitutes a subject—that is, by taking a fully qualified andoften multipart subject heading as our unit of analysis—wemitigate this problem. Analogically, this is akin to consid-ering the location of books on a library shelf. Although agiven book may cover many subjects, a classification systemshould be able to collocate books that are extremely similar,say books about occupational safety and health. The mostserious liability with this evaluation, then, is the fact thatwe have compressed multiple subject headings, say prices :international into single subjects. This flattening obscuresthe multivalence of documents. We turn to a more realisticassessment of document-class relations in Section 6.2.

6.2 Accuracy of the Document ClassifiersAlthough the keyword-based clusters appear to classify theEditor’s Desk documents very well, their discovery only solvedhalf of the problem required for the successful implementa-tion of a dynamic user interface such as the relation browser.The matter of roughly fourteen thousand unclassified doc-uments remained to be addressed. To solve this problem,we trained a statistical classifier, as described above in Sec-tion 5. For each document in the collection di, this classifiergives a pi, a k vector of probabilities or distances (depend-ing on the classification method used), where pik quantifiesthe strength of association between the ith document andthe kth class.

Estimating these associations from the Editor’s Desk datais difficult. Because we have only 1279 documents and 10classes, the number of training documents per class is rel-atively small. In addition to models fitted to the Editor’sDesk data, then, we constructing a fourth model, supple-menting the training sets of each class by querying the Googlesearch engine (http://www.google.com) and applying naiveBayes to the augmented test set. For each class, we cre-ated a query by submitting the three terms with the highestlog-odds ratio with that class. Further, each query was lim-ited to the domain www.bls.gov. For each class we retrievedroughly up to 400 documents from Google (the actual num-ber varied depending on the size of the result set returned

Table 4: Cross Validation Results for 4 ClassifiersMethod Av. Percent Accuracy SEPrind 59.07 1.07Naive Bayes 75.57 0.4SVM 75.08 0.68Naive Bayes (augmented) 58.16 0.32

by Google). This led to a training set of 4113 documents inthe ”augmented model,” as we call it below.

To test the ability of each classifier to locate documents cor-rectly, we first performed a 10-fold cross-validation on theEditor’s Desk documents. During each fold, 33% of the doc-uments (i.e. 422) were withheld during model fitting. These422 documents were then classified, and the total accuracywas computed for that run. Using this methodology, wecompared the performance of the four classification mod-els described above. Table 6.2 gives the results from crossvalidation.

Although naive Bayes is not significantly more accurate forthese data than the SVM classifier, we limit the remainder ofour attention to analysis of its performance. Our selectionof naive Bayes is due to the fact that it appears to workcomparably to the SVM approach for these data, while beingmuch simpler, both in theory and implementation. Crossvalidation on the augmented model appeared to decreaseclassification accuracy (accuracy= 58.16%). As we discussbelow, however, augmenting the training set appeared tohelp generalization during our second experiment.

The results of Table 6.2 are encouraging. However, the suc-cess of our classifiers on the Editor’s Desk documents thatinformed the cross validation study may not be good pre-dictors of the models’ performance on the remainder to theBLS website. To test the generality of the naive Bayes clas-sifier, we solicited input from 11 human judges who werefamiliar with the BLS website. The sample was chosen byconvenience, and consisted of faculty and graduate studentswho work on the GovStat project. However, none of thereviewers had prior knowledge of the outcome of the clas-sification before their participation. For the experiment, arandom sample of 100 documents was drawn from the entireBLS collection. On average each reviewer classified 83 docu-ments, placing each document into as many of the categoriesshown in Table 6.1 as he or she saw fit.

Results from this experiment suggest that room for improve-ment remains with respect to generalizing to the whole col-lection from the class models fitted to the Editor’s Desk doc-uments. In Table 6.2, we see, for each classifier, the numberof documents for which it’s first or second most probableclass was voted best or second best by the 11 human judges.

In the context of this experiment, we consider a first- orsecond-place classification by the machine to be accuratebecause the relation browser interface operates on a multi-way classification, where each document is classified intomultiple categories. Thus a document with the ”correct”class as its second choice would still be easily available toa user. Likewise, a correct classification on either the most

Table 5: Human-Model Agreement on 100 SampleDocs.

Human Judge 1st Choice

Model Model 1st Choice Model 2nd ChoiceN. Bayes (aug.) 14 24N. Bayes 24 1

Human Judge 2nd Choice

Model Model 1st Choice Model 2nd ChoiceN. Bayes (aug.) 14 21N. Bayes 21 4

Number of Human-Assigned ClassesMNumber of Human-Assigned ClassesM

Number of Human-Assigned Classes

FrequencyMFrequencyM

Freq

uenc

y

m00M0M

0

11M1M

1

22M2M

2

33M3M

3

44M4M

4

55M5M

5

66M6M

6

77M7M

7

m00M0M

0

1010M10M

10

2020M20M

20

3030M30M

30

4040M40M

40

5050M50M

50

6060M60M

60

7070M70M

70

Figure 4: Number of Classes Assigned to Docu-ments by Judges

popular or second most popular category among the humanjudges is considered correct, in cases where a given documentwas classified into multiple classes. There were 44 multi-class documents in our sample, as seen in Figure 4. Theremaining 56 documents were assigned to 1 or 0 classes.

Under this rationale, The augmented naive Bayes classifiercorrectly grouped 73 documents, while the smaller model(not augmented by a Google search) correctly classified 50.The p-value on the resultant χ2 test is p = 0.001, suggest-ing that increasing the training set improved the ability ofthe naive Bayes model to generalize from the Editor’s Deskdocuments to the collection as a whole. However, the im-provement afforded by the augmented model comes at somecost. In particular, the augmented model is significantlyinferior to the model trained solely on Editor’s Desk docu-ments if we concern ourselves only with documents selectedby the majority of human reviewiers—i.e. only first-choiceclasses. Limiting the ”right” answers to the left columnof Table 6.2 gives p = 0.02 in favor of the non-augmentedmodel. For the purposes of applying the relation browser

mjobsjobsMjobsM

jobs

benefitsunemploymentpricespricesMpricesM

prices

safetyinternationalspendingspendingMspendingM

spending

occupationscostscostsMcostsM

costs

productivityHuman ClassificationsMHuman ClassificationsM

Human Classifications

m0.000.00M0.00M

0.00

0.050.100.150.15M0.15M

0.15

0.200.25mjobsjobsMjobsM

jobs

benefitsunemploymentpricespricesMpricesM

prices

safetyinternationalspendingspendingMspendingM

spending

occupationscostscostsMcostsM

costs

productivityMachine ClassificationsMMachine ClassificationsM

Machine Classifications

m0.000.00M0.00M

0.00

0.050.100.10M0.10M

0.10

0.15

Figure 5: Distribution of Classes Across Documents

to complex digital library content (where documents will beclassified along multiple categories), the augmented model ispreferable. But this is not necessesarily the case in general.

It must also be said that 73% accuracy under a fairly liberaltest condition leaves room for improvement in our assign-ment of topics to categories. We may begin to understandthe shortcomings of the described techniques by consultingFigure 5, which shows the distribution of categories acrossdocuments given by humans and by the augmented naiveBayes model. Interestingly, the majority of reviewers putdocuments into only three categories, jobs, benefits, and oc-cupations. On the other hand, the naive Bayes classifierdistributed classes more evenly across the topics. This be-havior suggests areas for future improvement. Most impor-tantly, we observed a strong correlation among the threemost frequent classes among the human judges (for instance,there was 68% correlation between benefits and occupations.This suggests that improving the clustering to produce top-ics that were more orthogonal might improve performance.

7. CONCLUSIONS AND FUTURE WORKMany developers and maintainers of digital libraries sharethe basic problem pursued here. Given increasingly large,complex bodies of data, how may we improve access to col-lections without incurring extraordinary cost, and while alsokeeping systems receptive to changes in content over time.Data mining and machine learning methods hold a great dealof promise with respect to this problem. Empirical meth-ods of knowledge discovery can aid in the organization andretrieval of information. As we have argued in this paper,these methods may also be brought to bear on the designand implementation of advanced user interfaces.

This study explored a hybrid technique for aiding informa-tion architects as they implement dynamic interfaces suchas the relation browser. Our approach combines unsuper-vised learning techniques, applied to a focused subset of theBLS website. The goal of this initial stage is to discover themost basic and far-reaching topics in the collections. Basedon a statistical model of these topics, the second phase ofour approach uses supervised learning (in particular, a naiveBayes classifier, trained on individual words), to assign top-ical relations to the remaining documents in the collection.

In the study reported here, this approach has demonstratedpromise. In its favor, our approach is highly scalable. Italso appears to give fairly good results. Comparing threemodes of document representation—full-text, title only, andkeyword—we found 98% accuracy as measured by colloca-tion of documents with identical subject headings. Whileit is not surprising that author-generated keywords shouldgive strong evidence for such learning, their superiority overfull text and titles, was dramatic, suggesting that even smallamount of metadata can be very useful for data mining.

However, we also found evidence that learning topics from asubset of the collection may lead to overfitted models. Afterclustering 1279 Editor’s Desk documents into 10 categories,we fitted a 10-way naive Bayes classifier to categorize theremaining 14,000 documents in the collection. While wesaw fairly good results (classification accuracy of 75% withrespect to a small sample of human judges), this experimentforced us to reconsider the quality of the topics learned byclustering. The high correlation among human judgments inour sample suggests that the topics discovered by analysisof the Edtior’s Desk were not independent. While we donot desire mutually exclusive categories in our setting, wedo desire independence among the topics we model.

Overall, then, the techniques described here provide an en-couraging start to our work on acquiring subject metadatafor dynamic interfaces automatically. It also suggests thata more sophisticated modeling approach might yield betterresults in the future. In upcoming work we will experimentwith streamlining the two-phase technique described here.Instead of clustering documents to find topics and then fit-ting a model to the learned clusters, our goal is to expandthe unsupervised portion of our analysis beyond a narrowsusbset of the collection, such as The Editor’s Desk. Incurrent work we have defined algorithms to identify docu-ments likely to help the topic discovery task. Supplied witha more comprehensive training set, we hope to experimentwith Hyvarinen’s independent component analysis [3], a fac-tor anlaytic matrix factorization that has shown promise fortext mining applications.

Topic discovery and document classification have long beenrecognized as fundamental problems in information retrievaland other forms of text mining. What is increasingly clear,however, as digital libraries grow in scope and complexity,is the applicability of these techniques to problems at thefront-end of systems such as information architecture andinterface design. Finally, then, in future work we will buildon the user studies undertaken by Marchionini and Brunkin efforts to evaluate the utility of automatically populateddynamic interfaces for the users of digital libraries.

8. REFERENCES[1] M. Efron, G. Marchionini, and J. Zhang. Implications

of the recursive representation problem for automaticconcept identification in on-line governmentalinformation. In Proceedings of the ASIST SpecialInterest Group on Classification Research (ASISTSIG-CR), 2003.

[2] T. Hastie, R. Tibshirani, and J. Friedman. TheElements of Statistical Learning: Data Mining,Inference, and Prediction. Springer Series in Statistics.Springer, New York, 2001.

[3] A. Hyvarinen, J. Karhunen, and E. Oja. IndependentComponent Analysis. Wiley Interscience, 2001.

[4] A. K. Jain, M. N. Murty, and P. J. Flynn. Dataclustering: a review. ACM Computing Surveys,31(3):264–323, September 1999.

[5] Thorsten Joachims. A probabilistic analysis of theRocchio algorithm with TFIDF for text categorization.In Douglas H. Fisher, editor, Proceedings of ICML-97,14th International Conference on Machine Learning,pages 143–151, Nashville, US, 1997. MorganKaufmann Publishers, San Francisco, US.

[6] Thorsten Joachims. Text categorization with supportvector machines: learning with many relevantfeatures. In Claire Nedellec and Celine Rouveirol,editors, Proceedings of ECML-98, 10th EuropeanConference on Machine Learning, pages 137–142,Chemnitz, DE, 1998. Springer Verlag, Heidelberg, DE.

[7] Leonard Kaufman and Peter J. Rosseeuw. FindingGroups in Data: an Introduction to Cluster Analysis.Wiley, 1990.

[8] G. Marchionini, S. Haas, C. Plaisant, B. Shneiderman,and C. Hert. Toward a statistical knowledge network.In Proceedings of the National Conference on DigitalGovernment Research, pages 27–32, Boston, 2003.National Science Foundation.

[9] Andrew Kachites McCallum. Bow: A toolkit forstatistical language modeling, text retrieval,classification and clustering.http://www.cs.cmu.edu/ mccallum/bow, 1996.

[10] T. Mitchell. Machine Learning. McGraw Hill, 1997.

[11] E. Rasmussen. Clustering algorithms. In W. B. Frakesand R. Baeza-Yates, editors, Information Retrieval:Data Structures and Algorithms, pages 419–442.Prentice Hall, 1992.

[12] G. Strang. Linear Algebra and its Applications.International Thompson Publishing, 1988.

[13] R. Tibshirani, G. Walther, and T. Hastie. Estimatingthe number of clusters in a dataset via the gapstatistic, 2000.

[14] V. N. Vapnik. The Nature of Statistical LearningTheory. Springer, 2000.