The Mechanical Librarian

Recommending Journal Articles

in a Scientific Digital Library

Andre Vellino

andre.vellino@cnrc.ca

Group Leader, CISTI Research

Canada Institute for Scientific and Technical Information

Chef de groupe, Recherche ICIST

Institute canadien de l'information scientifique et technique

2

Outline of Talk

• The Mechanical Librarian

• How Recommenders Work

• Recommenders in Digital Libraries

• Problems for Science Article Recommenders and

Strategies for CISTI’s Recommender Research

• Synthese on CISTI Lab

• Alternative Approaches

• Future Work

Acknowledgements to: Glen Newton, Jeff Demaine and Greg Kresko &

Students : Dave Zeber, Matthew Rutledge-Taylor and Aurel Constantinescu

The Human (Reference)

Librarian

3

World Knowledge Experience

Authoritative

Trustworthy

References

Vocabularies

Databases

The Mechanical Librarian

The Web, they say, is leaving the era of search and entering

one of discovery. What's the difference? Search is what you

do when you're looking for something. Discovery is when

something wonderful that you didn't know existed, or didn't

know how to ask for, finds you.

Jeffrey M. O'Brien, Fortune Magazine4

Knowledge Discovery

Technologies

• Text Mining

– Enhances the researcher’s ability to

discover new and meaningful information

from existing text repositories

• Network Analysis

– Distills the structural relationships among

bibliographic elements to reveal trends

and patterns in science

• User Behaviour

– Infers “wisdom of the crowds” from

usage statistics

5

What is a “Recommender”?

• A recommender is a software system which attempts to predict

items that a user may be interested in, given information about

– the user's interests

– the content in the items

– the usage patterns of other users

• Items may be:

– Merchandise: movies, music, books

– Text: news, blogs, web pages, and, why not,

Scientific Journal Articles

Amazon Recommender

System

Explanations

User Ratings

Category Filter

Personalized

User

Control

Companies That Offer

Recommenders to Users

9

Books

Web SitesMovies

Music

Companies That Sell

Recommender Services

10

Product Merchandise Placement

Database Mining

Advertizing / Product Placement

Software as a Service Platform

Recommendation is Hard

Netflix Prize: $1M

• Netflix Prize

– To develop a recommender that improves quality of

recommendations by 10% over Netflix’s

– http://www.netflixprize.com/

• Current Leader Board

– BellKor (9.6%)

– … + 39 others

• NY Times Magazine Article

http://www.nytimes.com/2008/11/23/magazine/23Netflix-t.html

11

Good Recommendations

are REALLY Hard

12

13

Outline of Talk

• The Mechanical Librarian

• How Recommenders Work

• Recommenders in Digital Libraries

• Problems for Science Article Recommenders and

Strategies for CISTI’s Recommender Research

• Demonstration of Synthese on CISTI Lab

• Alternative Approaches

• Future Work

Taxonomy of

Recommender Systems

Collaborative Filtering

• Usage based, with item-ratings

– User-Based (“similar users”)

– Item-Based (“like items”)

• Algorithms

– Memory-based

– Model-based

Content-Based Filtering

• Content (text / waveform / pixel) analysis to

– Find “similar users”

– Find “similar items”

J. Breese, D. Heckerman, C. Kadie, et al. Empirical Analysis of Predictive Algorithms for Collaborative

Filtering. Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, 461, 1998.

15

How Collaborative

Filtering (CF) Works

• User-Based CF

– Given user A find all the other users {U} that have the most

“similar” item-rating patterns

– For each item I not yet rated by A, predict the likely rating A will

assign to I given the ratings for I given by {U}

– Present the Top-N ordered list of items {I} to the user

• Item-Based CF

– Given user A and the set of items {I} to which A has given

ratings, find all the other items {O} that are “similar” to {I}

– Present the Top-N ordered list of items {O} to the user

Sarwar, Badrul M., George Karypis, Joseph A. Konstan, and John Reidl. "Item-based

collaborative filtering recommendation algorithms." World Wide Web. 2001, 285-295.

16

Find “Nearest Neighbour”

and Predict Rating

• Find Nearest Neighbours (e.g. cosine similarity)

• Predict Rating (item i for user u)

– Weighted average of user’s ratings on N similar users

17

User-Based

Collaborative Filtering

5

2

Reader

?44Ted

5345Alice

434Carol

51Bob

BoltDark NightDoubtMilk

• Goal: predict the rating Ted will give to the movie “Bolt”

• Step 1 – eliminate the user-profiles of users who didn’t rate “Bolt”

• Step 2 – find Ted’s “K-nearest neighbours” who rated “Bolt” and at

least 2 other movies (Alice)

• R(Ted,Bolt) ~= 5.

5

Movies

Users

Things that can go wrong

with Collaborative Filtering

• False “product ratings” to artificially boost ranking (spamming)

• Losing the diversity in the “Long Tail” – converges to “Top N”.

18Fleder, D. and K. Hosanagar. 2008. Blockbuster culture's next rise or fall: The effect of

recommender systems on sales diversity. NET Institute Working Paper 07-10.

Content-Based

Recommenders

“These things are similar (in content) to that”.

• Depends only on a measure of similarity between the content in

the items (text, music, images)

• Typical Steps for Content Based Recommenders

1. Cluster the user’s purchased or highly-rated items by

content-similarity

2. Find other similar items not purchased or rated by the user

3. Recommend the “Top N” to the user

19

Search Engine as

“Content-Based

Recommender”

Collaborative filtering

“Similar Pages” is a

Content-Based

Recommender

What can go wrong with

Content Based

Recommenders

that use only Metadata

• Bad Men Do What Good Men Dream: A Forensic Psychiatrist Illuminates the Darker Side of Human Behavior

• Do Animals Dream?: Children's Questions about Animals Most Often asked of the Natural History Museum

• All I Do is Dream of You The other end of the leash : why we do what we do around dogs

• Why do Catholics do that : a guide to the teachings and practices of the Catholic Church

• Electric universe : the shocking true story of electricity

• The Island of Sheep

23

Outline of Talk

• The Mechanical Librarian

• How Recommenders Work

• Recommenders in Digital Libraries

• Problems for Science Article Recommenders and

Strategies for CISTI’s Recommender Research

• Demonstration of Synthese on CISTI Lab

• Alternative Approaches

• Future Work

Value of Recommenders

in a Digital Library

24

• For the Researcher

– Provide serendipity in a Browse / Search / Retrieve portal

• Broaden scope of search to cognate but otherwise disparate domains

• For the Library

– Increase customer loyalty by creating dynamic, adaptive,

customized services

• Alerts & notifications based on usage and collaborative filtering rather

than stored queries

• For Authors

– Given a draft article (with citations), find additional citations

• For Publishers & Journal Reviewers

– Given a submitted article, recommending peer-reviewers

Recommender Systems in

Digital Libraries

– Techlens (University of Minnesota) (2002)

• Uses ACM DL, full text Mixed Hybrid

– BibTip (University of Karlsruhe) (2003)

• Uses OPAC (Library Catalog) usage data for collaborative filtering

– IngentaConnect (2007)

• Uses Baynote (SaaS) customer tracking

– DSpace (2008)

• Content-based recommender based on user-bookmarks

– CiteULike (academic experiment 2008)

• Collaborative filtering on user bookmarks from CiteULike

– “bX” system from Ex Libris (2009)

• Uses SFX resolver logs

– NextBio (to be announced in March 2009)

• Life sciences search engine that uses collaborative filtering + ontologiesto suggest new content (trials / abstracts / data)

25

TechLens

26

“bX”

Recommender (Jan „09)

27

Features

• Uses log data from SFX resolvers

• Applies Collaborative Filtering

• Uses lots of aggregated data

• Developed w/ the Los Alamos National Laboratory.

Possible issues

• Infers identity of users only through IP address

• May not be accurate when http proxies are used

• Same IP address can have several “IR objectives”

• Identical resolved objects may not be recognized

28

Outline of Talk

• The Mechanical Librarian

• How Recommenders Work

• Recommenders in Digital Libraries

• Problems for Science Article Recommenders and

Strategies for CISTI’s Recommender Research

• Demonstration of Synthese on CISTI Lab

• Alternative Approaches

• Future Work

29

Typical Problems with CF

Recommenders in General

• Data Sparsity

– Ratio of Users / Items is low (~ 1:10)

– Number of Ratings per User is low

– Ratings matrix sparsity ~ 95%

• Cold Start Problem

– First-time users get poor or no recommendations because CF matrix has no entries

• Rating Items

– CF recommender must be trained (explicitly or implicitly) by providing ratings to items

• Principle of Induction

– People who exhibited similar behaviour in the past will tend to exhibit similar behaviour in the future.

30

Specific Problems for

Collaborative Filtering in

Science Digital Libraries

• Data Sparsity– Many More Articles & Far Fewer Users (10x)

– Fewer Item / Ratings (~ 99% sparsity)

• Rating Articles– Explicit ratings are more difficult to obtain

• DL users have less need to “express themselves” by explicitly rating items than movie watchers

– Implicit ratings depend on UI features of DL• No reliable method for inferring ratings from browsing and query

behaviour

• Principle of Induction (that past is a good predictor of the future) not necessarily true in digital libraries– Interest drift

– Context shifts

Recommender Research

Strategy @ CISTI

• Follow in footsteps of TechLens+

– Collaborative Filtering (CF) among users

– Seed CF recommender with citation matrix

– Extended with

• PageRank on Citations

• User Contexts

– Future Extensions

• Add Content-Based Filtering (“Fusion Mixed Hybrid” model)

• Distributed Multi-Dimensional Recommender

• Explanation-based interface

31

A. Vellino and D. Zeber. (2007) “A Hybrid, Multi-dimensional Recommender for Journal

Articles in a Scientific Digital Library.” Conference Proceedings on Web Intelligence and

Intelligent Agent Technology

Making a Reference Rating

32

33

Recommender Citation

Seeding

• Articles either cite or don’t cite other articles

• Some articles that are cited are not in collection

• Users’ “article collection profile” citations

TechLens approach to Cold Start / Data Sparsity problem

34

Outline of Talk

• The Mechanical Librarian

• How Recommenders Work

• Recommenders in Digital Libraries

• Problems for Science Article Recommenders and

Strategies for CISTI’s Recommender Research

• Demonstration of Synthese on CISTI Lab

• Alternative Approaches

• Future Work

Synthese Recommender

on CISTI Lab

35

36

Query Index

Add Important Articles to

“Basket” (1)

37

Add Important Articles to

“Basket” (2)

38

Add Important Articles to

“Basket” (3)

39

Add Important Articles to

“Basket” (4)

40

Query Again

41

Add More Articles to

“Basket” (1)

42

Add More Articles to

“Basket” (2)

43

Recommend Based on

Current “Basket”

44

View Recommendations

45

Evaluate Recommender

46

Search and Basket

History

47

Multiple Profiles

48

Synthese Performance

49

Ratings

Perc

enta

ge

0

5

10

15

20

25

30

35

1 2 3 4 5

Ratings of Recommendations

50

Recommender Citation

Seeding

Can we improve on 0 / 1 (Boolean) citation seeding?

51

Apply PageRank to

Citation Matrix

Aurel Constantinescu “Ranking Full-Text Articles using Citation Based Methods”

Master’s Thesis, University of Ottawa

PageRank algorithm applied to citations

52

PageRank-weighted

Citation matrix

• Apply Page Rank on Citations

– Use citation data (as in TechLens+)

– Apply PageRank to weight the citation-based “ratings”

• Done before but only at the Journal level (http://www.eigenfactor.org/)

0.30.2

0.60.30.5

0.50.7

0.60.2

0.40.5

0.4

p6p1 p5p2 p4p3

u2

p1

u1

p2

p4

p3

articles

citationsp7 p8

= constantusers

PageRank Experimental

Results

53A. Vellino “The Effect of PageRank on the Collaborative Filtering of Journal Articles”

NRC Research Report, 2008.

54

Outline of Talk

• The Mechanical Librarian

• How Recommenders Work

• Recommenders in Digital Libraries

• Problems for Science Article Recommenders and

Strategies for CISTI’s Recommender Research

• Demonstration of Synthese on CISTI Lab

• Alternative Approaches

• Future Work

What is a Holographic

Memory System?

• A Holographic Memory System (HMS) stores information in

a manner analogous to the storage of an image on a

holographic plate.

• HMS is composed of units called items

– Each item represents some content

• e.g, a concept, a word, a bibliographic item

– Items are analogous to points on the surface of

holographic film (or, plate)

– Each item stores information about the associations it

has with other items

T. A. Plate, 2003 Holographic Reduced Representations: Distributed Representations for

Cognitive Structures (Stanford, CA: CSLI Publications)

Holographic Memory

System (HMS)

Apple

Red

Spherical

Fruit

Each item stores information about

many other items in the system

HMS

Each point on the Holographic plate

stores information about many parts

of the image

Holography

HMS Recommender for

Journal Articles

• We compared DSHM and user-based CF on journal article

recommendation on 2 small collections

• 90% - 10% Cross Validation

• systematically removed one reference at a time

• tested whether recommender predicts the reference.

• compared DSHM and user-based CF

Medicine Biology

7495 articles 38,667 articles

0.55 references per article 1.15 references per article

M. F. Rutledge-Taylor, A. Vellino and R. L. West. “A Holographic Associative Memory

Recommender System” 3rd Int. Conference on Digital Information Management, London, 2008.

Experimental Results

58

• Advantages

– Holographic System outperformed standard user-based

CF on very sparse bibliographic datasets

– DSHM is better able to exploit the available information

– The uniformly consistent model of DSHM gives it good

potential for success on multi-dimensional datasets

• Disadvantages

– Requires a lot of computational resources

– Unclear about how it works on a large scale.

Holographic Recommender:

Discussion

60

Outline of Talk

• The Mechanical Librarian

• How Recommenders Work

• Recommenders in Digital Libraries

• Problems for Science Article Recommenders and

Strategies for CISTI’s Recommender Research

• Demonstration of Synthese on CISTI Lab

• Alternative Approaches

• Future Work

61

Multi-Dimensional Ratings

Matrix

G. Adomavicious, R. Sankaranarayanan, S. Sen, A. Tuzhilin, ACM Transactions on Information Systems 2005

Incorporating Contextual Information in Recommender Systems Using a Multidimensional Approach

62

Scaling Strategy:

Distributed

Recommenders

• Multiple ratings matrices decomposed by subject area

• Merge separate recommendations by subject

• Reduces matrix sparsity

• Improves accuracy of recommendations

S. Berkovsky, T.Kuflik, and F. Ricci Distributed Collaborative Filtering with

Domain Specialization Proceedings of Recommender Systems 2007

What predicts overall usefulness of a System?

0

0.1

0.2

0.3

0.4

0.5

0.6

Good Rec. Useful Rec. Trust

Generating

Rec.

Adequate

Item

Description

Ease of

Use

Co

rre

lati

on

Importance of Quality and

Trust

63Rashmi Sinha & Kirsten Swearingen – UC Berkeley

64

UI for Navigating

Recommendations

• Explanation-based

Recommendations

– Provide transparency

increase user trust

– Allow users to cluster by

type of reason

– Filter out unwanted

recommendations

P. Pu and L. Chen. Trust Building with Explanation Interfaces. In IUI ’06: Proceedings of

the 11th International Conference On Intelligent User Interfaces, pages 93–100

Conclusions

• Recommender technology is only 12 years old, but mature

enough for widespread commercial use.

• Digital Libraries / Web 2.0 Bibliographic applications are

beginning to use recommenders.

• Digital Libraries create new problems for recommenders

(“context drift” / “data sparsity” / “multiple dimensions”)

• Recommenders insufficiently understood in Digital Libraries.

• Recommender as mechanism for enhancing the process of

scientific discovery promising but still uncertain.

65

Thank You!

Questions?

http://lab.cisti-icist.nrc-cnrc.gc.ca/synthese/