Mendeley: Recommendation Systems for Academic Literature

Mendeley:Recommendation

Systems for AcademicLiterature

Kris Jack, PhDData Mining Team Lead

“All the time we are very conscious of the huge challenges that human society has now – curing cancer, understanding the brain for Alzheimer‘s [...].

But a lot of the state of knowledge of the human race is sitting in the scientists’ computers, and is currently not shared […] We need to get it unlocked so we can tackle those huge problems.“

➔ what's a recommender and what does it look like?

➔ what's Mendeley?

➔ the secrets behind recommenders

➔ recommenders @ Mendeley

Overview

What's a recommender and

what does it look like?

Definition:

A recommendation system (recommender) is a subclass of information filtering system that aims to predict a user's interest in items.

What's a recommender?

Recommendation Systems in the Wild

Recommendation Vs. Search

➔ search is a pull strategyvs.

➔ recommendation is a push strategy

Recommendation Vs. Search

search is like following a path...

Recommendation Vs. Search

recommendation is like being on a roller coaster...

A differentsense ofcontrol

What's Mendeley?

...a large data technology startup company

...and it's on a mission to change the way that

research is done!

What is Mendeley?

works like this:

1) Install “Audioscrobbler”

2) Listen to music

3) Last.fm builds your music profile and recommends you music you also could like... and it’s the world‘s biggest open music database

Last.fmMendeley

research libraries

researchers

papers

disciplines

music libraries

artists

songs

genres

Last.fmMendeley

...organise their research

Mendeley provides tools to help users...

...organise their research

...organise their research

...collaborate with one another

Mendeley provides tools to help users...

...organise their research

Tools of scientific discovery

Clean energyClean water

Sustainable food supplies

Pandemic diseases

Terrorist violence

Climate change

US National Academy of Engineering “Grand Challenges”:

Artificial Intelligence

...organise their research

...collaborate with one another

...discover new research

Mendeley provides tools to help users...

...organise their research

...organise their research

...collaborate with one another

...discover new research

Mendeley provides tools to help users...

...organise their research

1.4 million+ users; the 20 largest userbases:

University of CambridgeStanford University

MITUniversity of Michigan

Harvard UniversityUniversity of OxfordSao Paulo University

Imperial College LondonUniversity of Edinburgh

Cornell UniversityUniversity of California at Berkeley

RWTH AachenColumbia University

Georgia TechUniversity of Wisconsin

UC San DiegoUniversity of California at LA

University of FloridaUniversity of North Carolina

Real-time data on 28m unique papers:

Thomson Reuters’ Web of Knowledge(dating from 1934)

Mendeley after 16 months:

50m

Q1/2: How can a tool generate recommendations?

Q2/2: How can you measure the tool's performance?

The secrets behind recommenders

Q1/2: How can a tool generate recommendations?

Content-based Filtering Collaborative Filtering

Find items with similar characteristics (e.g. title, discipline) to what the user previously liked

Find items that users who are similar to you also liked (wisdom of the crowds)

TF-IDF, BM25, Bayesian classifiers, decision trees, artificial neural networks

User-based and item-based variations, matrix factorisation

Quickly absorbs new items (ovecomes cold start problem)

No need to understand item characteristics

Can make good recommendations from very few examples

Tends to give more novel recommendations

Hybrid tools too...

Q2/2: How can you measure the tool's performance?

➔ Cross validation with hold outs➔ get yourself a good ground truth➔ hide a fraction of your data from the system➔ try to predict the hidden fraction from the

remaining data➔ calculate precision and recall

➔ Let users decide➔ set up evaluations with real users (experimental)➔ track tool usage by users

2) Personalised Recommendations● given a user's profile (e.g. interests)● find new articles of interest to them

1) Related Research● given 1 research article● find other related articles

Recommenders@ Mendeley

Use Case 1: Related Research

Strategy

content-based approach (tf-idf with lucene implementation)search for articles with same metadata (e.g. title, tags)

Evaluation

cross-validation with hold outs on a ground truth data set

Use Case 1: Related Research

Q2/2 What are our results?

tag abstract mesh-term title general-keyword author keyword0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

tf-idf Precision per Field when Field is Available

metadata field

Pre

cisi

on

@ 5

Results 1) tags are the most informative field for finding related research

Use Case 1: Related Research

tag bestCombo abstract mesh-term title general-keyword author keyword0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

tf-idf Precision for Field Combos when Field is Available

metadata field(s)

pre

cisi

on

@ 5

abstract+author+general-keyword+tag+title

Results 2) tags outperform combinations of fields

How does Mendeley use recommendation

technologies?

Personalised Recommendations

2/2

2) Personalised Recommendations● given a user's profile (e.g. interests)● find new articles of interest to them

Use Case 2: Perso Recommendations

Strategy

collaborative filtering (item-based with apache mahout)recommend articles to researchers that would interest them

Evaluation

cross-validation with hold outs on a ground truth data set

Use Case 2: Perso Recommendations

Strategy

collaborative filtering (item-based with apache mahout)recommend articles to researchers that would interest them

Evaluation

cross-validation with hold outs on a ground truth data set

Output:Recommend 10 articles to each user

Input:User libraries

16 months ago

Test:10-fold cross validation50,000 user libraries

Results:<0.025 precision at 10

Test:10-fold cross validation50,000 user libraries

10 months ago (i.e. + 6 months)

Results:~0.1 precision at 10

Test:Release to a subset of users

10 months ago (i.e. + 6 months)

Results:~0.4 precision at 10

Article Recommendation Acceptance RatesA

ccep

tan

ce r

ate

(i.e

. acc

ept/

reje

ct c

l ick

s)

Number of months live

Pre

cis i

on a

t 10

art

icle

s

Number of articles in user library

Precision by Library Size

Test:10-fold cross validation50,000 user libraries

So, results comparable to non-distributed recommender

Completely distributed, so can easily run on EC2 within 24 hours...

➔ Recommendations can be complementary to search

➔ They can help users to discover interesting items

➔ They can exploit item metadata (content-based)

➔ They can exploit the 'wisdom of the crowds' (CF)

SummaryConclusions

➔ Crowd-sourced metadata can have a poweful informative value (e.g. article tags)

➔ Sometimes you need to let data grow

➔ Evaluations under lab conditions don't always predict real world results well

➔ Recommenders don't just have to be about making money … remember where we started...?

SummaryConclusions

“All the time we are very conscious of the huge challenges that human society has now – curing cancer, understanding the brain for Alzheimer‘s [...].

But a lot of the state of knowledge of the human race is sitting in the scientists’ computers, and is currently not shared […] We need to get it unlocked so we can tackle those huge problems.“

www.mendeley.com