Research recommendations at Mendeley

Research recommendations at Mendeley Marco Rossetti Data Scientist @ross85 24/11/2015 Elsevier, Amsterdam

2

Outline

Research recommendations at Mendeley

• What is Mendeley

• Recommender Systems at Mendeley – Why

– Data Sources

– Algorithms

– Business Logic & Analytics

– User Interface 24/11/2015

3

What is Mendeley

Research recommendations at Mendeley 24/11/2015

4

Mendeley builds tools tohelp researchers …

Research recommendations at Mendeley

Read &

Organize

Search &

Discover

Collaborate &

Network

Experiment &

Synthesize

24/11/2015

5

Read & Organize

Research recommendations at Mendeley

Reference management

Cite-as-you-

write

Full-text article search

Digitalised

annotations

24/11/2015

6

Search & Discover

Research recommendations at Mendeley

Mendeley Suggest

Literature

Search

Related Documents

24/11/2015

7

Collaborate & Network

Research recommendations at Mendeley

Research network

Groups

24/11/2015

8

Mendeley & Elsevier

Research recommendations at Mendeley 24/11/2015

9

Recommender Systems

Research recommendations at Mendeley 24/11/2015

10

Why Recommender Systemsat Mendeley?

Research recommendations at Mendeley

Vision: “To build a personalised research advisor that helps you to organise your work, contextualise it within the global body of research, and connect you with relevant researchers and artifacts.”

24/11/2015

11

Recommender Systemsat Mendeley – Related Documents

Research recommendations at Mendeley 24/11/2015

12

Recommender Systemsat Mendeley – Mendeley Suggest

Research recommendations at Mendeley https://www.mendeley.com/suggest/

24/11/2015

13

Recommender SystemComponents

Research recommendations at Mendeley

Algorithms

Business Logic and Analytics

User Experience

Data Sources Algorithms

Business Logic

& Analytics

User Interface

24/11/2015

14

• Mendeley – User Libraries

• What the users have in their libraries (what they read, what they annotate, what they highlight, what folders they have, etc. etc.)

– Articles metadata (title, authors, abstract, keywords, tags, etc. etc.) – Groups

• Scopus – Citation network

• Science Direct – Logs

• …

Data Sources

Research recommendations at Mendeley 24/11/2015

15

Algorithms

Research recommendations at Mendeley

1.  Collaborative filtering User-based

If Alice read X, Y, Z and Bob read X, Y, Z and W, we recommend W to Alice + Efficient for us because users << items - Only for users with enough articles in the library

Item-based Users who read X also read Y + Item-item similarity matrix is useful to model last n articles read - Expensive in our setting (millions of items)

24/11/2015

16

Algorithms [2]

Research recommendations at Mendeley

1.  Collaborative filtering (still) Matrix factorization

+ Best CF model in literature - A lot of latent factors, generate recommendations on a catalog of million of items is too slow

1 1 11 1 1? ? 1 ? 1 ?

1 1 11 1

1 1 1

U n x k

V k x m

X n x m

X ≈

24/11/2015

17

Algorithms [3]

Research recommendations at Mendeley

2.  Content-based I read articles about text mining, show me other stuff about text mining + Good for semi-cold users (users with only a few articles) - Overspecialisation: items recommended are too similar

3.  Popularity/Trending I work in Computer Science, show me popular/trending articles in Computer Science + Perfect for cold users - Non personalised, discipline too broad

24/11/2015

18

Algorithms [4]

Research recommendations at Mendeley

4.  Citation Network Articles similar to articles I cited

Articles that cite me

Articles from my co-author

+ Good for some kind of users

- Young researchers do not have (enough) publications

24/11/2015

19

Offline experiments

Research recommendations at Mendeley

Offline Evaluation of 100+ algorithms variations on an historical dataset

• Split data into training and testing based on timestamps: train until day X, try to predict what users will add in the next day/week/month

• Computed different metrics to measure different dimensions: • Accuracy (precision, recall, f-score, nDCG, MAP) • Diversity • Recency • Popularity • Consistency • Coverage

24/11/2015

20

Offline results

Research recommendations at Mendeley

Warm Users

Cold Users

24/11/2015

User Based CF Item Based CF Content Based

Citation Network Popularity Trending

Content Based Popularity Trending

21

Business Logic / Analytics

Research recommendations at Mendeley

• Business put some constraints that could have an impact on the recommendation experience

– Don’t show articles outside the user discipline – Show articles only with a minimum readership – Show only recommendations that you can explain (especially for people recommendations, a different matter)

• Analytics – Dashboard on the recommender statistics:

• Number of recommendations served • Number of users with recommendations • …

24/11/2015

22

User Interface

Research recommendations at Mendeley

• Original idea: One list fits all

Create a single list with the best recommendations for the user: use advanced methods to take into account every signal and provide what is best for you!

24/11/2015

23

User Interface [2]

Research recommendations at Mendeley

• However… – Different kinds of users can have different information needs!

– The same user in different contexts can have different information needs!

VS

24/11/2015

24

User Interface [3]

Research recommendations at Mendeley

• Solution: different lists! • Provide multiple lists that satisfy different information needs • More likely for a user to find something he is interested in

24/11/2015

25

Online Survey

Research recommendations at Mendeley

Survey with Mendeley Advisors (pre-launch)

24/11/2015

Based on all the articles

Good

Bad

Popular

Good

Bad

Based on the last article

Good

Bad

Trending

Good

Bad

26

Online Statistics

Research recommendations at Mendeley

Different statistics collected:

• overall and list

• click on title oradd to library • different metrics:

– # users – CR – CTR

24/11/2015

27

What’s next

Research recommendations at Mendeley

• New lists! – Based on your research interests – …

• Improve current lists

• Researchers you may want to follow

24/11/2015

28 Research recommendations at Mendeley 24/11/2015

Thank you!