ERCIM research project at CWI Focused on novel aspects of recommender systems evalua3on from an informa3on retrieval and a user centric perspec3ve. Dura3on and personnel January 2013 – March 2014 24 person months 2 postdocs Centrum Wiskunde & Informa:ca www.cwi.nl Alan Said, Alejandro Bellogín, Arjen De Vries, Benjamin Kille {alan.said, alejandro.bellogin, arjen.de.vries}@cwi.nl, [email protected] Informa3on Retrieval and UserCentric Recommender System Evalua3on UMAP 2013 – Rome, Italy Acknowledgement This work is carried out during the tenure of an ERCIM “Alain Bensoussan” Fellowship Programme. The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/20072013) under grant agreement no.246016. Usercentric Evalua:on A personalized offline evalua:on protocol mimicking reallife deployed recommender systems. • Each algorithm is trained once per user based on a personalized training and test split. • Candidate test items are selected based on a personal ra:ng value threshold (e.g. mean). Diversityoriented evalua:on methods focusing on: • nonaccuracybased evalua3on metrics • item featurecentric evalua3on • novelty IRbased Evalua:on A coherencebased func:on that is able to predict the user performance • Correlated with the “magic barrier” • Improves the total performance of the system when used to group users into difficult and easy ones Explore correla:ons between metrics typically used in offline experiments (precision) and in real applica3ons (e.g., CTR). Perform an analysis of how evalua3on in recommenda3on should be performed to obtain interpretable and comparable results. Related Events Workshop on Reproducibility and Replica1on in Recommender Systems Evalua1on. In conjunc3on with RecSys 2013 Workshop on Benchmarking Adap1ve Retrieval and Recommender Systems. In conjunc3on with SIGIR 2013 ACM TIST Special Issue on Recommender System Benchmarking Further Reading Ar1st popularity: do web and social music services agree? A. Bellogín et al. In ICWSM 2013 UserCentric Evalua1on of a KFurthest Neighbor Collabora1ve Filtering Recommender Algorithm A. Said et al. in CSCW 2013 Poster Abstract Informa1on Retrieval and UserCentric Recommender System Evalua1on A. Said, A. Bellogín et al. in UMAP 2013

Information Retrieval and User-centric Recommender System Evaluation

Download PDF Report

Upload
alan-said
View
336
Download
1

Embed Size (px)

DESCRIPTION

Poster describing the ERCIM-funded project on IR- and user-centric recommender system evaluation currently being undertaken in the Information Access group at CWI. Presented at UMAP 2013.

Citation preview

Page 1: Information Retrieval and User-centric Recommender System Evaluation

ERCIM research project at CWI

Focused on novel aspects of recommender systems evalua3on from an informa3on retrieval and a user-‐centric perspec3ve.

Dura3on and personnel §  January 2013 – March 2014 §  24 person months §  2 postdocs

Centrum Wiskunde & Informa:ca www.cwi.nl

Alan Said, Alejandro Bellogín, Arjen De Vries, Benjamin Kille {alan.said, alejandro.bellogin, arjen.de.vries}@cwi.nl, kille@dai-‐lab.de

Informa3on Retrieval and User-‐Centric Recommender System Evalua3on

UMAP 2013 – Rome, Italy

Acknowledgement

This work is carried out during the tenure of an ERCIM “Alain Bensoussan” Fellowship Programme. The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-‐2013) under grant agreement no.246016.

User-‐centric Evalua:on

A personalized offline evalua:on protocol mimicking real-‐life deployed recommender systems. •  Each algorithm is trained once per user based on a

personalized training and test split. •  Candidate test items are selected based on a personal ra:ng

value threshold (e.g. mean). Diversity-‐oriented evalua:on methods focusing on: •  non-‐accuracy-‐based evalua3on metrics •  item feature-‐centric evalua3on •  novelty

IR-‐based Evalua:on

A coherence-‐based func:on that is able to predict the user performance •  Correlated with the “magic barrier” •  Improves the total performance of the system when used to group

users into difficult and easy ones Explore correla:ons between metrics typically used in offline experiments (precision) and in real applica3ons (e.g., CTR). Perform an analysis of how evalua3on in recommenda3on should be performed to obtain interpretable and comparable results.

Related Events

Workshop on Reproducibility and Replica1on in Recommender Systems Evalua1on. In conjunc3on with RecSys 2013

Workshop on Benchmarking Adap1ve Retrieval and Recommender Systems. In conjunc3on with SIGIR 2013

ACM TIST Special Issue on Recommender System Benchmarking

Further Reading

Ar1st popularity: do web and social music services agree? A. Bellogín et al. In ICWSM 2013

User-‐Centric Evalua1on of a K-‐Furthest Neighbor Collabora1ve Filtering Recommender Algorithm

A. Said et al. in CSCW 2013 Poster Abstract Informa1on Retrieval and User-‐Centric Recommender System Evalua1on A. Said, A. Bellogín et al. in UMAP 2013