Effectiveness of Implicit Rating Data on Effectiveness of Implicit Rating Data on Characterizing Users in Complex Characterizing Users in Complex

Information SystemsInformation Systems

9th ECDL 2005

Vienna, Austria

Sep. 20, 2005Seonho Kim, Uma Murthy, Kapil Ahuja, Sandi Vasile,

Edward A. Fox

Digital Library Research Laboratory (DLRL)

Virginia Tech, Blacksburg, VA 26061 USA

ECDL 20052

Acknowledgements (Selected)

• Sponsors: AOL; NSF grants DUE-0121679 DUE-0435059; Virginia Tech; …

• Faculty/Staff: Lillian Cassel, Manuel Perez, …

• VT (Former) Students: Aaron Krowne, Ming Luo, Hussein Suleman, …

ECDL 20053

OverviewOverview

• Introduction

– Prior Work

– Web Trends and DL

• Data for User Studies

– Problem of Explicit Rating Data

– Implicit Rating Data in DLs

– Attributes of User Activity

– User Tracking Interface and User Model DB

• Questions and Experiments

– Questions to Solve

– Experiments, Hypothesis Tests, Data, Settings

– Results of Hypothesis Testing

– Data Types and Characterizing Users

• Future Work

• Conclusions

• References

ECDL 20054

Prior WorkPrior Work

• User study, User feedback– Pazzani et al. [1]: learned user profile from user feedback on the

interestingness of Web sites.

• Log analysis & standardization efforts– Jones et al [2]: a transaction log analysis of DL

– Gonçalves et al. [3]: defined an XML log standard for DLs.

• Implicit rating data– Nichols [4]: suggested the use of implicit data as a check on

explicit ratings.

– GroupLens[5]: employed “time consuming” factor for personalization.

ECDL 20055

Web Trends & DLWeb Trends & DL

• WWW Trends– One way Two way services

– e.g., Blogs, wikis, online journals, forums, etc.

– Passive anonymous observer visible individuals with personalities

– Same situation in Digital Libraries

– Research emphasis on “User Study”• Collaborative Filtering

• Personalization

• User Modeling

• Recommender system, etc.

ECDL 20056

Data for User StudiesData for User Studies

• Explicit Ratings– User interview

– User preference survey: demographic info, research area, majors, learning topics, publications

– User rating for items

• Implicit Ratings– “User activities”, e.g., browsing, clicking, reading,

opening, skipping, etc.

– Time

ECDL 20057

Problem of Explicit Rating Data in Problem of Explicit Rating Data in Digital LibrariesDigital Libraries

• Expensive to obtain• Patrons feel bothered• Limited questions• Terminology problems in describing research

interests and learning topic– Too broad area, and too narrow personal interests

– Term ambiguity

– New terminology in new areas

– Multiple terms for same area, multiple meanings of a term

Hard to figure out users’ interests and topics

ECDL 20058

Implicit Rating Data in Complex Implicit Rating Data in Complex Information SystemsInformation Systems

• Easy to obtain• Patrons don’t feel bothered, can concentrate on

their tasks• No terminology issues• Potential knowledge is included in data• More effective when hybrid, with explicit rating

data (Nichols [4], GroupLens[5])

ECDL 20059

User Tracking Interface User Tracking Interface and User Model DBand User Model DB

Retrieval System

User Model DB

-Load-Update-Save-Createexpand

ignore

type a query

read

browse

openclick

tracking info

Digital Library

ECDL 200510

Attributes of User ActivityAttributes of User Activity

• DGG (Domain Generalization Graph) for user activity attributes in DL

ANY

ANY ANY

User Activity

Direction Intention

Frequency Type

Entering a query implicitSending a query implicit

Reading implicitSkipping implicit

Selecting implicitExpanding a node implicit

Scrolling implicitDragging implicit

Entering user info. explicit

Entering a query perceivingSending a query rating

Reading perceivingSkipping rating

Selecting ratingExpanding a node rating

Scrolling perceivingDragging perceiving

Entering user info. rating

User Interest ANYDocument Topic ANY

High ANYLow ANY

Rating ANYPerceiving ANY

implicit ANYexplicit ANY

ECDL 200511

OverviewOverview

• Introduction

– Prior Work

– Web Trends and DL

• Data for User Studies

– Problem of Explicit Rating Data

– Implicit Rating Data in DLs

– Attributes of User Activity

– User Tracking Interface and User Model DB

• Questions and Experiments

– Questions to Solve

– Experiments, Hypothesis Tests, Data, Settings

– Results of Hypothesis Testing

– Data Types and Characterizing Users

• Future Work

• Conclusions

• References

ECDL 200512

Proposed User Grouping ModelProposed User Grouping Model

• User grouping is the most critical procedure for a recommender system.

• Suitable for dynamic and complex information systems like DLs

• Overcomes data sparseness

• Uses implicit rating data rather than explicit rating data

• User oriented recommender algorithm

• User interest-based community finding

• User modeling– User model (UM) contains complete statistics for recommender

system.

– Enhanced interoperability

ECDL 200513

Collecting User Interests for User Collecting User Interests for User GroupingGrouping

• Users with similar interests are grouped• Employs a Document Clustering Algorithm,

LINGO [10], to collect document topics• Users’ interests are collected implicitly during

searching and browsing.• A User Model (UM) contains her interests and

document topics.• Interests of a user are subset of document topics

proposed to her by Document Clustering.

ECDL 200514

Interest-based Recommender Interest-based Recommender SystemSystem

ECDL 200515

System Analysis with 5S ModelSystem Analysis with 5S Model

Interest-based Recommender System for DL

Society

Space

Structure

Stream Scenario

User Interface

User Model

PresentationPush service

FilteringRanking

HighlightingPersonalized

pages

Recommendation

Group Selection

Individual Selection

Interest GroupResearcher

Learner

Teacher

Class Group

Probability Space

Vector space

Collaboration space

Community

displays

Text AudioVideo

represented by

UM schema

User description

User interestsDocument topics

User groups

Statistics

participates

generates

refers

composed of

refers

Users

Users

ECDL 200516

User Model (UM)User Model (UM)

User ID

User Description Groups Statistics

Name Document Topic Score

User Interest Score

Group ID Score

E-mail

Address

Publications

User Interests

(implicit data-generated by user interface and recommender)

(implicit data-generated by recommender)

(explicit data-obtained from questionnaire)

ECDL 200517

Experiment - TasksExperiment - Tasks

• Subjects are asked to – answer a questionnaire to collect democratic

information

– list research interests to help us collect explicit rating data which is used for evaluation in the experiment

– search some documents in her research interests and browse the result documents to help us collect implicit rating data

ECDL 200518

Experiment - ParticipantsExperiment - Participants

• 22 Ph.D and MS students majoring in Computer Science

• CITIDEL [8] is used as a DL in “Computing” field• Data from 4 students were excluded as their

research domains are not included in CITIDEL

ECDL 200519

Experiment - InterfacesExperiment - Interfaces

• Specially designed user interfaces are required to capture user’s interactions

• JavaScripts

• Java Application

ECDL 200520

Results - Collected DataResults - Collected Data

• Example<Semi Structured Data<Cross Language Information Retrieval CLIR<Translation Model<Structured English Query<TREC Experiments at Maryland<Structured Document<Evaluation<Attribute Grammars<Learning<Web<Query Processing<Query Optimisers<QA<Disambiguation<Sources<SEQUEL<Fuzzy<Indexing<Inference Problem<Schematically Heterogeneity<Sub Optimization Query Execution Plan<Generation<(Other)(<Cross Language Information Retrieval CLIR)(<Structured English Query)(<TREC Experiments at Maryland)(<Evaluation)(<Query Processing)(<Query Optimisers)(<Disambiguation)

<Cross Language Information Retrieval CLIR<Machine Translation<English Japanese<Based Machine<TREC Experiments at Maryland<Approach to Machine<Natural Language<Future of Machine Translation<Machine Adaptable Dynamic Binary<CLIR Track<Systems<New<Tables Provide<Design<Statistical Machine<Query Translation<Evaluates<Chinese<USA October Proceedings<Interlingual<Technology<Syntax Directed Transduction<Interpretation<Knowledge<Linguistic<Divergences<(Other)(<Cross Language Information Retrieval CLIR)(<Machine Translation)(<English Japanese)(<TREC Experiments at Maryland)(<CLIR Track)(<Query Translation)

• Parenthesized topics mean they are rated positively

ECDL 200521

Questions to SolveQuestions to Solve

• Is implicit rating data really effective for user study? for characterizing users? especially in complex information systems like DLs?

• If we are to prove it statistically, what are the right hypotheses and what are the right settings for hypotheses testing?

ECDL 200522

Two Experiments in this StudyTwo Experiments in this Study

• Two hypothesis tests to prove the effectiveness of Implicit Rating Data on characterizing users in DL

• An ANOVA test for comparing implicit rating data types on distinguishing users in DL

ECDL 200523

Hypothesis TestsHypothesis Tests

• Hypotheses– H1: For any serious user with their own research

interests and topics, show repeated (consistent) output for the document collections referred to by the user.

– H2: For serious users who share common research interests and topics, show overlapped output for the document collections referred to by them.

– H3: For serious users who don’t share any research interests and topics, show different output for the document collections referred to by them.

ECDL 200524

Data Used for Hypothesis TestsData Used for Hypothesis Tests

• Data for Hypothesis Tests: Users’ learning topics and research interests are obtained “implicitly” by tracking users’ activities with user tracking interface while users need not be aware.

• Data collected by a user tracking system for 18 students at both Ph.D. and M.S. levels, in CS major, while using CITIDEL [6]

ECDL 200525

Setting for Hypothesis Test 1Setting for Hypothesis Test 1

• Let H0 be a null hypothesis of H1, thus H0 is: Means (μ) of the frequency of document topics ‘proposed’ by the Document Clustering Algorithm are NOT consistent for a user.

• Simplified A testing whether the population mean, μ, is statistically significantly greater than hypothesis mean, μ0.

ECDL 200526

Setting for Hypothesis Test 2Setting for Hypothesis Test 2• Let H0 be a null hypothesis of H2, thus H0 is: A user’s

average ratio of overlapped topics with other persons in her groups over her total topics which have been referred, in-group overlapping ratio, μ1, is the same as the average ratio of overlapped topics with other persons out of her groups over her total topics which have been referred, out-group overlapping ratio, μ2

DL system

a

b

c

d

e

f

a,b,c,d,e,f : users

User Groups

ECDL 200527

Setting for Hypothesis Test 2Setting for Hypothesis Test 2

G

kKK

G

k

n

i

N

Kjjji

nNn

OK

1

1 1 ,1,

2

)(

G

kKK

G

k

n

i

n

ijjji

nn

OK K

1

1 1 ,1,

1

)1(

• In-group overlapping ratio

• Out-group overlapping ratio

• Oi,j: user i’s topic ratio overlapped with user j’s topics over i’s total topics

G : total number of user group

nK : total number of users in group K

N : total number of users in the system

ECDL 200528

Setting for Hypothesis Test 2Setting for Hypothesis Test 2

• Simplified A testing whether μ1 is statistically significantly greater than μ2

• Hypothesis 3 can be proven and estimated together, by hypothesis test 2

ECDL 200529

Results of Test 1Results of Test 1

• Conditions: 95% confidence (test size α = 0.05), sample size ‘n’ < 25, standard deviation ‘σ’ unknown, i.i.d. random samples, normal distribution, estimated z-score T-test

• Test statistics: sample mean ‘ỹ’ = 1.1429, sample standard deviation ‘s’ = 0.2277, are observed from the experiment

• Rejection Rule is to reject H0 if the ỹ > μ0+zα/2 σ/√n

• From the experiment, ỹ = 1.1429 > μ0+zα/2 σ/√n = 1.0934

• Therefore decision is to Reject H0 and accept H1

• 95% Confidence Interval for μ is 1.0297 ≤ μ ≤1.2561

• P-value (confidence of H0) = 0.0039

ECDL 200530

Results of Test 2Results of Test 2• Conditions: 95% confidence (test size α = 0.05), two i.i.d.

random sample from a normal distribution, for two sample sizes n1 and n2, n1=n2 < 25, standard deviations of each sample σ1 and σ2 are unknown two-sample Welch T-test

• From the experiment, sample mean of μ1, ỹ1 = 0.103, sample mean of μ2, ỹ2 = 0.0215, Satterthwaite’s degree of freedom approximation dfs =16.2 and Welch score w0 = 4.64 > t16.2, 0.05 = 1.745

• Therefore decision is to Reject H0 and accept H2

• 95% Confidence Interval for μ1, μ2 and μ1 - μ2 are 0.0659 ≤ μ1 ≤ 0.1402, 0.0183 ≤ μ2 ≤ 0.0247 and 0.0468 ≤ μ1 - μ2≤ 0.1163, respectively

• P-value (confidence of H0) = 0.0003

ECDL 200531

Results of Hypothesis TestingResults of Hypothesis Testing

• Statistically proved that implicit rating data is effective in characterizing users in complex information systems.

ECDL 200532

Data Types and Characterizing Data Types and Characterizing UsersUsers

• Previous similar studies were based on explicit user answers to surveys on their preferences, research and learning topics basic flaw caused by the variety of academic terms.

• Purpose: Compare the effectiveness of different data types in characterizing users by using only automatically obtained objective data without using subjective users’ answers.

ECDL 200533

Data Types and Characterizing Data Types and Characterizing UsersUsers

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

selected topics proposed topics selected terms proposed terms

Implicit Rating Data Used

Overl

ap

pin

g R

ati

o

0

1

2

3

4

5

6

Avera

ge o

f B

IU /

BO

U

Between In-Group Users (BIU)Between Out-Group Users (BOU)Distinguishability : Average of BIU/BOU

– Topics: noun phrases logged in User Models generated by a document clustering system ‘LINGO’ from documents to which users referred

– Terms: single words found on user queries and topics

– ANOVA statistics F(3,64) = 4.86, p-value = 0.0042, LSD = 1.7531

ECDL 200534

Data Types and Characterizing Data Types and Characterizing UsersUsers

• The higher in-group overlapping ratio / out-group overlapping ratio is more effective in characterizing users.

• “Proposed topics” which have appeared during the use of a digital library were most effective, however the differences between data types were not significant except the “proposed terms”.

ECDL 200535

Future WorkFuture Work

• Large scale Experiment on NDLTD [7]

• User Model DB visualization

• Utilize implicit rating data for personalization and recommendation

ECDL 200536

ConclusionsConclusions

• Built a User Tracking system to collect Implicit rating data in DL.

• Statistically proved that implicit ratings is effective information in characterizing users in complex information systems like DLs.

• Compared the effectiveness of data types in characterizing users without depending on users’ subjective answers.

ECDL 200537

ReferencesReferences

• [1] Michael Pazzani, Daniel Billsus: Learning and Revising User Profiles: The Identification of Interesting Web Sites, Machine Learning 27, 1997, 313-331

• [2] Steve Jones, Sally Jo Cunningham, Rodger McNab: An Analysis of Usage of a Digital Library, in Proceedings of the 2nd ECDL, 1998, 261-277

• [3] Marcos André Gonçalves, Ming Luo, Rao Shen, Mir Farooq and Edward A. Fox: An XML Log Standard and Tools for Digital Library Logging Analysis. In Proceedings of Sixth European Conference on Research and Advanced Technology for Digital Libraries, Rome, Italy, September, 2002, 16-18

• [4] David M. Nichols: Implicit Rating and Filtering. In Proceedings of 5th DELOS Workshop on Filtering and Collaborative Filtering, Budapest Hungary, November 1997, 31-36

• [5] Joseph A. Konstan, Bradley N. Miller, David Maltz, Jonathan L. Herlocker, Lee R. Gordon and John Riedl, GroupLens: Applying Collaborative Filtering to Usenet News. In Communications of the ACM, Vol. 40, No. 3, 1997, 77-87

• [6] CITIDEL: Available at http://www.citidel.org/, 2005

• [7] NDLTD: Available at http://www.ndltd.org/, 2005

ECDL 200538

ReviewReview

• Introduction

– Prior Work

– Web Trends and DL

• Data for User Studies

– Problem of Explicit Rating Data

– Implicit Rating Data in DLs

– Attributes of User Activity

– User Tracking Interface and User Model DB

• Questions and Experiments

– Questions to Solve

– Experiments, Hypothesis Tests, Data, Settings

– Results of Hypothesis Testing

– Data Types and Characterizing Users

• Future Work

• Conclusions

• References