Exploration & Exploitation in Exploration & Exploitation in Adaptive Filtering Based on Adaptive Filtering Based on

Bayesian Active LearningBayesian Active Learning

Yi Zhang, Jamie CallanYi Zhang, Jamie Callan

Carnegie Mellon Univ.Carnegie Mellon Univ.Wei XuWei Xu

NEC Lab AmericaNEC Lab America

A Typical Adaptive Filtering SystemA Typical Adaptive Filtering System

Filtering System…

Accumulateddocs

document stream Delivered docs

FeedbackUser ProfileLearning

FirstRequest

initialization

(Binary Classifier)(Utility Function)

Commonly Used EvaluationCommonly Used EvaluationRelevant Non-Relevant

Delivered AR AN

Not Delivered BR BN

NBRBNARAUtility NRNR

If we assume user satisfaction is mostly influenced by what she/he has seen, then a simplified version for utility is:

NARAUtility NR

For example: Utility=2R+-N+ (Used in TREC9, TREC10, TREC11 Adaptive Filtering Track trec.nist.gov)

Common Approach in Common Approach in Adaptive FilteringAdaptive Filtering

• Set the dissemination threshold where the immediate utility gain of delivering a document is zero:

For example: in order to optimize Utility=2R+-N+, system delivers iff P(Rel)>=0.33

Because Uimmediate=2P(Rel)-P(Nrel)>=0

Problem with Current Problem with Current Adaptive Filtering ResearchAdaptive Filtering Research

• Why deliver a document to the user? 1. Satisfies the information need immediate 2. Get user feedback so the system can

improve its model of the user’s information need, thus satisfy the information need better in the future

• Current research in adaptive filtering: underestimates the utility gain of delivering a document by ignoring the second effect

– Related work: active learning, Bayesian experimental design

Solution: Explicitly Model the Solution: Explicitly Model the Future Utility of Delivering a Future Utility of Delivering a

DocumentDocument

)()()( dUNdUdU futurefutureimmediate

Nfuture : number of discounted documents in the future

•Exploitation: estimation of the immediate utility of delivering a new document based on model learned

• Exploration: estimation of the future utility of delivering a new document by considering the improvement of the model learned if we can get user feedback obout the document.

Exploitation: Estimate UExploitation: Estimate Uimmediate immediate

Using Bayesian InferenceUsing Bayesian Inference

dDPdUDdU ttimmediatettimmediate )|()|()|( 11

Let P(|Dt-1) be the posterior distribution of model parameters given training data set Dt-1.

Using Bayesian Inference, we have:

Ay is the credit/penalty defined by the utility function that model user satisfaction

Y=R if relevant, y=N if none relevant

y

tytimmediate dyPAdU ),|()|(

Exploration: Utility Divergence to Exploration: Utility Divergence to Measure Loss (1)Measure Loss (1)

• If we use while the true model is , we incur some loss (utility divergence (UD)):

),(),()||()||(

UUUDLoss

Document Space

deliver:^

deliver :

Exploration: Utility Divergence Exploration: Utility Divergence to Measure Loss (2)to Measure Loss (2)

• We do not know . However, based on our beliefs about its distribution, we can estimate the expected loss of using :

• Thus we can measure the quality of Training data D as the expected loss if we use the estimator

),(),( )|(

UDEDLoss DP

),()(^

DDLossDLoss

D

^

)|( DP

The Whole ProcessThe Whole Process

• Step 1:

• Step 2:x

y

y

future

yxDLossDxyPDLoss

DxU

)),((),|()(

)|(

dDPdU

DdU

ttimmediate

ttimmediate

)|()|(

)|(

1

1

Adaptive Filtering: Logistic Regression to Adaptive Filtering: Logistic Regression to Find Dissemination ThresholdFind Dissemination Threshold

)exp(1

1),|(

10 xwwxrelevantyP

X: score* indicates how well each document matches the profile

Metropolis-Hasting algorithm to sample I for integration.

*scoring function is learned adaptively using Rocchio algorithm

Experimental Data Sets and Experimental Data Sets and Evaluation MeasuresEvaluation Measures

TREC9 OHSUMED

TREC10 Reuter’s

relevant 51 9795

total +300,000 +800,000

Relevant% 0.016% 1.2%

Initialization 2 relevant documents + topic description

Rel

redRel_DeliveRecall

Delivered

redRel_DelivePrecision

5.01

)5.0,Rel*2

T9Umax(

11

29

SUT

NRUT

Trec-10 Filtering Data: Trec-10 Filtering Data: Reuters DatasetReuters DatasetBayesian Active

Bayesian Immediate

Norm.Exp.

MLN-E

T9U 3534 3149 2926 3015

T11SU 0.448 0.445 0.436 0.439

Precision 0.463 0.481 0.464 0.496

Recall 0.251 0.234 0.227 0.212

Docs/Profile 4527 3895 2792 3380

•Active learning is very effective on TREC10 dataset

Trec-9 Filtering Data: Trec-9 Filtering Data: OHSUMED DatasetOHSUMED Dataset

Bayesian Active

Bayesian Immediate

Norm.Exp.

MLN-E

T9U 11.32 11.54 6.59 11.79

T11SU 0.353 0.360 0.329 0.362

Precision 0.300 0.325 0.256 0.339

Recall 0.231 0.203 0.264 0.177

Docs/Profile 31 25 46 20

• On average, only 51 out of 300000 are relevant documents.

• Active learning didn’t improve utility on TREC9 dataset. But it didn’t hurt either. (The algorithm is robust)

Related WorkRelated Work• Related Work

– active learning• Uncertainty about the label of document

– Request the label of the most uncertain document– Minimize the uncertainty about future labels

• Uncertainty about the model parameters (KL divergence, variance)

– Bayesian Experimental Design• Improvement of the utility of the model

– Information Retrieval• Mutual Information between document and label

Contribution and Future Contribution and Future WorkWork

• Our Contribution– Derivation of Utility Divergence to measure

model quality– Combining immediate utility and future utility

gain in adaptive filtering task– Empirically robust algorithm

• Future Work– High dimensional space

• Computational issues: variational algorithms, Gaussian approximations, Gibbs sampling, …

• Number of training data needed– Other active learning applications

• Online marketing• Interactive retrieval• …

The EndThanks