Statistical Models for Web Search Click Log Analysis

Fan Guo Chao LiuCarnegie Mellon University Microsoft Research-Redmond

Search Results for “CIKM”

04/20/23 2CIKM'09 Tutorial, Hong Kong, China

# of clicks received

Adapt ranking to user clicks?



Tools needed for non-trivial cases



One of the most extensive (yet indirect) surveys of user experience.

For researchers: Help understand human interaction with IR

results Design and calibrate novel models and

hypotheses For practitioners:

Measure, monitor and improve search engine performance.

Attract more page views and clicks, boost profit 04/20/23 CIKM'09 Tutorial, Hong Kong, China 5

Introduce problems and applications in web search click modeling.

Present latest development of click models in web search.

Provide examples and discuss trade-offs for model design, implementation and evaluation.

04/20/23 CIKM'09 Tutorial, Hong Kong, China 6


Ph.D. Student (exp. 2011), Computer Science Department, Carnegie Mellon University

Advisor: Christos Faloutsos Dissertation topic: graph

mining for large bioinformatics image databases

2008, M.S., CMU 2005, B.E., Tsinghua

University, Beijing, China

Researcher, Internet Services Research Center (ISRC), MSR-Redmond.

Research focus: large-scale search/browsing log analysis for effective Web information access.

2007, Ph.D., UIUC2005, M.S., UIUC Advisor: Jiawei Han Dissertation on statistical

debugging and automated failure analysis

2003, B.S., Peking University, China


IntroductionDesigning click modelsBayesian click modelsSelected topics on click modelsConclusion


Introduction Web search click logs Interpret clicks as relevance feedback Building statistical models for clicks Applications of click models

Designing click models Bayesian click models Selected topics on click models Conclusion


Click-throughBrowser actionDwelling timeExplicit judgmentOther page elements


Auto-generated data keeping important information about search activity.

1204/20/23 CIKM'09 Tutorial, Hong Kong, China

Position URL Click1 cikm2008.org 1

2 www.cikm.org 03 www.cikm.org/2002 04 www.fc.ul.pt/cikm2007 05 www.comp.polyu.edu.hk/conference/cikm2009 16 cikmconference.org 07 Ir.iit.edu/cikm2004 08 www.informatik.uni-trier.de/~ley/db/conf/cikm/index.html 09 www.tzi.de/CIKM2005 0

10 www.cikm.com 0

Query cikmSession

IDf851c5af178384d12f3d

A real world example


How large is the click log? search logs: 10+ TB/day

In existing publications:▪ [Craswell+08]: 108k sessions▪ [Dupret+08] : 4.5M sessions (21 subsets * 216k

sessions)▪ [Guo +09a] : 8.8M sessions from 110k unique queries▪ [Guo+09b]: 8.8M sessions from 110k unique queries▪ [Chapelle+09]: 58M sessions from 682k unique

queries▪ [Liu+09a]: 0.26PB data from 103M unique queries


How large is one ?





Clicks are good… Are these two

clicks equally “good”?

Non-clicks may have excuses: Not relevant Not examined



Higher positions receive more user attention (eye fixation) and clicks than lower positions.

This is true even in the extreme setting where the order of positions is reversed.

“Clicks are informative but biased”.


[Joachims+07]

Normal Position

Perc

enta

ge

Reversed Impression

Perc

enta

ge

“Clicked > Skipped Above” [Joachims02]


Preference pairs:#5>#2, #5>#3, #5>#4.

Use Rank SVM to optimize the retrieval function.

Limitation: Confidence of

judgments Little implication to

user modeling

1

2345

67

8




Given a set of web search click logs: Predict clicks: output the

probability of click vectors given a new order of URLs.


210 possibilities!

Given a set of web search click logs: Estimate relevance: measures how

good a URL is with regard to the information need of the query/user.

04/20/23 23

Relevance score = 0.5

CIKM'09 Tutorial, Hong Kong, China

The probability of a click if the document appears at the top position. Relevance score = 0.5 indicates that on

average, the document will be clicked once per 2 sessions.

Bayesian click models characterize relevance using a probability distribution

2404/20/23Relevance score

Densi

ty f

unct

ion


Effective: aware of the position-bias and address it properly

Scalable: linear complexity for both time and space, easy to parallel

Incremental: flexible for model update based on new data





Optimizing the retrieval function Ranking alternation based on clicks

[Liu+09b]


0.90

0.10

0.08

0.05

0.20

0.72

Optimizing the retrieval function Ranking alternation based on clicks As a feature to a learning-to-rank

system (e.g., RankNet [Burges+05] )


Online advertising User model for sponsored search

auctions


Online advertising User model for sponsored search

auctions Click through rate (CTR) prediction

[Zhu+10]


Search engine evaluation Pskip [Wang+09]:

click-through-rate above last clicks; dwelling time features could also be incorporated.


Search engine evaluation Pskip [Wang+09]: click-through-rate above

last clicks;

Search relevance score [Guo+09c]: average relevance score weighted by chance of examination


User behavior analysis A preliminary work showing different

user behavior patterns for navigational and informational queries [Guo+09c]


Introduction Designing click models

Basic user hypotheses Modeling the first click Extending to multiple clicks Summary of model design

Bayesian click models Selected topics on click models Conclusion


A document must be examined before a click.

The (conditional) probability of click upon examination depends on document relevance.


The click probability could be decomposed: Global component: the examination

probability which reflects the position-bias Local component: depends on the (query,

URL) pair only

The building block for every existing model!


The first document is always examined.

First-order Markov property: Examination at position (i+1) depends on

examination and click at position i only

Examination follows a strict linear order:


Position i Position (i+1)

The first document is always examined.

First-order Markov property: Examination at position (i+1) depends on

examination and click at position i only

Examination follows a strict linear order:


Position i Position (i+1)

Limitation: examination/click rate monotonically decreases with rank, which is not always true.

Some models do not follow this hypothesis (e.g., UBM)


Web search data in [Guo+09a]

Ads click data in [Zhu+10]





Put together two hypotheses:

Formal model specification: P(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rui

P(E1=1) =1, P(Ei+1=1|Ei=0) = 0

P(Ei+1=1|Ei=1, Ci=0)=104/20/23 CIKM'09 Tutorial, Hong Kong, China 41

Cascade Model = [Craswell+08]

examination hypothesiscascade hypothesis

modeling a single click

The user behavior chart:


Examine the URL

Click?

Yes

No See Next URL?

Done

Yes

Index for URL at position i

First click in Click Chain Model [Guo+09b] as well asDynamic Bayesian Network model [Chapelle+09]


The chance that user may

immediately abandon

examination w/o a click.

Examine the URL

Click?

Yes

No See Next URL?

Done

Yes

Done

No

First click in User Browsing Model [Dupret+08]


Examine the URL

Click?

Yes

No

Done

Yes

Noi ←i+1

See Next URL?

Position-dependent parameters





Generalize the cascade model to 1+ clicks: P(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rui

P(E1=1) =1, P(Ei+1=1|Ei=0) = 0

P(Ei+1=1|Ei=1, Ci=0)=1

P(Ei+1=1|Ei=1, Ci=1)= λi


λ:global parameters characterizing user browsing

behavior

Generalize the cascade model to 1+ clicks:


DCM Algorithms: Input: for each query session, the query

term, with (URL, clicked) tuple for all top-10 positions.

Output: relevance for each (query, URL) pair;global parameters for user behavior

Method: approximate* maximum-likelihood estimation.

04/20/23 CIKM'09 Tutorial, Hong Kong, China 48*Footnote: the algorithm maximizes a lower bound of log-likelihood function.


Position URL Click1 cikm2008.org 12 www.cikm.org 03 www.cikm.org/2002 04 www.fc.ul.pt/cikm2007 05 www.comp.polyu.edu.hk/... 16 cikmconference.org 07 Ir.iit.edu/cikm2004 08 www.informatik.uni-trier.de... 09 www.tzi.de/CIKM2005 0

10 www.cikm.com 0

Last clicked position

Query cikm

Session ID f851c5af178384d12f3d


Position URL Click1 cikm2008.org 02 www.cikm.org 13 www.cikm.org/2002 04 www.fc.ul.pt/cikm2007 05 cikmconference.org 06 www.comp.polyu.edu.hk/... 17 Ir.iit.edu/cikm2004 08 www.informatik.uni-trier.de... 09 www.tzi.de/CIKM2005 1

10 www.cikm.com 0

Last clicked position

Query cikm

Session ID ab8dee4c4dd21e6aaf03

The estimation formula for relevance:

empirical CTR measured before last clicked position

The estimation formula for global (user behavior) parameters:

empirical probability of “clicked-but-not-last”


Keep 3 counts for each (query, URL) pair

Then


Details

The examine-next probability depends on the relevance of the URL clicked:


Not what I want, go to examine the

next

Aha, this is the right one, and I’m done!

The examine-next probability depends on the relevance of the URL clicked: P(Ei+1=1|Ei=1, Ci=1)= α2(1-rui

) + α3rui

P(Ei+1=1|Ei=1, Ci=0)= α1

where 0 < α1 ≤ 1, 0 ≤ α3< α2≤ 1


The full picture:


There is a subtle difference between the relevance of the URL snippet and the landing page.


hmmm…, this looks

pretty nice

errr…, it’s way out of

date

Conclusion: attractive, but not satisfactory.

The examine-next probability depends on the “satisfaction score”: P(Ei+1=1|Ei=1, Ci=1)= γ(1-sui

) + 0sui

P(Ei+1=1|Ei=1, Ci=0)= γ

where 0 < γ ≤1The click probability is associated

with “attractiveness score”: P(Ci=1|Ei=1)= aui


The full picture:


The examine-next probability depends on both the preceding clicked position r, and the distance to this position d.


r = 0d = 1

Position URL Click1 cikm2008.org 02 www.cikm.org 13 www.cikm.org/2002 04 www.fc.ul.pt/cikm2007 05 cikmconference.org 06 www.comp.polyu.edu.hk/... 1… … …



r = 0d = 2




r = 2d = 1




r = 2d = 2




r = 2d = 3


The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. Users would lose patience when they

browse through without issuing a click. The probability monotonically drops as d

increases and r remains the same.


The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. P(Ei=1|C1:i-1)= βri,di

55 parameters are needed for top-10 positions (0≤r<r+d≤10).

Cascade hypothesis is not assumed.04/20/23 CIKM'09 Tutorial, Hong Kong, China 65

where ri = max{j| j <i , Cj=1}, di = i - ri

The full picture:






Probability of examine the first URL


Model P(E1)

Cascade 1DCM 1CCM 1*

DBN 1*

UBM β0,1

* Footnote: it is flexible to add another parameter to specify this probability.

Probability of click upon examination


Model P(Ci=1|Ei=1)

Cascade rdi

DCM rdi

CCM rdi

*

DBN adi

UBM rdi*Footnote: the mean of the relevance distribution, detailed in the next part

Probability of examine-next w/o a click


Model P(Ei+1=1|Ei=1,Ci=0)

Cascade 1DCM 1CCM α1

DBN γUBM βri+1,di+1

*

*Footnote: the probability does not depend on Ei

Probability of examine-next after a click



Cascade --DCM αi

CCM α2(1-rdi) + α3rdi

DBN γ(1-sdi)

UBM βi,1

Probability of examine-next after a click



Cascade --DCM αi

CCM α2(1-rdi) + α3rdi

DBN γ(1-sdi)

UBM βi,1

Size of parameter sets


Model # of global params

Cascade 0DCM 9CCM 3DBN 1UBM 55

Inference and estimation algorithms


Model

Single-Pass

Details

DCM Maximizing a lower bound of LL, fastest

CCMNo iteration needed,

thanks to the Bayesian framework

DBN EM-based, iterative algorithms

UBM EM-based, usually takes ~30 iterations to converge

Inference and estimation algorithms


Model

Single-Pass

Details

DCM Maximizing a lower bound of LL, fastest

CCMNo iteration needed,

thanks to the Bayesian framework

DBN EM-based, iterative algorithms

UBM EM-based, usually takes ~30 iterations to converge

Introduction Designing click models Bayesian click models

Bayesian framework and the rationale

Bayesian Browsing Model: a case study

Click Chain Model in a nutshell Selected topics on click models Conclusion


p(H)=0.8

Frequentist

Bayesian

0 1

Prior Posterior

10


p(H) p(H)

“probability” of p(H)

Prior Posterior


Density Function(not normalized)

x x2 x3 x3(1-x) x4(1-x)

Prior Posterior



x1(1-x)0 x2(1-x)0 x3(1-x)0

x3(1-x)1 x4(1-x)1

The graphical model for coin-toss


X

C1

C2

C3

C4

C5

The graphical model for coin-toss


X

C1

C2

C3

C4

C5


Prior


x1

(1-x)0

(1-0.6x)0

(1+0.3x)1

(1-0.5x)0

(1-0.2x)0

…

x1

(1-x)1

(1-0.6x)0

(1+0.3x)1

(1-0.5x)0

(1-0.2x)0

…

x2

(1-x)1

(1-0.6x)0

(1+0.3x)2

(1-0.5x)0

(1-0.2x)0

…

x3

(1-x)1

(1-0.6x)1

(1+0.3x)2

(1-0.5x)0

(1-0.2x)0

…

x3

(1-x)1

(1-0.6x)1

(1+0.3x)2

(1-0.5x)1

(1-0.2x)0

…

Representation of relevance A probability distribution on

[0,1] for each (query, URL) pair

The density function is in a polynomial form over a small set of linear factors.

The coefficients of such linear factors are shared between different (query, URL) pairs.


x3

(1-1x)1

(1-0.6x)1

(1+0.3x)2

(1-0.5x)1

(1-0.2x)0

…

Inference: Go over each query session

once, update the exponents for corresponding (query, URL) pair impressed*

Analytical or numerical integration may be needed to compute the normalization constant.

04/20/23 CIKM'09 Tutorial, Hong Kong, China 84*Footnote: by virtue of the Bayes theorem and conditional independence relationship/assumption

Key problems: Which is the right factor to update?

How to estimate all the coefficients?


Modeling Benefits: Confidence for the URL relevance estimate Relative judgments: probability of URL i is

more relevant to the query than URL j Easy to interpret: coefficients in linear

factors reflect position-bias and user browsing patterns

Computational Benefits: Single-pass, linear algorithms; no

iterations Paralleled version is easy to implement







For a specific query session, let

where 1 ≤ i ≤ M=10.

04/20/23 88

S1

S2

S3

SM

…

E1

E2

E3

EM

…

C1

C2

C3

CM

…


04/20/23 89

S1

S2

S3

SM

…

E1

E2

E3

EM

…

C1

C2

C3

CM

…

Relevance

Examination

Click


Compute the posterior distributionConditional independence

relationship induced from the graphical model

04/20/23 90

How many times the URL j was clicked

How many times URLj was not clicked when it is at position (r + d) with the preceding click at position rCIKM'09 Tutorial, Hong Kong, China

Details

9104/20/23

Only top M=3 positions are shown, 3 query sessions and 4 distinct URLs.

41

4

3

1 3

31 2

Position 1 2 3

Query Session 3

Query Session 2

Query Session 1


9204/20/23

Initialize M(M+1)/2+1 counts for each URL

URL Clicks r=0d=1

r=0d=2

r=0d=3

r=1d=1

r=1d=2

r=2d=1

4 0 0 0 0 0 0 0


9304/20/23

Update counts for URL 4 If not impressed, do nothing; If clicked, increment “clicks” by 1; Otherwise, locate the right r and d to

increment.

URL Clicks r=0d=1

r=0d=2

r=0d=3

r=1d=1

r=1d=2

r=2d=1

4 0 0 0 0 0 0 0CIKM'09 Tutorial, Hong Kong, China

9404/20/23


increment.

URL Clicks r=0d=1

r=0d=2

r=0d=3

r=1d=1

r=1d=2

r=2d=1


9504/20/23


increment.

URL Clicks r=0d=1

r=0d=2

r=0d=3

r=1d=1

r=1d=2

r=2d=1


9604/20/23

The posterior for URL 4

Interpretation: The larger the probability of examination,

the stronger the penalty for a non-click.

URL Clicks r=0d=1

r=0d=2

r=0d=3

r=1d=1

r=1d=2

r=2d=1

4 1 0 0 0 0 0 1


Keep 2 counts for each parameter (one for click, and the other one for non-click)


Parameter Click Non-click Parameter Click Non-Click

β0,1 0 0 β1,1 0 0

β0,2 0 0 β1,2 0 0

β0,3 0 0 β2,1 0 0

For each position in a query session, locate the right r and d to increment.


Parameter Click Non-click Parameter Click Non-Click

β0,1 1 0 β1,1 0 1

β0,2 0 0 β1,2 0 1

β0,3 0 0 β2,1 0 0



Parameter Click Non-click Parameter

Click Non-Click

β0,1 1 1 β1,1 0 1

β0,2 1 0 β1,2 0 1

β0,3 0 0 β2,1 0 1




Click Non-Click

β0,1 1 2 β1,1 1 1

β0,2 1 0 β1,2 0 1

β0,3 0 0 β2,1 1 1

Maximum-Likelihood Estimate:



Click Non-Click

β0,1 1 2 β1,1 1 1

β0,2 1 0 β1,2 0 1

β0,3 0 0 β2,1 1 1

Let

Initializing and updating the counts: Time: Space:

04/20/23 102

Linear to the size of the click log

Almost constant storage requiredCIKM'09 Tutorial, Hong Kong, China

Details

Let

Initializing and updating the counts: Time: Space:

Computing relevance scores using numerical integration with B bins: Time: Space:


Details

Step 1: Step 1: initialize counting statistics; Step 2: Step 2: scan through the click log

once and update the counts for both inference and estimation

Step 3: Step 3: compute parameter values; Step 4: Step 4: use numerical integration to

obtain relevance scores.

Step 2 also applies for (linear) incremental computation!







The user behavior model:


Graphical model:


Relevance

Examination

Click

S1

S2

S3

SM

…

E1

E2

E3

EM

…

C1

C2

C3

CM

…


Details

Number of user behavior parameters

Number of distinct factors for (query, URL)

Number of counts needed for parameters


CCM UBM

3 55

CCM UBM

22 56

CCM UBM

5 110

Introduction Designing click models Bayesian click models Selected topics on click models

Scaling click models for Petabyte-scale data

Click model evaluation

Tailoring user goals to click models Conclusion


Data collected in 8 weeks Job k includes data between week 1 and

k Both time and space costs are

prohibitive for a single node.


A Simple Task: counting # impression for each (query, URL) pair


Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Output

Count Count Count Count

Machine #1

Machine #2 Machine #3 Machine #4

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Output


“Map” puts all of the same Pairs onto one machine. This allows you to group by various fields in

subsequent processes.

Machine #1



Map = Bucket: the intermediate key is (query, URL) pair


Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Output


“Count” carries out standard increment-by-1 over each distinct Pair.

Machine #1


“Count” REDUCES the amount of data since each Pair has only one output value


Map = Bucket: the intermediate key is (query, URL) pair

Reduce = Count: it accepts a list of (key, value) tuple, and outputs the final result for each distinct key


Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Extent

GetPairs

Map

Sort

Output


MAPMAP

REDUCEREDUCE

Machine #1


04/20/23 119

0 for clicks0 for clicks332 52 51 4 61 4 6


Map: scan the click log Intermediate key: (query, URL) Value: the index of linear factors

(0~55 for top-10 positions)

Reduce: scan the list of (key, value) The key indicates which exponent vector

to update The value indicates the index of the

element in the exponent vector to increment


Linearly increasing computation loadNear-constant elapsed time

04/20/23121

Single machine computation load

Elapse time on SCOPE

• 3 hours• 265 TB log data• 1.15 billion (query, url) pairs







04/20/23 123

Impression Data

Click Data


04/20/23 124

Impression Data

Click Data

Relevance Scores

Global Parameters

M=10


Relevance

New Impression Vector from an Existing Query

04/20/23 125

Global params

Predicted Examination

Predicted ClicksCIKM'09 Tutorial, Hong Kong, China

Data are collected from a commercial search engine after query term normalization and spam removal.

For each query term, split query sessions evenly into training and test sets according to the timestamp.

Top frequent/infrequent query terms are removed.


Most popular metrics: Average test data log-likelihood (LL)

(probability of accurately predicting the click vector, 2^10 possibilities)[Guo+09a, Guo+09b, Liu+09a, Zhu+10]

Perplexity of prediction for each position(2^{average entropy} of click/no-click binary prediction for each position independently)[Dupret+08, Guo+09a, Guo+09b, Zhu+10]


Other Metrics: Click-through-rate (CTR) prediction

(Especially for predicting CTR@1)[Chapelle+09, Zhu+10]

Predicting first/last clicked positions[Guo+09a, Guo+09b]

Position-bias sanity check(plot the click rate curve for top-10 positions v.s. the ground truth)[Guo+09a, Guo+09b]


Average Log-likelihood Random guess: log(2-10) = -3.01 Optimal value: 0

12904/20/23

Model CCM UBM DCM

LL -1.171 -1.264 -1.302

Improve-ment Ratio

9.7% 14%


13004/20/23

Better

Worse


13104/20/23

Better

Worse


Average Perplexity over top 10 positions Random guess: 2 Optimal value: 1


Model CCM UBM DCM

Perplexity

-1.1479

1.1577 1.1590

Improve-ment Ratio

7.5% 8.3%


Worse

Better



For 1M query sessions, the estimated time in seconds:

* Time for CCM and BBM includes computing posterior mean and variance using numerical integration w/ 100 bins.

** UBM converges in 34 iterations.

DCM CCM* BBM* UBM**

80 150 165 5,000






Queries could be categorized into 2 sets: Navigational: to find the link to an

existing website, e.g., bing; Informational: more exploration, multiple

clicks may arise, e.g., iron man.


Different user goals result in different browsing and click patterns.

The straightforward mixture-modeling approach is not practical. [Dupret+08]

Solution: Classify query terms a priori based on user

goals. Fitting and learning 2 sets of model

parameters for navigational and informational queries.


Two-way classification for query terms based on click data using… Median position of click distribution Mean position of click distribution Average # clicks per query session …

Pick the one which has best click prediction If a position receives 50% of the click,

then navigational, else informational04/20/23 CIKM'09 Tutorial, Hong Kong, China 139

Improvement of click prediction for DCM: Log-Likelihood: 4.0% Perplexity: 1.3%

Examination/Click position-bias:


Introduction Designing click models Bayesian click models Selected topics on click models Conclusion


Click models A statistical tool to leverage valuable

user implicit feedback in terabyte/petabyte search logs.

Provide click prediction as well as relevance estimates.

Application domains include learning to rank, measuring search performance, online advertising, user behavior analysis…


Click models Different model designs reflect various

assumption of user behaviors to explain the position-bias.

The modeling choice may depend on the application scenario.


Click models Efficient, single-pass, parallelizable

algorithms are desired in real-world applications.

Bayesian framework could be applied to click models for both modeling benefits and computational benefits.

Click Chain Model and Bayesian Browsing Model represent state-of-the-art examples.


Bigger Context Query reformulations Personalization

Richer inputs Universal search Diverse user feedback

Click model v.s. Human judgments04/20/23 CIKM'09 Tutorial, Hong Kong, China 145

[Burges+05]: C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. ICML’05.

[Chapelle+09]: O. Chapelle and Y. Zhang. A dynamic Bayesian network click model for web search ranking. WWW’09.

[Craswell+08]: N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. WSDM ’08.

[Dean+04]: J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. OSDI’04.

[Dupret+08]: G. Dupret and B. Piwowarski. A user browsing model to predict search engine click data from past observations. SIGIR’08.


[Guo+09a]: F. Guo, C. Liu, and Y.-M. Wang. Efficient multiple-click models in web search. WSDM’09.

[Guo+09b]: F. Guo, C. Liu, A. Kannan, T. Minka, M. Taylor, Y.-M. Wang, and C. Faloutsos. Click chain model in web search. WWW’09.

[Guo+09c]: F. Guo, L. Li, and C. Faloutsos. Tailoring click models to user goals. WSCD’09.

[Joachims02]: T. Joachims. Optimizing search engines using clickthrough data. KDD’02.

[Joachims+07]: T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Accurately interpreting clickthrough data as implicit feedback, ACM TOIS, 25(2), 2007.


[Lee+05]: U. Lee, Z. Liu, and J. Cho. Automatic identification ofuser goals in web search. WWW’05.

[Liu+09a]: C. Liu, F. Guo, and C. Faloutsos. BBM: Deriving click models from petabyte-scale data. KDD’09.

[Liu+09b]: C. Liu, M. Li, and Y.-M. Wang. Post-rank reordering: resolving preference misalignments between search engines and end users. CIKM’09.

[Richardson+07]: M. Richardson, E. Dominowska, and R. Ragno. Predicting clicks: estimating the click-through rate for new ads. WWW’07.

[Zhu+10]: Z. Zhu, W. Chen, T. Minka, C. Zhu and Z. Chen. A novel click model and its applications to online advertising. To appear in WSDM’10.



MSR, Search LabAnitha Kannan MSR, Cambridge

Tom Minka

Carnegie Mellon University

Christos Faloutsos Li-Wei HeMSR, ISRC-RedmondMSR, Cambridge

Nick Craswell


Yi-Min WangMSR, ISRC-Redmond

MSR, CambridgeMike Taylor

MSR, ISRC-RedmondEthan Tu


Documents

Statistical Models for Web Search Click Log Analysis