Introduction to Information Retrieval - IPTricardo/ficheiros/IR-Evaluation.pdf · Introduction to Information Retrieval Ricardo Campos Evaluation. What is Information Retrieval? What

Introduction to Information Retrieval

Ricardo Campos

Evaluation

What is Information Retrieval?



AGENDA

What is this talk about?

Batch/Lab

Evaluation2

Relevance

1

User-Study

Evaluation3

Metrics

5

Online

Evaluation4

Q&A

6




•

•


R

R

R

R

R

R


R

R

R

R

R


R

R

R

R

R


R

R

R

R

R


R

R

R

R












•

•

•


Batch evaluation / laboratory;•

•

•








•

•

•


•










•

•

•




User-study evaluation;

•

•

•





•

•


•

•

Online evaluation;•


•

•




𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =# 𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝐷𝑜𝑐𝑠 𝑅𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑

# 𝑅𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑 𝐷𝑜𝑐𝑠

𝑅𝑒𝑐𝑎𝑙𝑙 =# 𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝐷𝑜𝑐𝑠 𝑅𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑

# 𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝐷𝑜𝑐𝑠


Gold Standard

Condittion

Positive

Condition

Negative

Test

Ou

tco

me Test Outcome Positive TP FP (Type I

Error)

Test Outcome

Negative

FN (Type II

Error)

TN

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑇𝑃

𝑇𝑃 + 𝐹𝑃

𝑅𝑒𝑐𝑎𝑙𝑙 =𝑇𝑃

𝑇𝑃 + 𝐹𝑁


•

•


RetrievedRelevant

Relevant Retrieved Precision

1.0

Recall

1.0


RetrievedRelevant

Relevant Retrieved Precision

1.0

Recall

1.0


R

R

R

R

R

R













Documents

Introduction to Information Retrieval - IPTricardo/ficheiros/IR-Evaluation.pdf · Introduction to Information Retrieval Ricardo Campos Evaluation. What is Information Retrieval? What