E.G.M. PetrakisInformation Retrieval Models1 Classic IR Models Boolean model simple model based on set theory queries as Boolean expressions adopted

E.G.M. Petrakis Information Retrieval Models 1

Classic IR Models

Boolean model simple model based on set theory queries as Boolean expressions adopted by many commercial systems

Vector space model queries and documents as vectors in an M-dimensional

space M is the number of terms find documents most similar to the query in the M-

dimensional space

Probabilistic model a probabilistic approach assume an ideal answer set for each query iteratively refine the properties of the ideal answer set


Document Index Terms

Each document is represented by a set of representative index terms or keywords requires text pre-processing (off-line) these terms summarize document contents adjectives, adverbs, connectives are less useful the index terms are mainly nouns (lexicon look-

up) Not all terms are equally useful

very frequent terms are not useful very infrequent terms are not useful neither terms have varying relevance (weights) when

used to describe documents


Text Preprocessing

Extract terms from documents and queriesdocument - query profile

Processing stagesword separation sentence splittingchange terms to a standard form (e.g.,

lowercase)eliminate stop-words (e.g. and, is, the, …)reduce terms to their base form (e.g., eliminate

prefixes, suffixes)construct term indices (usually inverted files)


Text Preprocessing Chart

from Baeza – Yates & Ribeiro – Neto, 1999


Inverted Index

άγαλμααγάπη…δουλειά…πρωί…ωκεανός

index posting list

(1,2)(3,4)

(4,3)(7,5)

(10,3)

123456789

1011

………

documents


Basic NotationDocument: usually text

D: document collection (corpus)d: an instance of D

Query: same representation with documentsQ: set of all possible queriesq: an instance of Q

Relevance: R(d,q)binary relation R: D x Q {0,1}d is “relevant” to q iff R(d,q) = 1 or degree of relevance: R(d,q) [0,1] or probability of relevance R(d,q) = Prob(R|d,q)


Term Weights

T = {t1, t2, ….tM } the terms in corpus N number of documents in corpusdj a document

dj is represented by (w1j,w2j,…wMj) where

wij > 0 if ti appears in dj

wij = 0 otherwise

q is represented by (q1,q2,…qM)R(d,q) > 0 if q and d have common terms


Term Weighting

t2

wMNwM1tM

w1Nw12w11t1

dN….d2d1 docsterms

w2i


Document Space (corpus)

q

D

query

relevant document

non-relevant document


Boolean Model

Based on set theory and Boolean algebraBoolean queries: “John” and “Mary” not “Ann”terms linked by “and”, “or”, “not”terms weights are 0 or 1 (wij=0 or 1)query terms are present or absent in a documenta document is relevant if the query condition is

satisfied

Pros: simple, in many commercial systemsCons: no ranking, not easy for complex

queries


Query Processing

For each term ti in query q={t1,t2,…tM}1) use the index to retrieve all dj with wij > 02) sort them by decreasing order (e.g., by term

frequency) Return documents satisfying the query

condition Slow for many terms: involves set

intersections Keep only the top K documents for each

term at step 2 or Do not process all query terms


Vector Space Model

Documents and queries are M – dimensional term vectorsnon-binary weights to index termsa query is similar to a document if their

vectors are similarretrieved documents are sorted by

decreasing order a document may match a query only

partially SMART is the most popular

implementation

http://www-a2k.is.tokushima-u.ac.jp/member/kita/NLP/IR.html

http://www-a2k.is.tokushima-u.ac.jp/member/kita/NLP/IR.html


Query – Document Similarity

M

i id

M

i iq

M

i idiq

ww

ww

dq

dqdqSim

1

2

1

2

1

||||),(

Similarity is defined as the cosine of the angle between document and query vectors

θ

q

d


Weighting Scheme

tf x idf weighting scheme

wij: weight of term ti associated with document dj

tfij frequency of term ti in document dj

max frequencytfli is computed over all terms in dj

tfij: normalized frequency

idfi: inverse document frequency

ni: number of documents where term ti occurs

iidf

nN

ijtf

freq

freqw

ili

ij

ij logmax

=


Weight Normalization

Many ways to express weights E.g., using log(tfij) The weight is normalized in [0,1]

Normalize by document length

M

ik kj

iijij

tf

idftfw

2))log(1(

))log(1(


M

k kj

ijij

w

ww

1

2'

Normalization by Document Length

The longer the document, the more likely it is for a given term to appear in it

Normalize the term weights by document length (so longer documents are not given more weight)


Comments on Term Weighting

tfij: term frequency – measures how well a term describes a documentintra document characterization

idfi: terms appearing in many documents are not very useful in distinguishing relevant from non-relevant documentsinter document characterization

This scheme favors average terms


Comments on Vector Space Model

Pros:at least as good as other modelsapproximate query matching: a query

and a document need not contain exactly the same terms

allows for ranking of resultsCons:

assumes term independency


Document Distance

Consider documents d1, d2 with vectors u1, u2

their distance is defined as the length AB

)),(1(2

=))cos(1(2

=)2/sin(2

=),(tan

21

21

ddsimilarity

θ

θ

ddcedis

-

-


Probabilistic Model

Computes the probability that the document is relevant to the queryranks the documents according to their

probability of being relevant to the queryAssumption: there is a set R of relevant

documents which maximizes the overall probability of relevance R: ideal answer set

R is not known in advanceinitially assume a description (the terms) of Riteratively refine this description


Basic Notation

D: corpus, d: an instance of DQ: set of queries, q: an instance of Q

P(R | d) : probability that d is relevant : probability that d is not

relevant

q} orelevant t is d ,q ,d | q){(d, R QD

q} orelevant tnot is d ,q ,d | q){(d, R QD

)( |dRP


Probability of Relevance

P(R|d): probability that d is relevant

Bayes rule

P(d|R): probability of selecting d from RP(R): probability of selecting R from DP(d): probability of selecting d from D

)()()(

=)(dP

RPd|RPR|dP


Document Ranking

Take the odds of relevance as the rank

Minimizes probability of erroneous judgment

are the same for all docs

)()(

)()(

)(

)()(

RPRd|P

RPd|RP

|dRP

R|dPd|qSim

)(),( RPRP

)()(

=)(Rd|P

d|RPd|qSim


Ranking (cont’d)

Each document is represented by a set of index terms t1,t2,..tM assume binary terms wi for terms ti

d=(w1,w2,…wM) wherewi=1 if the term appears in d

wi=0 otherwise

Assuming independence of index terms

dt idt i

ii|R)tP(|RtPd|RP )()(


Ranking (conted)

By taking logarithms and by omitting constant terms

R is initially unknown

)R|P(t

)R|P(t-1logww+

R)|P(t-1

R)|P(tlogww

~)R|P(d

R)|P(d=Sim(d/q)

i

iid1 q

i

iid1 q

M

i i

M

i i


Initial Estimation

Make simplifying assumptions such as

where ni: number of documents containing ti and N: total number of documents

Retrieve initial answer set using these values

Refine answer iteratively

Nn

R|tP|RtP iii =)( ,5.0=)(


Improvement

Let V the number of documents retrieved initially

Take the fist r answers as relevant From them compute Vi: number of documents

containing ti

Update the initial probabilities:

Resubmit query and repeat until convergenceV-NV-n

=)RP(t ,VV

=R)P(t iii

ii ||


Comments on Probabilistic Model

Pros: good theoretical basis

Cons: need to guess initial probabilitiesbinary weights independence assumption

Extensions:relevance feedback: humans choose relevant

docsOKAPI formula for non – binary weights


Comparison of Models

The Boolean model is simple and used used almost everywhere. It does not allow for partial matches. It is the weakest model

The Vector space model has been shown (Salton and Buckley) to outperform the other two models

Various extensions deal with their weaknesses


Query Modification

The results are not always satisfactorysome answers are correct, others are notqueries can’t specify user’s needs precisely

Iteratively reformulate and resubmit the query until the results become satisfactory

Two approachesrelevance feedbackquery expansion


Relevance Feedback

Mark answers as relevant: positive examplesirrelevant: negative examples

Query: a point in document spaceat each iteration compute new query pointthe query moves towards an “optimal

point” that distinguishes relevant from non-relevant document

the weights of query terms are modified “term reweighting”


Rochio Vectors

q0 q1

q2

optimal query


Rochio Formula

Query point

di: relevant answerdj: non-relevant answern1: number of relevant answersn2: number or non-relevant answersα, β, γ: relative strength (usually

α=β=γ=1) α = 1, β = 0.75, γ = 0.25: q0 and relevant

answers contain important information

21

12

n

1i1

0 -n

j ji dn

dn

qq


Query Expansion

Adds new terms to the query which are somehow related to existing termssynonyms from dictionary (e.g., staff,

crew)semantically related terms from a

thesaurus (e.g., “wordnet”): man, woman, man kind, human…)

terms with similar pronunciation (Phonix, Soundex)

Better results in many cases but query defocuses (topic drift)


Comments

Do all togetherquery expansion: new terms are added

from relevant documents, dictionaries, thesaurus

term reweighing by Rochio formulaIf consistent relevance judgments

are provided2-3 iterations improve resultsquality depends on corpus


Extensions

Pseudo relevance feedback: mark top k answers as relevant, bottom k answers as non-relevant and apply Rochio formula

Relevance models for probabilistic modelevaluation of initial answers by humansterm reweighting model by Bruce Croft,

1983


Text Clustering

The grouping of similar vectors into clusters

Similar documents tend to be relevant to the same requests

Clustering on M-dimensional space M number of terms


Clustering Methods

Sound methods based on the document-to-document similarity matrixgraph theoretic methodsO(N2) time

Iterative methods operating directly on the document vectorsO(NlogN) or O(N2/logN) time


Sound Methods

1. Two documents with similarity > T (threshold) are connected with an edge [Duda&Hart73]

clusters: the connected components (maximal cliques) of the resulting graph

problem: selection of appropriate threshold T


Zahn’s method [Zahn71]

Find the minimum spanning tree For each doc delete edges with length l > lavg

lavg: average distance if its incident edges

Or remove the longest edge (1 edge removed => 2 clusters, 2 edges removed => 3 clusters

Clusters: the connected components of the graph

the dashed edge is inconsistent and is deleted


Iterative Methods

K-means clustering (K known in advance)

Choose some seed points (documents)possible cluster centroids

Repeat until the centroids do not changeassign each vector (document) to its

closest seed compute new centroids reassign vectors to improve clusters


Cluster Searching

The M-dimensional query vector is compared with the cluster-centroidssearch closest cluster retrieve documents with similarity > T


References

"Modern Information Retrieval", Richardo Baeza-Yates, Addison Wesley 1999

"Searching Multimedia Databases by Content", Christos Faloutsos, Kluwer Academic Publishers, 1996

Information Retrieval Resources http://nlp.stanford.edu/IR-book/information-retrieval.html

TREC http://trec.nist.gov/ SMART http://en.wikipedia.org/wiki/SMART_

Information_Retrieval_System LEMOUR http://www.lemurproject.org/ LUCENE http://lucene.apache.org/

http://en.wikipedia.org/wiki/SMART_