1

Boolean and Vector Space Retrieval Models

Many slides in this section are adapted from Prof Raymond Mooney (UTexas), Prof. Joydeep Ghosh (UT ECE) who in turn adapted them from Prof. Dik Lee (Univ. of Science and Tech, Hong Kong)

2

Retrieval Models• A retrieval model specifies the details

of:– Document representation– Query representation– Retrieval function

• Determines a notion of relevance.• Notion of relevance can be binary or

continuous (i.e. ranked retrieval).

3

Classes of Retrieval Models

• Boolean models (set theoretic)– Extended Boolean

• Vector space models (statistical/algebraic)

– Generalized VS– Latent Semantic Indexing

• Probabilistic models

4

Other Model Dimensions• Logical View of Documents

– Index terms– Full text– Full text + Structure (e.g. hypertext)

• User Task– Retrieval– Browsing

5

Retrieval Tasks• Ad hoc retrieval: Fixed document corpus, varied

queries.• Filtering: Fixed query, continuous document

stream.– User Profile: A model of relative static preferences.– Binary decision of relevant/not-relevant.

• Routing: Same as filtering but continuously supply ranked lists rather than binary filtering.

6

Common Preprocessing Steps• Strip unwanted characters/markup (e.g. HTML

tags, punctuation, numbers, etc.).• Break into tokens (keywords) on whitespace.• Stem tokens to “root” words

– computational comput• Remove common stopwords (e.g. a, the, it, etc.).• Detect common phrases (possibly using a domain

specific dictionary).• Build inverted index (keyword list of docs

containing it).

7

Boolean Model• A document is represented as a set of

keywords.• Queries are Boolean expressions of keywords,

connected by AND, OR, and NOT, including the use of brackets to indicate scope.– [[Rio & Brazil] | [Hilo & Hawaii]] & hotel & !Hilton]

• Output: Document is relevant or not. No partial matches or ranking.

8

• Popular retrieval model because:– Easy to understand for simple queries.– Clean formalism.

• Boolean models can be extended to include ranking.

• Reasonably efficient implementations possible for normal queries.

Boolean Retrieval Model

9

Boolean Models Problems• Very rigid: AND means all; OR means any.• Difficult to express complex user requests.• Difficult to control the number of documents

retrieved.– All matched documents will be returned.

• Difficult to rank output.– All matched documents logically satisfy the query.

• Difficult to perform relevance feedback.– If a document is identified by the user as relevant or

irrelevant, how should the query be modified?

10

Example

• Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia?

• Could grep all of Shakespeare’s plays for Brutus and Caesar, then strip out lines containing Calpurnia?– Slow (for large corpora)– NOT Calpurnia is non-trivial– Other operations (e.g., find the phrase Romans and

countrymen) not feasible

11

Term-document incidence

1 if play contains word, 0 otherwise

Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth

Antony 1 1 0 0 0 1

Brutus 1 1 0 1 0 0Caesar 1 1 0 1 1 1

Calpurnia 0 1 0 0 0 0Cleopatra 1 0 0 0 0 0

mercy 1 0 1 1 1 1

worser 1 0 1 1 1 0

12

Incidence vectors

• So we have a 0/1 vector for each term.• To answer query: take the vectors for Brutus,

Caesar and Calpurnia (complemented) bitwise AND.

• 110100 AND 110111 AND 101111 = 100100.

13

Answers to query

• Antony and Cleopatra, Act III, Scene ii• Agrippa [Aside to DOMITIUS ENOBARBUS]: Why, Enobarbus,• When Antony found Julius Caesar dead,• He cried almost to roaring; and he wept• When at Philippi he found Brutus slain.

• Hamlet, Act III, Scene ii• Lord Polonius: I did enact Julius Caesar I was killed i' the• Capitol; Brutus killed me.

14

Bigger corpora

• Consider n = 1M documents, each with about 1K terms.

• Avg 6 bytes/term incl spaces/punctuation – 6GB of data in the documents.

• Say there are m = 500K distinct terms among these.

15

Can’t build the matrix

• 500K x 1M matrix has half-a-trillion 0’s and 1’s.• But it has no more than one billion 1’s.

– matrix is extremely sparse.• What’s a better representation?

– We only record the 1 positions.

Why?

16

Inverted index

• For each term T, must store a list of all documents that contain T.

• Do we use an array or a list for this?

BrutusCalpurniaCaesar

1 2 3 5 8 13 21 342 4 8 16 32 64128

13 16

What happens if the word Caesar is added to document 14?

17

Inverted index

• Linked lists generally preferred to arrays– Dynamic space allocation– Insertion of terms into documents easy– Space overhead of pointers

BrutusCalpurniaCaesar

2 4 8 16 32 64 1282 3 5 8 13 21 34

13 161

Dictionary PostingsSorted by docID (more later on why).

18

Inverted index construction

TokenizerToken stream. Friends Romans Countrymen

Linguistic modules

Modified tokens. friend roman countryman

Indexer

Inverted index.

friendromancountryman

2 42

13 161

More onthese later.

Documents tobe indexed. Friends, Romans, countrymen.

19

• Sequence of (Modified token, Document ID) pairs.

I did enact JuliusCaesar I was killed

i' the Capitol; Brutus killed me.

Doc 1

So let it be withCaesar. The noble

Brutus hath told youCaesar was ambitious

Doc 2

Term Doc #I 1did 1enact 1julius 1caesar 1I 1was 1killed 1i' 1the 1capitol 1brutus 1killed 1me 1so 2let 2it 2be 2with 2caesar 2the 2noble 2brutus 2hath 2told 2you 2caesar 2was 2ambitious 2

Indexer steps

20

• Sort by terms. Term Doc #ambitious 2be 2brutus 1brutus 2capitol 1caesar 1caesar 2caesar 2did 1enact 1hath 1I 1I 1i' 1it 2julius 1killed 1killed 1let 2me 1noble 2so 2the 1the 2told 2you 2was 1was 2with 2

Term Doc #I 1did 1enact 1julius 1caesar 1I 1was 1killed 1i' 1the 1capitol 1brutus 1killed 1me 1so 2let 2it 2be 2with 2caesar 2the 2noble 2brutus 2hath 2told 2you 2caesar 2was 2ambitious 2

Core indexing step.

21

• Multiple term entries in a single document are merged.

• Frequency information is added.

Term Doc # Freqambitious 2 1be 2 1brutus 1 1brutus 2 1capitol 1 1caesar 1 1caesar 2 2did 1 1enact 1 1hath 2 1I 1 2i' 1 1it 2 1julius 1 1killed 1 2let 2 1me 1 1noble 2 1so 2 1the 1 1the 2 1told 2 1you 2 1was 1 1was 2 1with 2 1

Term Doc #ambitious 2be 2brutus 1brutus 2capitol 1caesar 1caesar 2caesar 2did 1enact 1hath 1I 1I 1i' 1it 2julius 1killed 1killed 1let 2me 1noble 2so 2the 1the 2told 2you 2was 1was 2with 2

Why frequency?Will discuss later.

22

• The result is split into a Dictionary file and a Postings file.

Doc # Freq2 12 11 12 11 11 12 21 11 12 11 21 12 11 11 22 11 12 12 11 12 12 12 11 12 12 1

Term N docs Tot Freqambitious 1 1be 1 1brutus 2 2capitol 1 1caesar 2 3did 1 1enact 1 1hath 1 1I 1 2i' 1 1it 1 1julius 1 1killed 1 2let 1 1me 1 1noble 1 1so 1 1the 2 2told 1 1you 1 1was 2 2with 1 1

Term Doc # Freqambitious 2 1be 2 1brutus 1 1brutus 2 1capitol 1 1caesar 1 1caesar 2 2did 1 1enact 1 1hath 2 1I 1 2i' 1 1it 2 1julius 1 1killed 1 2let 2 1me 1 1noble 2 1so 2 1the 1 1the 2 1told 2 1you 2 1was 1 1was 2 1with 2 1

23

• Where do we pay in storage?

Doc # Freq2 12 11 12 11 11 12 21 11 12 11 21 12 11 11 22 11 12 12 11 12 12 12 11 12 12 1

Term N docs Tot Freqambitious 1 1be 1 1brutus 2 2capitol 1 1caesar 2 3did 1 1enact 1 1hath 1 1I 1 2i' 1 1it 1 1julius 1 1killed 1 2let 1 1me 1 1noble 1 1so 1 1the 2 2told 1 1you 1 1was 2 2with 1 1

Pointers

Terms

Will quantify the storage, later.

24

The index we just built

• How do we process a query?– What kinds of queries can we process?

• Which terms in a doc do we index?– All words or only “important” ones?

• Stopword list: terms that are so common that they’re ignored for indexing.– e.g., the, a, an, of, to …– language-specific.

Today’s focus

25

Query processing

• Consider processing the query:Brutus AND Caesar– Locate Brutus in the Dictionary;

• Retrieve its postings.

– Locate Caesar in the Dictionary;• Retrieve its postings.

– “Merge” the two postings:

12834

2 4 8 16 32 641 2 3 5 8 13 21

BrutusCaesar

26

341282 4 8 16 32 64

1 2 3 5 8 13 21

The merge

• Walk through the two postings simultaneously, in time linear in the total number of postings entries

12834

2 4 8 16 32 641 2 3 5 8 13 21

BrutusCaesar2 8

If the list lengths are m and n, the merge takes O(m+n)operations.Crucial: postings sorted by docID.

27

Boolean queries: Exact match

• Queries using AND, OR and NOT together with query terms– Views each document as a set of words– Is precise: document matches condition or not.

• Primary commercial retrieval tool for 3 decades. • Professional searchers (e.g., Lawyers) still like

Boolean queries:– You know exactly what you’re getting.

28

Example: WestLaw http://www.westlaw.com/

• Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992)

• About 7 terabytes of data; 700,000 users• Majority of users still use boolean queries• Example query:

– What is the statute of limitations in cases involving the federal tort claims act?

– LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM

• Long, precise queries; proximity operators; incrementally developed; not like web search

29

More general merges

• Exercise: Adapt the merge for the queries:Brutus AND NOT CaesarBrutus OR NOT Caesar

Can we still run through the merge in time O(m+n)?

30

Statistical Models• A document is typically represented by a bag of

words (unordered words with frequencies).• Bag = set that allows multiple occurrences of the

same element.• User specifies a set of desired terms with optional

weights:– Weighted query terms: Q = < database 0.5; text 0.8; information 0.2 >– Unweighted query terms: Q = < database; text; information >– No Boolean conditions specified in the query.

31

Statistical Retrieval • Retrieval based on similarity between query

and documents.• Output documents are ranked according to

similarity to query.• Similarity based on occurrence frequencies

of keywords in query and document.• Automatic relevance feedback can be supported:

– Relevant documents “added” to query.– Irrelevant documents “subtracted” from query.

32

Issues for Vector Space Model• How to determine important words in a document?

– Word sense?– Word n-grams (and phrases, idioms,…) terms

• How to determine the degree of importance of a term within a document and within the entire collection?

• How to determine the degree of similarity between a document and the query?

• In the case of the web, what is a collection and what are the effects of links, formatting information, etc.?

33

The Vector-Space Model• Assume t distinct terms remain after preprocessing;

call them index terms or the vocabulary.• These “orthogonal” terms form a vector space.

Dimension = t = |vocabulary| • Each term, i, in a document or query, j, is given a

real-valued weight, wij.

• Both documents and queries are expressed as t-dimensional vectors: dj = (w1j, w2j, …, wtj)

34

Graphic RepresentationExample:D1 = 2T1 + 3T2 + 5T3

D2 = 3T1 + 7T2 + T3

Q = 0T1 + 0T2 + 2T3

T3

T1

T2

D1 = 2T1+ 3T2 + 5T3

D2 = 3T1 + 7T2 + T3

Q = 0T1 + 0T2 + 2T3

7

32

5

• Is D1 or D2 more similar to Q?• How to measure the degree of

similarity? Distance? Angle? Projection?

35

Document Collection• A collection of n documents can be represented in the vector

space model by a term-document matrix.• An entry in the matrix corresponds to the “weight” of a term

in the document; zero means the term has no significance in the document or it simply doesn’t exist in the document.

T1 T2 …. Tt

D1 w11 w21 … wt1

D2 w12 w22 … wt2

: : : : : : : :Dn w1n w2n … wtn

36

Term Weights: Term Frequency

• More frequent terms in a document are more important, i.e. more indicative of the topic. fij = frequency of term i in document j

• May want to normalize term frequency (tf) across the entire corpus: tfij = fij / max{fij}

37

Term Weights: Inverse Document Frequency

• Terms that appear in many different documents are less indicative of overall topic.

df i = document frequency of term i

= number of documents containing term i idfi = inverse document frequency of term i,

= log2 (N/ df i)

(N: total number of documents)• An indication of a term’s discrimination power.• Log used to dampen the effect relative to tf.

38

TF-IDF Weighting• A typical combined term importance indicator is tf-

idf weighting:wij = tfij idfi = tfij log2 (N/ dfi)

• A term occurring frequently in the document but rarely in the rest of the collection is given high weight.

• Many other ways of determining term weights have been proposed.

• Experimentally, tf-idf has been found to work well.

39

Computing TF-IDF -- An Example

Given a document containing terms with given frequencies: A(3), B(2), C(1)Assume collection contains 10,000 documents and document frequencies of these terms are: A(50), B(1300), C(250)Then:A: tf = 3/3; idf = log(10000/50) = 5.3; tf-idf = 5.3B: tf = 2/3; idf = log(10000/1300) = 2.0; tf-idf = 1.3C: tf = 1/3; idf = log(10000/250) = 3.7; tf-idf = 1.2

40

Query Vector• Query vector is typically treated as a

document and also tf-idf weighted.• Alternative is for the user to supply weights

for the given query terms.

41

Similarity Measure• A similarity measure is a function that computes

the degree of similarity between two vectors.

• Using a similarity measure between the query and each document:– It is possible to rank the retrieved documents in the

order of presumed relevance.– It is possible to enforce a certain threshold so that the

size of the retrieved set can be controlled.

42

Similarity Measure - Inner Product

• Similarity between vectors for the document di and query q can be computed as the vector inner product:

sim(dj,q) = dj•q = wij · wiq

where wij is the weight of term i in document j and wiq is the weight of term i in the query• For binary vectors, the inner product is the number of matched query terms

in the document (size of intersection).• For weighted term vectors, it is the sum of the products of the weights of the

matched terms.

t

i 1

43

Properties of Inner Product

• The inner product is unbounded.

• Favors long documents with a large number of unique terms.

• Measures how many terms matched but not how many terms are not matched.

44

Inner Product -- Examples

Binary:– D = 1, 1, 1, 0, 1, 1, 0

– Q = 1, 0 , 1, 0, 0, 1, 1

sim(D, Q) = 3

retrie

val

database

archite

cture

computer

textmanagem

ent

informatio

n

Size of vector = size of vocabulary = 70 means corresponding term not found

in document or query

Weighted: D1 = 2T1 + 3T2 + 5T3 D2 = 3T1 + 7T2 + 1T3

Q = 0T1 + 0T2 + 2T3

sim(D1 , Q) = 2*0 + 3*0 + 5*2 = 10 sim(D2 , Q) = 3*0 + 7*0 + 1*2 = 2

45

Cosine Similarity Measure• Cosine similarity measures the cosine of the angle

between two vectors.• Inner product normalized by the vector lengths.

D1 = 2T1 + 3T2 + 5T3 CosSim(D1 , Q) = 10 / (4+9+25)(0+0+4) = 0.81D2 = 3T1 + 7T2 + 1T3 CosSim(D2 , Q) = 2 / (9+49+1)(0+0+4) = 0.13

Q = 0T1 + 0T2 + 2T3

t3

t1

t2

D1

D2

Q

D1 is 6 times better than D2 using cosine similarity but only 5 times better using inner product.

t

i

t

i

t

i

ww

ww

qdqd

iqij

iqij

j

j

1 1

22

1

)(

CosSim(dj, q) =

46

Naïve ImplementationConvert all documents in collection D to tf-idf

weighted vectors, dj, for keyword vocabulary V.Convert query to a tf-idf-weighted vector q.For each dj in D do Compute score sj = cosSim(dj, q)Sort documents by decreasing score.Present top ranked documents to the user.

Time complexity: O(|V|·|D|) Bad for large V & D !|V| = 10,000; |D| = 100,000; |V|·|D| = 1,000,000,000

47

Comments on Vector Space Models

• Simple, mathematically based approach. • Considers both local (tf) and global (idf) word

occurrence frequencies.• Provides partial matching and ranked results.• Tends to work quite well in practice despite

obvious weaknesses.• Allows efficient implementation for large

document collections.

48

Problems with Vector Space Model• Missing semantic information (e.g. word sense).• Missing syntactic information (e.g. phrase structure,

word order, proximity information).• Assumption of term independence (e.g. ignores

synonomy).• Lacks the control of a Boolean model (e.g.,

requiring a term to appear in a document).– Given a two-term query “A B”, may prefer a document

containing A frequently but not B, over a document that contains both A and B, but both less frequently.