Query processing:optimizations

Paolo FerraginaDipartimento di Informatica

Università di Pisa

Reading 2.3

Augment postings with skip pointers (at indexing time)

How do we deploy them ? Where do we place them ?

1282 4 8 41 48 64

311 2 3 8 11 17 21

3111

41 128

Sec. 2.3

Using skips

1282 4 8 41 48 64

311 2 3 8 11 17 21

3111

41 128

Suppose we’ve stepped through the lists until we process 8 on each list. We match it and advance.

We then have 41 and 11 on the lower. 11 is smaller.

But the skip successor of 11 on the lower list is 31, sowe can skip ahead past the intervening postings.

Sec. 2.3

Placing skips

Tradeoff: More skips shorter skip spans more

likely to skip. But lots of comparisons to skip pointers.

Fewer skips few pointer comparison, but then long skip spans few successful skips.

Sec. 2.3

Placing skips

Simple heuristic: for postings of length L, use L evenly-spaced skip pointers.

This ignores the distribution of query terms. Easy if the index is relatively static; harder if

L keeps changing because of updates.

This definitely used to help; with modern hardware it may not unless you’re memory-based The I/O cost of loading a bigger postings list

can outweigh the gains from quicker in memory merging!

Sec. 2.3

Faster query = caching?Two opposite approaches:

I. Cache the query results (exploits query locality)

II. Cache pages of posting lists (exploits term

locality)

Query processing:phrase queries and positional

indexes

Paolo FerraginaDipartimento di Informatica

Università di Pisa

Reading 2.4

Phrase queries

Want to be able to answer queries such as “stanford university” – as a phrase

Thus the sentence “I went at Stanford my university” is not a match.

Sec. 2.4

Solution #1: Biword indexes

For example the text “Friends, Romans, Countrymen” would generate the biwords friends romans romans countrymen

Each of these biwords is now an entry in the dictionary

Two-word phrase query-processing is immediate.

Sec. 2.4.1

Longer phrase queries

Longer phrases are processed by reducing them to bi-word queries in AND

stanford university palo alto can be broken into the Boolean query on biwords, such as

stanford university AND university palo AND palo alto

Need the docs to verify+They are combined with other solutions

Can have false positives!Index blows up

Sec. 2.4.1

Solution #2: Positional indexes

In the postings, store for each term and document the position(s) in which that term occurs:

<term, number of docs containing term;doc1: position1, position2 … ;doc2: position1, position2 … ;etc.>

Sec. 2.4.2

Processing a phrase query

“to be or not to be”. to:

2:1,17,74,222,551; 4:8,16,190,429,433; 7:13,23,191; ...

be: 1:17,19; 4:17,191,291,430,434;

5:14,19,101; ...

Same general method for proximity searches

Sec. 2.4.2

Query term proximity

Free text queries: just a set of terms typed into the query box – common on the web

Users prefer docs in which query terms occur within close proximity of each other

Would like scoring function to take this into account – how?

Sec. 7.2.2

Positional index size

You can compress position values/offsets Nevertheless, a positional index expands

postings storage by a factor 2-4 in English

Nevertheless, a positional index is now commonly used because of the power and usefulness of phrase and proximity queries … whether used explicitly or implicitly in a ranking retrieval system.

Sec. 2.4.2

Combination schemes

BiWord + Positional index is a profitable combination

Biword is particularly useful for particular phrases (“Michael Jackson”, “Britney Spears”)

More complicated mixing strategies do exist!

Sec. 2.4.3

Soft-AND

E.g. query rising interest rates

Run the query as a phrase query

If <K docs contain the phrase rising interest

rates, run the two phrase queries rising

interest and interest rates

If we still have <K docs, run the “vector space

query” rising interest rates (…see next…)

“Rank” the matching docs (…see next…)

Sec. 7.2.3

Query processing:other sophisticated queries

Paolo FerraginaDipartimento di Informatica

Università di Pisa

Reading 3.2 and 3.3

Wild-card queries: *

mon*: find all docs containing words beginning with “mon”. Use a Prefix-search data structure

*mon: find words ending in “mon” Maintain a prefix-search data structure for

reverse terms.

How can we solve the wild-card query pro*cent ?

Sec. 3.2

What about * in the middle?

co*tion We could look up co* AND *tion and

intersect the two lists Expensive

se*ate AND fil*erThis may result in many Boolean AND queries.

The solution: transform wild-card queries so that the *’s occur at the end

This gives rise to the Permuterm Index.

Sec. 3.2

Permuterm index

For term hello, index under: hello$, ello$h, llo$he, lo$hel, o$hell,

$hellowhere $ is a special symbol.

Queries: X lookup on X$ X* lookup on $X* *X lookup on X$* *X* lookup on X* X*Y lookup on Y$X* X*Y*Z ??? Exercise!

Sec. 3.2.1

Permuterm query processing

Rotate query wild-card to the right P*Q Q$P*

Now use prefix-search data structure Permuterm problem: ≈ 4x lexicon size

Empirical observation for English.

Sec. 3.2.1

K-gram indexes

The k-gram index finds terms based on a query consisting of k-grams (here k=2).

mo

on

among

$m mace

among

amortize

madden

arond

Sec. 3.2.2

K-gram for wild-cards queries

Query mon* can now be run as

$m AND mo AND on

Gets terms that match AND version of our

wildcard query.

Must post-filter these terms against query.

Sec. 3.2.2

Isolated word correction

Given a lexicon and a character sequence Q, return the words in the lexicon closest to Q

What’s “closest”? Edit distance (Levenshtein distance) Weighted edit distance n-gram overlap

Useful in query-mispellings

Sec. 3.3.2

Edit distance

Given two strings S1 and S2, the minimum number of operations to convert one to the other

Operations are typically character-level Insert, Delete, Replace, (Transposition)

E.g., the edit distance from dof to dog is 1 From cat to act is 2 (Just 1 with transpose.) from cat to dog is 3.

Generally found by dynamic programming.

Sec. 3.3.3

Let E(i,j) = edit distance between T1,j and P1,i.

DynProg for Edit Distance

E(i,0)=E(0,i)=i

E(i, j) = E(i–1, j–1) if Pi=Tj

E(i, j) = min{E(i, j–1),

E(i–1, j),

E(i–1, j–1)}+1 if Pi Tj

Example

T0 1 2 3 4 5 6

p t t a p a

P

0 0 1 2 3 4 5 61 p 1 0 1 2 3 4 52 a 2 1 1 2 2 3 43 t 3 2 1 1 2 3 44 t 4 3 2 1 2 3 4

Weighted edit distance

As above, but the weight of an operation depends on the character(s) involved Meant to capture OCR or keyboard errors,

e.g. m more likely to be mis-typed as n than as q

Therefore, replacing m by n is a smaller edit distance than by q

Requires weight matrix as input Modify dynamic programming to handle

weights

Sec. 3.3.3

k-gram overlap for Edit Distance

Enumerate all the k-grams in the query string as well as in the lexicon

Use the k-gram index (recall wild-card search) to retrieve all lexicon terms matching any of the query k-grams

Threshold by number of matching k-grams If the term is L chars long If E is the number of allowed errors (E*k, k-grams are killed) At least (L-k+1) – E*k of the query k-grams must match a

dictionary term to be a candidate answer.

Sec. 3.3.4

Context-sensitive spell correction

Text: I flew from Heathrow to Narita. Consider the phrase query “flew form

Heathrow” We’d like to respond

Did you mean “flew from Heathrow ”?

because no docs matched the query phrase.

Sec. 3.3.5

Zone indexes

Paolo FerraginaDipartimento di Informatica

Università di Pisa

Reading 6.1

Parametric and zone indexes

Thus far, a doc has been a term sequence

But documents have multiple parts: Author Title Date of publication Language Format etc.

These are the metadata about a document

Sec. 6.1

Zone

A zone is a region of the doc that can contain an arbitrary amount of text e.g., Title Abstract References …

Build inverted indexes on fields AND zones to permit querying

E.g., “find docs with merchant in the title zone and matching the query gentle rain”

Sec. 6.1

Example zone indexes

Encode zones in dictionary vs. postings.

Sec. 6.1

Tiered indexes

Break postings up into a hierarchy of lists Most important … Least important

Inverted index thus broken up into tiers of decreasing importance

At query time use top tier unless it fails to yield K docs If so drop to lower tiers

Sec. 7.2.1

Example tiered index

Sec. 7.2.1