December 16, 2003 INEX 2003 -- Ray R. Larson Cheshire II at INEX 2003: Component and Algorithm Fusion Ray R. Larson School of Information Management and

December 16 , 2003 INEX 2003 -- Ray R. Larson

Cheshire II at INEX 2003: Cheshire II at INEX 2003: Component and Algorithm FusionComponent and Algorithm Fusion

Cheshire II at INEX 2003: Cheshire II at INEX 2003: Component and Algorithm FusionComponent and Algorithm Fusion

Ray R. LarsonRay R. LarsonSchool of Information Management and School of Information Management and

Systems Systems

University of California, BerkeleyUniversity of California, Berkeley


OverviewOverviewOverviewOverview

• Cheshire II feature overview Cheshire II feature overview – Logistic Regression Ranking and Boolean Logistic Regression Ranking and Boolean

OperationsOperations

• Additions from INEX ‘02Additions from INEX ‘02– XML Schemas and Element RetrievalXML Schemas and Element Retrieval– CORI, Okapi BM-25 ranking algorithmsCORI, Okapi BM-25 ranking algorithms– Result Set sorting, merging and ranking Result Set sorting, merging and ranking

operatorsoperators

• Evaluation ResultsEvaluation Results


Overview of Cheshire IIOverview of Cheshire IIOverview of Cheshire IIOverview of Cheshire II• It supports SGML and XML with components and component indexesIt supports SGML and XML with components and component indexes• It is a client/server applicationIt is a client/server application• Uses the Z39.50 Information Retrieval Protocol, support for SRW, OAI, Uses the Z39.50 Information Retrieval Protocol, support for SRW, OAI,

SOAP, SDLIP also implementedSOAP, SDLIP also implemented• Server supports a Relational Database GatewayServer supports a Relational Database Gateway• Supports Boolean searching of all serversSupports Boolean searching of all servers• Supports probabilistic ranked retrieval in the Cheshire search engine as Supports probabilistic ranked retrieval in the Cheshire search engine as

well as Boolean and proximity searchwell as Boolean and proximity search• Search engine supports ``nearest neighbor'' searches and relevance Search engine supports ``nearest neighbor'' searches and relevance

feedbackfeedback• GUI interface on X window displays and Windows NTGUI interface on X window displays and Windows NT• WWW/CGI forms interface for DL, using combined client/server CGI WWW/CGI forms interface for DL, using combined client/server CGI

scripting via WebCheshirescripting via WebCheshire• Scriptable clients using Tcl and (new) PythonScriptable clients using Tcl and (new) Python• Store SGML/XML as files or “Datastore” databaseStore SGML/XML as files or “Datastore” database


XML Element ExtractionXML Element ExtractionXML Element ExtractionXML Element Extraction

• A new search “ElementSetName” is A new search “ElementSetName” is XML_ELEMENT_XML_ELEMENT_

• Any Xpath, element name, or regular Any Xpath, element name, or regular expression can be included following the expression can be included following the final underscore when submitting a present final underscore when submitting a present requestrequest

• The matching elements are extracted from The matching elements are extracted from the records matching the search and the records matching the search and delivered in a simple format..delivered in a simple format..


XML ExtractionXML ExtractionXML ExtractionXML Extraction

% zselect sherlock372 {Connection with SHERLOCK (sherlock.berkeley.edu) database 'bibfile' at port 2100 is open as connection #372}% zfind topic mathematics{OK {Status 1} {Hits 26} {Received 0} {Set Default} {RecordSyntax UNKNOWN}}% zset recsyntax XML% zset elementset XML_ELEMENT_Fld245% zdisplay{OK {Status 0} {Received 10} {Position 1} {Set Default} {NextPosition 11} {RecordSyntax XML 1.2.840.10003.5.109.10}} {<RESULT_DATA DOCID="1"><ITEM XPATH="/USMARC[1]/VarFlds[1]/VarDFlds[1]/Titles[1]/Fld245[1]"><Fld245 AddEnty="No" NFChars="0"><a>Singularitâes áa Cargáese</a></Fld245></ITEM><RESULT_DATA> … etc…


Boolean Search CapabilityBoolean Search CapabilityBoolean Search CapabilityBoolean Search Capability• All Boolean operations are supportedAll Boolean operations are supported

– ““zfind author x and (title y or subject z) not subject zfind author x and (title y or subject z) not subject A”A”

• Named sets are supported and stored on the Named sets are supported and stored on the serverserver

• Boolean operations between stored sets are Boolean operations between stored sets are supportedsupported– ““zfind SET1 and subject widgets or SET2”zfind SET1 and subject widgets or SET2”

• Nested parentheses and truncation are supportedNested parentheses and truncation are supported– ““zfind xtitle Alice#”zfind xtitle Alice#”


Probabilistic RetrievalProbabilistic RetrievalProbabilistic RetrievalProbabilistic Retrieval• Uses Logistic Regression ranking method developed Uses Logistic Regression ranking method developed

at Berkeley (W. Cooper, F. Gey, D. Dabney, A. at Berkeley (W. Cooper, F. Gey, D. Dabney, A. Chen) with new algorithm for weigh calculation at Chen) with new algorithm for weigh calculation at retrieval timeretrieval time

• Z39.50 “relevance” operator used to indicate Z39.50 “relevance” operator used to indicate probabilistic searchprobabilistic search

• Any index can have Probabilistic searching Any index can have Probabilistic searching performed:performed:– zfind topic @ “cheshire cats, looking glasses, march hares zfind topic @ “cheshire cats, looking glasses, march hares

and other such things”and other such things”– zfind title @ caucus raceszfind title @ caucus races

• Boolean and Probabilistic elements can be Boolean and Probabilistic elements can be combined:combined:– zfind topic @ government documents and title zfind topic @ government documents and title

guidebooksguidebooks


∑=

+=6

10),|(

iii XccDQRP

Probability of relevance is based onProbability of relevance is based onLogistic regression from a sample set of documentsLogistic regression from a sample set of documentsto determine values of the coefficients (TREC). to determine values of the coefficients (TREC). At retrieval the probability estimate is obtained by:At retrieval the probability estimate is obtained by:

For the 6 For the 6 XX attribute measures shown on the next slide attribute measures shown on the next slideNote that we did NOT retrain the coefficients this yearNote that we did NOT retrain the coefficients this year

Probabilistic Retrieval: Logistic Probabilistic Retrieval: Logistic RegressionRegression

Probabilistic Retrieval: Logistic Probabilistic Retrieval: Logistic RegressionRegression


Probabilistic Retrieval: Logistic Probabilistic Retrieval: Logistic Regression attributesRegression attributes

Probabilistic Retrieval: Logistic Probabilistic Retrieval: Logistic Regression attributesRegression attributes

MX

n

nNIDF

IDFM

X

DLX

DAFM

X

QLX

QAFM

X

j

j

j

j

j

t

t

M

t

M

t

M

t

log

log1

log1

log1

6

15

4

13

2

11

=

−=

=

=

=

=

=

∑

∑

∑ Average Absolute Query FrequencyAverage Absolute Query Frequency

Query LengthQuery Length

Average Absolute Component FrequencyAverage Absolute Component Frequency

Document LengthDocument Length

Average Inverse Component FrequencyAverage Inverse Component Frequency

Inverse Component FrequencyInverse Component Frequency

Number of Terms in common between Number of Terms in common between query and Component -- logged query and Component -- logged


Combining Boolean and Combining Boolean and Probabilistic Search ElementsProbabilistic Search Elements

Combining Boolean and Combining Boolean and Probabilistic Search ElementsProbabilistic Search Elements

• Two original approaches:Two original approaches:– Boolean ApproachBoolean Approach

– Non-probabilistic “Fusion Search” Set merger approach Non-probabilistic “Fusion Search” Set merger approach is a weighted merger of document scores from separate is a weighted merger of document scores from separate Boolean and Probabilistic queries Boolean and Probabilistic queries €

P(R | Q,D) = P(R | Qbool,D)P(R | Qprob ,D)

P(R | Qbool,D) =1: if Boolean eval successful for D

0 : Otherwise

⎧ ⎨ ⎩


Ranking Methods added since Ranking Methods added since INEX ‘02INEX ‘02

Ranking Methods added since Ranking Methods added since INEX ‘02INEX ‘02

• CORICORI -- From Jamie Callan: Simple -- From Jamie Callan: Simple implementation of a weighting scheme for implementation of a weighting scheme for distributed search. Very effective for distributed search. Very effective for distributed search collection selection. Not distributed search collection selection. Not used for official INEX runs.used for official INEX runs.

• OKAPI BM-25OKAPI BM-25 -- From Steve Robertson. -- From Steve Robertson. This is now seems to be the “default” This is now seems to be the “default” retrieval algorithm in experimental IRretrieval algorithm in experimental IR

• New operators (later) let us mix and match New operators (later) let us mix and match ranking methods and Boolean operationsranking methods and Boolean operations


Okapi BM25Okapi BM25Okapi BM25Okapi BM25

• Where:Where:• QQ is a query containing terms is a query containing terms TT• K K is is kk11((1-((1-bb) + ) + b.dlb.dl//avdlavdl))• kk11, b , b and and kk33 are parameters , usually 1.2, 0.75 and 7-1000are parameters , usually 1.2, 0.75 and 7-1000• tftf is the frequency of the term in a specific document is the frequency of the term in a specific document• qtf qtf is the frequency of the term in a topic from which is the frequency of the term in a topic from which QQ was derived was derived• dl dl and and avdl avdl are the document length and the average document length are the document length and the average document length

measured in some convenient unitmeasured in some convenient unit• ww(1) (1) is the Robertson-Sparck Jones weight.is the Robertson-Sparck Jones weight.

∑∈ +

+++

QT qtfk

qtfk

tfK

tfkw

3

31)1( )1()1(

⎟⎠⎞

⎜⎝⎛

++−−+−

⎟⎠⎞

⎜⎝⎛

+−+

=

5.0

5.05.0

5.0

log)1(

rRnN

rnrR

r

w


INEX ‘02 Fusion SearchINEX ‘02 Fusion SearchINEX ‘02 Fusion SearchINEX ‘02 Fusion Search

QueryResults

Sort/Merge

FinalRanked

List

• Merge multiple resultsets and sort new setMerge multiple resultsets and sort new set– Sort by index name/key (ATTRIBUTE)Sort by index name/key (ATTRIBUTE)– Sort by rank (ELEMENTS)Sort by rank (ELEMENTS)

• Merges ranked results and Boolean resultsMerges ranked results and Boolean results

– Sort by XML/SGML Tag contents (TAG)Sort by XML/SGML Tag contents (TAG)


Merging and Ranking OperatorsMerging and Ranking OperatorsMerging and Ranking OperatorsMerging and Ranking Operators• Extends the capabilities of merging to include merger Extends the capabilities of merging to include merger

operations in queries like Boolean operatorsoperations in queries like Boolean operators• Fuzzy Logic Operators (not used for INEX)Fuzzy Logic Operators (not used for INEX)

– !FUZZY_AND!FUZZY_AND– !FUZZY_OR!FUZZY_OR– !FUZZY_NOT!FUZZY_NOT

• Containment operators: Restrict components to or Containment operators: Restrict components to or with a particular parent with a particular parent – !RESTRICT_FROM!RESTRICT_FROM– !RESTRICT_TO!RESTRICT_TO

• Merge OperatorsMerge Operators– !MERGE_SUM!MERGE_SUM– !MERGE_MEAN!MERGE_MEAN– !MERGE_NORM!MERGE_NORM


Query Generation - COQuery Generation - COQuery Generation - COQuery Generation - CO• # 91 TITLE = Internet traffic# 91 TITLE = Internet traffic

• (topicshort @+ {Internet traffic internet, web, traffic, (topicshort @+ {Internet traffic internet, web, traffic, measurement, congestion}) !MERGE_NORM (alltitles measurement, congestion}) !MERGE_NORM (alltitles @+ {Internet traffic}) !MERGE_NORM (kwd @+ @+ {Internet traffic}) !MERGE_NORM (kwd @+ {Internet traffic}) !MERGE_NORM (topicshort @ {Internet traffic}) !MERGE_NORM (topicshort @ {Internet traffic internet, web, traffic, measurement, {Internet traffic internet, web, traffic, measurement, congestion}) !MERGE_NORM (alltitles @ {Internet congestion}) !MERGE_NORM (alltitles @ {Internet traffic}) !MERGE_NORM (kwd @ {Internet traffic})traffic}) !MERGE_NORM (kwd @ {Internet traffic})

• TARGETPATH = XML_ELEMENT_articleTARGETPATH = XML_ELEMENT_article


INEX CO RunsINEX CO RunsINEX CO RunsINEX CO Runs


Query Generation - SCASQuery Generation - SCASQuery Generation - SCASQuery Generation - SCAS

• #66 TITLE = #66 TITLE = /article[./fm//yr < /article[./fm//yr < '2000’] //sec[about(.,'"search engines"')]'2000’] //sec[about(.,'"search engines"')]

• ((date < '2000')) !RESTRICT_FROM ((date < '2000')) !RESTRICT_FROM ((sec_words @ {"search engines"} !((sec_words @ {"search engines"} !MERGE_MEAN (sec_words {$search MERGE_MEAN (sec_words {$search engines$})))engines$})))

• TARGETPATH = XML_ELEMENT_secTARGETPATH = XML_ELEMENT_sec


Query Generation -- SCASQuery Generation -- SCASQuery Generation -- SCASQuery Generation -- SCAS

• This run uses Logistic regression matching This run uses Logistic regression matching combined with Boolean phrase matching combined with Boolean phrase matching and MERGE_MEAN partial result and MERGE_MEAN partial result combinations FUZZY_AND and combinations FUZZY_AND and FUZZY_OR operators were used in FUZZY_OR operators were used in combining AND and OR elements within combining AND and OR elements within an "about" predicate. Containment an "about" predicate. Containment operators were used to constrain component operators were used to constrain component searches within ancestor elements, E.g.:searches within ancestor elements, E.g.:


INEX SCAS RunsINEX SCAS RunsINEX SCAS RunsINEX SCAS Runs

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.


Future PlansFuture PlansFuture PlansFuture Plans

• Bug fixes -- incorrect query generation for some Bug fixes -- incorrect query generation for some SCAS queries, for example…SCAS queries, for example…– TITLE = //article[about(.,'security +biometrics') AND TITLE = //article[about(.,'security +biometrics') AND

about(.//sec,'"facial recognition"')]about(.//sec,'"facial recognition"')]

– Submitted : (topicshort @ {security biometrics} !Submitted : (topicshort @ {security biometrics} !MERGE_MEAN (topicshort @ {biometrics biometrics MERGE_MEAN (topicshort @ {biometrics biometrics biometrics biometrics}) ) !FUZZY_AND (biometrics biometrics}) ) !FUZZY_AND (sec_titlesec_title @ @ {"facial recognition"} !MERGE_MEAN ({"facial recognition"} !MERGE_MEAN (sec_titlesec_title {$facial recognition$})){$facial recognition$}))

– Should have included Should have included sec_words sec_words and Boolean subquery and Boolean subquery for biometrics merged with ranked subqueryfor biometrics merged with ranked subquery


Future PlansFuture PlansFuture PlansFuture Plans

• Add Language Model ranking for Add Language Model ranking for componentscomponents

• Retrain Logistic Regression coefficients on Retrain Logistic Regression coefficients on INEX assessment data -- and experiment INEX assessment data -- and experiment with including new variables, such as with including new variables, such as relative component sizerelative component size

• Find bugs in Okapi BM-25Find bugs in Okapi BM-25• Find more bugs ahead of time, and be more Find more bugs ahead of time, and be more

consistent in runs!consistent in runs!

Documents

December 16, 2003 INEX 2003 -- Ray R. Larson Cheshire II at INEX 2003: Component and Algorithm Fusion Ray R. Larson School of Information Management and