ZING: the Next Generation of Z39.50

ZING: the Next Generation of ZING: the Next Generation of Z39.50Z39.50

ADLUG Users Group MeetingADLUG Users Group Meeting

San Sebastian, 8-10 September, 2004San Sebastian, 8-10 September, 2004

ÁdámÁdám HorváthHorváth

NSZLNSZL

33 ZING: the Next Generation of Z39.50 - Ádám Horváth - ZING: the Next Generation of Z39.50 - Ádám Horváth - NSZLNSZL

ContentsContents

Why does Z39.50 need renewal Why does Z39.50 need renewal

What is ZINGWhat is ZING

Technical aspectsTechnical aspects

Social contextSocial context

Amicus/LibriVision and ZINGAmicus/LibriVision and ZING

Experimental SRW client of NSZLExperimental SRW client of NSZL


Why does Z39.50 need renewalWhy does Z39.50 need renewal

There are problems with Z39.50There are problems with Z39.50Not popular with the web communityNot popular with the web community– Connection-based sessionsConnection-based sessions– Binary encodingBinary encoding– Transmitted directly over TCP/IPTransmitted directly over TCP/IP

ComplicatedComplicated


The complexity of Z39.50The complexity of Z39.50

11 Native Services11 Native Services– Init, Search, Present, DeleteResultSet, Init, Search, Present, DeleteResultSet,

AccessControl, ResourceControl, AccessControl, ResourceControl, TriggerResourceControl, ResourceReport, TriggerResourceControl, ResourceReport, Scan, Sort, Segment.Scan, Sort, Segment.

7 Extended Services7 Extended Services– PersistentResultSet, PersistentQuery, PersistentResultSet, PersistentQuery,

PeriodicQuerySchedule, ItemOrder, PeriodicQuerySchedule, ItemOrder, DatabaseUpdate, ExportSpecification, DatabaseUpdate, ExportSpecification, ExportInvocationExportInvocation


The complexity of Z39.50The complexity of Z39.50

A 156 page standardA 156 page standardVocabulary from the OSI communityVocabulary from the OSI community– Target, Origin, APDU, A-association, Z-Target, Origin, APDU, A-association, Z-

association...association...Records described with ASN.1Records described with ASN.1Attribute SetsAttribute Sets


Strengths of Z39.50Strengths of Z39.50

Result SetsResult SetsAbstractionAbstraction– Abstract Access Points (Attribute Sets)Abstract Access Points (Attribute Sets)– Abstract Record SchemasAbstract Record Schemas

ExplainExplain


ContentsContents








What is ZING?What is ZING?

ZING: ZING: ZZ39.50 39.50 IInternational: nternational: NNext ext GGenerationeneration

New web based Search and Retrieve New web based Search and Retrieve protocol familyprotocol family

Keeps the strengths of Z39.50Keeps the strengths of Z39.50Eliminates the complexity of Z39.50Eliminates the complexity of Z39.50


What is ZING?What is ZING?

Protocol familyProtocol family– SRW - Search/Retrieve Web ServiceSRW - Search/Retrieve Web Service– SRU - Search and Retrieve URL ServiceSRU - Search and Retrieve URL Service– CQL - Common Query LanguageCQL - Common Query Language– ZeeRex - Z39.50 Explain, Explained and Re-ZeeRex - Z39.50 Explain, Explained and Re-

Engineered in XMLEngineered in XML– ZOOM : the Z39.50 Object Orientation ModelZOOM : the Z39.50 Object Orientation Model– ez3950 : simple implementation over SOAP ez3950 : simple implementation over SOAP

using XERusing XER

Protocol familyProtocol family– SRW - Search/Retrieve Web ServiceSRW - Search/Retrieve Web Service– SRU - Search and Retrieve URL ServiceSRU - Search and Retrieve URL Service– CQL - Common Query LanguageCQL - Common Query Language– ZeeRex - Z39.50 Explain, Explained and Re-ZeeRex - Z39.50 Explain, Explained and Re-

Engineered in XMLEngineered in XML– ZOOM : the Z39.50 Object Orientation ModelZOOM : the Z39.50 Object Orientation Model– ez3950 : simple implementation over SOAP ez3950 : simple implementation over SOAP

using XERusing XER


SRW SRW Search/Retrieve Web ServiceSearch/Retrieve Web Service

What is SRWWhat is SRW– XML oriented protocolXML oriented protocol– Performs searches and other information Performs searches and other information

retrieval operationsretrieval operations


SRW and Z39.50SRW and Z39.50

Z39.50 concepts retained in SRWZ39.50 concepts retained in SRW– Result setsResult sets– Abstract access pointsAbstract access points– Abstract record schemasAbstract record schemas– ExplainExplain– DiagnosticsDiagnostics



SRW differences from Z39.50SRW differences from Z39.50– Result sets named by serverResult sets named by server– No explicit concept of connection, session No explicit concept of connection, session

or stateor state– No distinction between server and databaseNo distinction between server and database– Single record syntax: XMLSingle record syntax: XML• Schemas are XML schemas: DC, MODS, Schemas are XML schemas: DC, MODS,

MarcXMLMarcXML

– Simplified explainSimplified explain



SRW differences from Z39.50 (cont’d)SRW differences from Z39.50 (cont’d)– String queryString query• SRW querySRW query

– <SRW:query>(<SRW:query>(titletitle == information information ANDAND titletitle == retrieval) retrieval) </SRW:query></SRW:query>

• Z39.50 queryZ39.50 query– @and@and @attr 6=1 @attr 3=3 @attr 2=3@attr 6=1 @attr 3=3 @attr 2=3 @attr 1=4@attr 1=4

”information" ”information" @attr 6=1 @attr 3=3 @attr 2=3@attr 6=1 @attr 3=3 @attr 2=3 @attr 1=4@attr 1=4 ”retrieval"”retrieval"


SRW operationsSRW operations

searchRetrieve OperationsearchRetrieve Operation– searchRetrieveRequestsearchRetrieveRequest– searchRetrieveResponsesearchRetrieveResponse

scan Operationscan Operation– scanRequestscanRequest– scanResponsescanResponse

explain Operationexplain Operation– explainRequestexplainRequest– explainResponseexplainResponse


SRW sample search requestSRW sample search request

<SOAP:Envelope <SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">

<SOAP:Body><SOAP:Body>

<SRW:searchRetrieveRequest <SRW:searchRetrieveRequest xmlns:SRW="http://www.loc.gov/zing/srw/">xmlns:SRW="http://www.loc.gov/zing/srw/">

<SRW:version>1.1</SRW:version><SRW:version>1.1</SRW:version>

<SRW:query>(dc.author exact "jones" and dc.title >= <SRW:query>(dc.author exact "jones" and dc.title >= "smith")</SRW:query>"smith")</SRW:query>

<SRW:startRecord>1</SRW:startRecord><SRW:startRecord>1</SRW:startRecord>

<SRW:maximumRecords>10</SRW:maximumRecords><SRW:maximumRecords>10</SRW:maximumRecords>

<SRW:recordSchema>info:srw/schema/1/mods-v3.0</SRW:recordsSch<SRW:recordSchema>info:srw/schema/1/mods-v3.0</SRW:recordsSchema>ema>

</SRW:searchRetrieveRequest></SRW:searchRetrieveRequest>

</SOAP:Body></SOAP:Body>

</SOAP:Envelope></SOAP:Envelope>


SRU SRU Search/Retrieve URL ServiceSearch/Retrieve URL Service

Search requestSearch request– Coded in the URL and sent as HTTP GETCoded in the URL and sent as HTTP GET

ResponseResponse– SRW without the SOAP wrapperSRW without the SOAP wrapper

Example for a search requestExample for a search request

http://www.loc.gov/sru.cgi? http://www.loc.gov/sru.cgi? operation=searchRetrieve&query=dc.title=cat&maximumRecordsoperation=searchRetrieve&query=dc.title=cat&maximumRecords=10&recordSchema=DC&sortKeys=title,dc&startRecord=1=10&recordSchema=DC&sortKeys=title,dc&startRecord=1


CQL - Common Query LanguageCQL - Common Query Language

Loosely based on CCL searchLoosely based on CCL searchCan be simple for every day useCan be simple for every day use– This is a valid query: This is a valid query: programming languageprogramming language

Can be very complexCan be very complex– dc.author=(kern* or ritchie) and (bath.title dc.author=(kern* or ritchie) and (bath.title

exact "the c programming language" or exact "the c programming language" or dc.title=elements prox///4 dc.title=elements prox///4 dc.title=programming) and subject dc.title=programming) and subject any/relevant "style design analysis"any/relevant "style design analysis"



Many shortcutsMany shortcuts– Index and relation can be omittedIndex and relation can be omitted• E.g.: E.g.: “bird dog”“bird dog”• Full version: Full version: cql.serverChoice scr “bird dog”cql.serverChoice scr “bird dog”

– Relationship typesRelationship types• all all

– E.g.: E.g.: dc.title all “bird dog”dc.title all “bird dog”– Full version: Full version: dc.title = “bird”dc.title = “bird” andand dc.title = “dog”dc.title = “dog”

• any any – E.g.: E.g.: dc.title any “bird dog”dc.title any “bird dog”– Full version: Full version: dc.title = “bird”dc.title = “bird” oror dc.title = “dog”dc.title = “dog”



Context setsContext sets– Contexts may be applied to indexes, Contexts may be applied to indexes,

relations, relation modifiers and boolean relations, relation modifiers and boolean modifiers. They are of the form modifiers. They are of the form prefix.valueprefix.value, , where prefix is a short name for a unique where prefix is a short name for a unique context set identifier, defined either in the context set identifier, defined either in the query or by the server.query or by the server.

– PurposePurpose• Make a distinction between two identical index Make a distinction between two identical index

names: names: dc.titledc.title, , foo.titlefoo.title



RelationsRelations– <, >, <=,<, >, <=, and and >=>=

retain their regular meaningsretain their regular meanings– ==• For word adjacency, when the term is a list of For word adjacency, when the term is a list of

words.words.• Otherwise, for exact equality of valueOtherwise, for exact equality of value

– allall, , anyany• ““all of these items" or "any of these items”all of these items" or "any of these items”• E.g.: E.g.: dc.title all “bird dog” dc.title any “bird dog”dc.title all “bird dog” dc.title any “bird dog”



Relations (cont’d)Relations (cont’d)– exactexact• is used for exact string matchingis used for exact string matching

– scrscr• is used to mean "server choice relation”is used to mean "server choice relation”

– within within • may be used with a search term that has multiple may be used with a search term that has multiple

dimensionsdimensions

– enclosesencloses• database's term fully encloses the search termdatabase's term fully encloses the search term



Pattern matchingPattern matching– ?? matches any single character matches any single character– ** matches any sequence of characters matches any sequence of characters

E.g.: E.g.: ?in?s*r?in?s*r – ^̂ word anchoring word anchoring• ^bird^bird must be the first in field must be the first in field• bird^bird^ must be the last in field must be the last in field• ^bird^^bird^ must be the only word in field must be the only word in field


ContentsContents








Bath profile in context setBath profile in context set

Defines indexesDefines indexes– It uses the DC context set when appropriateIt uses the DC context set when appropriate– Creates some bath indexes: bath.issnCreates some bath indexes: bath.issn

Defines relationsDefines relations– exact, =, exact, =, <, <=, >=, >, <, <=, >=, >, withinwithin

Defines usage of wildcards and anchorsDefines usage of wildcards and anchorsDefines schemasDefines schemas– MarcXML, DCMarcXML, DC


DevelopersDevelopers

Principal Editor of the SRW 1.1 Principal Editor of the SRW 1.1 Specification Specification – Rob SandersonRob Sanderson

WSDL/Schema Editor WSDL/Schema Editor – Matthew DoveyMatthew Dovey



Additional Contributors to Version 1.1: Additional Contributors to Version 1.1: – Eliot Christian Eliot Christian – Ray DenenbergRay Denenberg– Adam DickmeissAdam Dickmeiss– Janifer GatenbyJanifer Gatenby– Ralph LeVanRalph LeVan– Mike TaylorMike Taylor– Theo Van VeenTheo Van Veen



Additional Contributors to the Additional Contributors to the Intellectual Development of SRW: Intellectual Development of SRW: – Larry DixsonLarry Dixson– Sebastian HammerSebastian Hammer– Poul Henrik Jorgensen Poul Henrik Jorgensen – Alan KentAlan Kent– Bill MoenBill Moen– Pat StevensPat Stevens


Early implementersEarly implementers

Library of CongressLibrary of CongressKoninklijke Bibliotheek Koninklijke Bibliotheek OCLC/PICA OCLC/PICA OCLC Research OCLC Research Index Data Index Data The European Library projectThe European Library projectNational Széchényi LibraryNational Széchényi Library


ToolsTools

CQL-JavaCQL-Java– CQL parser for JavaCQL parser for Java

Python CQL ParserPython CQL Parser– A Python Library for parsing CQL and A Python Library for parsing CQL and

XCQL. Includes CQL to RPN transformation XCQL. Includes CQL to RPN transformation for PyZ3950for PyZ3950

YAZ Proxy from IndexdataYAZ Proxy from Indexdata– SRU/SRW server to allow any Z39.50 server SRU/SRW server to allow any Z39.50 server

to also support the ZING protocolsto also support the ZING protocols


InformationInformation

http://www.loc.gov/zinghttp://www.loc.gov/zing


ContentsContents








ZING compliant AmicusZING compliant Amicus

Developing a ZING server for AmicusDeveloping a ZING server for Amicus– It could be fully ZING compliantIt could be fully ZING compliant

The use of a YAZ ProxyThe use of a YAZ Proxy– The proxy translates SRW/SRU into Z39.50The proxy translates SRW/SRU into Z39.50– LicenseLicense• Amicus users can use it freely (Amicus users can use it freely (GPL license)GPL license)• LibriCore would need special agreement LibriCore would need special agreement

– Problems in matching CQL and attribute Problems in matching CQL and attribute setssets


CQL and attribute setsCQL and attribute sets

RelationsRelations– ““all” and “any” have no equivalent relation all” and “any” have no equivalent relation

attributeattributeRelation modifiersRelation modifiers– fuzzy has no equivalent relation attributefuzzy has no equivalent relation attribute


CQL and attribute setsCQL and attribute sets

IndexesIndexes– CQL is based on DC indexesCQL is based on DC indexes– Not all of them have USE attribute equivalentsNot all of them have USE attribute equivalents• E.g.: dc.coverage, dc.rights, etc.E.g.: dc.coverage, dc.rights, etc.

Pattern matchingPattern matching– AnchorsAnchors• ‘‘Position last’ has no equivalent position attributePosition last’ has no equivalent position attribute

– Wildcard Wildcard **• CQL search terms like CQL search terms like b*db*d cannot be expressed by cannot be expressed by

truncation attributestruncation attributes


LibriVision as a ZING clientLibriVision as a ZING client

LibriVision was designed as a LibriVision was designed as a multiprotocol productmultiprotocol product

It might be easy to incorporate the SRW It might be easy to incorporate the SRW or SRU protocolor SRU protocol

Tools are availableTools are available It could be developed by LibriCoreIt could be developed by LibriCore


ContentsContents









Web based clientWeb based clientTools usedTools used– YAZ Proxy, PHP, Sablotron, Expat, Java, YAZ Proxy, PHP, Sablotron, Expat, Java,

SaxonSaxonPlatformsPlatforms– WindowsWindows– LinuxLinux



Flow of informationFlow of information

Web

client

Web

client

YA

Z P

roxy

YA

Z P

roxy

Z39.50

Z39.50

serverserver

SRW SearchRetrieve RequestSRW SearchRetrieve Request Z39.50 Search RequestZ39.50 Search Request

SRW SearchRetrieve ResponseSRW SearchRetrieve Response

Z39.50 Present RequestZ39.50 Present Request

Z39.50 Search ResponseZ39.50 Search Response

Z39.50 Present ResponseZ39.50 Present Response



The target databaseThe target database– AmicusAmicus• The Z server used to be ZEDLIB made by The Z server used to be ZEDLIB made by

CrossnetCrossnet– Anonymous login (solved)Anonymous login (solved)– Present problem (solved)Present problem (solved)



Extra featureExtra feature– FRBR display is integrated into the clientFRBR display is integrated into the client

Main purposeMain purpose– To learn about this new protocol in depthTo learn about this new protocol in depth

Live demoLive demo– http://w3.oszk.hu/zingamicus/index.phphttp://w3.oszk.hu/zingamicus/index.php


[email protected]@oszk.hu

Thank you for your attentionThank you for your attention

Technology

ZING: the Next Generation of Z39.50