Oseegenius & Olisuite - Technicalities

Copyright 2011 @CULT. All rights reserved

National Széchényi Library of HungaryMonday, June 15, 2015

Architecture and main characteristics

Copyright 2009-2010 @CULT. All rights reserved

Andrea GazzariniSoftware Architect, @Cult


http://www.atcult.it [email protected]

http://people.apache.org/map.html?person=agazzarini

https://twitter.com/agazzarini

https://www.linkedin.com/in/andreagazzarini

http://andreagazzarini.blogspot.it

https://github.com/agazzarini

http://www.slideshare.net/AndreaGazzarini

https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-essentials


What is OseeGenius? (1/2)

What is ?

A search platformCompletely written in Java using *only* open source components.

An extensible search frameworkEvery module or component has been implemented with extensibility in mind. As consequence of that, the platform provides a lot of extension points for plugging / implementing / enabling custom features.

Search Service Oriented Architecture (SSOA)Search services are requestor-unaware. For instance, they can serve an OPAC, a Library Management System or whatever application that needs search capabilities.


What is OseeGenius? (2/2)

What is ?

CohesiveResponsibilites are clearly defined within each module. That means a high decoupled system where indexing, searching and visualization aremanaged by focused, distinct and decoupled modules.

And last but not least...an OPAC! The Online Public Access Catalog, from a framework perspective, is just a concrete implementation instance of the underlying framework, specifically an instance that manages bibliographic records.

Copyright 2011 @CULT. All rights reserved 5

The OseeGenius Family (1/2)

-I

-S

-Z

-W

(ndexer): The module that holds the indexing logic.

(earcher): The module that exposes search services.

(3950): BATH profile compliant Z3950 proxy / adapter

(eb): End-user front-end application & REST services.


The OseeGenius Family (2/2)

-I--I-Data

-S--S-

-W--W- -Z--Z-

1

Data is indexed in

2 Then it is synchronized in

3

Then it is exposed by means of

4

Offline Index is continuously updated

5

Changes are periodically synchronized within the online index

HTML / HTTP XML / HTTP

CLIENT Z3950

Z3950 / TCP

LMSLMS

The natural evolution of the platform is the integration of the search services (-I- and -S-) with the Library Management System


OseeGenius: Search Services Domain

Search Services Domain


What is OseeGenius -I- ? (1/2)

What is -I ?

The “indexer”The module that manages the underlying persistent storage.It is mainly responsible to manipulate (i.e. add, remove or update) thedata that will be later available for search.

Apache Solr basedThe indexer module is built on top of Apache Solr, the popular IR platform. We added some extension for enhancing indexing capabilities in terms of supported formats like:

- MARC record indexing (file or stream based)- Digital record indexing (e.g. Microsoft Office or LibreOffice docs, PDF)- RDF indexing


What is OseeGenius -I- ? (2/2)

Standard pluggable architectureOseeGenius enhances Solr with its own extensions by exclusively means of Solr provided extension points: that means the running Solr is exactly the same you can download from its website.


Supported formats

MARC: bibliographic records can be manipulated by means of a declarative pre-processing meta-language (e.g. subfields concatenation, trailing punctuation removal);

Rich documents: by means of a dedicated declarative meta-language for manipulating digital documents (e.g. metadata extraction, keyword indexing, fulltext indexing)

Text documents (e.g. XML, HTML, CSV, FLAT files)

RDF: RDF/XML, N-TRIPLES, TRIG, N3, N-QUADS, TURTLE

Database records: RDBMS, OODBMS and NoSQL

Web sites (crawling)

Other content (e.g. Emails, RSS)


Indexing MARC records

The following are just three examples: OseeGenius -I- includes at the moment about 150 extraction functions.

topic = alpha(650:653:654)

“Create a topic attribute on OseeGenius by extracting, from the current MARC record, all the alphabetical subfields of tags 650, 653 and 654.”

“Create a topic attribute on OseeGenius by extracting, from the current MARC record, all the alphabetical subfields of tags 650, 653 and 654.”

topical_subject = value(650a)

“Create a topical_subject attribute on OseeGenius by extracting the raw value of tag 650, subfield a”

“Create a topical_subject attribute on OseeGenius by extracting the raw value of tag 650, subfield a”

title = removeTrailingPunctuation(value(245a))

“Extracts the value of 245$a then remove any trailing punctuation”“Extracts the value of 245$a then remove any trailing punctuation”


What is OseeGenius -S- ? (1/2)

What is -S ?

The “Searcher”The module that exposes search services. It has a “read-only” view of the underlying data, periodically or near-real-timesynchronized with the changes applied by the indexer module.

Apache Solr...OseeGenius -S- is also built on top of Apache Solr. Here, differently from the indexer module, the extensions we added are exclusively related with the search domain. ...again with some extensionFor example, there's a plugin for adding real-time holdings data about a givenset of search results


What is OseeGenius -S- ? (2/2)

Standard pluggable architectureOseeGenius enhances Solr with its own extensions by exclusively means of Solr provided extension points: that means the running Solr is exactly the same you can download from its website.


(Some) Search features

• Simple (google-like) and advanced search• Holdings data• Faceted search• More like this• Spellchecking• Highlighting• Autocompletion• Translation• Geospatial search• Controlled query expansion• MARCXML Output Response Writer


Search Services Sample Scenarios

Sample usage scenarios of search services


Scenario #1: OPAC Search Engine (1/2)

Example usage scenarios of search services

Scenario #1: OPAC (OseeGenius -W-)


Scenario #1: OPAC Search Engine (2/2)

In this scenario:

Data is indexed using OseeGenius -I-

Data becomes searchable by means of OseeGenius -S-

Data becomes accessible to end users or client applications by means of OseeGenius -W- or OseeGenius -Z-

Typical usages

OPAC or in general search and discovery front-end applications

OPAC-to-OPAC integration

Mobile or custom client applications


Scenario #2: Bibliographic Search Engine (1/5)


Scenario #2: Search Engine



INDEX

(245a=History AND 1XX=Potter) OR (260b=Addison Wesley)(245a=History AND 1XX=Potter) OR (260b=Addison Wesley)

SEARCH

000 00694nam a2200241 i 4500008 971205s1997 it j 000 0 ita c020 a 880921191X082 1 a 853.8100 1 a Collodi, Carlo.245 13 a Le avventure di Pinocchio / c C. Collodi ; illustrazioni di A. Mussino260 a Firenze : b Giunti, c 1997.440 0 a Collana favolosa / [Giunti]521 a Letteratura per ragazzi700 1 a Mussino, Attilio.






MARC RECORDS


MARC Query Language (MQL)

(245a=History AND 1XX=Potter) OR (260b=A??is*n Wesle*)(245a=History AND 1XX=Potter) OR (260b=A??is*n Wesle*)

245a=* AND 245c=Potter AND 260b=G?unt?245a=* AND 245c=Potter AND 260b=G?unt?

001=(27892 27290 29900 246923)001=(27892 27290 29900 246923)

001=(27892 OR 27290 OR 29900 OR 246923)001=(27892 OR 27290 OR 29900 OR 246923)

005=[1995 TO 199502231]005=[1995 TO 199502231]

100a=Morgan AND 100e=(interviewer OR collector)100a=Morgan AND 100e=(interviewer OR collector)


Marc Query Language results (example)

Search metadata

Search result (first record)



In this scenario:

Data is indexed using OseeGenius -I-

Data becomes searchable by means of OseeGenius -S-

Data becomes accessible to REST clients

Typical usages

Deep integration with Library Management Systems

Rich client applications

Mobile applications


Scenario #3: Integrated System (1/3)


Scenario #3: Deeply Integrated System



-I--I-

-S--S-

-W--W- -Z--Z-3 Then it is consumed here

HTML / HTTP XML / HTTP

CLIENT Z3950

Z3950 / TCP

LMSLMS

Cataloguing data is indexed here

1

2 Then it is exposed here



In this scenario:

The LMS acts both as data supplier and consumer

Data is available in near-real-time

Some kind of “publication policies” are needed


Search Services Deployment Scenarios

Search Services Deployment Scenarios


Scenario #1: Master / Slave

MASTERMASTER

INDEX 1I

S

Data

Index

Read

SLAVE #1SLAVE #1 SLAVE #2SLAVE #2 SLAVE #3SLAVE #3

Synchronize Synchronize

Search Search

Round-robin

INDEX 1 INDEX 1 INDEX 1

Useful for small / medium organizations Periodical (e.g. daily) updates No Near-Real-Time


Scenario #2: Cloud Mode

CLUSTER

NODE #1NODE #1

SHARD1

SHARD2

I S

L

SHARD3

NODE #2NODE #2

SHARD1

SHARD2 L

SHARD3

NODE #3NODE #3

SHARD1

SHARD2

SHARD3 L

R

R

R

RR

R

ZOOKEEPERENSEMBLE

ZK #1ZK #1

ZK #2ZK #2

ZK #nZK #n

Discover search & index endpoints

High volumes Near real-time Sharding Replication


OseeGenius Frontend Modules

Frontend modules


What is OseeGenius -W- ? (1/3)

What is -W ?

A Web Graphical User InterfaceAt some point, an end-user needs some kind of graphical user interface tointeract with all things we described earlier

A REST-like interfaceAll services that are accessible in the graphical interfaces are also exposedas REST services. REST services are likely to be the optimal way to integrateOseeGenius with mobile or thirdy-party applications.



A pure Java applicationThat means NO restrictions about the hosting operating system. Any OS where Java (6 or later) can be installed is good.

A fully compliant JEE Web ApplicationIt can be deployed on whatever Application Server or Servlet Engine conforming to JEE (4-5) specs. At the moment it has been deployed on:

- JBoss Application Server 5.x / 6.x- Tomcat 6.x – 8.x - Jetty 6.x – 8.x



A pluggable frameworkAlthough it is a web application, it is structured as an extensible framework, meaning with that it provides a lot of extension points, allowing us to plug in your specific search & business logic.Features, in OseeGenius -W- can be activated and deactived by system administrator simply changing configuration. Some examples:

• Holdings data viewer• More Like This viewer• Cover providers chain• Third party extensions (e.g. Syndetics)• Account management• Circulation (loans, holds, fines) • Browsing (plain indexes and authorities)• Community (e.g. Tags, reviews, bibliography)


Skins


Simple Search

The most immediate kind of search: just type something in a text field...

...and get the corresponding results...


Advanced Search


Holdings data


Cover Provider Chain

Internal Cover Provider

Syndetics

OpenLibrary

Google Books

Not Available Provider

Find the cover locally

I have a Syndetics subscription, give me the cover

OpenLibrary, could you give me this cover?

Google, I need this cover, please...

Not found?

Not found?

Not found?

Not found?

Display a “Not available” image...


Highlighting (1/2)


Highlighting (2/2)


Autocompletion

Context-sensitive Suggested voices / headings depend on the selected index. For instance if you're looking for authors, OseeGenius will propose only names that match / start with / contain the user entered terms

“Starts with” or “contains” The matching algorithm is fully customizable


Tabbed Search (1/3)

What is it?A feature inherited from all most popular internet browsers like Firefox, Chrome and Internet Explorer. It allows users to maintain several searches by opening different tabs.

Local or FederatedOseeGenius tabs can hold local or federated search results;

Open, close and reopen laterYou can have unlimited tabs in a given session; they can be closed and reopened later;

Dedicated contextEach tab wraps a search context with its specific capabilities (e.g. RSS feeds, social share, faceting, navigation, federated browsing, export, print, send by email)


Tabbed Search (2/3)


Tabbed Search (3/3)


Faceted Search (1/3)

Faceted search improves the user search experience by enabling aggregation on top of search results. That allows users to have an immediate view about things like:

• How many books are written in Italian?• What are the most 10 popular geographical subjects in my catalog?• What are the most relevant publication date ranges in my catalog?






Federated Search

What is it?The federated search add-on enables OseeGenius to search across external search engines, get and merge the incoming results. External resources are always accessed within the OseeGenius -W- interface, delivering to the end users a common layout and graphical user interfaces to work with.

How?OseeGenius -W- includes a proxy towards PazPar2 by IndexData, the most popular open-source and web-oriented metasearching middleware. User can create search tabs and fill them with results coming from a predefined (at installation and configuration time) set of external catalogs.

What kind of catalogs can be integrated?Targets can be theoretically everything. Pazpar2 allows you to plug-in standard Z39.50, SRU and SRW endpoints but also non standard resources by using a sophisticated screen-scraping approach (this specific feature requires a subscription)


Third party content

What is it? Allows document information enrichment from third party vendors (e.g. Syndetics). Note that usually those additional content is not free of charge so a valid subscription is required in those cases;

Where?Additional data is usually inserted in each document detail, but it can be theoretically put everywhere

Asynchronous loadingAdditional content is loaded (where possible) asynchronously. That's because users, typically, don't like to wait too much in front of a blank page;

Integration is a matter of compromiseAdditional content services are different from vendor to vendor, so the specific integration needs to be discussed as it is strictly dependent from the vendor API and customer requirements.


Classification Browsing (1/4)

What is it?A navigation component that helps in browsing an arbitrary classification system. By default OseeGenius comes with a Dewey Decimal Classification browser; other decimal classifications (e.g Library of Congress Call Number) can be easily plugged-in;

How does it look like?It is (usually) represented by a tree component where each node is a Dewey class. Labels on nodes are internazionalizable.

What does the tree contain?It contains only valid classes: that means there are no empty nodes. At tree building time, classes that don't have any document associated are removed;

Standard or custom classification systems Custom classification systems can be also plugged-in but require a customisation








View Perspectives

What is it?The same result lists can be see under different perspectives:

Short:few attributes per record Long: many attributes per record Grouped: similar records are grouped MARC: the underlying record MARC is shown


Export, download or send

What is it?Bibliographic records (i.e. search results) in the user interface can be selected, exported or downloaded.

Exported?Selected records can be exported to services like RefWorks or EndNote;

Downloaded?Selected records can be downloaded in several formats like: MARC (binary or XML), Excel, PDF. As additional feature, records can be also sent by email

...and instead of downloading a bunch of records... OseeGenius -W- interface is completely stateless; it offers a REST-like interface so any page you see is uniquely identified by a URL. If you type that URL whenever and wherever you want, you will get back always the same page.


Internationalization

i18nAll labels in OseeGenius are internationalized;

Italian and EnglishTwo languages (italian and english) are built-in;

New languagesLanguages can be plugged-in easily; just a matter of translation;

Which language?When users open their browser with OseeGenius, the selected language will be determined by their locale; after that, using the i18n widget toolbar each user can choose and change his preferred language;


OseeGenius -W- Plugins

Plugins are an important part of the OseeGenius -W- architecture. They can be considered as its extension points, letting us to implement and extends the application with new features.

The plugin architecture separates between interface and implementations so the first declares and abstract behaviour while an implementor wraps the concrete logic for doing a certain task.

The following pages show a showcase example of the Account plugin, which is the plugin that enables users login, logout, private area and so on.


User Area: Login


User Area: Workspace


User Area: Reviews


User Area: Wish Lists


What is OseeGenius -Z- ?

What is -Z ?

Z39.50It is the Z39.50 interface of OseeGenius. It is a module that acts as a Z39.50 endpoint fully compliant with BATH 2.0 specification, extending Osee Genius -S- connectivity by providing that additional protocol so popular among library systems.

Pure Java standalone serverOseeGenius -Z- is completely written in Java so doesn't require any native component. It can be started on any OS / Platform where Java can run.

...just a maskThis module is just an adapter so It doesn't have any search capability. Search logic is completely hold in OseeGenius -S-


Olisuite: Technical Overview

Technical Overview


Fully compliant JEE applicationThe Java EE technology is the de-facto standard for delivering secure, robust, scalable multi-platform applications and services.

Three-tiered In Olisuite the presentation, application processing, and data management layers are physically and logically separated. That allows a high level of cohesion and decoupling of the overall system.

Leverage of JEE containers for non functional requirementsNon functional requirements are a crucial concern of every context, regardless the owning institution is a small, medium or a big enterprise. Instead of dealing with those requirements, Olisuite delegates them to the hosting JEE container. JEE technology greatly addresses concerns such as transaction management, scalability, high availability, load balancing and security.

Olisuite: technical overview (1/2)


Proven open source technologies: Enterprise Java Beans Java Server Faces (JSF) Apache Struts Apache Solr Various Apache frameworks (e.g. Hibernate, Logging)

Three-tiered Every module or component has been implemented with extensibility in mind. As consequence of that, the platform provides a lot of extension points for plugging / implementing / enabling custom features

Tested in production with 2 different JEE ServersAlthough Olisuite can be deployed in any JEE 5 compliant Application Server, we already tested two production deployments using JBoss 5.x /6.x Application Server and Oracle Weblogic 10.3.x

Olisuite: technical overview (2/2)


Logic LayerLogic Layer

Services LayerServices Layer

Integration LayerIntegration Layer

Domain ModelDomain Model

Presentation LayerPresentation Layer

RDBMS

RMI / IIOP

HTTP

Self-check stationSelf-check station

3M

JDBC

SOAP

WS ClientWS Client

Olisuite: context diagram


Deployment Scenario #1: single machine

Olisuite Deployment Scenarios

Scenario #1: single machine






RDBMS

RMI / IIOP

HTTP


3M

JDBC

SOAP

WS ClientWS Client

Deployment Scenario #1: single machine


Deployment Scenario #2: clustered logic layer


Scenario #2: clustered logic layer


LOGIC LAYER CLUSTER

Node #1Node #1





RMI / IIOP

HTTP


3M

SOAP

WS ClientWS Client

Node #2Node #2




Node #nNode #n



Domain ModelDomain Model...

Deployment Scenario #2: clustered logic layer


Deployment Scenario #3: clustered service


Scenario #2: clustered service


LOGIC LAYER CLUSTER

Node #1Node #1




RMI / IIOP


3M

SOAP

WS ClientWS Client

...

Node #2Node #2




Node #2Node #2




PRESENTATION LAYER CLUSTER

Node #1Node #1 Node #2Node #2 Node #nNode #n...

LOAD BALANCER

HTTP

Deployment Scenario #3: clustered service



Architecture and main characteristics

Thank you!