LOD2 Webinar: SIREn

LOD2 Webinar . 24.06.2014 . Page 1 http://lod2.eu

Creating Knowledge out of Interlinked Data



http://lod2.eu

LOD2 is a large-scale integrating project co-funded by the European Commission within the FP7 Information and Communication Technologies Work Programme. This 4-year project comprises leading Linked Open Data technology researchers, companies, and service providers. Coming from across 12 countries the partners are coordinated by the Agile Knowledge Engineering and Semantic Web Research Group at the University of Leipzig, Germany.

LOD2 will integrate and syndicate Linked Data with existing large-scale applications. The project shows the benefits in the scenarios of Media and Publishing, Corporate Data intranets and eGovernment.



http://lod2.eu

Once per month the LOD2 webinar series offer a free webinar about tools and services along the Linked Open Data Life Cycle.

Stay with us and learn more about acquisition, editing, composing, connected applications – and finally publishing Linked Open Data.



• Nested Data Model• SIREn Overview• Getting Started with the SIREn Elasticsearch Plugin• Demo

Agenda



• Model becoming prevalent: JSON, XML, Avro, …– Can be arbitrarily nested and large

– No strict schema / structure enforced

• Schema-less brings– Flexibility

– Ease of development

• Developers do not have to invest significant modelling effort upfront

Schema-Less Nested Data Model



• Lucene, Solr and Elasticsearch plugin for indexing and searching JSON

• Rich data model (JSON)– Nested objects, nested arrays, datatypes

– Generic architecture compatible with various nested data models: JSON, JSON-LD, XML, Avro, ...

• Schema-agnostic– SIREn does not require any schema definition to index and search data

– Schema definition can change across records

• Designed from the ground up for high performance and scalability

Introducing SIREn



Introducing SIREn



• Document-oriented search and analytics engine– JSON Document– Based on Apache Lucene

• Distributed, Replication– High Performance and Availability

• REST API

Elasticsearch - Overview



• Index = Collection of Documents– Can have multiple shards and replicas

• Type = A set of documents sharing the same schema– Similar to a DB table

• Document = JSON object– Uniquely identified (index/type/id)– Similar to a DB record

Elasticsearch – Basic Concepts



SIREn – Basic Concepts

• JSON object = Tree– Different mapping available

• Node = An element of the tree– Can have a parent and one or more children– Contains data: text, numeric, boolean



SIREn – Basic Concepts

JSON Object Tree

{ "name" : "Elasticsearch", "category_code" : "search", "funding_rounds" : [ { "round_code" : "a", "raised_amount" : 10000000, "funded_year" : 2012, "investments" : [ { "name" : "Data Collective", "type" : "financial-org" }, … ] }, … ] }

{ "name" : "Elasticsearch", "category_code" : "search", "funding_rounds" : [ { "round_code" : "a", "raised_amount" : 10000000, "funded_year" : 2012, "investments" : [ { "name" : "Data Collective", "type" : "financial-org" }, … ] }, … ] }

name : Elasticsearch

funding_rounds

round_code : a

raised_amount : 10000000

funded_year : 2012

investments

name : Data Collective

type : financial-org

category_code : search

investments

name : ...



http://sirendb.com/downloads/

(Elasticsearch Distribution Coming Soon)

Getting Started with Elasticsearch & SIREn



SIREn Elasticsearch Distribution

├── dist│ ├── siren-core-1.3.jar│ └── siren-qparser-1.3.jar├── docs│ ├── apidoc│ └── siren-1.3-javadoc.jar├── example├── CHANGES.txt├── LICENSE.md├── README.md└── THIRD-PARTY.txt

├── dist│ ├── siren-core-1.3.jar│ └── siren-qparser-1.3.jar├── docs│ ├── apidoc│ └── siren-1.3-javadoc.jar├── example├── CHANGES.txt├── LICENSE.md├── README.md└── THIRD-PARTY.txt

SIREn LibrariesSIREn

Libraries

SIREn JavadocSIREn

Javadoc

Elasticsearch Distribution

Elasticsearch Distribution



SIREn Elasticsearch Distribution

├── bin│ ├── elasticsearch│ └── ...├── config│ ├── elasticsearch.yml│ └── logging.yml├── lib│ ├── elasticsearch-1.2.0.jar│ └── ...├── plugins│ └── siren-plugin│ ├── siren-elasticsearch-1.3.jar│ └── ...├── datasets├── NOTICE.txt└── README.md

├── bin│ ├── elasticsearch│ └── ...├── config│ ├── elasticsearch.yml│ └── logging.yml├── lib│ ├── elasticsearch-1.2.0.jar│ └── ...├── plugins│ └── siren-plugin│ ├── siren-elasticsearch-1.3.jar│ └── ...├── datasets├── NOTICE.txt└── README.md

ExecutablesExecutables

ConfigurationConfiguration

SIREn PluginSIREn Plugin



Starting Elasticsearch

$ ./bin/elasticsearch

[INFO ][node ] [Valerie Cooper] version[1.2.0], pid[12842], …[INFO ][node ] [Valerie Cooper] initializing ...[INFO ][plugins] [Valerie Cooper] loaded [siren-plugin], sites [][INFO ][node ] [Valerie Cooper] initialized[INFO ][node ] [Valerie Cooper] starting ...…[INFO ][node ] [Valerie Cooper] started

$ ./bin/elasticsearch

[INFO ][node ] [Valerie Cooper] version[1.2.0], pid[12842], …[INFO ][node ] [Valerie Cooper] initializing ...[INFO ][plugins] [Valerie Cooper] loaded [siren-plugin], sites [][INFO ][node ] [Valerie Cooper] initialized[INFO ][node ] [Valerie Cooper] starting ...…[INFO ][node ] [Valerie Cooper] started



Create an Index and Map SIREn’s Field

$ curl -XPUT 'http://localhost:9200/test/' –d '{ "mappings" : { "companies" : { "properties" : { "_siren_source" : { "index" : "analyzed", "analyzer" : "hybrid-json", "postings_format" : "Siren10AFor", "store" : "no", "type" : "string" } }, "_siren" : {} } }}'

$ curl -XPUT 'http://localhost:9200/test/' –d '{ "mappings" : { "companies" : { "properties" : { "_siren_source" : { "index" : "analyzed", "analyzer" : "hybrid-json", "postings_format" : "Siren10AFor", "store" : "no", "type" : "string" } }, "_siren" : {} } }}'

SIREn’s Field

SIREn’s Analyzer

SIREn’s Posting Format



Indexing a Document

$ curl -XPUT 'http://localhost:9200/test/companies/1' –d '{ "name" : "Elasticsearch", "category_code" : "search", "funding_rounds" : [ { "round_code" : "a", "raised_amount" : 10000000, "funded_year" : 2012, "investments" : [ { "name" : "Data Collective", "type" : "financial-org" } ] } ]}'

$ curl -XPUT 'http://localhost:9200/test/companies/1' –d '{ "name" : "Elasticsearch", "category_code" : "search", "funding_rounds" : [ { "round_code" : "a", "raised_amount" : 10000000, "funded_year" : 2012, "investments" : [ { "name" : "Data Collective", "type" : "financial-org" } ] } ]}'



• SIREn distinguishes 5 types of values in the JSON model:– Field, String, Long, Double, Boolean

Customising Datatype Analysis

{ "name" : "Elasticsearch", "category_code" : "search", "funding_rounds" : [ { "round_code" : "a", "raised_amount" : 10000000, "funded_year" : 2012, "investments" : [ { "name" : "Data Collective", "type" : "financial-org" } ] } ]}

{ "name" : "Elasticsearch", "category_code" : "search", "funding_rounds" : [ { "round_code" : "a", "raised_amount" : 10000000, "funded_year" : 2012, "investments" : [ { "name" : "Data Collective", "type" : "financial-org" } ] } ]}

json:field

xsd:long

xsd:string



Customising Datatype Analysis

# config/elasticsearch.yml

siren.analysis.datatype:http://json.org/field:

index_analyzer: keywordhttp://www.w3.org/2001/XMLSchema#string:

index_analyzer: simplesearch_analyzer: simpleFolding

http://www.w3.org/2001/XMLSchema#long:index_analyzer: long

index.analysis.analyzer:simpleFolding:

type: customtokenizer: letterfilter: [lowercase, asciifolding]

# config/elasticsearch.yml

siren.analysis.datatype:http://json.org/field:

index_analyzer: keywordhttp://www.w3.org/2001/XMLSchema#string:

index_analyzer: simplesearch_analyzer: simpleFolding

http://www.w3.org/2001/XMLSchema#long:index_analyzer: long

index.analysis.analyzer:simpleFolding:

type: customtokenizer: letterfilter: [lowercase, asciifolding]



Searching: Node Query

$ curl -XGET 'localhost:9200/test/companies/_search' –d '{ "query" : { "hybrid-json" : { "node" : { "query" : "search", } }}'



funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...

SIREn’s Query Type

SIREn’s Query

Full-Text Query:•Boolean, Phrase, Range, Fuzzy, Regexp, Proximity, ...







funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...




$ curl -XGET 'localhost:9200/test/companies/_search' –d '{ "query" : { "hybrid-json" : { "node" : { "query" : "search", "attribute" : "category_code", } }}'



funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...







funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...



• Query operators for Ancestor-Descendant and Parent-Child relationships

Searching: Twig Query




• Consists of a root query


Boolean




• Consists of a root query and one or more child


Boolean

PhraseMUST




• Consists of a root query and one or more child and descendant queries


Boolean

PhraseMUST

BooleanSHOULD




• Consists of a root query and one or more child and descendant queries

• Can be nested to form complex tree structure


Boolean

PhraseMUST

TwigNOT

RangeMUST

BooleanSHOULD




$ curl -XGET 'localhost:9200/test/companies/_search' –d '{ "query" : { "hybrid-json" : { "twig" : { "root" : "funding_rounds" "child" : [{ "occur" : "must", "node" : { "query" : "2012", "attribute" : "funded_year" } }] } }}'



funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...

Twig Query Operator Root Query

Child Query







funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...







funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...







funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...







funding_rounds

round_code : a


funded_year : 2012

investments




investments

name : ...



Searching: Proximity Query

$ curl -XGET 'localhost:9200/test/companies/_search' –d '{ "query" : { "hybrid-json" : { "boolean" : { "inOrder" : true, "slop" : 0, "clause" : [{ "twig" : { "root" : "funding_rounds", "descendant" : [{ "level" : 2, "node" : { "query" : "Data AND Collective" } }] } },{ "twig" : { "root" : "funding_rounds", "descendant" : [{ "level" : 2, "node" : { "query" : "Index AND Ventures" } }] } }]}}}}'



funding_rounds

investments




funding_rounds

investments

name : Index Ventures


Order Constraint

33

2211

44

Proximity Constraint

Boolean Clauses







funding_rounds

investments




funding_rounds

investments



33

2211

44







funding_rounds

investments




funding_rounds

investments



33

2211

44







funding_rounds

investments




funding_rounds

investments



33

2211

44



Demo



• SIREn’s Key Feature– Dynamic/Schema-less Data Management– Nested Data– High performance and scalability– Powerful search operators– Elasticsearch/Solr integration

• Contact– http://sirendb.com– SindiceTech– [email protected]

Conclusion



Credits

Jingle R.E.M., Martin Kaltenböck, Florian Kondert

Coordination Thomas Thurner

Martin Kaltenböck

Moderation Martin Kaltenböck

Presented by Renaud Delbru, Harish Kumar



http://lod2.eu

Hope you enjoyed staying with us – if you need more detailed information, visit us at www.lod2.eu and let us know how we can improve to meet your expectations!

Don’t forget to register for our next webinar

20.12. 2011 - Virtuoso (Open Link Software) 24.01. 2012 - OntoWiki (University of Leipzig, Germany)

Have a great day and don’t forget ...



http://lod2.eu

Technology

LOD2 Webinar: SIREn