Tutorial on grid-powered data aggregation and accessing datasets

12/12/2013Athens, Greece

Open Data in Agriculture

Hands-on with data infrastructures that can power your agricultural data

products

Supported by EU projects

Nikos ManolisAgro-Know Technologies

Tutorial on grid-powered data aggregation and accessing datasets

Slide 3 of 63

There is a lot of data

Slide 4 of 63

Need for data aggregation and harmonization

Slide 5 of 63

Objectives

This presentation aims to provide information on:

How to use a service for aggregating datasets How to get already processed datasets How to search processed datasets with a search API

• Educational – GLN API (21008 metadata recs)• Bibliographic – ABN API (451602 metadata recs)

Slide 6 of 63

The agDataHarvester service

• Implements the OAI-PMH protocol to harvest metadata records from open data providers– REST-based API– Harvested dataset available through HTTP

Slide 7 of 63

agDataHarvester parameters

{ "document_type": "harvesting_target", "harvesting_target": { "name":"Repository name", "description":”Short Repository Description", "url":"OAI-PMH target URL", "type":"metadata format prefix", "frequency":hours }}

Slide 8 of 63

param.json

{ "document_type": "harvesting_target", "harvesting_target": { "name":"Indian Academy of Science", "description":"Indian Academy of Science", "url":"http://repository.ias.ac.in/cgi/oai2", "type":"mets", "frequency":24 }} curl -X POST -d@INDUS.json http://'demo001':aginfra@agro.ipb.ac.rs/agcouchdb

curl -X POST -d@param.json http://'demo001':aginfra@agro.ipb.ac.rs/agcouchdb{ "ok": true, "id": " 5c56a3fa18fa21d2a85fd63cc9eb78ac ", "rev": "1-

19ef1210376df8f1695a32b53ecb963a" }

Slide 9 of 63

Get details on the datasethttp://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list?dataset.process_parameter_id=5c56a3fa18fa21d2a85fd63cc9eb78ac

Slide 10 of 63

Get details on the datasethttp://agro.ipb.ac.rs/agcouchdb/_design/datasets/_view/list_by_process?key=agdataharvester

{"id": "6796259b52d79e4797e210c06e6a0aee","key": "6796259b52d79e4797e210c06e6a0aee","value": {

"_id": "6796259b52d79e4797e210c06e6a0aee","_rev": "1-d55d7bc90d26db64dae328c9328e4e4a","document_type": "harvesting_target","harvesting_target": {

"name": “WorldBank","description": "The World Bank - Open Knowledge Repository","url": ""https://openknowledge.worldbank.org/oai/request","type": “mets","frequency": 24

},"document_publisher": {

"address": "83.212.96.169","author": "demo001","utc_datetime": "Wed Dec 11 11:58:45 2013","utc_timestamp": 1386763125

}}}

Slide 11 of 63

The agWorkflow service

http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list?dataset.process=agworkflow&dataset.type=oai_lom&dataset.accuracy=true

I want all datasets with educational resources processed by the agINFRA powered aggregation workflow !

http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list?dataset.process=agworkflow&dataset.type=oai_agris&dataset.accuracy=true

I want all datasets with bibliographic resources processed by the agINFRA powered aggregation workflow !

Slide 12 of 63

Is there a way to search on available datasets ?

Slide 13 of 63

Search API

• REST-based queries over harmonized information (result of metadata processing)

• Two data models supported – akif: describing educational resources for

agriculture, http://domain/search-api/v1/akif/?q=*

– agrif: describing bibliographic resources for agriculture (mainly from FAO’s data), http://domain/search-api/v1/agrif/?q=*

Slide 14 of 63

Search options

• Simple searchhttp://domain/search-api/v1/akif/?q=tomato

• Searching within specific fieldshttp://BASE_URL/search-api/v1/akif/?

languageBlocks.en.description=tomato

• Temporalhttp://BASE_URL/search-api/v1/akif/?creationDate=2013-04-16

• Fetching specific items http://BASE_URL/search-api/v1/akif/COLLECTION/20296

Slide 15 of 63

Managing results

• Sorting resultse.g ?q=*&sort_by=creationDate&sort_order=desc

• Facetse.g ?facets=set&facet_size=3

• Paginatione.g ?q=sea&page_size=25&page=3

Nikos Manolis

Agro-Know Technologies

manolisn@agroknow.gr