Open Data Repositories and Big Data ARD Prasad DRTC, Indian Statistical Institute Bangalore

Open Data Repositories and

Big Data

ARD PrasadDRTC, Indian Statistical Institute

Bangalore

Our Interest

We are working on (semantics of Big Data)

Structured Data

Hosting data repositories

Metadata of Big data

Ontology

Linked Open Data (LOD) Presently NOT working on

Unstructured Data (especially on social networks)

Data Analytics

Semantic Web

Semantic web can be realized only when the web provides answers to queries than web pages

Scope

The Semantic Web is a Web of Data ... The collection of Semantic Web technologies (RDF, OWL, SKOS, SPARQL, etc.) provides an environment where application can query that data, draw inferences using vocabularies, etc. Linked data (http://www.w3.org/standards/semanticweb/data)

Metadata Of books/Documents (Dublin Core etc.) Products Events Processes Organizations Individuals (FOAF) Keywords (Ontology) Preservation (PREMIS) DATA

Looking Back

Open Access to Information (OAI)

•A Fairly successful movement, resulted in•Open Access Repositories (> 2000)•Open Access Journals (> 5000)

•Partially bridging digital divide in Social, Physical, Natural Sciences and Humanities,

Nature of Publications

Many publications use data. Actual article may not have complete data used

• For lack of space• Author might have overlooked the data• Author deliberately did not present data - so that others can not verify the data

For Example

Some suspect that Sigmund Freud's data is of fictious persons, it is not just fictitious names

If data is available ...

• Others may draw different conclusions contradictory to that of the author

• Others may deal with other facets of the data• Data Transparency supplements the Objectivity

and self corrective characteristics of Science

If “Case history of patients” is openly available, it will contribute significantly to medical research

Digital Divide• Social Sciences do not require laboratory

infrastructure• However, physical and natural sciences do

require expensive infrastructure• If experimental data is available to scientists that

do not have infrastructure, it will significantly reduce digital divide in Physical and Natural Sciences

ODA is a step toward transparency and quality in science

For Example

• Human Genome data• Data from Accelerator Labs (CERN)• Recent controversy about particle moving faster

than light• Not surprisingly, astronomy data is openly

available even before the OA movement

Features of Open Data Repositories

• Metadata: specify who is the owner, creator etc

• license the data to waive your rights to facilitate bulk download Open Data

• Technology Tools: automate data extraction

• Ontology: Index data

Licences

Creative Commons licenses (apart from CCZero), GPL, BSD, etc are NOT quite appropriate for

open data licences

Open Data Licences

• Open Data Commons Public Domain Dedication and Licence (PDDL)

• Dedicate to the Public Domain (all rights waived)• Open Data Commons Attribution License• Attribution for data(bases)• Open Data Commons Open Database

License (OdbL)• Attribution-ShareAlike for data(bases)• Creative Commons CCZero• Dedicate to the Public Domain (all rights waived)

Amazon Web Services (AWS)

Public Data Sets on AWS• Annotated Human Genome Data provided by ENSEMBL– The Ensembl project produces genome databases for human

as well as almost 50 other species, and makes this information freely available.

• Various US Census Databases from The US Census Bureau– Demographic data– US Censuses– Summary information about Business and Industry– Economic Household Profile Data.

• UniGene provided by the National Center for Biotechnology Information

Data Repositories by Governments

Many countries are hosting their data on data.gov in various formats like RDF, XSL, JSON, CSV etc.

Ex: www.data.gov.in (India) Www.data.gov (USA) Www.data.gov.au (Australia) Www.data.gov.uk (UK)

Registry of Data Repositories

Popular Data Registries:- Databib and re3data.org

Databib connects to 978 data repositories and databases (agriculture,Geo-sciences,social Sciences,Biological sciences)

re3data.org currently lists 634 research data repositories from different disciplines and 586 of these are described in detail using the re3data.org schema.

In future, Databib and re3data.org are likely to get merged into one service.

Note: The registry entries provide URL to the data repository and also a brief description of it. Manually one has to visit and download the data from the data repository. Again, no protocol to expose metadata of data providers

Digital Curation

• Collecting verifiable digital assets• Providing digital asset search and retrieval• Certification of the trustworthiness and integrity

of the collection content• Semantic and ontological continuity and

comparability of the collection content• Use of open standards (formats) for term

preservation and future proofing by migration of data

Technology

• Data repositories are much larger than OA repositories

• Cloud Computing is a good solution (AWS uses)• Hadoop

Resource Description in terms of Metadata and Ontology

RDF: Resource Description Framework SKOS: Simple Knowledge Organization

System OWL: Web Ontology Language

SPARQL: SPARQL Protocol and RDF Query Language

NoSQL DBMS

Key / Value Based Redis, MemcacheDB, etc.

Column Based Cassandra, HBase, etc

Document Based MongoDB, Couchbase, etc

Graph Based AllegroGraph, Neo4J, etc.

DBpedia Data Set

Multi-domain ontology derived from Wikipedia 3.77 million “things” (entities - Entitypedia) 400 million “facts” Uses YAGO (Yet Another Great Ontology)

Entitypedia

Multilingual controlled vocabulary Entity matching Data quality and type checking Entity type specific services Semantic or faceted search and navigation on

entities Summarization of entities and concepts

DRTC Projects

Living Knowledge (European Commission funded project on semantic web based on SRR's Analytico Synthetic Classification, completed)

ITPAR: India-Trento Program for Advanced Research

(Govts. Of India & Italy; Work on DERA, Ongoing)

AgInfra: Agriculture data (European Commission funded, Agriculture data, ongoing)

Immediate Plan

An international workshop on Big Data with ICSU/CODATA

Thank You