Open Data and Linked Data

  • Published on
    15-Apr-2017

  • View
    12.706

  • Download
    0

Transcript

  • Open Data and Linked Datathe what and how of linked open data

    James G. Boram KimLiST Inc.

    JGKim@LiSTInc.kr

    March 5th, 2016

  • Prologue

  • Photography: Jason Madara, WIRED UK 02:13, 2013.

    Tim OReillyFounder and CEO, OReilly Media

  • Data is the Next Intel Inside.

    Photography: Jason Madara, WIRED UK 02:13, 2013.

    Tim OReillyFounder and CEO, OReilly Media

    Every significant Internet application to date has been backed by a specialized database. [] Much as the rise of proprietary software led to the Free Software movement, we expect the rise of proprietary databases to result in a Free Data movement within the next decade. What is Web 2.0, Sep. 2005.

  • Tim OReillyFounder and CEO, OReilly Media

    John BattelleCEO, Co-founder, and Chairman,

    NewCo

    Photography: James Duncan Davidson, Web 2.0 Summit, 2010.

  • Data is the Intel Inside of the Next Generation of Applications

    Tim OReillyFounder and CEO, OReilly Media

    John Battelle CEO, Co-founder, and Chairman,

    NewCo

    Photography: James Duncan Davidson, Web 2.0 Summit, 2010.

    Collective intelligence applications depend on managing, understanding, and responding to massive amounts of user-generated data in real-time. The subsystems of the emerging Internet operating system are increasingly data subsystems: location, identity (of people, products, and places), and the skeins of meaning that tie them together. This leads to new levers of competitive advantage: Data is the Intel Inside of the next generation of computer applications. Web Squared: Web 2.0 Five Years On, Oct. 2010.

  • Infographic: visually, http://visual.ly/open-data-movement, 2011.

    http://visual.ly/open-data-movement

  • Barack Obama44th President of the United States

    Photography: Kevin S. OBrien, U.S. Navy, 2009.

  • Open Government

    Barack Obama44th President of the United States

    Photography: Kevin S. OBrien, U.S. Navy, 2009.

    My administration is committed to creating an unprecedented level of openness in Government. We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration. Openness will strengthen our democracy and promote efficiency and effectiveness in Government. Memorandum for the Heads of Executive Departments and Agencies, Transparency and Open Government, Jan. 2009.

  • Tim OReillyFounder and CEO, OReilly Media

    Photography: Eric Laycock, Esri, 2011.

  • Government as a Platform

    Tim OReillyFounder and CEO, OReilly Media

    Photography: Eric Laycock, Esri, 2011.

    This is the right way to frame the question of Government 2.0. How does government itself become an open platform that allows people inside and outside government to innovate? How do you design a system in which all of the outcomes arent specified beforehand, but instead evolve through interactions between the technology provider and its user community? [] Thats Government 2.0: technology helping build the kind of government the nations founders intended: of, for and by the people. Gov 2.0: The Promise of Innovation, Forbes, Aug. 2009.

  • Todd Park2nd United States Chief Technology Officer

    Photography: U.S. Department of Labor, 2012.

  • Open Data Policy

    Todd Park2nd United States Chief Technology Officer

    Photography: U.S. Department of Labor, 2012.

    Making information resources accessible, discoverable, and usable by the public can help fuel entrepreneurship, innovation, and scientific discovery all of which improve Americans lives and contribute significantly to job creation. Sylvia M. Burwell, Steven VanRoekel, Todd Park, and Dominic J. Mancini, Memorandum for the Heads of Executive Departments and Agencies, Open Data Policy Managing Information as an Asset, May. 2013.

  • Joel GurinPresident and Founder,

    Center for Open Data Enterprise

    Photography: Techonomy, 2014.

  • Open Data Movement

    Joel GurinPresident and Founder,

    Center for Open Data Enterprise

    Photography: Techonomy, 2014.

    The Open Data movement began with democratic goals, fuelled by the idea that governments should make the data they collect available to the taxpayers whove paid to collect it. But in addition to its social benefits, Open Data has created tremendous new business opportunities. Open Data Now, McGraw-Hill Education, 2014.

  • Why Open Data? Open Data and Social Impact

    Sketchnote: Open Government Partnership, 2013.

  • Why Open Data? Driving Growth, Ingenuity, and Innovation

    Sketchnote: Open Government Partnership, 2013.

    Data is the new capital of the global economy, and as organisations seek renewed growth, stronger performance and more meaningful customer engagement, the pressure to exploit data is immense. [] As a result, we foresee that open data, and not simply big data, will be a vital driver for growth, ingenuity and innovation in the UK economy. There are four key aspects to our vision:

    1. Every business wil have a strategy to exploit the rapidly growing estate of open data. 2. Businesses will increasingly open up their data to revolutionise the way they compete. 3. Businesses will use open data to inspire customer engagement. 4. Businesses will work with the Government to establish a new paradigm in data

    responsibility and privacy. Open data: Driving growth, ingenuity and innovation, Deloitte, 2012.

  • Why Open Data? Large Amount of Economic Value

    Making data more liquid (open, widely available, and in shareable formats) has the potential to unlock large amount of economic value (approx. $3 trillion annually), by improving the efficiency and effectiveness of existing processes; making possible new products, services, and markets; and creating value for individual consumers and citizens. Open data: Unlocking innovation and performance with liquid information, McKinsey Global Institute, Oct. 2013.

    McKinsey & CompanyMcKinsey Global Institute

    More open data for more users . . .

    40+Number of countries with government open data platforms*

    90,000+Data sets on data.gov (US site)*

    1.4 millionPage views for the UK open data site in the summer of 2013

    102Cities that participated in 2013 International Open Data Hackathon Day

    1 million+Data sets made open by governments worldwide

    * As of 2013

  • Why Open Data? Large Amount of Economic Value

    While sources differ in their precise estimates of the economic potential of Open Data, all are agreed that it is potentially very large. In countries which were early movers in Open Data, there is already evidence of significant businesses having developed to exploit that potential. Leading governments have recognised that their role is not simply to publish data they are supporting the whole value chain of the use of data []. Open Data for Economic Growth, The World Bank, Jun. 2014.

    The World BankIBRD IDA

  • Screenshot: Open Data 500, http://www.opendata500.com/.

    http://www.opendata500.com/

  • Screenshot: Open Data 500, http://www.opendata500.com/.

    http://www.opendata500.com/

  • Screenshot: Open Data 500, http://www.opendata500.com/.

    http://www.opendata500.com/

  • Joel GurinPresident and Founder,

    Center for Open Data Enterprise

    Photography: The GovLab, 2013.

  • Joel GurinPresident and Founder,

    Center for Open Data Enterprise

    Defining Data Categories

    OPEN DATABusiness Reporting And

    Other Business Data(e.g., ESG data and comsumer complaints)

    BIG DATA OPEN GOV

    Non-Public Data

    for marketing, business analysis, national security

    CitizenEngagement

    Programsnot based on

    data (e.g., petition

    websites)

    Large Datasetsfrom scientific research, social media, or other non-government

    sources

    Public Datafrom state, local,

    federal government (e.g., budget

    data)Large Public Government

    Datasets(e.g., weather, GPS, Census,

    SEC, healthcare)

    Photography: The GovLab, 2013.Diagram: From Joel Gurin, Open Data Now, McGraw-Hill Education, 2014.

  • Definition

  • Rufus PollockPresident and Co-Founder

    Open Knowledge (Foundation)

    Photography: Sebastiaan ter Burg, http://www.flickr.com/photos/31013861@N00/14860905785/, 2014.

    http://www.flickr.com/photos/31013861@N00/14860905785/

  • Rufus PollockPresident and Co-Founder

    Open Knowledge (Foundation)

    Photography: Sebastiaan ter Burg, http://www.flickr.com/photos/31013861@N00/14860905785/, 2014.

    http://www.flickr.com/photos/31013861@N00/14860905785/

  • Rufus PollockPresident and Co-Founder

    Open Knowledge (Foundation)

    Photography: Sebastiaan ter Burg, http://www.flickr.com/photos/31013861@N00/14860905785/, 2014.

    http://www.flickr.com/photos/31013861@N00/14860905785/

  • Rufus PollockPresident and Co-Founder

    Open Knowledge (Foundation)

    Photography: Sebastiaan ter Burg, http://www.flickr.com/photos/31013861@N00/14860905785/, 2014.

    http://www.flickr.com/photos/31013861@N00/14860905785/

  • Open Definition

    Rufus PollockPresident and Co-Founder

    Open Knowledge (Foundation)

    Photography: Sebastiaan ter Burg, http://www.flickr.com/photos/31013861@N00/14860905785/, 2014.Source: Open Definition 2.1, http://opendefinition.org/od/2.1/, 2015.

    Knowledge is

    OPENif ANYONE is

    FREE to ACCESS, USE,MODIFY, and SHARE it subject, at most, to measures that preserve

    PROVENANCE and OPENNESS.

    http://www.flickr.com/photos/31013861@N00/14860905785/http://opendefinition.org/od/2.1/

  • Open Definition

    Rufus PollockPresident and Co-Founder

    Open Knowledge (Foundation)

    Photography: Sebastiaan ter Burg, http://www.flickr.com/photos/31013861@N00/14860905785/, 2014.Source: Open Definition 2.1, http://opendefinition.org/od/2.1/, 2015.

    Data and Content are

    OPENif ANYONE is

    FREE to ACCESS, USE,MODIFY, and SHARE it subject, at most, to measures that preserve

    PROVENANCE and OPENNESS.

    http://www.flickr.com/photos/31013861@N00/14860905785/http://opendefinition.org/od/2.1/

  • How Data are Open or Closed, based on four characteristics

    Source: From Open data: Unlocking innovation and performance with liquid information, McKinsey Global Institute, Oct. 2013.

    Completely Closed

    Completely Open

    More Liquid

    Degree of Access Everyone has access Access to data is to a subset ofindividuals or organizations

    Machine-Readability Available in formats that can be easily retrieved and processed by computersData in formats not easily retrieved and

    processed by computers

    Cost No cost to obtain Offered only at a significant fee

    Rights Unlimited rights to reuseand redistribute dataRe-use, republishing, or

    distribution of data is forbidden

  • Methods

  • Tim Berners-LeeThe Inventor of the World Wide Web

    Photography: Bret Hartman, TED, 2014.

  • 5-Star Deployment Scheme for Open Data

    Tim Berners-LeeThe Inventor of the World Wide Web

    Photography: Bret Hartman, TED, 2014.

    In order to encourage people especially government data owners along the road to good linked data, I have developed this star rating system. Linked Open Data (LOD) is Linked Data which is released under an open license, which does not impede its reuse for free. [] Linked Data does not of course in general have to be open. []However, if it claims to be Linked Open Data then it does have to beopen, to get any star at all. Linked Data, Design Issues, 2010.

  • Image: Science for all, 2015.

  • The Tip of the Iceberg

    Image: Science for all, 2015.

    All those pages on websites are only tips of icebergs:

    The real data is hidden in databases, XML files, Excel sheets, You only have access to what the Web page designers allow you to see.

    [] Various data sources expose their data via Web Services or APIs, each with a different API, a different logic, a different structure. Mashups are forced to reinvent the wheel many times because there is no standard way getting to the data. Ivan Herman, High Level Intro to Semantic Web, Feb. 2012.

  • Tim Berners-LeeThe Inventor of the World Wide Web

    Open Data, Now!

  • Conformant Licenses

    Screenshot: http://opendefinition.org/licenses/ Screenshot: http://licenses.opendefinition.org/

    http://opendefinition.org/licenses/http://licenses.opendefinition.org/

  • Screenshot: http://opendatacommons.org/

    http://opendatacommons.org/

  • Licenses for the Database and its Contents

    Screenshot: http://opendatacommons.org/

    The database and its contents may have separate rights. [] Different types of subject matter (e.g., code, content, or data) necessitate differences in licensing. Licenses designed for one type of subject matter as CC licenses (lower than 4.0) were designed for content, and F/OSS licenses for code arent always best suited to licensing another type of subject matter. Licenses FAQ, Open Data Commons, 2010.

    http://opendatacommons.org/

  • Creative Commons Rights Expression Language (CC REL)

    Source: https://www.w3.org/Submission/ccREL/

    https://www.w3.org/Submission/ccREL/

  • Open Data Rights Statement Vocabulary

    Screenshot: https://alpha.openaddressesuk.org/about/terms/ Screenshot: http://schema.theodi.org/odrs/

    https://alpha.openaddressesuk.org/about/terms/http://schema.theodi.org/odrs/

  • Open Data Rights Statement Vocabulary

    Screenshot: https://alpha.openaddressesuk.org/about/terms/ Screenshot: http://schema.theodi.org/odrs/

    https://alpha.openaddressesuk.org/about/terms/http://schema.theodi.org/odrs/

  • Screenshot: The Next Web, TED, 2009.

    Tim Berners-LeeThe Inventor of the World Wide Web

  • Tim Berners-LeeThe Inventor of the World Wide Web

    Raw (Structured) Data, Now!

    Screenshot: The Next Web, TED, 2009.

  • No Ontological Commitment, No Machine-Understandability.

    The term ontological commitment is used as a general term in both philosophy and in information systems to refer to the essential elements of an ontology. An ontological commitment in describing ontological comparisons is taken to refer to a subset of elements of an ontology that it shares with all other ontologies based upon the same theory or conceptualization. Citizendium, 2013.

    Diagram: John R. Brews, Citizendium, 2013.

  • Open Formats

  • Screenshot: http://data.okfn.org/

    http://data.okfn.org/

  • Frictionless Data

    Diagram: Open Knowledge, http://blog.okfn.org/2013/04/24/frictionless-data-making-it-radically-easier-to-get-stuff-done-with-data/, 2013.

    http://blog.okfn.org/2013/04/24/frictionless-data-making-it-radically-easier-to-get-stuff-done-with-data/

  • Frictionless Data

    Diagram: Open Knowledge, http://data.okfn.org/roadmap

    http://data.okfn.org/roadmap

  • Data Package Standards & Tools

    Diagram: Open Knowledge, http://blog.okfn.org/2013/04/24/frictionless-data-making-it-radically-easier-to-get-stuff-done-with-data/, 2013. Screenshot: http://data.okfn.org/tools

    http://blog.okfn.org/2013/04/24/frictionless-data-making-it-radically-easier-to-get-stuff-done-with-data/http://data.okfn.org/tools

  • CSV on the Web

    Screenshot: https://www.w3.org/TR/tabular-metadata/ Screenshot: http://www.w3.org/TR/tabular-data-model/ Screenshot: http://www.w3.org/TR/csv2json/ Screenshot: https://www.w3.org/TR/csv2rdf/

    https://www.w3.org/TR/tabular-metadata/http://www.w3.org/TR/tabular-data-model/http://www.w3.org/TR/csv2json/https://www.w3.org/TR/csv2rdf/

  • Core & Community Datasets

    Screenshot: http://data.okfn.org/data Screenshot: https://github.com/datasets

    http://data.okfn.org/datahttps://github.com/datasets

  • Screenshot: https://github.com/datasets/country-codes/blob/master/data/country-codes.csv

    https://github.com/datasets/country-codes/blob/master/data/country-codes.csv

  • Screenshot: http://dat-data.com/

    http://dat-data.com/

  • Tim Berners-LeeThe Inventor of the World Wide Web

    Photography: Paul Clarke, 2014.

  • Linked Data

    Tim Berners-LeeThe Inventor of the World Wide Web

    Photography: Paul Clarke, 2014.

    1. Use URIs as names for things 2. Use HTTP URIs so that people can look up those names 3. When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL) 4. Include links to other URIs so that they can discover more things

    Ill refer to the steps above as rules, but they are expectations of behavior. Breaking them does not destroy anything, but misses an opportunity to make data interconnected. This in turn limits the ways it can later be reused in unexpected ways. It is the unexpected re-use of information which is the value added by the Web. Linked Data, Design Issues, 2010.

  • Hypertext Transfer Protocol Uniform Resource Identifiers

    Tim Berners-LeeThe Inventor of the World Wide Web

    Photography: Paul Clarke, 2014.

    The first rule, to identify things with URIs, is pretty much understood by most people doing semantic Web technology. [] The second rule, to use HTTP URIs, is also widely understood. The only deviation has been a constant tendency for people to invent new URI schemes such as XRIs, DOIs, and so on for various reasons. Typically, these involve not wanting to commit to the established Domain Name System (DNS) for delegation of authority but to construct something under separate control. Sometimes it has to do with not understanding that HTTP URIs are names []. Linked Data, Design Issues, 2010.

  • Hypertext Transfer Protocol Uniform Resource Identifiers

    HTTP URIs, in the Web architecture, have been used to denote documents. However, with the growth of the Semantic Web, which uses URIs to denote anything at all, the urge to use and practice of using HTTP URIs for arbitrary things grew steadily. What HTTP URIs Identify, Design Issues, 2010.

    Diagram: What do HTTP URIs Identify?, Design Issues, 2002.

    KEY 1 Car

    KEY 2

    KEY 3

    KEY 4

    KEY 5

    URI 1

    URI 2

    URI 3

    URI 4

    URI 5

    URI 0

    URI 6

  • Resource Description Framework

    Subject ObjectPredicate

    Triple

    URI 1 URI 3 / Value

    URI 2

    - COL 1 COL 2 COL 3 COL 4 COL 5 COL 6

    KEY 1 Car

    KEY 2

    KEY 3

    KEY 4

  • Resource Description Framework

    Subject 1Object 1Subject 2

    Predicate 1

    Triple 1

    URI 1 URI 3

    URI 2

    - COL 1 COL 2 COL 3 COL 4 COL 5 COL 6

    KEY 1 Car

    KEY 2

    KEY 3

    KEY 4

    Object 2Predicate 2

    URI 5 / Value

    URI 4

    Triple 2

    - COL 7

    Car Tire

    Graph

  • Linked Open Vocabularies (LOV)

    Screenshot: http://lov.okfn.org/

    http://lov.okfn.org/

  • Dereferenceable Uniform Resource Identifiers

    Tim Berners-LeeThe Inventor of the World Wide Web

    Photography: Paul Clarke, 2014.

    The third rule, that one should serve information on the Web against a URI, is, in 2006, well followed for most ontologies, but, for some reason, not for some major datasets. [] Large datasets provide a SPARQL query service, but the basic linked data should be provided as well. Many research and evaluation projects in the few years of the Semantic Web technologies produces ontologies, and significant data stores, but the data, if available at all, is buried in a zip archive somewhere, rather than being accessible on the Web as linked data. Linked Data, Design Issues, 2010.

  • Dereferenceable Uniform Resource Identifiers

    Diagram: From Architecture of the World Wide Web, Volume One, W3C, 2004.

  • Why Linked Data?

    Source: Tom Heath, How to Publish Linked Data on the Web, 2008.

    Ease of Discovery Ease of Consumption

    - standards-based data sharing Reduced Redundancy

    - avoid duplication Added Value

    - build ecosystems around your data/content

  • Diagram: The Linking Open Data cloud diagram, http://lod-cloud.net/, 2014.

    http://lod-cloud.net/

  • Open Data EcosystemFigure 1. The open data ecosystem

    Supplies data toUses data to deliver to

    Source: Deloitte LLP

    Business data

    Businessdata

    Businessdata

    Governmentdata

    Citizendata

    Citizendata

    Government data

    Citizen data

    Citizen

    Governmentdata

    Government

    Business

    There are three principal constituencies in any successful open data ecosystem: government, business and citizen. Each constituency supplies data to itself and to others. In turn, businesses and government use the data to deliver services demanded by all constituencies. The three classes of open data supplied by the constituencies and used to deliver services are:

    Open government data data produced, collected or paid for by the public sector, subject to restrictions relating to sub judice, national security, commercial sensitivity and privacy. In addition, special commercial arrangements also being made for certain trading funds, including Companies House, the Ordnance Survey, the Meteorological Office and HM Land Registry, which together form the newly created Public Data Group.10

    Open business data data produced or collected by the private sector and published freely and openly, subject to restrictions that individual businesses decide to put in place.

    Open citizen data the personal and non-personal data of individual citizens published into the open domain.

    Open data Driving growth, ingenuity and innovation 9

    Diagram: From Open data: Driving growth, ingenuity and innovation, Deloitte, 2012.

  • Epilogue

  • Diagram: John Snow, 1854.

  • Diagram: Florence Nightingale, Notes on Matters Affecting the Health, Efficiency, and Hospital Administration of the British Army, 1858.

  • Photography: "Dont Panic the Truth about Population, BBC, 2013.

    Hans RoslingCo-founder and Chairman,

    Gapminder Foundation

  • Photography: "Dont Panic the Truth about Population, BBC, 2013.

    Hans RoslingCo-founder and Chairman,

    Gapminder Foundation

    Animation: Mesmerizing Animation Shows How Much Healthier The World Has Become, Business Insider, 2014.

  • Reality Mining: Serendipitous Reuse

    Figure 1. Normalized data from Fluwatch (influenza cases, lab tests, ILI reports from sentinel physicians) and Google (number of clicks on an keyword-triggered influenza link).

    Results Over the flu-season period, the Google campaign received a total of 54,507 impressions and 4,582 clicks (Figure 1). Among all the ad campaign measures, the number of clicks on the ad was found to have the best correlation with traditional surveillance measures, which is why I show only correlation data for clicks. In general, clicks correlated better with flu events than ILI reports from sentinel physicians (Table 1). Internet clicks also were a timelier marker than ILI-SPR, in that they performed better to predict the flu events of the following week, whereas correlation coefficients in the ILI-SPR method were better for the current week than for the following week. All correlations were significant on a P

  • Thank You.