Linked Data Universe - Large Scale Computing Tasks for the HPI FutureSOC-Lab

Harald SackInternet Technologies and Systems (ITS) Future Internet TechnologiesHasso-Plattner-Institute for IT Systems Engineering

5th Annual Symposium on Future Trends in Service-Oriented ComputingJune 16th, 2010Hasso-Plattner-Institute for IT Systems EngineeringPotsdam

Linked Open Data Universe

JHarald Sack, 5th Annual Symposium on Future Trends in Service-Oriented Computing, June 16th, 2010, HPI, Potsdam

2010

2

The Web is huge....

To be more precise, the WWW is rather huge...•more than 25 x 109 documents in

Search engine indexes (TNL Blog: Google has 24 billion items index, considers MSN search nearest competitor, September 2005)

•Google Web Crawler found more than 1012 documents(The Official Google Blog: We knew the Web was Big....., Juli 25, 2008)

•New Google Search Index Caffeine comprises 100 Million Gigabytes of datai.e. 1017 Byte (SMX Video: Google’s Matt Cutts On Caffeine Launch, June 9, 2010,http://searchengineland.com/smx-video-googles-matt-cutts-on-caffeine-launch-43933)

•And then, there is also the DeepWeb (Darkweb) ...and it is supposed to be up to 500 time larger than the Surface Web(Bergman, 2001)

http://www.tnl.net/blog/2005/09/27/google-has-24-billion-items-index-considers-msn-search-nearest-competitor/




http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html

http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html


2010

3

The Web is growing...

Multimedia, Real-Time Data, Sensor Data, ....

in 06/2010: 7 TB/day

in 05/2010: •24 h of video upload / minute•2 billion streamed videos per day

in 06/2010: 7 TB/day


2010

3

The Web is growing...

Multimedia, Real-Time Data, Sensor Data, ....

in 06/2010: 7 TB/day

in 05/2010: •24 h of video upload / minute•2 billion streamed videos per day

in 06/2010: 7 TB/day


2010

4

How to find something on the Web?

in 06/2010: 7 TB/day


2010

5

The ‘Web of Data‘

Semantic Web Technologies

• Interoperable and machine understandabledata semantics

• Based on formal knowledge representations

• Creating a ‘Web of Data‘


2010

6

• Topic: Semantic Web and Linked Data

•Problems and Experiments

•Application: Exploratory Multimedia Search



2010

7

Semantic Web and Linked Data

From World Wide Web to Web of Data„The Web was designed as an information space, with the goal that it should be useful not only for human-human communication, but also that machines would be able to participate and help… “

Prerequisites:

• Content can be read and interpreted correctly (=understood) by machines

Tim Berners-Lee, Semantic Web Roadmap, Sept 1998

Semantic Web• (natural language) web content is

explicitely annotated with semantic metadata

• semantic metadata encode the meaning (semantics) of web content and can be read andinterpreted correctly my machine

Natural Language Processing• Technology from traditional Information

Retrieval (WWW Search Engines)


2010

8


Understanding Web Content - I




2010

8






2010

8





?...

?

text: „FAB“

fabulous

Entity MappingDisambiguation

?


2010

8





Fabio CapelloManager ofUK National

Football Team

?...

?

text: „FAB“

fabulous


?


2010

8





Fabio CapelloManager ofUK National

Football Team

?...

?

text: „FAB“

fabulous


?

David JamesGoal Keeper of

UK NationalFootball Team


2010

9


Understanding Web Content - II

text: „FAB“

Fabio Capello

Entity Mapping


2010

9



text: „FAB“

Fabio Capello

Entity Mapping

Soccer Manager

is a


2010

9



text: „FAB“

Fabio Capello

Entity Mapping

Soccer Manager

is a

Person

is a


2010

10


Understanding Web Content - III

Fabio Capello (entity)


2010

10




Soccer Manager

is a

(class)

Class-membership has type


2010

10




Soccer Manager

is a

(class)

Class-membership has type

Person

is a

(class)

superclass

subclass

is subclass of


2010

11


Understanding Web Content - IV

Fabio Capello

Soccer Manager

Person

is a

is aEntities

Classes


2010

11



Fabio Capello

Soccer Manager

Person

is a

PlacehasBirthPlace

is aEntities

Classes


2010

11



Fabio Capello

Soccer Manager

Person

is a

PlacehasBirthPlaceDate hasBirthDate

is aEntities

Classes


2010

11



Fabio Capello

Soccer Manager

Person

is a


is a

hasBirthDate1946-06-18

is a

Entities

Classes


2010

11



Fabio Capello

Soccer Manager

Person

is a


is a


is a

San Canzian d‘IsonzohasBirthPlace

is a

Entities

Classes


2010

12



2010

13


Fabio Capello http://dbpedia.org/resource/Fabio_Capello

URI - Uniform Resource Identifier

http://dbpedia.org/page/Fabio_Capello



2010

14


http://dbpedia.org/resource/Fabio_Capello

http://en.wikipediapedia.org/resource/Fabio_Capello






2010

15

Semantic Web and Linked Datahttp://dbpedia.org/resource/Fabio_Capello




2010

16


http://dbpedia.org/resource/Fabio_Capello

RDF Resource Description Framework

:Fabio_Capello dbpp:birthPlace :San_Canzian_d%27Isonzo .:Fabio_Capello dbpp:birthDate “1946-06-18“ .:Fabio_Capello rdfs:type dbpo:SoccerManager .:Fabio_Capello rdfs:type dbpo:Person ....

:Fabio_Capello rdf:type dbpo:SoccerManager .

RDF Tripel RDF Subject RDF Property RDF Object




2010

17


http://dbpedia.org/ontology/soccer_manager

dbpo:SoccerManager rdf:type owl:class .dbpo:SoccerManager rdfs:subClassOf dbpo:Person .dbpo:SoccerManager rdfs:label “Soccer Manager“ .dbpp:birthPlace rdf:type rdf:Property .dbpp:birthPlace rdfs:domain dbpo:Person .dbpp:birthPlace rdfs:range dbpo:Place .dbpp:birthDate rdf:type rdf:Property .dbpp:birthDate rdfs:domain :Person .dbpp:birthDate rdfs:range xsd:date ....

RDF Schema

Person PlacehasBirthPlaceDate hasBirthDate

Soccer Manager

is a




2010

18

Semantic Web and Linked DataUnderstanding Web Content - V

Fabio Capello

LivingPeople

PersonDate


hasBirthDate

is a

is a

is a

DeadPeople∩ =∅

logical constraint

is a

+ Rules (Description Logics)

∀x.∃y.hasDeathDate(x,y) ∧ Person(x) ∧ Date(y) → DeadPeople(x)


2010

19


SELECT DISTINCT ?l ?l2 ?g FROM <http://dbpedia.org> WHERE { ?s dbpp:nationalteam ?o . ?s rdfs:label?l FILTER langMatches( lang(?l), "EN" ) . ?s dbpp:nationalgoals ?g FILTER(?g>10). ?s dbprop:nationalteam ?nat . ?nat rdfs:label ?l2 FILTER langMatches( lang(?l2), "EN" ).} ORDER BY DESC(?g)

Select all players of a soccer nationalteam that have scored more than 10 goals while inthe team

http://dbpedia.org

http://dbpedia.org

http://dbpedia.org/property/nationalteam

http://dbpedia.org/property/nationalteam


2010

20


Select all players of a soccer nationalteam that have scored more than 10 goals while in the team


2010

21


(RDF)

(URI)

M.Hausenblas, Quick Linked Data Introduction, http://www.slideshare.net/mediasemanticweb/quick-linked-data-introduction

Linked Data■ Term was originally coined by Tim Berners-Lee

(Tim Berners-Lee, Linked Data, 2006, http://www.w3.org/DesignIssues/LinkedData.html)

The Web of data is abouta dataand namingmodel on the Web

http://www.w3.org/DesignIssues/LinkedData.html

http://www.w3.org/DesignIssues/LinkedData.html


2010

22


Linked Data

■ Technical Principles

□ use URIs to identify things uniquely (not only documents...)

□ use HTTP URIs (URLs) so that these things can be referred to and looked up ("dereferenced") by people and user agents

□ use RDF as an universal data model to provide useful information about these things

□ include links to other, related URIs in the exposed data to improve discovery of other related information on the Web.

http://en.wikipedia.org/wiki/Dereferenceable_Uniform_Resource_Identifier

http://en.wikipedia.org/wiki/Dereferenceable_Uniform_Resource_Identifier

http://en.wikipedia.org/wiki/User_agents




2010

23


Linked Data□ The application lf the Linked Data principles leads to the creation of a

,Web of Data‘


2010

24


Linking Open Data■ Public available structured data should be published as Linked Data

■ Various data sources should be interlinked

LOD-WikiPage: http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData/

http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData/

http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData/


2010

25


Linked Data Achievments■ Extension of the Web with a

data commons (14b RDF triples = facts)

■ Vibrant global RTD community

■ Industrial uptake starting(BBC, Thomson, Reuters, etc.)

■ Emerging governmental adoption in sight

■ Establishing Linked Data as a deployment path for the Semantic Web


2010

26


Linked Data Challenges■ Coherence

relatively few, expensively maintained links

■ Qualitypartly low quality data and inconsistencies

■ Performancestill substantial penalties compared torelational database technologies

■ Data consumptionlarge scale processing, schema mapping anddata fusion still in its infancy

■ UsabilityMissing direct end user tools and network effect


2010

27


Sören Auer:"Linked Data: Now what?"ESWC2010 Panel Discussion


2010

28


•Problems and Experiments




2010

29

Problems and Experiments


2010

30



2010

31


A. Hoigan et al: Weaving the Pedantic Web, LDOW 2010


2010

32


Experiment Summary (1) Crawling the Semantic Web

(2) Structural Analysis

(3) Content-based Analysis

(4) Data Cleansing

(5) Heuristics for Ranking Semantic Web Data

(6) Augmenting Semantic Web Infrastructure


2010

33


So what? ■ Interesting Facts to find out about

Semantic Web & Linked Data

■How big is the Semantic Universe?

■ # tripel

■ # documents

■ # interlinking

■ Linking Open Data is only registered vocabulary/data in the LOD-Wiki→ 14b RDF triples

■What else is out there ... and how much of it?

■ ...and how do we get it?


2010

34


(1) Crawling the Semantic Web■Of course we are not the first to be out there...

■ SwoogleLi Ding et al: Finding and Ranking Knowledge on the Semantic Web, ISWC 2005.

■ Scutter/Slug Leigh Dodds: Slug: A Semantic Web Crawler, 2006

■ Sindice Giovanni Tumarello et al: Sindice.com - weaving the open linked data, ISWC 2007

→ 2.1b RDF triples

■ SWSE Andreas Harth et al: SWSE: Objects before Documents,

Semantic Web Challenge 2008, ISWC 2008


■ FalconsG.Cheng et al.:Falcons: Searching and Browsing Entities on the Semantic Web, WWW17 2008.



2010

35


(1) Crawling the Semantic Web■ First experiments:

■ Adapting & Improving Slug Crawler

■ for parallelization (48 Cores) and

■ lots of RAM (256GB - 2TB)

■ first test run: >1GB RDF data/1h

■What‘s new:

■ crawl not only RDF/RDFS and OWL resources

■ include (X)HTML with RDFa extensions and

■ dynamic documents with (semantic) sitemaps

■What‘s next...?


2010

36


(2) Analyzing the Semantic Web I - Structural Analysis■ Again we are not the first to be out there...

■ Structural Analysis of the ,early‘ WWW

IN44m nodes

SCC56m nodes

OUT44m nodes

unconnected components

unconnected components

tunnels

appendices

appendices

A. Broder et al.: Graph structure in the Web. In Comput. Netw. 33, 1-6 (Jun. 2000), 309-320.


2010

37


(2) Analyzing the Semantic Web I - Structural Analysis■ Again we are not the first to be there...

■ Structural Analysis of the ,early‘ Semantic Web

Weiyi Ge et al.: Object Link Structure in the Semantic Web, ESWC 2010

■ Experimental Setup

■ 18m RDF documents (Falcons crawl 2009)

■ 110m nodes with 190m edges■ Analysis of RDF link graph

■ average node degree: ≈3.4

■ effective diameter: ≈11.5

■ Largest connected component: ≈88% of all nodes


2010

38


(3) Analyzing the Semantic Web II - Content-Based Analysis■ Again we are not the first to be there...

http://pedantic-web.org/

A. Hoigan et al: Weaving the Pedantic Web, LDOW 2010

■ 150k documents with more than 12m RDF triples

■ Discovered categories of symptoms:

■ incomplete → dead links

■ incoherent → no correct interpretation (local)

■ hijack → no correct interpretation (remote)

■ inconsistent → contradictions

http://pedantic-web.org

http://pedantic-web.org


2010

39



Urbani et al: OWL Reasoning with WebPIE: Calculating the Closure of 100 Billion Triples, ESWC 2010■ Artificial Benchmark dataset used

Leigh University Benchmark (LUBM) with 100b RDF triples

■ Computing the transitive closure (= reasoning)

■ Making implicit knowledge explicit

Fabio Capello San Canzian d‘IsonzohasBirthPlace


2010

39







Fabio Capello

Person

is a



2010

39







Fabio Capello

Person

is a

PlacehasBirthPlace



2010

39







Fabio Capello

Person

is a

PlacehasBirthPlace


class membershipcan be deduced


2010

40


(4) Analyzing the Semantic Web III - Data Cleansing■ trying to clean out Linked Open Data and possibly also (partially) the

Semantic Web...

(1) Identify inconsistencies and ambiguities by (automated) content-based analysis

(2)Solve inconsistencies & ambiguities

■ if possible by reasoning

■ else by crowdsourcing (game-based evaluation, etc.)

Cleaning out the Augean stables...AUGEAN-STABLES: Extremely nasty and smelly warehouses of filth, straw and manure


2010

41


(5) Analyzing the Semantic Web IV - Data Ranking■ Linked Data provides (unbiased) knowledge

■ unbiased = no distinction of what is important, what is not important

■ e.g., Albert Einstein■ > 600 facts (triples)■ > 80 properties■ no ranking■ no relevance

http://dbpedia.org/page/Albert_Einstein

http://dbpedia.org/page/Ernest_Hemingway

http://dbpedia.org/page/Ernest_Hemingway


2010

42


(5) Analyzing the Semantic Web IV - Data Ranking■We have developed heuristics for ranking objects and properties,

e.g.

:Albert_Einstein

:AmericanVegetarian

rdf:type

:Scientistrdf:type


2010

42



e.g.

:Albert_Einstein

:AmericanVegetarian

rdf:type

:Alfred_Kleiner

rdf:type

:Scientistrdf:type


2010

42



e.g.

:Albert_Einstein

:AmericanVegetarian

rdf:type

:Alfred_Kleiner

rdf:type

:Scientistrdf:type :Bill_Cosby

rdf:type


2010

42



e.g.

:Albert_Einstein

:AmericanVegetarian

rdf:type

:Alfred_Kleiner

rdf:type

:Scientistrdf:type :Bill_Cosby

rdf:type

:doctoralAdviser

considered to be relevant


2010

43


(6) Semantic Web Infrastructure - Tripel Stores■ RDF(S) Data is stored in Triple Stores

■ Basic idea:

■ Use 1 table with 3 columns (s,p,o)

■ For every row / row combinationcreate index structures for fast access(spo, sop, pos, pso, ops, osp)

■ Drawback: many self-joins needed(memory consumption)


2010

44


Experiment Summary (1) Crawling the Semantic Web

(2) Structural Analysis

(3) Content-based Analysis

(4) Data Cleansing

(5) Heuristics for Ranking Semantic Web Data

(6) Augmenting Semantic Web Infrastructure


2010

45


•Problem Defintion and Experiments




2010

46

http://www.yovisto.com

Application: Exploratory Multimedia Search

Yovisto semantic video search engine

■specialized on academic video content, e.g., lecture recordings

■enables to search within the content of video

■ automated video analysis: video scene cut detection, intelligent character recognition, complemented by collaborative user annotation

■more than 8.000h of video

Semantic Metadata:

■ Ontology: http://www.yovisto.com/ontology/0.9/

■ DBpedia, FOAF, DublinCore, MPEG-7, Tagging

■ RDFa annotation

■ public SPARQL Endpoint: http://sparql.yovisto.com/J. Waitelonis, H. Sack: Augmenting Video Search with Linked Open Data, in Proc. of International Conference on Semantic Systems 2009 (i-semantics 2009), September, 2-4, 2009, Graz, Journal of Universal Computer Science



http://www.yovisto.com/ontology/0.9/

http://www.yovisto.com/ontology/0.9/

http://sparql.yovisto.com

http://sparql.yovisto.com

http://typo3/https://www3.hpi.uni-potsdam.de/fileadmin/hpi/FG_ITS/papers/Harald/ISemantics09-Final.pdf




http://www.i-semantics.tugraz.at/

http://www.i-semantics.tugraz.at/


2010

47 ■Semantic Annotation

timeMetadata Extraction



2010



e.g., person xy

location yz

event abc

Entity Recognition/ Mapping



2010



e.g., bibliographical data,geographical data,encyclopedic data, ..

e.g., person xy

location yz

event abc

Entity Recognition/ Mapping



2010

Exploratory Search• Is a kind of investigation task, where the user is

(a) not familiar with the domain of the search result,i.e. before entering appropriate keywords, she needs to learn about the domain

(b) not sure about the way how to reach search destination (concerning search process and search technology)

(c) not really sure about what she’s looking for, i.e. “Can you please find something out about ... ?”.

48

White, R.W., Kules, B., Drucker, S.M., and schraefel, M.C.Supporting Exploratory Search, Introduction to Special Section of Communications of the ACM, Vol. 49, Issue 4, (2006), pp. 36-39.

„Which modern philosophers build on the theories of the greek philosopher Plato?“



2010

29

history

search term

related resources with properties

Waitelonis, Sack: Augmenting Video Search with Linked Open Data, in Proc. I-Semantics , Graz 2009.


2010

50


•Problem Defintion and Experiments



Thank you for your Attention!