Managing Completeness of Web Data

Fariz DarariPhD Supervisor: Werner Nutt

Supported by the project MAGIC, funded by the province of Bolzano

Fariz Darari (unibz) Managing Completeness of Web Data Oct 20, 2015 1 / 38

About Us

Research Group

Sorted by distance to Werner’s office :)

Bozen-Bolzano

Motivation

Completeness statements are already there

However . . .

Completeness statements are availablebut only in natural languageUnclear what data completeness & query completeness meanNo techniques to check whether data completeness entailsquery completeness

Solution Ideas

Completeness statements are availablebut only in natural language

Solution: RDF-ize completeness statements

Unclear what data completeness & query completeness meanSolution: Formalize data completeness & query completeness

No techniques to check whether data completeness entailsquery completeness

Solution: Develop techniques to check whether data completenessentails query completeness

Solution Ideas

Solution: RDF-ize completeness statementsUnclear what data completeness & query completeness mean

Solution: Formalize data completeness & query completeness

Solution Ideas

Solution: RDF-ize completeness statementsUnclear what data completeness & query completeness mean

Solution: Formalize data completeness & query completeness

Solutions

Background: RDF

Grd = { (resDogs,dir , tarantino),(resDogs,act , tarantino) }

Background: SPARQL

SELECTQsdir = ({ ?m }, { (?m,dir , tarantino) })

ASKQadir = ({ }, { (?m,dir , tarantino) })

CONSTRUCT

Qcdir = ({ (?m,dir , tarantino) }, { (?m,dir , tarantino) })

Story: Incomplete Data Source

An incomplete data source of Reservoir Dogs,Gdbp = (Ga

dbp,Gidbp):

Gadbp = {(resDogs,dir , tarantino)}

Story: Incomplete Data Source

An incomplete data source of Reservoir Dogs,Gdbp = (Ga

dbp,Gidbp):

Gidbp = {(resDogs,dir , tarantino), (resDogs,act , tarantino)}

Story: Completeness Statement

From (Gadbp,G

idbp), we can say that DBpedia is complete

for movies directed by Tarantino:

Cdir = Compl((?m,dir , tarantino) | ∅)

However, it is not complete for actors in movies directed by Tarantino:

Cact = Compl((?m,act , ?a) | (?m,dir , tarantino))

Story: Completeness Statement

From (Gadbp,G

idbp), we can say that DBpedia is complete

for movies directed by Tarantino:

Cdir = Compl((?m,dir , tarantino) | ∅)

However, it is not complete for actors in movies directed by Tarantino:

Story: Query Completeness

Consequently, when we ask for all movies directed by Tarantinoover DBpedia:

Qdir = ({?m}, {(?m,dir , tarantino)})

the query completeness Compl(Qdir ) is obtained.

Story: Query Completeness

However, if we ask for all movies directed by and starring Tarantino:

Qdir+act = ({?m}, {(?m,dir , tarantino), (?m,act , tarantino)})

the query completeness Compl(Qdir+act) is not obtained.

Incomplete Data Source

Definition (Incomplete Data Source)An incomplete data source is a pair of two graphs

G = (Ga,Gi), where Ga ⊆ Gi .

We call Ga the available graph and Gi the ideal graph.

Completeness Statement

Definition (Completeness Statement)Let P1 be a non-empty BGP and P2 a BGP.

A completeness statement is defined as

Compl(P1 | P2)

where we call P1 the pattern and P2 the condition of the statement.

Satisfaction of Completeness Statements

To a statementC = Compl(P1 | P2),

we associate the CONSTRUCT query

QC = (P1,P1 ∪ P2).

Then, we say:

C is satisfied by an incomplete data source G = (Ga,Gi),written G |= C, if

JQCKGi ⊆ Ga.

Completeness Statements in RDF

lv:dataset a void:Dataset;c:hasComplStmt lv:csAct.

lv:csAct c:hasPattern [c:subject [c:varName "m"];c:predicate s:actor;c:object [c:varName "a"]];

c:hasCondition [c:subject [c:varName "m"];c:predicate s:director;c:object lmdb:Quentin_Tarantino].

Query Completeness

Definition (Query Completeness)Let Q be a query. We write

Compl(Q)

to say that Q is complete.

An incomplete data source G = (Ga,Gi) satisfies Compl(Q),written G |= Compl(Q), if

JQKGi = JQKGa .

Completeness Entailment

Problem Definition (Completeness Entailment)Let C be a set of completeness statements and Q a query.

We say that C entails the completeness of Q, written

C |= Compl(Q),

if any incomplete data source satisfying C also satisfies Compl(Q).

Intuition: Completeness Entailment

Consider the set Cdir ,act = {Cdir ,Cact } of completeness statementsand the query Qdir+act = ({ ?m },Pdir+act) where

Pdir+act = { (?m,dir , tarantino), (?m,act , tarantino) }.

Consider the set Cdir ,act = {Cdir ,Cact } of completeness statementsand the query Qdir+act = ({ ?m },Pdir+act).

Pdir+act = { (m,dir , tarantino), (m,act , tarantino) }

Therefore,

JQCdir KPdir+act∪ JQCact KPdir+act

{ (m,dir , tarantino), (m,act , tarantino) } =Pdir+act .

Thus,Cdir ,act |= Compl(Qdir+act).

Therefore,

{ (m,dir , tarantino), (m,act , tarantino) } =

Pdir+act .

Therefore,

{ (m,dir , tarantino), (m,act , tarantino) } =Pdir+act .

Prototypical Graph

Definition (Prototypical Graph)Let Q = (W ,P) be a query.

The freeze mapping id is defined as a mappingfrom each variable ?v in P to a new IRI v .

Instantiating the graph pattern P with id yields the graph

P := id P,

which we call the prototypical graph of Q.

Transfer Operator

Definition (Transfer Operator)For any set C of completeness statements and a graph G,we define the transfer operator TC that computes the unionof the evaluation over G of all CONSTRUCT queriesof the statements in C:

TC(G) =⋃

Completeness Entailment Theorem

Pdir+act = TCdir,act (Pdir+act)

Theorem (Completeness of Basic Queries)Let C be a set of completeness statements andQ = (W ,P) a basic query. Then,

C |= Compl(Q) if and only if P = TC(P).

Query Class: DISTINCT Queries

Give us all Oscar-winning things:

Qawd = (Wawd ,Pawd)d =

({?m}, { (?m,award ,oscar), (?m,award , ?aw) })d

Complete for all Oscar-winning things:

Cos = Compl((?m,award ,oscar) | ∅)

{Cos } |= Compl(Qawd) holds?

Query Class: OPT Queries

Give us all movies, and their awards, if any:

Qmaw = ({ ?m, ?aw }, ((?m,a,Movie) OPT (?m,award , ?aw)))

Complete for all movies and their awards:

Caw = Compl((?m,a,Movie), (?m,award , ?aw) | ∅)

{Caw } |= Compl(Qmaw ) holds?

Query Class: Queries under RDFS Semantics

Give us all films:

Qfilm = ({ ?m }, { (?m,a,Film) })

Complete for all movies:

Cmovie = Compl((?m,a,Movie) | ∅)

Films are the same as movies:

Sfm = {(Film, subclass,Movie), (Movie, subclass,Film)}

{Cmovie } |= Compl(Qfilm) wrt. Sfm holds?

Federated Completeness Statements

Timestamped Completeness Statements

Conclusions

Completeness statements can now be represented in RDFWe know how completeness statements can entail querycompleteness in different query classes anddifferent settings of completeness statements

Future Work

Completeness statements for queries with negationCompleteness statements as session annotationsfor RDF streamsStatistical completeness reasoning

Publications

Fariz Darari, Werner Nutt, Giuseppe Pirrò, Simon Razniewski: CompletenessStatements about RDF Data Sources and Their Use for Query Answering.ISWC 2013.

Fariz Darari, Radityo Eko Prasojo, Werner Nutt: CORNER: A CompletenessReasoner for SPARQL Queries Over RDF Data Sources. ESWC Posters andDemos 2014.

Fariz Darari, Simon Razniewski, Werner Nutt: Bridging the Semantic Gapbetween RDF and SPARQL using Completeness Statements. ISWC Postersand Demos 2014.

Fariz Darari, Radityo Eko Prasojo, Werner Nutt: Expressing No-ValueInformation in RDF. ISWC Posters & Demos 2015.

The latest results (timestamped statements and efficient completenessreasoning with 1 million statements) have been submitted to a journal.

Compl((myDaSePresentation, slide, ?s) | ∅)

Thank You!

Managing Completeness of Web Data

Technology

Data Integrity and Completeness using IDEA · Data Integrity and Completeness using IDEA ISACA Accra Chapter's IT Audit, Information Security and Risks Insights Alisa Hotel, North

Completeness Statements about RDF Data Sources and Their Use for Query Answering ... · 2014. 2. 18. · Completeness Statements about RDF Data Sources and Their Use for Query Answering

Managing Data Growth

Managing Data Resources Lecture 5 Managing Data Resources Lecture 5

Managing Data Resources

Data completeness analysis in the Malaysian Educational

Network Correlated Data Gathering With Explicit Communication : NP-Completeness and Algorithms

MAGIK: Managing Completeness of Data · Completeness is a central aspect of data quality which only re-cently has received increased attention in research (cf. [1, 2]) In particular,

Data Qualit and Completeness in the National Cancer Data Rep

Managing DataManaging Data Resources - Webs€¦ · Chapter Chapter 7 7 Managing Data Resources Managing Data Resources ORGANIZING DATA IN A TRADITIONAL FILE ENVIRONMENT Problems

ON COMPLETENESS OF HISTORICAL RELATIONAL DATA …

Strategies to Assess and Ensure Encounter Data Completeness & Accuracy · 2020-01-01 · Ensure Encounter Data Completeness & Accuracy ... and Health Plans base their analytic decision

Completeness, Coverage & Equivalence in Scientific Data

Enabling Fine-grained RDF Data Completeness Assessmentnutt/Publications/16-completeness-icwe.pdf · Enabling Fine-grained RDF Data Completeness Assessment Fariz Darari , Simon Razniewski,

Managing data smartly

Test District Fleet Data for Completeness

Managing and Consuming Completeness Information for Wikidata Using COOL-WD

Managing Data

Enabling Fine-Grained RDF Data Completeness Assessment · Enabling Fine-Grained RDF Data Completeness Assessment 171 Fig.1. Wikidata is actually complete for all the Apollo 11 crew

CORNER: A Completeness Reasoner for SPARQL Queries over RDF Data Sources