View
11.559
Download
0
Category
Preview:
Citation preview
Today:
Marc C. Hadfield, FounderVital AIhttp://vital.ai marc@vital.ai 917.463.4776
MetaQL:Queries Across NoSQL, SQL, Sparql, and Spark
<intro>
Marc C. Hadfield, Founder Vital AIhttp://vital.aimarc@vital.ai
MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Quick Overview
agenda
MetaQL Intro
Motivation
Domain Models (Schema)
MetaQL DSL
MetaQL Implementations
Examples
MetaQL
Leverage Domain Model (Schema)
Compose Queries in Code: Typed
Execute Queries on Databases, Interchangeably
Minimize TCO: Separation of Concerns
Developer Efficiency
Query Framework
Executable JVM Code! (Groovy Closure)
MetaQL Origin
Across many data-driven application implementations, a desire for:
Reusable Processes, Tools: Stop re-inventing the wheel.
Tools to manage “schema” across an application & organization.
Tools to combine Semantic Web, NOSQL, and Hadoop/Spark.
Team Collaboration: Human Labor is usually limiting factor.
sample
Recipient
Sender EMail
hasRecipient
hasSender
sample
Recipient
Sender EMail
hasRecipient
hasSender
ARC
ARC
sample
Recipient
Sender EMail
hasRecipient
hasSender
notEqual
type:PersonAddress:john@example.org
type:Person
type:hasSender
type:hasRecipient
type:Email
sample MetaQL graph query
GRAPH { value segments: ["mydata"] ARC { node_constraint { Email.class } constraint { "?person1 != ?person2" } ARC_AND { ARC { edge_constraint { Edge_hasSender.class } node_constraint {
Person.props().emailAddress.equalTo(“john@example.org") }
node_constraint { Person.class } node_provides { "person1 = URI" } } ARC { edge_constraint { Edge_hasRecipient.class } node_constraint { Person.class } node_provides { "person2 = URI" } } } } }
Internet of Things
Amazon Echo
Internet of Things
Coffee
Internet of Things:
Batch and Stream
Processing
Amazon Echo
Amazon Echo Service
haley-app webserviceVert.X
Vital Prime
Database
DataScript
Hadoop - HDFS
Apache SparkStreaming, MLLIB, NLP, GraphX
Aspen Datawarehouse
Analytics Layer
Serving Layer
Haley DeviceRaspberry Pi
Voice to Text API
Cognitive Application
NLP and Inference to process User request.
Query Knowledge in DB
Streaming Prediction Models:
“Should I really have more Coffee?”
External APIs…
Demo Examples
Vital Prime
Database
Vert.XVital-Vertx
JavaScript WebAppVitalService-JS
PredictionModels
DataScript
https://github.com/vital-ai/vital-examples
Demo Example
https://demos.vital.ai/enron-js-app/index.htmlhttps://github.com/vital-ai/vital-examples/tree/master/enron-js-app
Demo Example
Demo Example
Demo Example
Recipient EMailhasRecipient
Cytoscape Plugin
https://github.com/vital-ai/vital-cytoscapehttp://cytoscape.org/
Cytoscape Plugin
Cytoscape Plugin
Cytoscape Plugin
Cytoscape Plugin
Cytoscape Plugin: Wordnet Data, “wine, vino”
where are we using MetaQL?
Financial Services Healthcare
Internet-of-Things Start-Ups, Recommendation Apps
motivation for MetaQL
application architecture
Batch and Stream
Processing
Web / Mobile Application
Application Server
TransactionalDatabase
Hadoop - HDFS
Apache SparkStreaming, MLLIB, GraphX
Analytics Layer
Serving Layer
Key/ValueCache
External APIs Exrernal API Services
Multiple Databases + Analytics +
External APIs
enterprise application architecture
Dashboard
Application Server
Enterprise Datawarehouse
Data Silo Data Silo Data Silo Data Silo Data Silo ∞Many Many Many Data Models…
volume, velocity, variety
polyglot persistance = multiple database technologies
…but we also have very many data models.
many databases, many data models, changing rapidly.
too many moving parts for a developer to reasonably manage! need fewer APIs to learn!
what happens when changes occur?
Task
Infrastructure DevOps
Data Scientists
Business +Domain Experts
Developers
Roles
what changes?
Data Model Changes New Data Sources
Infrastructure Change Switch Databases
New Prediction Models / Features New Service APIs…
Many Interdependencies…
Example: Change in the taxonomy of a categorization service breaks all the logic tied to the old categories.
total cost of ownership
How much code changes when we modify our data model to include new sources?
How to minimize by decoupling dependencies?
When we switch database technologies?
Domain Model as “Contract”
Infrastructure DevOps
Data Scientists
Business +Domain Experts
Developers DomainModel
Everyone to agree (or at least be aware) of the definition of Domain Concepts.
Ue semantics to map “views”.
MetaQL Abstraction
Infrastructure DevOps
Data Scientists
Business +Domain Experts
Developers DomainModel
MetaQL
Abstraction to give breathing room to Infrastructure.
Infrastructure / DevOps
Database Types: • Key/Value • Document • RDF Graph • NOSQL • Relational • Timeseries
ACID vs. BASE
Optimizing Query Generation
Tuning Secondary Indices
Update MetaQL DSL for new DB features
CAP Theorem
Domain Model (Schema)
Domain Model Implementation
Combine: SQL-style Schema with Hadoop Data Serialization Schema (Avro, Thrift, Protocol Buffers, Kyro, Parquet) add Semantics: the “Meaning” of objects
Not a table “person”, but define the concept of Person to be used throughout an application. The implementation decides how to store “Person” data in it’s database.
Domain Model Implementation
Domain Model definition resolves: RDF vs Property Graph model Object Relational Impedance Mismatch
Use OWL to capture Domain Model: SubClasses SubProperties
Multiple Inheritance
Marginal technology performance gains are hugely outweighed by Human productively gains, and wider choice of tools.
Compromise across modeling paradigms .
Domain Model Implementation
Example: Healthcare Application: URI<Person123> IS_A: • Patient • BillableAccount • InsuredEntity Same URI across three domain concepts: Diagnostics Records, Billing System, Insurance System.
Implementation Note: We generate code for the JVM using “traits” as a way to implement multiple inheritance (Groovy, Scala, Java8). The trait is used as a semantic marker to link to the Domain Model.
Domain Model - Core Classes
Node NodeEdge
HyperNode
HyperEdge
Properties: • URI • Primary Type • Types
Edges/HyperEdges: • Source URI • Destination URI
Edges: • Peer • Taxonomy
Class Instances contain Properties.
Protege OWL Editor
VitalSigns: Domain Model Dev Kit
$ vitalsigns generate -o ./domain-ontology/enron-dataset-1.0.0.owl
$ ls domain-groovy-jarenron-dataset-groovy-1.0.0.jar
$ ls domain-json-schemaenron-dataset-1.0.0.js
OWL can be compiled into JVM code statically (create an artifact for maven), or done dynamically at runtime.
Development with the Domain Model
Code Completion from Domain Model
Development with the Domain ModelVitalSigns vs = VitalSigns.get()
Musician john = new Musician().generateURI(“john")
john.name = "John Lennon"
john.birthday = "October 9, 1940"^xsd.xdatetime("MMMM d, yyyy”)
MusicGroup thebeatles = new MusicGroup().generateURI("thebeatles")
thebeatles.name = "The Beatles"
// try to assign the wrong property, throws an exception
try { thebeatles.birthday = "January 1, 1970"^xsd.xdatetime("MMMM d, yyyy”)
} catch(Exception ex) { println ex } // no such property exception
vs.addToCache( thebeatles.addEdge_hasMember(john) )
// use cache to resolve queriesthebeatles.getMembers().each{ println it.name }
// use database to resolve queriesthebeatles.getMembers(ServiceWide).each{ println it.name }
Implicit MetaQL Queries
VitalService API
• Open/Close Endpoint • Create/Remove Segment • Create/Read/Update/Delete Object • Queries (MetaQL as input closure) • Service Operations (MetaQL as input closure) • callFunction (DataScript) • init Transaction/Commit/Rollback
A “Segment” is a Database (container of objects)
MetaQL
VitalSigns: Domain Model Manager • MetaQL DSL • Prediction Model DSL • Pipeline Transformation DSL (ETL)
(in development)
A tricky bit is find the best way to express the DSL within the allowed grammar of the host language (Groovy). It’s an ongoing effort.
Query Types
AGGREGATION
PATH
GRAPH
SELECT
Query Elements
• constraints: node_constraint, edge_constraint, … • comparators (equalTo, greaterThan, …) • provides, ?reference • AND, OR • OPTIONAL • Sort Criteria
SELECT query
SELECT {
value limit: 100value offset: 0value segments: ["mydata"]
constraint { Person.class }
constraint { Person.props().name.equalTo("John" ) }
}
GRAPH query
GRAPH { value segments: ["mydata"] ARC { node_constraint { Email.class } constraint { "?person1 != ?person2" } ARC_AND { ARC { edge_constraint { Edge_hasSender.class } node_constraint {
Person.props().emailAddress.equalTo(“john@example.org") } node_constraint { Person.class } node_provides { "person1 = URI" } } ARC { edge_constraint { Edge_hasRecipient.class } node_constraint { Person.class } node_provides { "person2 = URI" } } } } }
GRAPH query (2)
GRAPH {
value segments: [VitalSegment.withId('wordnet')]value inlineObjects: true
ARC {node_bind { "node1" }node_constraint { SynsetNode.expandSubclasses(true) }node_constraint { SynsetNode.props().name.contains_i("happy") }
ARC { edge_bind { "edge" } node_bind { "node2" } }
} }
Code iterating over Results can use bind names to reference objects in each solution: node1, edge, node2.
<—- inline objects
PATH query
def forward = truedef reverse = false
PATH {value segments: segmentsvalue maxdepth: 5 value rootURIs: [URIProperty.withString(inputURI)]
if( forward ) {ARC {
value direction: 'forward'// accept any edge: edge_constraint { }// accept any node: node_constraint { }
}}if( reverse ) {
ARC {value direction: 'reverse'// accept any edge: edge_constraint { }// accept any node: node_constraint { }}
}}
AGGREGATION query
SUM Product.props().cost
AVERAGE Person.props().birthday
COUNT_DISTINCT Document.props().active
FIRST { DISTINCT Document.props().title, expandProperty : false, order: Order.ASC }
Part of a SELECT query
Service Operations DSL
Insert
Update
Delete
Service Operations
INSERT {value segment: 'testing'
insert(MusicGroup.class, provides: "thebeatles") { MusicGroup thebeatles ->thebeatles.name = "The Beatles"thebeatles.URI = "thebeatles"
}insert(Musician.class, provides: "john") {
Musician john ->john.name = "John"john.URI = "john"
}insert(Edge_hasMember) { Edge_hasMember member ->
member.sourceURI = ref("thebeatles").toString()member.destinationURI = ref("john").toString()member.URI = "edge1"
}}
<— Using “provides” values
Transactions
def xid = service.startTransaction()
service.save(xid, person123)
service.commitTransaction(xid)
Implemented at the service level:
MetaQL Implementations
MetaQL
ExecutableQuery
Query Generator
Sparql/RDF Implementation
G S P O
Quad Store
Franz Allegrograph
Sparql/RDF Implementation
VitalGraphQuery q = builder.query {GRAPH {
value segments: ["documents"]ARC {
node_constraint { Person.class }node_constraint { Person.props().emailID.equalTo(“k.lay@enron.com" ) }
ARC {node_constraint { EMailMessage.class }edge_constraint { Edge_hasEMailMessage.class }
} } }
}.toQuery()
println "Query: " + q.toSparql()
Sparql/RDF Implementation
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>PREFIX vital-core: <http://vital.ai/ontology/vital-core#>PREFIX p0: <http://vital.ai/ontology/enron-emails#>
SELECT DISTINCT ?s1 ?d2 ?e2FROM <segment:customer__app__documents>WHERE { { ?s1 p0:hasEmailID ?value1 . ?s1 rdf:type ?value2 . FILTER ( ?value2 = p0:Person && ?value1 = “k.lay@enron.com"^^xsd:string ) { ?d2 rdf:type ?value3 . ?e2 rdf:type ?value4 . FILTER ( ?value3 = p0:EMailMessage && ?value4 = p0:Edge_hasEMailMessage ) ?e2 vital-core:hasEdgeSource ?s1 . ?e2 vital-core:hasEdgeDestination ?d2 . } }}
Spark-SQL / Dataframe
URI P V
Segment RDD Property RDD
K V
Experimenting with: new Dataframe Optimizer: Catalyst, new Dataframe DSL for query generation, and using GraphX for isolated Graph Query cases
Generate “Bad” queries, with optimizer fixing them and Spark partitioning RDDs, as long as Spark is aware of Schema.
Key/Value Implementation
K V
URI —> Serialized Object
Lucene/SOLR Implementation
DocID
1
2
3
P1
V1
V1
P2
V2
V2
P3
V3
V3
P4
V4
V4
Inverted Index of Property Values…
NoSQL BigTable Implementation
DynamoDB (HBase, Cassandra, Accumulo, …)
ROWID
1
2
3
C1
K1=V1
K1=V1
K1=V1
C2
K1=V1
K1=V1
K1=V1
C3
K1=V1
K1=V1
K1=V1
C4
K1=V1, K1=V1
K1=V1, K1=V1
K1=V1, K1=V1
URI P V
Per Segment object table
Per Segment property table
+ Secondary Indices
+ Secondary Indices
SQL Implementation
SQL, Hive-SQL, Redshift, …
G S P O
Per Segment Table
with Partitioning (Hive)
implementation
DSL Documentation to be posted: http://www.metaql.org/
VitalSigns, VitalService, MetaQL https://dashboard.vital.ai/
Vital AI github: https://github.com/vital-ai/ Sample Code
Spark Code: Aspen, Aspen-Datawarehouse
Documentation Coming!
closing thoughts
Separation of Concerns yields the Agility needed to keep up with rapidly evolving Data.
“Domain Model as Contract” provides a framework for consistent interpretation of Data across an application.
MetaQL provides a framework for the consistent access and query of Data across an application.
Context: Data-Driven Application / Cognitive Applications:
Thank You!
Marc C. Hadfield, FounderVital AIhttp://vital.ai marc@vital.ai 917.463.4776
Pipeline DSL (ETL)
PIPELINE { // WorkflowPIPE { // a Workflow Component with dependencies
TRANSFORM { // Joins across Datasets IF (RULE { } ) // Boolean, Query, Construct, … THEN { RULE { } } ELSE { RULE { } } }PIPE { … } // dependent PIPE} // Output Dataset
PIPE { … }
}
Influenced by Spark Pipeline and Google Dataflow Pipeline
Schema Upgrade/Downgrade
UPGRADE {
upgrade(oldClass: OLD_Person.class,newClass: NEW_Person.class ) {
person_old, person_new -> person_new.newName = person_old.oldName }}
DOWNGRADE {
downgrade(newClass: NEW_Person.class,oldClass: OLD_Person.class ) {
person_new, person_old -> person_old.oldName = person_new.newName }}
Multiple Endpoints
def service1 = VitalService.getService(profile:”kv-users”)def service2 = VitalService.getService(profile:”posts-db”)def service3 = VitalService.getService(profile:”friendgraph-db”)
// given user URI:user123@email.org
// get user object from service1
// find friends of user in friendgraph via service3
// find posts of friends in posts-db
// update service1 with cache of user-to-friends-postings
// send postings of friends to user in UI
Recommended