Multiplatform Spark solution for Graph datasources by Javier Dominguez

17 NOV 2016 @ BIG DATA SPAIN

@StratioBD

MULTIPLATFORM SOLUTION FOR GRAPH DATASOURCES

Multiplatform Spark solution for Graph datasourcess, Stratio Stratio

Javier Domínguez

Javier Dominguez Montes

CTO SKILLS

PROFILE

JAVIER DOMÍNGUEZ

Studied computer engineering at the ULPGC. He is passionate about Scala, Python and all Big Data technologies

and is currently part of the Data Science team at Stratio Big Data,

working with ML algorithms, profiling analysis based around Spark.

LET'S HAVE FUN!

INDEX

1

2

3

4

INTRODUCTION

MULTIPLATFORM SOLUTION FOR GRAPH DATASOURCES

DEMO

THE END

Graph use cases Results

What's next?

Dataset

Main process explanation

Notebooks show off

DataStores

Machine learning

Business example

INTRODUCTION

@StratioBD

500 GB - 2 TB

4 TB - 8 TB

20 GB - 100 GB

80’S 2000 2010 2015 2020

CUSTOMER DATA WILL GROW OVER 100X

100 TB

> 10 PB

VALUE IS THE DATA VALUE IS UNDERSTANDING THE DATA

DO NOT STAY ON THE SURFACE OF KNOWLEDGE

MULTIPLATFORM SOLUTION FOR GRAPH DATASOURCES

• Graph use cases

• DataStores

• Machine learning

@StratioBD

Example of how to exploit a massive database from different stages and through several graph technologies

MACHINE LEARNING LIFE CYCLE WITH BIG DATA

Machine Learning life cycle

Show how a data sciencist is able to take advantage of a Graph Database through different datasources and technologies thanks to our solution.

Use as a example a masive dataset.

Query the datasource from different technologies like:

• GraphX• GraphFrames• Neo4j

And finally apply Machine Learning over our information!

BIG DATA SPAIN USE CASE

USE CASES

USE CASES

Making use of a masive graph datasource implies make batch queries over it.We will need to maken them with our distributed technologies... The easier the better

Batch Queries

Motifs filter example

import org.graphframes._val g: GraphFrame = Graph(usersRdd,relationshipsRdd0)

// Search for pairs of vertices with edges in both directions between themval motifs: Dataframe = g.find("(person_1)-[relation]->(person_2); (person_2)-[abilities]->(technology)")motifs.show()

// More complex queries can be expressed by applying filters.motifs.filter("person_1.name = 'Javier' AND technology.name = 'Neo4j'")

Most of our clients or teammates will need to have fast and easy access to the information.We would need a way to make easy queries and of course a graphic representation of our data!

We would need of course microservices like REST operations over our datastore.

Online queries

USE CASES

DATASTORES

SparkApache Spark is a fast and generic engine for large-scale data processing.

GraphX

Spark API for the management and distributed calculation of graphs. It comes with a great variety of graph algorithms: Connected componentes PageRank Triangle count SVD++

GraphFramesIt aims to provide both the functionality of GraphX and extended functionality taking advantage of Spark DataFrames. This extended functionality includes motif finding and highly expressive graph queries.

DATASTORES

Neo4j

Neo4j is a highly scalable native graph database that leverages data relationships as first-class entities.Big data alone used to be enough, but enterprise leaders need more than just volumes of information to make bottom-line decisions. You need real-time insights into how data is related.

DATASTORES

MACHINE LEARNING

MACHINE LEARNING

It's possible to quickly and automatically produce models that can analyze bigger, more complex data and deliver faster, more accurate results – even on a very large scale. The result? High-value predictions that can guide better decisions and smart actions in real time without human intervention.

Machine learning

SVD

Will relate all the existing object in our dataset and infer possible new behaviors.

DEMO

• Dataset

• Main process explanation

• Notebooks show off

@StratioBD

STRATIO INTELLIGENCE

Integration of different Open Source libraries of distributed machine learning algorithms.

Development environment adapted to each data scientist.

Real-time decision based on models based on machine learning algorithms

Integrated with all components of the Stratio Big Data Platform

Comprehensive knowledge lifecycle management

DATASET

Freebase aimed to create a global resource that allowed people (and machines) to access common information more effectively.

This model is based on the idea of converting the declarations of the resources in expressions with the subject-predicate-object which are called triplets.

Subject: It's the resource, what we are describing.Predicate: Could be a property or a relationship with the object value. Object value: Propertie's value or the related subject.

<'Cristiano Ronaldo'> <'Scores in 2014/2015'> 61 .<'Cristiano Ronaldo'> <'Born in'> 'Portugal' .

Freebase Google

Total triplets: 1.9 Billion

DATASET

PROCESS EXPLANATION

PROCESS EXPLANATION

Transforms

CastRDF Dataset

GraphFrames Batch query

Neo4jGraphXExtracts sample & transforms Online

query

SVDK-core

Decomposition Strongly connected graph

Apply algorithms

Behavior Inference

Graph

Subject equality

PROCESS EXPLANATION

A k-core of a graph G is a maximal connected subgraph of G in which all vertices have degree at least k. Equivalently, it is one of the connected components of the subgraph of G formed by repeatedly deleting all vertices of degree less than k.

Objective

Remove all nodes with fewer connections.At the end, we want only the most representative and connected elements in our grah.In our use case we used K = 5.

K-Core process

PROCESS EXPLANATION

NOTEBOOKS SHOW OFF

BUSINESS EXAMPLE

Jaccard Graph Clustering

Node Clusterization based on concrete relations optimized for Big Data environments.

We've developed an straightforward functionality which is able to detect patterns and clusterize data in a graph database thanks to daily machine learning processes.

Neo4j

Scala Graph functionalities

Jaccard Indexation

Connected Componentes

Java

HDFS / Parquet

Spark / GraphX

40BJaccard distance calculation

in everyday process

400Knodes graph clustering

BANK USE CASE

THE END

• Results

• What's next?

@StratioBD

WHAT'S NEXT?

Semantic search engineInclude ElasticSearch for making text searchs as a search engine.

Apply more Machine Learning algorithms

• Connected components: As we've already done, try to cluster information thanks to their relationships.• PageRank: Measure the importance of a subject.• Triangle counting: Check posible triangle relationships inside our dataset to avoid redundancy.

New Graph use cases

• Fraud detection• Recommendation System • Profiling

THANK YOU

UNITED STATESTel: (+1) 408 5998830

EUROPETel: (+34) 91 828 64 73

contact@stratio.com

www.stratio.com

@StratioBD

people@stratio.comWE ARE HIRING

@StratioBD