Joins in a distributed world - Lucian Precup

Joins in a distributed world

@LucianPrecup

@dmconf Barcelona 2015 #dmconf15

2015-11-21

whoami

• CTO of Adelean (http://adelean.com/)

• Integrate

– search, nosql and big data technologies – search, nosql and big data technologies

• to support

– ETL, BI, data mining, data processing and data

visualization

• use cases

2015-11-21 2@LucianPrecup #dmconf15 Barcelona

Poll - How many of you …

• Use a NoSQL database within your current application / information system?

• Which NoSQL database?

• Use a relational database within your current application / information system?

• Use a relational database within your current application / information system?

• Which RDBMS?

• Store the data into more than one system?

• Distribute data on several servers?

• Use an ORM framework (hibernate, doctrine, …) ?


Distributed databases

• Issues

– CAP theorem

– Two phase commit

– Distributed Transactions


Objective

• Show what distributed databases are able or

not able to do about joins and why

• Give YOU the ability to implement joins in

your applications (your service layers)your applications (your service layers)


Joins

• A join combines the output from two sources

• A join condition – the relationship between

the sources

• Join trees• Join trees


The "Query Optimizer"

SELECT DISTINCT offer_status FROM offer;

SELECT offer_status FROM offer GROUP by offer_status;

≡

SELECT offer_status FROM offer GROUP by offer_status;

SELECT A.ID as ID1, A.X, B.ID as ID2, B.Y

FROM A LEFT OUTER JOIN B

ON A.ID = B.ID

WHERE ID2 IS NOT NULL

SELECT A.ID as ID1, A.X, B.ID as ID2, B.Y

FROM A INNER JOIN B

ON A.ID = B.ID≡


The query optimizer - decisions

• Access paths– The optimizer must choose an access path to retrieve data from each table in

the join statement. For example, choose between a full table scan or an index scan.

• Join methods– To join each pair of row sources, the database must decide how to do it:

nested loop, sort merge, hash joins. Each join method has specific situations in which it is more suitable than the others.which it is more suitable than the others.

• Join types– The join condition determines the join type: an inner join retrieves only rows

that match the join condition, an outer join retrieves rows that do not match the join condition.

• Join order– To execute a statement that joins more than two tables, the database joins

two tables and then joins the resulting row source to the next table and so on.

• Calculating the cost of a query plan– Based on metrics : I/Os (single block I/O, multiblock I/Os), estimated CPU,

functions and expressions


Join methods

• Nested Loops Joins


Join methods

• Hash Joins


Join methods

• Sort Merge Joins

1

3

1

2


7

9

2

7

8

9

Join types

• Inner Joins– Equijoins

– nonequijoins

• Outer Joins– Nested Loop Outer Joins

– Hash Join Outer Joins– Hash Join Outer Joins

– Sort Merge Outer Joins

– Full Outer Joins

– Multiple Tables on the Left of an Outer Join

• Semijoins– ~ IN or EXISTS clause

• Antijoins– ~NOT IN or NOT EXISTS


Optimizing tricks

• Push filters as low as possible in the execution tree

• Cache small tables into memory

• Calculate statistics on each datasources and use them

• Think use cases

• “Fortunately, many of the lessons learned over the years for optimizing and executing join queries can be directly translated to the distributed world.” (see Nicolas

Bruno et all. in the references below)


Push filters

Filter 1

Join‘

Filter 1’ Filter 1’’

Table 1 Table 2 Table 1 Table 2

JoinFilter 1’ Filter 1’’


Take advantage of small tables

Join Filter

with data from

Table 2

Table 1

Table 2

Table 1

Table 2


Use statistics

Join 2

Join 3

Join 4

Table 1 Table 2

Join 1

Table 3

Table 3

Join 3

Table 2

Table 1


Other join optimisations (technical)

• Join algorithms– Different ways to implement a logical join operator : nested-loop joins,

sort-based joins, hash-based joins.

– Auxiliary data structures: secondary indexes, join indexes, bitmap indexes, bloom filters (e.g., indexed loop join, indexed sort-merge join, and distributed semi-join).

• Bloom Filters• Bloom Filters– especially useful when the amount of memory needed to store the

filter is small relative to the amount of data in the data set, and when most data is expected to fail the membership test.

• Partition-Wise Joins– A partition-wise join is a join optimization that divides a large join of

two tables, one of which must be partitioned on the join key, into several smaller joins.

• Full partition-wise join : Both tables must be equipartitioned on their join keys. We can then divide a large join into smaller joins between two partitions.

• Partial partition-wise joins : Only one table is partitioned on the join key. The other table may or may not be partitioned.


The "Query Optimizer"NoSQLSQL/RDBMS ��Power to the DBA


The "Query Optimizer"SQL/RDBMS ��Power to the DBA NoSQL ��Power to the developer

192015-11-21 @LucianPrecup #dmconf15 Barcelona

The "Query Optimizer"SQL/RDBMS NoSQL / Distributed / Multi-Model


Distributing data

• Clusters of nodes

• Partitioning the data (sharding)

• Partition keys (random or functional)

• Joining heterogeneous systems

Persons

(Oracle)

Contracts

(SQL Server)

Logs

(Elasticsearch)


Partitions (shards)

• "Random" partitioning

• "Functional" routing key � colocated joins

Data A Data B Data C

• "Functional" routing key � colocated joins

Customers by

customer_id

Customers by

customer_id

Contracts by

customer_id

Contracts by

customer_id

Contracts by

customer_id


The join graph topology

• Partitioning schemes: a partition function (e.g., hash partitioning, range partitioning, random partitioning, and custom partitioning), a partition key, and a partition count.

• Data exchange operators: initial partitioning, repartitioning, full merge, partial repartitioning and partial merge

• Merging schemes: random merge, sort merge, concat-merge and sort concat-merge.

• Distribution policies: include distribution with duplication and distribution without duplication


Joins from a functional perspective

• Think use cases : Model and distribute the data according to the queries you are expecting– Enrich a table with data from lookup tables, I need objects of

type A filtered by criteria on properties of type B

• Loosen the generality– I only want star schemas, I only want to bulk load data at night – I only want star schemas, I only want to bulk load data at night

and query it all day, I only want to run a few really expensive queries not millions of tiny ones

• Polyglot persistence : store your data in multiple ways– Graph databases for relations, document store for business

objects, key value for lookup properties

• Services and micro-services : you access the data through the services layer– Then you can implement the joins in many ways


The service layer

• Seamless integration for applications

• Caches the data model and exposes a business

model (services fit for business needs)

Service Layer

Customers Contracts …

Application 1 Application 2 …


Implementing your own joins• According to use cases and capacities of each module, joins

can be implemented at different levels

NoSQL Front End

Service Layer

Source 1

The datasource

executes the join

Join executed by the

NoSQL database

(~E.g. Parent Child in

Elasticsearch)

Parallel reading of the

two datasources and

join done by the Batch

(~Sort Merge Join)

Parallel reading of the

two datasources and

join done by the Batch

(~Sort Merge Join)

26

NoSQL Front EndSource 1

Source 2

Batch

(init or delta)

Real time data

injection

Join implemented by the

Service layer at query time

(~"My Own Custom Join")

Join implemented by the

Service layer at query time

(~"My Own Custom Join")

JMS

Queue

"GET" calls to the second

data source

(~Nested Loops Join)

"GET" calls to the second

data source

(~Nested Loops Join)

Read one source, hash it

then stream the other one

(~HashJoin)

Read one source, hash it

then stream the other one

(~HashJoin)

2015-11-21

Joins with NoSQL databases

Normalized database Document

{"film" : {

"id" : "183070",

"title" : "The Artist",

"published" : "2011-10-12",

"genre" : ["Romance", "Drama", "Comedy"],

"language" : ["English", "French"],

"persons" : ["persons" : [

{"person" : { "id" : "5079", "name" : "Michel

Hazanavicius", "role" : "director" }},

{"person" : { "id" : "84145", "name" : "Jean

Dujardin", "role" : "actor" }},

{"person" : { "id" : "24485", "name" : "Bérénice

Bejo", "role" : "actor" }},

{"person" : { "id" : "4204", "name" : "John

Goodman", "role" : "actor" }}

]

}}


SQL vs. NoSQL : the issue with joins :-)

• Let’s say you have two relational entities: Persons and Contracts– A Person has zero, one or more Contracts

– A Contract is attached to one or more Persons (eg. the Subscriber, the Grantee, …)

• Need a search services : • Need a search services : – S1: getPersonsDetailsByContractProperties

– S2: getContractsDetailsByPersonProperties

• Simple solution with SQL:SELECT P.* FROM P, C WHERE P.id = C.pid AND C.a = 'A‘

SELECT C.* FROM P, C WHERE P.id = C.pid AND P.a = 'A'


The issue with joins - solutions• Solution 1

– Store Persons with Contracts together for S1{"person" : { "details" : …, … , "contracts" : ["contract" :{"id" : 1, …}, …] }}

– Store Contracts with Persons together for S2{"contract" : { "details" : …, …, "persons" : ["person" :{"id" : 1, "role" : "S", …}, …]}}

• Issues with solution 1: – A lot of data duplication

– Have to get Contracts when indexing Persons and vice-versa

• Solution 2• Solution 2– Use the joins provided by the NoSQL system (Eg. Elasticsearch’s Parent/Child)

• Issues with solution 2:– Works in one way but not the other (only one parent for n children, a 1 to n relationship)

• Solution 3– Store Persons and Contracts separately

– Launch two NoSQL queries and join the results into your application to get the final response

– For S1 : First get all Contract ids by Contract properties, then get Persons by Contract ids (terms query or mget)

– For S2 : First get all Persons ids by Person properties, then get Contracts by Person ids (terms query or mget)

– The response to the second query can be returned “as is” to the client (pagination, etc.)


Optimizing tricks. Distributed.

• Model the data according to use cases

• Duplicate the data if different modeling

schemas are needed

• Loosen the generality• Loosen the generality

• Choose a good partitioning key

• Colocate joins as much as possible. The less

redistribution the better.

• Implement your own joins :-)


References

• Back to the future : SQL 92 for Elasticsearch? - Lucian Precup - NoSQL Matters Dublin 2014 (https://2014.nosql-matters.org/dub/wp-content/uploads/2014/09/lucian_precup_back_to_the_future_sql_92_for_elasticsearch.pdf)

• Oracle Database Online Documentation - Database SQL Tuning Guide (https://docs.oracle.com/database/121/TGSQL/tgsql_join.htm#TGSQL242)

• Advanced Join Strategies for Large-Scale Distributed Computation - Nicolas Bruno, YongChul Kwon, Ming-Chuan Wu - VLDB

• Advanced Join Strategies for Large-Scale Distributed Computation - Nicolas Bruno, YongChul Kwon, Ming-Chuan Wu - VLDB (http://www.vldb.org/pvldb/vol7/p1484-bruno.pdf)

• “Distributed joins are hard to scale”- Interview with Dwight Merriman by Roberto V. Zicari (http://www.odbms.org/blog/2011/02/distributed-joins-are-hard-to-scale-interview-with-dwight-merriman/)

• Joins and aggregations in a distributed NoSQL DB - Max Neunhöffer –NoSQL Matters Dublin 2014 (https://2014.nosql-matters.org/dub/wp-content/uploads/2014/09/NeunhoefferDublin.pdf)


Data & Analytics

Joins in a distributed world - Lucian Precup