DruidKylin

2017-08-05

OLAP

Druid

Kylin

OLAP

ROLAP (Relational OLAP)

RDBMS/

Presto / Impala / SparkSQL / Drill

MOLAP (Multi-dimensional OLAP)

Kylin / Druid

Search Engines

Elasticsearch / Solr

ROLAP (Presto / SparkSQL)

SQL

MOLAP (Kylin, Druid)

OLAP

Search Engine (ES)

Join

OLAP

1.

2. /

3.

OLAP

SQL

KylinDruid

Presto / Spark Kylin Druid ES

SQL

OLAP

Druid

Kylin

Druid

OLAP

40100Datasource20 TB

Kafka

150TP99~1

Druid/JVM

JVM

Coordinator/Overlord 2 x 8g VMs 6g Heap

Broker 8 x 8core 16g VMs 11g Heap, 3g Non-Heap

Historical 19 x 40core 128g 12disk 12g Heap, 10g Non-Heap

MiddleManager 19 x 40core 128g 8 x 6g Heap Peons

Tranquility Clients 2 x 40core 128g 2~3g per JVM

Druid

DataSourceQPSLatencyIngestion Rate

/CPUI/OJVM

DashboardOLAP

Kafka ESEmitter

druid

Spark Streaming

DruidDashboard

Druid

Filter

Agg

Grouping

Tier

historical historical

historical historical

broker broker

Load Balance

_default_tier

historical historical

historical historical

broker broker

Load Balance

workload

(1/2)

Historicalsys cpu

NUMAzone reclaim modedirect page scan

echo 0 > /proc/sys/vm/zone_reclaim_mode

(2/2)

CentOS 6.5

echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag

SQL

OLAP

Druid

Kylin

Kylin

KylinHadoopOLAPSQL

KylinMOLAPCube

CubeCube Cube

Kylin

Kylin

OLAP

20+Project350Cube

8000Cube140TB

130TP99~1

Kylin

RS RS

job jobHA

A

query queryB

query query

HBase

RS

RS RSRS

Cube

1. 2. 3. 4. Union

1. 2. HBaseHA 3. 4.

1. 2. 3. 4. JMX

3Kylin Committer

Cube