Upload
cask-data-inc
View
964
Download
0
Embed Size (px)
Citation preview
Using Apache Kylin for Large Scale Data Analytics @
eBay
http://kylin.io
Agenda Big Data @ eBay What’s Apache Kylin? Cube Segments &
Streaming Q&A
Big Data @ eBay
$28BGMV VIA MOBILE
(2014)
266MMOBILE APP
GLOBALLY
1BLISTINGS CREATED
VIA MOBILE
157MACTIVE BUYERS
25MACTIVE SELLERS
800MACTIVE LISTINGS
8.8MNEW LISTINGS
EVERY WEEK
CHECKOUT
WATCH LIST
SHARE
LOCAL
PREFERENCES
BID
100’s PB 9 Quarters of historical
data
100MQUERIES/MONTH
20TBBehavior Data Capture
every day
CLICK
SEARCHDA
TA
Hadoop@ eBay
1-10 nodes2007
100+ nodes
1000s + core1 PB
2010
20111000+ node10,000+ core10+ PB
3000+ node10,000+ core
50+ PB2012
2013/201410,000 nodes150,000+ cores170 PB +> 2000 users2009
50+ nodes
Extreme OLAP Engine for Big DataApache Kylin is an open source Distributed Analytics Engine from eBay that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets
What’s Kylin
kylin / ˈkiːˈlɪn / 麒麟--n. (in Chinese art) a mythical animal of composite form
• Open Sourced on Oct 1st, 2014• Accepted as Apache Incubator Project on Nov 25th, 2014
Business Needs for Big Data Analysis
Sub-second query latency on billions of rows ANSI SQL for both analysts and engineers Full OLAP capability to offer advanced functionality Seamless Integration with BI Tools
Support of high cardinality and high dimensions High concurrency – thousands of end users Distributed and scale out architecture for large data
volume
Apache Kylin AdoptionExternal
25+ contributors in community
Adoption:In Production: eBay, Baidu MapsEvaluation: Huawei, Bloomberg Law, British Gas, JD.com
Microsoft, Tableau… eBay Use Cases
Case Cube Size Raw RecordsUser Session Analysis 14+TB 75+ billion rowsTraffic Analysis 21+ TB 20+ billion rowsBehavior Analysis 560+ GB 1.2+ billion rows
Kylin Architecture Overview
9
Cube Builder (MapReduce…)
SQL
Low Latency - SecondsRouting
3rd Party App(Web App, Mobile…)
Metadata
SQL-Based Tool(BI Tools: Tableau…)
Query Engine
HadoopHive
REST API JDBC/ODBC
Online Analysis Data Flow Offline Data Flow
Clients/Users interactive with Kylin via SQL
OLAP Cube is transparent to users
Star Schema Data Key Value Data
Data Cube
OLAPCubes(HBase)
SQL
REST Server
Data
Sou
rce
Abst
racti
on Engine
Abstraction
Stor
age
Abst
racti
on
Cube Storage in HBase
OLAP Cube – Balance between Space and Time
time, item
time, item, location
time, item, location, supplier
time item location supplier
time, location
Time, supplier
item, locationitem, supplier
location, supplier
time, item, supplier
time, location, supplier
item, location, supplier
0-D(apex) cuboid
1-D cuboids
2-D cuboids
3-D cuboids
4-D(base) cuboid
Base vs. aggregate cells; ancestor vs. descendant cells; parent vs. child cells
1. (9/15, milk, Urbana, Dairy_land) - <time, item, location, supplier>2. (9/15, milk, Urbana, *) - <time, item, location>3. (*, milk, Urbana, *) - <item, location>4. (*, milk, Chicago, *) - <item, location>5. (*, milk, *, *) - <item>
Cuboid = one combination of dimensions Cube = all combination of dimensions (all cuboids)
Dynamic data management framework. Formerly known as Optiq, Calcite is an Apache incubator project, used by
Apache Drill and Apache Hive, among others. http://optiq.incubator.apache.org
Querying the Cube
Metadata SPI Provide table schema from Kylin metadata
Optimization Rules Translate the logic operators into kylin operators
Relational Operators Find right cube Translate SQL into storage engine API call Generate physical execute plan by linq4j java implementation
Result Enumerator Translate storage engine result into java implementation result.
SQL Function Add HyperLogLog for distinct count Implement date time related functions (i.e. Quarter)
Query Engine based on Calcite
Full Cube Pre-aggregate all dimension combinations “Curse of dimensionality”: N dimension cube has 2N cuboid.
Partial Cubes To avoid dimension explosion, we divide the dimensions into
different aggregation groups 2N+M+L 2N + 2M + 2L
For cube with 30 dimensions, if we divide these dimensions into 3 group, the cuboid number will reduce from 1 Billion to 3 Thousands
230 210 + 210 + 210
Tradeoff between online aggregation and offline pre-aggregation
Optimizing Cube Builds – Full Cube vs. Partial Cube
Optimizing Cube Builds: Partial Cubes
Optimizing Cube Building – Incremental Building
Optimizing Cube Builds: Improved Parallelism• The current algorithm
– Many MRs, the number of dimensions– Huge shuffles, aggregation at reduce side, 100x of total cube size
Full Data
0-D Cuboid
1-D Cuboid
2-D Cuboid
3-D Cuboid
4-D CuboidMR
MR
MR
MR
MR
Optimizing Cube Builds: Improved Parallelism
• The to-be algorithm, 30%-50% faster– 1 round MR– Reduced shuffles, map side aggregation, 20x total cube
size– Hourly incremental build done in tens of minutes
Data Split
Cube Segment
Data Split
Cube Segment
Data Split
Cube Segment
……
Final Cube
Merge Sort(Shuffle)
mapper mapper mapper
Historical DataRecent Data
Kafka Topics
SQL Query
Optimizing Cube Builds: Streaming
Inverted Index
Hybrid Storage Interface
Cube Segments
Use Case: SEO Operational Dashboard• eBay Site
– ebay.com, ebay.co.uk, ebay.de• Buyer Country
– US, CN, RU• Search Engine
– Google, Bing, Yahoo!• Referrer
– google.com, google.co.uk• Page
– Search, View Item, Product• User Experience
– Desktop, Mobile APP, mWeb
• Visits, GMB $, GMB share, conversion rate, bounce rate, # of view items, # of bought items etc.
Dimensions
Measurements
Q & A