View
30
Download
0
Embed Size (px)
Citation preview
HBase in Practice
Lars George – Partner and Co-Founder @ OpenCore
DataWorks Summit 2017 - Munich
NoSQL is no SQL is SQL?
About Us• Partner & Co-Founder at OpenCore • Before that• Lars: EMEA Chief Architect at Cloudera (5+ years)
• Hadoop since 2007• Apache Committer & Apache Member• HBase (also in PMC)
• Lars: O’Reilly Author: HBase – The Definitive Guide• Contact• [email protected] • @larsgeorge
Website: www.opencore.com
Agenda• Brief Intro To Core Concepts• Access Options• Data Modelling• Performance Tuning• Use-Cases• Summary
Introduction To Core Concepts
HBase Tables• From user perspective, HBase is similar to a database, or spreadsheet• There are rows and columns, storing values• By default asking for a specific row/column combination returns the current
value (that is, that last value stored there)
HBase Tables• HBase can have a
different schema per row• Could be called
schema-less• Primary access by
the user given row key and column name • Sorting of rows and
columns by their key (aka names)
HBase Tables• Each row/column coordinate is tagged with a version number, allowing
multi-versioned values• Version is usually
the current time(as epoch)• API lets user ask
for versions (specific, by count,or by ranges)• Up to 2B versions
HBase Tables• Table data is cut into pieces to distribute over cluster• Regions split table into
shards at size boundaries• Families split within
regions to groupsets of columnstogether• At least one of
each is needed
Scalability – Regions as Shards• A region is served by exactly
one region server• Every region server serves many
regions• Table data is spread over servers• Distribution of I/O
• Assignment is based on configurable logic• Balancing cluster load
• Clients talk directly to region servers
Column Family-Oriented• Group multiple columns into
physically separated locations• Apply different properties to each
family• TTL, compression, versions, …
• Useful to separate distinct data sets that are related• Also useful to separate larger blob
from meta data
Data Management• What is available is tracked in three
locations• System catalog table hbase:meta• Files in HDFS directories• Open region instances on servers
• System aligns these locations• Sometimes (very rarely) a repair may
be needed using HBase Fsck
• Redundant information is useful to repair corrupt tables
HBase really is….• A distributed Hash Map• Imagine a complex, concatenated key including the user given row key and
column name, the timestamp (version)• Complex key points to actual value, that is, the cell
Fold, Store, and Shift• Logical rows in tables are
really stored as flat key-value pairs• Each carries full coordinates
• Pertinent information can be freely placed in cell to improve lookup
• HBase is a column-family grouped key-value store
HFile Format Information• All data is stored in a custom (open-source) format, called HFile• Data is stored in blocks (64KB default)• Trade-off between lookups and I/O throughput• Compression, encoding applied _after_ limit check
• Index, filter and meta data is stored in separate blocks• Fixed trailer allows traversal of file structure• Newer versions introduce multilayered index and filter structures• Only load master index and load partial index blocks on demand
• Reading data requires deserialization of block into cells• Kind of Amdahl’s Law applies
HBase Architecture• One Master and many Worker servers• Clients mostly communicate with workers• Workers store actual data• Memstore for accruing• HFile for persistence• WAL for fail-safety
• Data provided as regions• HDFS is backing store• But could be another
HBase Architecture (cont.)
HBase Architecture (cont.)• Based on Log-Structured Merge-Trees (LSM-Trees)• Inserts are done in write-ahead log first• Data is stored in memory and flushed to disk on regular intervals or based
on size• Small flushes are merged in the background to keep number of files small• Reads read memory stores first and then disk based files second• Deletes are handled with “tombstone”
markers• Atomicity on row level no matter how
many columns • Keeps locking model easy
Merge Reads• Read Memstore & StoreFiles
using separate scanners• Merge matching cells into
single row “view”• Delete’s mask existing data• Bloom filters help skip
StoreFiles
• Reads may have to span many files
APIs and Access Options
HBase Clients• Native Java Client/API• Non-Java Clients• REST server• Thrift server• Jython, Groovy DSL• Spark
• TableInputFormat/TableOutputFormat for MapReduce• HBase as MapReduce source and/or target• Also available for table snapshots
• HBase Shell• JRuby shell adding get, put, scan etc. and admin calls
• Phoenix, Impala, Hive, …
Java APIFrom Wikipedia:• CRUD: “In computer programming, create, read, update, and delete are the four basic functions of persistent storage.”• Other variations of CRUD include • BREAD (Browse, Read, Edit, Add, Delete)• MADS (Modify, Add, Delete, Show)• DAVE (Delete, Add, View, Edit)• CRAP (Create, Retrieve, Alter, Purge)
Java API (cont.)• CRUD• put: Create and update a row (CU)• get: Retrieve an entire, or partial row (R)• delete: Delete a cell, column, columns, or row (D)
• CRUD+SI• scan: Scan any number of rows (S)• increment: Increment a column value (I)
• CRUD+SI+CAS• Atomic compare-and-swap (CAS)• Combined get, check, and put operation • Helps to overcome lack of full transactions
Java API (cont.)• Batch Operations• Support Get, Put, and Delete• Reduce network round-trips• If possible, batch operation to the server to gain better overall throughput
• Filters• Can be used with Get and Scan operations• Server side hinting• Reduce data transferred to client• Filters are no guarantee for fast scans
• Still full table scan in worst-case scenario• Might have to implement your own
• Filters can hint next row key
Data ModelingWhere’s your data at?
Key Cardinality• The best performance is gained from using row keys• Time range bound reads can skip store files• So can Bloom Filters
• Selecting column families reduces the amount of data to be scanned• Pure value based access
is a full table scan• Filters often are too, but
reduce network traffic
Key/Table Design• Crucial to gain best performance• Why do I need to know? Well, you also need to know that RDBMS is only working
well when columns are indexed and query plan is OK• Absence of secondary indexes forces use of row key or column name sorting• Transfer multiple indexes into one• Generate large table -> Good since fits architecture and spreads across cluster
• DDI• Stands for Denormalization, Duplication and Intelligent Keys• Needed to overcome trade-offs of architecture• Denormalization -> Replacement for JOINs• Duplication -> Design for reads• Intelligent Keys -> Implement indexing and sorting, optimize reads
Pre-materialize Everything• Achieve one read per customer request if possible• Otherwise keep at lowest number• Reads between 10ms (cache miss) and 1ms (cache hit)• Use MapReduce or Spark to compute exacts in batch• Store and merge updates live• Use increment() methods
Motto: “Design for Reads”
Tall-Narrow vs. Flat-Wide Tables• Rows do not split• Might end up with one row per region
• Same storage footprint• Put more details into the row key • Sometimes dummy column only• Make use of partial key scans
• Tall with Scans, Wide with Gets• Atomicity only on row level
• Examples• Large graphs, stored as adjacency matrix (narrow)• Message inbox (wide)
Sequential Keys<timestamp><more key>: {CF: {CQ: {TS : Val}}}
• Hotspotting on regions is bad!• Instead do one of the following:• Salting
• Prefix <timestamp> with distributed value• Binning or bucketing rows across regions
• Key field swap/promotion• Move <more key> before the timestamp (see OpenTSDB)
• Randomization• Move <timestamp> out of key or prefix with MD5 hash
• Might also be mitigated by overall spread of workloads
Key Design Choices• Based on access pattern, either use
sequential or random keys• Often a combination of both is needed• Overcome architectural limitations
• Neither is necessarily bad• Use bulk import for sequential keys and
reads• Random keys are good for random access
patterns
Checklist• Design for Use-Case
• Read, Write, or Both?• Avoid Hotspotting
• Hash leading key part, or use salting/bucketing• Use bulk loading where possible• Monitor your servers!• Presplit tables
• Try prefix encoding when values are small• Otherwise use compression (or both)
• For Reads: Restrict yourself• Specify what you need, i.e. columns, families, time range
• Shift details to appropriate position• Composite Keys• Column Qualifiers
Performance Tuning1000 knobs to turn… 20 are important?
Everything is Pluggable• Cell• Memstore• Flush Policy• Compaction
Policy• Cache• WAL• RPC handling• …
Cluster Tuning• First, tune the global settings
• Heap size and GC algorithm• Memory share for reads and writes• Enable Block Cache• Number of RPC handlers• Load Balancer• Default flush and compaction strategy• Thread pools (10+)
• Next, tune the per-table and family settings• Region sizes• Block sizes• Compression and encoding• Compactions• …
Region Balancer Tuning• A background process in the HBase
Master is tracking load on servers• The load balancer moves regions
occasionally• Multiple implementations exists• Simple counts number of regions• Stochastic determines cost• Favored Node pins HDFS block
replicas
• Can be tuned further• Cluster-wide setting!
RPC Tuning• Default is one queue for
all types of requests• Can be split into
separate queues for reads and writes• Read queue can be
further split into reads and scans
Stricter resource limits, but may avoid cross-starvation
Key Tuning• Design keys to match use-case• Sequential, salted, or random
• Use sorting to convey meaning• Colocate related data• Spread load over all servers• Clever key design can make use
of distribution: aging-out regions
Compaction Tuning• Default compaction settings are aggressive• Set for update use-case
• For insert use-cases, Blooms are effective• Allows to tune down compactions• Saves resources by reducing write amplification
• More store files are also enabling faster full table scans with time range bound scans• Server can ignore older files
• Large regions may be eligible for advanced compaction strategies• Stripe or date-tiered compactions• Reduce rewrites to fraction of region size
Use-CasesWhat works well, what does not, and what is so-so
Placing the Use-Case
Big Data Workloads
Lowlatency
Batch
Random Access Full ScanShort Scan
HDFS + MR(Hive/Pig)
HBase
HBase + Snapshots -> HDFS + MR/Spark
HDFS + SQL
HBase + MR/Spark
Big Data Workloads
Lowlatency
Batch
Random Access Full ScanShort Scan
HDFS + MR/Spark(Hive/Pig)
HBase
HBase + Snapshots -> HDFS + MR/Spark
HDFS + SQL
HBase + MR/Spark
Current Metrics
Graph data
Simple Entities
Hybrid Entity Time series + Rollup serving
Messages
Analytic archive
Hybrid Entity Time series+ Rollup generation
Index building
Entity Time series
SummaryWrapping it up…
What matters…• For optimal performance, two things need to be considered:• Optimize the cluster and table settings• Choose the matching key schema• Ensure load is spread over tables and cluster nodes
• HBase works best for random access and bound scans• HBase can be optimized for larger scans, but its sweet spot is short burst scans (can
be parallelized too) and random point gets
• Java heap space limits addressable space• Play with region sizes, compaction strategies, and key design to maximize result
• Using HBase for a suitable use-case will make for a happy customer…• Conversely, forcing it into non-suitable use-cases may be cause for trouble
Questions?
Thank You!@larsgeorge