Embed Size (px)
Citation preview
State of the Database
@HBase h)p://hbase.apache.org
2015-‐09-‐28
Nick Dimiduk (@xefyr) h)p://n10k.com
#apachebigdata
Agenda
o State of the Project o State of the SoMware o State of the Ecosystem o Latest Releases o Bonus Content! o Q & A
STATE OF THE PROJECT Who we are, what we do, why we do it
Project: Vision
Simple, steady, and powerful: “A first class high performance horizontally scalable data storage engine for Big Data, suitable as the store of record for mission cri]cal data.”
Project: Usage
o Data access for medium-‐ and high-‐scale services o Hundreds of enterprises and startups o Some of the largest Internet companies in the world
o Running major produc]on workloads since 2011 o Use-‐cases
o messaging, security, measurement/“IoT”, collabora]on, digital media, digital adver]sing, telecommunica]ons, computa]onal biology, clinical informa]cs/healthcare, insurance
Project: Goals
o Availability: Always more, always faster o Stability and operability o Scaling up, scaling down o Up-‐to-‐date with “commodity” hardware o Mul]-‐tenancy o Diversity of ecosystem
STATE OF THE SOFTWARE Regarding the codebase
State of the SoMware
o Mature codebase o 100+ contributors (40+ commi)ers) o 1.1M lines of code (each ac]ve branch) o est. 1200+ human-‐years’ effort
o Clusters sizes from 10 to 1000+ machines o that we know of!
o Runs on HDFS, MapR, Gluster, GPFS, Lustre o HBase as a Service
o AWS/EMR, HDInsight, Qubole, Google (sort-‐of)
SoMware: Releases
SoMware: Seman]c Versioning
MAJOR-‐MINOR-‐PATCH[-‐iden]fier]
o Client/Server wire compa]bility o Server/Server feature compa]bility o API compliance guarantees o ABI compliance guarantees
h)p://hbase.apache.org/book.html#hbase.versioning
SoMware: Ac]ve Development
o Smaller regions, more regions o Less write amplifica]on o 1M+ region clusters
o Stability o ProcedureV2 o Assignment improvements/stability
o Backup, restore tools o Built on snapshots, easier opera]ons
SoMware: Ac]ve Development
o Adap]on: Workloads o HBase as Medium Object Store (MOB)
o Tunable Availability o Region replicas o TIMELINE consistency
o Coprocessor API stability o Less GC, more RAM (off-‐heap)
SoMware: Ac]ve Development
o Mul]-‐tenancy o Table groups o Quotas o Priori]es
o Improved machine u]liza]on o More RAM (100’s of GB) o IOPS o Be)er concurrency
STATE OF THE ECOSYSTEM The whole enchilada
State of the Ecosystem
o OpenTSDB o Transac]on Managers
o Themis, Tephera, Omid2, LeanXcale o Graph engines
o Titan, Giraph, Zen, S2Graph o Myriad SQL’s o Other Hadoop components o Google Cloud Bigtable
Ecosystem: SQL
Ecosystem: Hadoop Components
o YARN-‐2928 Applica]on Timeline Service o HIVE-‐9452 HBase to store Hive metadata o AMBARI-‐5707 Ambari Metrics System
LATEST RELEASES Come and get it!
Release: 0.94
o Last (final?) release: 0.94.27, 2015-‐03-‐26 o “ancient history”
o No new deployments o Exis]ng users highly encouraged to upgrade
o Requires down]me to upgrade 😫 😡 (╯°□°)╯（ ┻━┻
Release: 0.98
o Last release: 0.98.14, 2015-‐08-‐31 o “legacy”
o Most produc]on deploys (probably) o Largest produc]on clusters (probably) o New features back-‐ported when possible
Release 1.x
o Last release: 1.1.2, 2015-‐09-‐01 o “stable”
o Produc]on deploys moving here o Ac]ve development
o Rolling upgrade from 0.98.x 😄 😍 ヽ(´ー｀)ﾉ
Release 1.0
o Released 1.0.0, 2015-‐02-‐24 o Adop]ng seman]c versioning
o Patch releases don’t quite follow spec yet o Client / Server API cleanup
o Interfaces, builder pa)ern, @InterfaceAudience o Region Replicas
o Trade Consistency, resources for Availability
github.com/ndimiduk/hbase-‐1.0-‐api-‐examples
Region Replicas
o Mul]ple Region Servers host each region o Primary + N read replicas (usually N=2) o Primary is authority on reads and writes o Replicas tail replicate edits, offer TIMELINE view
o Client’s choice o Read primary only for “classic” strong consistency o Fan-‐out reads for faster, poten]ally TIMELINE results
Release 1.1
o Release 1.1.0, 2015-‐05-‐15 o Async RPC client o Scanner improvements
o RPC chunking, heartbeat messages, API o RPC thro)ling
o quotas for per user, table, namespace o Compac]on thro)ling, monitoring o ProcedureV2
o Improved opera]onal reliability
ProcedureV2
o Distributed, fault-‐tolerant opera]ons o Mul]ple steps on mul]ple machines o Roll-‐back in case of failure
o Coordina]on of long-‐running procedures o Compac]ons, splits, &c.
o Progress tracking o No]fica]ons across mul]ple machines o Current status inquiries
Branch-‐1.2
o Next up in 1.x line o Java 8 support o Na]ve checksums o SyncTable
o Flush-‐per-‐store o ProcV2 all the things! o (More) Compac]on improvements
o Region normalizer
Region Normalizer o An]-‐entropy for region size
o Converge towards uniform size o Compliments balancer working toward uniform distribu]on
o Managed by Master, runs in the background (like balancer) o Pluggable normaliza]on strategies (“simple” default) o Use-‐cases
o Merge away regions from expired ]meseries data o Smooth uneven bulk loads o Correct operator ini]al split guesses o Ease upgrades from ancient versions (0.92/1g vs. today/20g)
Thanks!
@HBase h)p://hbase.apache.org
2015-‐09-‐28
Nick Dimiduk (@xefyr) h)p://n10k.com
#apachebigdata
BONUS CONTENT! Ask and you shall receive
Agenda
o Replica]on o Filters o Coprocessors
Replica]on
o Keep data synchronized between clusters o Supports mulOple desOnaOons o Cyclical graphs supported o Configurable at Column Family granularity
o Uses WAL shipping to propagate data o Replica]on state, status stored in ZooKeeper o General purpose interface for asynchronously shipping edits from a cluster o Other HBase clusters, Region Replicas, SOLR/Elas]cSearch
hbase.apache.org/book.html#_cluster_replica]on
Filters o Addi]onal applied to reads
o Use in conjunc]on with specifying start, end rows, &c. o Run on the Region Servers
o Included in GET, SCAN request o Explicitly exclude data based on criteria
o I.E., value >= 10 o Implicitly exclude data by hin]ng seeks
o INCLUDE_AND_NEXT_COL, NEXT_ROW, SEEK_NEXT_USING_HINT
o Operate on data read from BlockCache
Filters
o 30+ Filters included in distribu]on o Mini-‐language for use in ThriM, REST
o "(PrefixFilter ('row2') AND (QualifierFilter (>=, 'binary:xyz'))) AND (TimestampsFilter ( 123, 456))"
o hbase.apache.org/book.html#thriM.filter_language
o Simple interface, Implement your own!
public class PageFilter extends FilterBase { public PageFilter(long pageSize) {…} public boolean filterRowKey(Cell c) { return false; } public ReturnCode filterKeyValue(Cell c) { return ReturnCode.INCLUDE; } public boolean filterAllRemaining() { return this.rowsAccepted >= this.pageSize; } public filterRow() { this.rowsAccepted++; return this.rowsAccepted > this.pageSize; } }
Coprocessors
o Extension points for HBase o Think Linux Kernel Module, not Stored Procedure o I.E., customize compac]ons, Table constraints
o Observers o pre-‐ and post-‐execu]on logic o I.E., MasterObserver#preTruncateTable, RegionObserver#postScannerNext
o Endpoints o Cluster RPC extensions o I.E., RowCountEndpoint, BulkDeleteEndpoint
public class RowCountEndpoint implements ExampleProtos.RowCountService {
public void getRowCount(…) { Scan = new Scan();
InternalScanner scanner =
env.getRegion().getScanner(scan); …
do { count++;
} while (scanner.next()); // return count
} }
Thanks!
@HBase h)p://hbase.apache.org
2015-‐09-‐28
Nick Dimiduk (@xefyr) h)p://n10k.com
#apachebigdata
