View
140
Download
0
Category
Tags:
Preview:
DESCRIPTION
HBase Tracing. Elliott Clark. Who Am I. Elliott Clark eclark@apache.org HBase PMC HBase Committer HTrace Mantainer Cloudera Engineer CDH Distribution. What’s Going on During Failovers. Motivation. Want to improve MTTR First need to know what’s going on Google’s Dapper. Options. - PowerPoint PPT Presentation
Citation preview
1
HBase TracingElliott Clark
2
Who Am I
• Elliott Clark• eclark@apache.org
• HBase PMC• HBase Committer• HTrace Mantainer• Cloudera Engineer
• CDH Distribution
3
What’s Going on During Failovers
4
Motivation
• Want to improve MTTR• First need to know what’s going on• Google’s Dapper
5
Options
• Log all the things• We already do that.• Our logs are way too verbose as it is.• Hard to combine multiple logs
• HRegionServer• HMaster• Client
• Guess• Zipkin• Not Invented Here
6
Spoiler:Guessing Didn’t Win
7
About Zipkin
• Created by Twitter• Thanks
• Modeled after Dapper• Scala• Finagle
• Functional• Netty• Non-blocking
• Thrift• Cool UI• Pluggable Storage Backend
8
Dapper’s Terminology
• Span• Segment of a remote call• Contains Annotations• Has a parent trace• Can have multiple children spans• Can have multiple annotations
• Annotations• Trace
• Grouping of spans
9
Can’t use Cassandra and look at myself in the mirror
10
Hooking up Zipkin, HTrace, and HBase
Implementation
11
Zipkin HBase Storage
• Replaces need for Cassandra• Functional• Created a non-blocking client shim• Fixed Key Length
• ID’s are long• Mapping table
• ID -> Name• Name -> ID• Inspiration from OpenTSDB
• Open source on Github
12
HTrace
• Way to instrument Java Code• Used In HBase trunk• Coming to an install near you in 0.96.0• Pluggable where it emits Spans
• To a file• To sys.out
13
HTrace and Zipkin
• Modularize HTrace• Create htrace-zipkin• Create ZipkinSpanReceiver
• Receives spans from HTrace instrumented code• Converts to Thrift objects• Sends to Zipkin
• Background Flush• Batching
14
Let’s hope this works.
Demo
15
Recommended