Upload
kiyoto-tamura
View
201
Download
1
Embed Size (px)
DESCRIPTION
Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?
Citation preview
Fluentd:Towards Unified Logging Layer
Kiyoto Tamura@kiyototamura
Treasure Data, Inc.
• Not an ops engineer myself
• “marketing guy”
• My laptop isn’t very hipster
Why you shouldn’t listen to me
• Fluentd maintainer and
community manager
• Former programmer
• data nerd
Why you should listen to me
why Fluentd?
I was a data scientist before it was cool to be one.
http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-and-challenges/fulltext
Acquire Data (or so you think)
WUT!? Invalid UTF8?
Fix the encoding issue…
Yell at the engineers
Some columns are missing!?
Run the script…DIVISION BY
ZERO!!!
Hmm…
Logging.priority=> :not_super_high
analytics.priority=> :very_high
analytics.needs? :logs=> true
outage.priority=> :super_high
outage.needs? :logs=> [“no”, “shit”]
Unified Logging Layer
overview
127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}
Parse as JSON!
?
[“05/Feb/2012:17:11:55”,“web.access”,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}]
?
web.mongodb
web.file
web.hdfs
web.s3
web.mysql
Fluentd Design Patterns(credit: @tagomoris’s talk at RedDotRubyConf)
Real World Use Cases
• Automate monitoring!
• App and System metrics
• JSON everywhere
• 2000+ node• ~1B events/day• Forwarder-
Aggregator
Demo
Fluentd
Treasure Data
Twitter API#NED #ARG
Postgres + d3.js
Collect Raw Tweets
Tokenize + Count
serve + visualize
{ “install”: “gem install fluentd”, “website”: “www.fluentd.org”, “github” : “fluent/fluentd”, “twitter”: “@fluentd”}