Take care of your logs with ELK - entwickler.de · Take care of your logs withELK @MattKetmo –...

ELKTake care of your logs with

@MattKetmo – IPC Berlin 2015

Matthieu Moquet@MattKetmo

web engineer at

Logs

Why logs?Debug

Understand code Execution workflow

Metrics Monitoring

Which logs?

Syslog

Access Logs

Application Logs

How to view logs?

ssh  example.org  tail  -­‐f  /var/log/some.log

multitail colortail rainbow

grc …

You should stop doing this!

! Not easily readable ! No aggregations ! Bad performances on concurrent processes

Dataviz > Log messages

Send your logs

AWAY

Open Source

ELK

Elasticsearch Logstash Kibana

3 independent softwaresFlexible

Performant Scalable

« Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents »

— Wikipedia

elasticsearch

elasticsearch

Aggregations

Real-Time Data

Distributed

RESTful API

Schema-Free

High Availability

elasticsearch

Download at https://www.elastic.co/downloads/elasticsearch

bin/elasticsearch

elasticsearch

See it running…

curl  -­‐X  GET  http://localhost:9200/

elasticsearch

Index a document

curl  -­‐XPUT  http://localhost:9200/users/user/mattketmo  -­‐d  '{      "name":  "Matthieu  Moquet",    "age":  26}'

elasticsearch

Get a document

curl  -­‐XGET  http://localhost:9200/users/user/mattketmo

elasticsearchUse aggregations

curl  -­‐XPOST  "http://localhost:9200/users/user/_search"  -­‐d'  {        "size":  0,          "aggregations":  {                "age_avg":  {                        "avg":  {                                  "field":  "age"                          }                  }          }  }'

Logstash

Log Processing / ETL

Input

Output

Filters

echo  "[ALERT]  Some  error  message"      |  sed  -­‐e  "s/^\[\(.*\)\]  \(.*\)/\1,\2/"      >  output.csv  !

cat  output.csv  ALERT,Some  error  message

Logstash

Logstash

gelf syslog

nginx logs varnish logs

udp …

date dns

geoip grok

urldecode …

elasticsearch redis

graphite nagios zabbix

…

Inputs Filters Outputs

LogstashDownload at https://www.elastic.co/downloads/logstash

bin/logstash  -­‐e  '  input  {        stdin  {}    }    output  {        stdout  {}    }'

LogstashDownload at https://www.elastic.co/downloads/logstash

bin/logstash  -­‐f  /path/to/logstash.conf

Logstashinput  {        ...    }  filter  {      ...  }  output  {      elasticsearch_http  {          host  =>  "elasticsearch.tld"          port  =>  9200          index  =>  "logstash-­‐%{+YYYY.MM.dd}"      }  }

Curator

https://github.com/elastic/curator

Allow you to remove old indexes (via a CRON)

curator  -­‐-­‐host  10.0.0.2  delete  indices  \        -­‐-­‐older-­‐than  30  -­‐-­‐time-­‐unit  days  \        -­‐-­‐timestring  '%Y.%m.%d'

Heka

« Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing. »

http://hekad.readthedocs.org

«!Fluentd is an open source data collector for unified logging layer!»

Kibana

! Data visualization web app !Many graphes (histogram, pie chart, geo map, …) ! Built in HTML / CSS / Javascript

KibanaDownload at https://www.elastic.co/downloads/kibana

bin/kibana

upstream  es_backend  {          server  127.0.0.1:9200;          keepalive  64;  }  !server  {      listen  80;      server_name    kibana.tld;      root  /var/www/kibana;      try_files  $uri  $uri/  index.html  @elasticsearch;  !    location  @elasticsearch  {          proxy_pass  http://es_backend;          proxy_read_timeout  90;          proxy_redirect  off;          proxy_http_version  1.1;          proxy_set_header  Connection  "";          proxy_set_header    X-­‐Real-­‐IP    $remote_addr;          proxy_set_header    X-­‐Forwarded-­‐For  $proxy_add_x_forwarded_for;          proxy_set_header    Host  $http_host;          proxy_pass_header  Access-­‐Control-­‐Allow-­‐Origin;          proxy_pass_header  Access-­‐Control-­‐Allow-­‐Methods;          proxy_hide_header  Access-­‐Control-­‐Allow-­‐Headers;          add_header  Access-­‐Control-­‐Allow-­‐Headers  'X-­‐Requested-­‐With,  Content-­‐Type';          add_header  Access-­‐Control-­‐Allow-­‐Credentials  true;      }  }

Kibana 3 & nginx

Basic pipeline

Indexer

Our current pipeline

Shipper + Indexer

Broker

Kafka pipeline

1M+ messages per second

Logstash >= 1.5.0

How to forward logs?

Forwarding Syslog

RSYSLOGThe rocket-fast system for log processing

*.*  @127.0.0.1:514;RYSLOG_ForwardFormat

Logstash input

input  {        udp  {          port  =>  514          type  =>  syslog      }    }  

Logstash filter

filter  {      if  [type]  ==  "syslog"  {          grok  {              match        =>  ["message",  "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp}  %{SYSLOGHOST:syslog_hostname}  %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:  %{GREENDYDATA:syslog_message}"]              add_field  =>  ["received_at",  "%{@timestamp}"]              add_field  =>  ["received_from",  "%{host}"]              add_tag      =>  ["rsyslog"]          }      }  }

Forwarding Access Log

logstash-forwarderA tool to collect logs locally in preparation for processing elsewhere

https://github.com/elastic/logstash-forwarder

logstash-­‐forwarder  -­‐config  conf.json

logstash-forwarder...  {      "paths":  [            "/var/log/nginx/access.log"      ],      "fields":  {  "type":  "nginx-­‐access"  }  }  ...

logtailSmall Perl script to read HTTP access logs

and send it directly to Redis !

https://github.com/shtouff/logtail

Forwarding Application Logs

Monologhttps://github.com/Seldaek/monolog

PSR-3php-fig.org/psr/psr-3

github.com/php-fig/log

<?php  !namespace  Psr\Log;  !interface  LoggerInterface  {          public  function  log($level,  $message,  array  $context  =  array());  !        //  Shortcuts          public  function  emergency($message,  array  $context  =  array());          public  function  alert($message,  array  $context  =  array());          public  function  critical($message,  array  $context  =  array());          public  function  error($message,  array  $context  =  array());          public  function  warning($message,  array  $context  =  array());          public  function  notice($message,  array  $context  =  array());          public  function  info($message,  array  $context  =  array());          public  function  debug($message,  array  $context  =  array());  }

<?php  !namespace  Psr\Log;  !class  LogLevel  {          const  EMERGENCY  =  'emergency';          const  ALERT  =  'alert';          const  CRITICAL  =  'critical';          const  ERROR  =  'error';          const  WARNING  =  'warning';          const  NOTICE  =  'notice';          const  INFO  =  'info';          const  DEBUG  =  'debug';  }

Usage: PHPps4,4.!-(%.B6#6&6EUP6EE%+d.-(%.B6#6&6EUm'#>&%+UI)+%'<m'#>&%+d.!$$.j+%')%.'.&6E.*,'##%&.h,'#>&%+.\.#%g.I)+%'<m'#>&%+V?4'),$)6$b6-+W&6E?D.P6EE%+55nOQqkq1Yd.h&6E.\.#%g.P6EE%+V?#'<%?Yd.h&6E/[4-(,m'#>&%+Vh,'#>&%+Yd.!$$.O>>.+%*6+>(.)6.),%.&6E.h&6E/[g'+#"#EV?l66?Yd.h&6E/[%++6+V?u'+?Yd

Usage: Symfonymonolog:          handlers:                  main:                          type:  fingers_crossed                          action_level:  warning                          handler:  file                  file:                          type:  stream                          level:  debug                          path:  /var/log/symfony.log                  syslog:                          type:  syslog                          level:  error

Usage: Symfony Controllerps4,4.!#'<%(4'*%.O*<%Uu-#>&%UO44u-#>&%Uj6#)+6&&%+d.!-(%.Ib<M6#bUu-#>&%Ul+'<%g6+=u-#>&%Uj6#)+6&&%+Uj6#)+6&&%+d.!*&'((.l66j6#)+6&&%+.%])%#>(.j6#)+6&&%+.@.....4-!&"*.M-#*)"6#.!'+O*)"6#VY.....@.........h),"(/[E%)V?&6EE%+?Y/["#M6V?2]%*-)"#E.M66.!'+?Yd.!........+%)-+#.h),"(/[+%#>%+V?iO44u-#>&%$l66$!'+W,)<&W)g"E?Yd.....G.G

Usage: Symfony Service

-(%.:(+UP6EUP6EE%+k#)%+M'*%d.-(%.:(+UP6EUq-&&P6EE%+d.!*&'((.O*<%I%+L"*%.@.....4+6)%*)%>.hM66d.....4+6)%*)%>.h&6EE%+d.!....4-!&"*.M-#*)"6#.JJ*6#()+-*)Vl66.hM66D.P6EE%+k#)%+M'*%.h&6EE%+.\.#-&&Y.....@.........h),"(/[M66.\.hM66d.........h),"(/[&6EE%+.\.h&6EE%+.s5.#%g.q-&&P6EE%+VYd.....G.G

Avoid "M.V#-&&.v\\.h&6EE%+Y

Usage: Symfony Service-(%.:(+UP6EUP6EE%+k#)%+M'*%d.-(%.:(+UP6EUq-&&P6EE%+d.!*&'((.O*<%I%+L"*%.@.....4+6)%*)%>.hM66d.....4+6)%*)%>.h&6EE%+d.!....4-!&"*.M-#*)"6#.JJ*6#()+-*)Vl66.hM66Y.....@.........h),"(/[M66.\.hM66d.........h),"(/[&6EE%+.\.#%g.q-&&P6EE%+VYd.....G.!....4-!&"*.M-#*)"6#.(%)P6EE%+VP6EE%+k#)%+M'*%.h&6EE%+Y.....@.........h),"(/[&6EE%+.\.h&6EE%+d.....G.G

40+ HandlersAmqpHandler

ErrorLogHandler FirePHPHandler

GelfHandler MailHandler

NewRelicHandler NullHandler

RedisHandler StreamHandler

SyslogUdpHandler ...

SyslogHandler

<6#6&6E5...,'#>&%+(5......(b(&6E5.........)b4%5.(b(&6E.........&%L%&5.g'+#"#E

Ops problem now!

GelfHandlermonolog:          handlers:                  main:                          type:  fingers_crossed                          action_level:  warning                          handler:  file                  gelf:                          type:  gelf                          level:  notice                          publisher:                                  hostname:  %logstash_host%

Logstash input

input  {      gelf  {          port  =>  12201          type  =>  gelf      }  }

Logstash filter

filter  {}

Logstash output

output  {      elasticsearch  {          host  =>  ["127.0.0.1"]          port    =>  9200          index  =>  "logstash-­‐%{+YYYY.MM.dd}"      }  }

Log

{      "message":  "Lorem  ipsum",      "level":  "200",      "level_name":  "info",      "@timestamp":  "1432825193000"  }

Wait! There is more…

ChannelsUseful to group logs by category

Make it easy to filter Use different rules / handlers

Channels

- assetic - doctrine - event - php - (php_error) - profiler - request - …

Symfony is shipped with many default channels

Usage: Symfony

monolog:          channels:                  -­‐  api                  -­‐  business_domain                  -­‐  super_feature                  -­‐  rabbitmq

Create as many as you want

Usage: Symfony

monolog:          handlers:                  main:                          type:  stream                          path:  /var/log/symfony.log                          channels:  !doctrine                  doctrine:                          type:  stream                          path:  /var/log/doctrine.log                          channels:  doctrine

Usage: Symfony

monolog:          handlers:              default_notice:                      type:  gelf                      level:  NOTICE                      channels:  [request,  security]              default_info:                      type:  gelf                      level:  INFO                      channels:  [!request,  !security]

Usage: Symfony

services:          my_service:                  class:  Acme\Class\Name                  arguments:  ["@logger"]                  tags:                      -­‐  name:  monolog.logger                          channel:  acme

Channels

Visualize proportions Easy filtering

ContextMetadata for your logs

Context

interface  LoggerInterface  {          public  function  log(                  $level,                    $message,                    array  $context  =  array()          );  }

Use it! Abuse it!

Context

h&6EE%+/["#M6V?;(%+.we8.,'(.&6E6-)?Yd

h&6EE%+/["#M6V?;(%+..&6E6-)?D.N?-(%+J">?.\[.e8RY

Instead of building dynamic log messages

Use static strings and add data into the context

@.....A)%<4&')%A5.A&6E()'(,/XAD.....A(%))"#E(A.5.AWWWAD.....A<'44"#E(A5.@.........A(b(&6EA5.AWWWAD.........A,))4A5.AWWWAD.........AE%&MA5.@.............A>b#'<"*J)%<4&')%(A5.N@.................A*)])A5.@.....................A<')*,A5.A*)])JXAD.....................A<')*,J<'44"#EJ)b4%A5.A()+"#EAD.....................A<'44"#EA5.@.........................A)b4%A5.A()+"#EAD.........................A"#>%]A5.A#6)J'#'&bK%>A.....................G.................G.............GRD.............A4+64%+)"%(A.5.@.................Ai)"<%()'<4A.5.@.A)b4%A.5.A>')%AD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A<%(('E%A.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A'#'&bK%>A.GD.................AM'*"&")bA.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A)b4%A.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.GD.................A*)])J-(%+J">A.5.@.A)b4%A.5.A()+"#EAD.A"#>%]A.5.A#6)J'#'&bK%>A.G.............G.........G.....G.G

ProcessorAdd metadata for ALL your logs

ProcessorsGitProcessor

IntrospectionProcessor MemoryPeakUsageProcessor

MemoryProcessor MemoryUsageProcessor

ProcessIdProcessor PsrLogMessageProcessor

TagProcessor UidProcessor

WebProcessor

ProcessorCurrent user ID

User-Agent Locale

Country Code IP

Current route Application name / type

Environment (dev / staging / prod) Request UUID

API client …

4-!&"*.M-#*)"6#.JJ"#L6=%V'++'b.h+%*6+>Y.@.....h%#L.....\.h),"(/[*6#)'"#%+/[E%)V?=%+#%&?Y/[E%)2#L"+6#<%#)VYd.....h*6#)%]).\.h),"(/[*6#)'"#%+/[E%)V?*6#)%])?Yd.

....h+%*6+>N?%])+'?RN?%#L?R.....\.h%#Ld.

....h+%*6+>N?%])+'?RN?&6*'&%?R..\.h*6#)%])/[E%)P6*'&%VYd.

....h+%*6+>N?%])+'?RN?<%>"'?R...\.h*6#)%])/[E%)B%>"'VYd.

....h+%*6+>N?%])+'?RN?*&"?R.....\.h*6#)%])/["(j&"VY.s.Z.5.9d.

"M.Vh),"(/[*6#)'"#%+/["(I*64%O*)"L%V?+%C-%()?YY.@.........h+%C-%().\.h),"(/[*6#)'"#%+/[E%)V?+%C-%()J()'*=?Y/[E%)B'()%+Q%C-%()VYd.

"M.Vh+%C-%()/[,%'>%+(/[,'(V?0/Q%C-%()/k>?YY.@.............h+%C-%()k>.\.h+%C-%()/[,%'>%+(/[E%)V?0/Q%C-%()/k>?Yd.............h+%*6+>N?%])+'?RN?+%C-%()J">?R.\.h+%C-%()k>d.........G.

........h+%*6+>N?%])+'?RN?+6-)%?R.\.h+%C-%()/[E%)V?J+6-)%?Yd.

........h+%*6+>N?%])+'?RN?*&"%#)J"4?R.\.h+%C-%()/[E%)j&"%#)k4VYd.

....G.

"M.V#-&&.v\\.h-(%+.\.h-(%+B'#'E%+/[E%)j-++%#);(%+VYY.@.........h+%*6+>N?%])+'?RN?-(%+J">?R.\.h-(%+/[E%)k>VYd.....G.

$$.WWW.

+%)-+#.h+%*6+>d.G

Context + Processor

h4+6t%*).\.#%g.:+6t%*)Ve8D.?M66!'+?Yd.!h&6EE%+/["#M6V?:+6t%*).*+%')%>?D.N.....?4+6t%*)?.\[.h4+6t%*).RYd.

Use processor to pretty format the context

Context + Processor*&'((.:+6t%*):+6*%((6+..@.....4-!&"*.M-#*)"6#.JJ"#L6=%V'++'b.h+%*6+>(Y.....@........."M.Vv"((%)Vh+%*6+>(N?*6#)%])?RN?4+6t%*)?RYY.@.............+%)-+#d.........G.!........h4+6t%*).\.h+%*6+>(N?*6#)%])?RN?4+6t%*)?Rd........."M.Vvh4+6t%*)."#()'#*%6M.:+6t%*)Y.@.............+%)-+#d.........G.!........h+%*6+>(N?*6#)%])?RN?4+6t%*)?R.\.N.............?">?...\[.h4+6t%*)/[E%)k>VYD.............?#'<%?.\[.h4+6t%*)/[E%)q'<%VYD.........R.!........+%)-+#.h+%*6+>(d.....G.G

ELK is very powerful & flexible

Other usages

Business Metrics

Business Metrics

http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/

PacketBeat: Performance Management

Suricate: Security

https://home.regit.org/tag/suricata/

Graphana: Monitoring

Thank You

Slides available at

moquet.net/talks/ipc-2015-elk

Leave feedbacks at @MattKetmo