Application Logging With The ELK Stack

Application Logging With The ELK Stack

@bwaine - #DPC15

Monday, 29 June 15

2

Ben Andersen-Waine

Software Engineer Contractor

Deployed ELK To Prod Numerous Times

Monday, 29 June 15

Logging?

Monday, 29 June 15

System Logs

Monday, 29 June 15

5

Monday, 29 June 15

Application Log

Monday, 29 June 15

Debug Information - Errors (connections, uncaught exceptions, resource exhaustion)

Narrative Information - Methods Calls, Event Triggers

Business Events - Purchases, Logins, Registrations, Unsubscribes

7

Application Log

Monday, 29 June 15

ssh webserver@mydomain.nettail -f /var/log/nginx/my-site.access.logtail -f /var/log/my.application.log

ssh data@mydomain.nettail -f /var/log/mysql/mysql.log

ssh q@mydomain.nettail -f /var/log/rabbitmq/nodename.log

8

Keeping Track Of All This....

Monday, 29 June 15

9

The Elk Stack

Monday, 29 June 15

Monday, 29 June 15

1) Monolog2) Everything else....

11

PHP Logging Tools

Monday, 29 June 15

1) Monolog: Loggers And Handlers2) Monolog: Tags & Formatters3) Logging business events

12

Basic Logging Examples

Monday, 29 June 15

use Monolog\Logger;use Monolog\Handler\FingersCrossedHandler;use Monolog\Handler\StreamHandler;

$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;

$appLog = new Logger('AppLog');

$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level);

$appLog−>pushHandler($fcHandler);$appLog−>debug('LOGGING!');

EG1: Loggers And Handlers

13

Monday, 29 June 15

// Set A Log Level$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;

// Create A Logger$appLog = new Logger('AppLog');

14

Monday, 29 June 15

$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);

$fcHandler= new FingersCrossedHandler($strHandler, $level);

// Create Handlers

$appLog−>pushHandler($fcHandler);

$appLog−>debug('Start Logging!');$appLog−>emergency('Something Terrible Happened');

// Push The Handler And Start Logging

15

Monday, 29 June 15

EG 2: Tagging Formatting

$appLog = new Logger('AppLog');

$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "application");

$strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler));

$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])

$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");

16

Monday, 29 June 15

// Create A Logger$appLog = new Logger('AppLog');

$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "app");

// Create A Handler & Formatter

// Set Formatter Onto Handler$strHandler−>setFormatter($formatter);

$appLog−>pushHandler($strHandler));

//Push Handler Onto Logger

17

Monday, 29 June 15

$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");

// Capture A Unique Id, Create A Tag Processor, Push

18

Monday, 29 June 15

2009 - RFC 5424 - Syslog Protocol

Code / Severity

0 Emergency: system is unusable1 Alert: action must be taken immediately2 Critical: critical conditions3 Error: error conditions4 Warning: warning conditions5 Notice: normal but significant condition6 Informational: informational messages7 Debug: debug-level messages

https://tools.ietf.org/html/rfc542419

Log Levels

Monday, 29 June 15

2013 - PSR03 - PHP Logging Interface Standard

http://www.php-fig.org/psr/psr-3/

20

PSR3

Monday, 29 June 15

EG 3: Event Logginguse Monolog\Logger;use Symfony\Component\EventDispatcher\EventDispatcher;

$dispatcher = new EventDispatcher();

$dispatcher−>addListener( "business.registration.post", function () use ($busLog) { $busLog−>info("Customer registered"); });

$dispatcher−>dispatch("business.registration.post");

Monday, 29 June 15

Logstash Architecture

1. Logstash Shipper ships logs to logstash

2. Logstash processes them

3. Logstash Inserts Into Elastic Search

4. Kibana exposes a web interface to Elastic Search data

Monday, 29 June 15

Logstash Architecture

Monday, 29 June 15

Why not rate the talk now BEFORE the demo?

24

https://joind.in/talk/view/14235

Monday, 29 June 15

ELK Demo

25

1) Discover Data (search / diagnose)2) Visualize Data 3) Produce A Dashboard 4) Demonstrate ‘the new hotness’ of Kibana 4

Monday, 29 June 15

26

https://github.com/LoveSoftware/getting-started-with-the-elk-stack

Monday, 29 June 15

Monday, 29 June 15

Monday, 29 June 15

Monday, 29 June 15

Monday, 29 June 15

Logstash Config

31

Monday, 29 June 15

Logstash Collecting{ "network": { "servers": [ "logs.logstashdemo.com:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash−forwarder.crt" }, "files": [ { "paths": [ "/var/log/nginx/helloapp.access.log" ], "fields": { "type": "nginx−access" } } ] }

32

Monday, 29 June 15

Logstash Processing

input { lumberjack { port => 5000 ssl_certificate => "/etc/pki/tls/certs/logstash−forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash−forwarder.key"}

}

Input

33

Monday, 29 June 15

Logstash ProcessingFilteringfilter { if [type] == "nginx−access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ] } } }

34

Monday, 29 June 15

Logstash ProcessingOutput

output { elasticsearch { host => localhost }}

35

Monday, 29 June 15

Groking grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }

https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns

http://grokdebug.herokuapp.com/

55.3.244.1 GET /index.html 15824 0.043

%{IP:client}%{WORD:method}%{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

Monday, 29 June 15

37

Hey Ben.... Have you got time for that

gratuitously flashy geo data demo?

Monday, 29 June 15

Monday, 29 June 15

Logging IdeasRelease MarkerError rates of various applications over timeLatency in various percentiles of each application tierHTTP Responses: 400 series responsesHTTP Responses: 500 series responsesAuto git blame production errorsAuth and Syslogs

39

Monday, 29 June 15

Go Forth And Log....BUT

Remember log rotation

Beware running out of space

Beware file logging on NFS

40

Monday, 29 June 15

Questions?

41

Monday, 29 June 15

https://joind.in/talk/view/14235

42

Monday, 29 June 15