View
769
Download
0
Tags:
Embed Size (px)
Citation preview
How to ditch the debugger and use logging instead
Matthew Skelton & Rob Thatcher, Skelton Thatcher Consulting
/dev/winter 2015,
Saturday 24th January, Cambridge, UK
#devwinter
Event tracingVagrant + ELK
Demo: no debugging
Skelton ThatcherIntroduction and Services Overview
‘Cloud’ changes the way we mustdesign, deliver, and operate
our software systems
Changes for ‘cloud’
Event tracing
Event type IDsTransaction tracing
Adjusting logging levels
Event type IDs
How many event types in your application?
Event type IDs
Event type IDs
enum
Human-readable sets: unique values, sparse, immutable
C#, Java, Python, node(Ruby, PHP, …)
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
ItemAddedToBasket = 60001,
ItemRemovedFromBasket = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}
Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
ItemAddedToBasket = 60001,
ItemRemovedFromBasket = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}
ItemAddedToBasket = 60001
ItemAddedToBasket = 60001
ItemAddedToBasket = 60001
Transaction tracing
Monolith to microservices:debugger does not have the full view
Distributed systems
Even with remote debugger, it’s boring to attach and detach
Remote debugging?
Transaction tracing
Unique-ish identifier assigned to web request
Passed through downstream layers
What about APM?
APM gives us application insightBUT
How much do we learn? Is APM available on the Dev box?
It’s not just ‘an Ops problem’!
Minimal viable tracing?
Helps us to understand how the software really works
Small overhead is worth it
Which log level is right?
DEBUG, INFO, WARNING, ERROR, CRITICAL
Which log level is right?
Log level should *not* be fixed at compile or build time!
<?xml version="1.0" encoding="utf-8" ?>
<Application.EventMappings>
<Events>
<Event Type="CacheServiceStarted">
<Severity Type="Information"/>
</Event>
<Event Type="PageCachePurged">
<Severity Type="Debug"/>
<Enabled="false"/>
</Event>
<Event Type="DatabaseConnectionTimeOut">
<Severity Type="Error"/>
</Event>
</Events>
</Application.EventMappings>
Tune log levels
Tune log levels
Tune log levels
Tune log levels
http://bit.ly/TuneLoggingLevels
Dev and Ops collaboration*
* and testers too!
Event tracing
Use enumerations (or closest thing)
Technical and Domain event types
Distributed systems: debuggers less useful
Trace calls with ‘unique-enough’ handles
Tune log levels via config
Vagrant + ELK
Log AggregationElasticsearch, Logstash & Kibana
Vagrant
Log Aggregation
Multiple log sourcesMany logs in one place
SearchableTime Sequenced
What is ELK?
http://www.exploringnature.org/db/detail.php?dbID=43&detID=956
Vagrant
Ruby CLI to deploy Virtual environments
Deploy locally to AWS or Azure etc.
Four Simple Steps
$ vagrant box add ubuntu/trusty64$ vagrant init trusty64$ vagrant up$ vagrant ssh
RapidRepeatable
Provisioning via scripts ++
• Listening to: • Local nginx logs• Local Syslog• (Application logs)
http://bit.ly/velk-demo
Vagrant + ELK Demo VM
Elasticsearch, Logstash, Kibana
Recap
Aggregated Logging:Vagrant + ELK
Centralised logs, time sequenced, searchableVagrant :quickly spin up & manage machines
ELK : collect, store, display & search logsDeveloper-friendly!
Demo: ditching the debugger
Vagrant on developer laptopELK for log aggregationWeb app: PHP (Slim)
Event type IDs
Demo: background
Use VirtualBoxshared folders
Browse to Kibana UI via NAT-ed interface
Event type IDsLog aggregation with ELK
Demo
Demo: summary
Super-easy to hook application logs into ELK
Sequences of events emerge
Compare timings of events: past and present
Recapitulation
How to ditch the debuggerUse Vagrant + ELK on development VM
Use a set of event type IDs (enums)Log distinct states of execution
Both technical and domain eventsConsistent event naming helps with searchCompare previous and current executions New logging-fu: make friends with Ops!
LinksTune logging levels in Production without recompiling code: http://bit.ly/TuneLoggingLevels
VELK (Vagrant + ELK) demo VM: http://bit.ly/velk-demo
GOOS book (section on logging: p.233-5):http://www.growing-object-oriented-software.com/