39
Workshops and Conference: May 9-11, 2016 2016 Stockholm Let us know what you think! Click “Engage” to rate a session. If you rate 12 sessions you get a cool GOTO prize!

Monitoring Microservices

Embed Size (px)

Citation preview

Page 1: Monitoring Microservices

Workshops and Conference: May 9-11, 2016

2016

Stockholm

Let us know

what you think!

Click “Engage”to rate a session.If you rate 12 sessionsyou get a cool GOTO prize!

Page 3: Monitoring Microservices
Page 4: Monitoring Microservices

VisualisationMonitoring Tracing

0255075

100

Page 5: Monitoring Microservices

Monitoring

0255075

100

Page 6: Monitoring Microservices

Traditional 3-tier architectureIncoming traffic

Load balancers

Application servers

Database & replica

Page 7: Monitoring Microservices

Microservice architecture

Public APIWeb UI

NoSQL serversDatabase

Message Broker

Services

Page 8: Monitoring Microservices
Page 9: Monitoring Microservices

Microservices should be treated like cattle not pets

Page 10: Monitoring Microservices
Page 11: Monitoring Microservices

USE Method* - for every resource, check: • utilization, • saturation, and • errors

RED Method - for every service, check request: • rate, • error (rate), and • duration (distributions)

* http://www.brendangregg.com/usemethod.html

An alternative view

Page 12: Monitoring Microservices

Okay, but how?var rpcDurations = prometheus.NewHistogram(prometheus.HistogramOpts{

Name: "rpc_durations_histogram_microseconds",

Help: "RPC latency distributions.",

Buckets: prometheus.LinearBuckets(0, 100, 20),

})

func init() {

prometheus.MustRegister(rpcDurations)

}

func handleRequest(w http.ResponseWriter, r *http.Request) {

begin := time.Now()

...

rpcDurations.WithLabelValues(r.Method).Observe(

float64(time.Since(begin).Nanoseconds()))

}

Page 13: Monitoring Microservices
Page 14: Monitoring Microservices

There must be a better way…

Kubeproxy

Replicas

incoming traffic from other services

Page 15: Monitoring Microservices

Demo Time

Page 16: Monitoring Microservices

https://github.com/weaveworks/flux

Try it out!

Page 17: Monitoring Microservices

Monitoring

0255075

100

Page 18: Monitoring Microservices

Visualisation

Page 19: Monitoring Microservices
Page 20: Monitoring Microservices

Weave Scope

Page 21: Monitoring Microservices

Connection Tracking/home/weave # conntrack -E [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41066 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41066 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=36236 dport=32778 src=172.17.0.8 dst=192.168.99.100 sport=80 dport=36236 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41068 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41068 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52996 dport=32776 src=172.17.0.6 dst=192.168.99.100 sport=80 dport=52996 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41070 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41070 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=52998 dport=32776 src=172.17.0.6 dst=192.168.99.100 sport=80 dport=52998 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41072 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41072 [ASSURED] [DESTROY] tcp 6 src=192.168.99.100 dst=192.168.99.100 sport=57975 dport=32777 src=172.17.0.7 dst=192.168.99.100 sport=80 dport=57975 [ASSURED] [DESTROY] tcp 6 src=172.17.0.10 dst=10.128.0.1 sport=41074 dport=80 src=172.17.0.1 dst=172.17.0.10 sport=42525 dport=41074 [ASSURED]

/home/weave # cat /proc/net/tcp sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode 0: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 16810 1 ffff8800d79c1800 100 0 0 10 0 1: 0100007F:EB74 0100007F:0FC8 06 00000000:00000000 03:0000016D 00000000 0 0 0 3 ffff8800ae3f6e80 2: 0100007F:EB69 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 307011 1 ffff8800cf467040 21 4 30 10 -1 3: 0100007F:EB7B 0100007F:0FC8 06 00000000:00000000 03:00000D27 00000000 0 0 0 3 ffff8800d7a47538 4: 0100007F:EB7C 0100007F:0FC8 06 00000000:00000000 03:0000110E 00000000 0 0 0 3 ffff8800cf656c70 5: 0100007F:EB67 0100007F:0FC8 01 00000000:00000000 00:00000000 00000000 0 0 306868 1 ffff8800d79c1040 21 4 27 10 -1 6: 0100007F:EB76 0100007F:0FC8 06 00000000:00000000 03:00000556 00000000 0 0 0 3 ffff8800d37ac748 7: 0100007F:EB7F 0100007F:0FC8 06 00000000:00000000 03:000014F7 00000000 0 0 0 3 ffff8800d87f0c70

Page 22: Monitoring Microservices

all connections

from /proc

conntrack!

Connection Tracking

load balanced

Page 23: Monitoring Microservices

Demo Time

Page 24: Monitoring Microservices

https://github.com/weaveworks/scope

Try it out!

https://weave-scope-slack.herokuapp.com

v0.13.1

https://www.weave.works/products/weave-scope

https://scope.weave.works

Page 25: Monitoring Microservices

Visualisation

Page 26: Monitoring Microservices

Tracing

Page 27: Monitoring Microservices

Distributed Tracing

Page 28: Monitoring Microservices

Not a new topic

• Lots of literature• Existing open source

projects• e.g. Zipkin, originally from

Twitter

Page 29: Monitoring Microservices

• Challenge: detecting causality between incoming and outgoing requests

• Existing solutions require propagation of some unique ID (dapper, zipkin)

• This requires application-specific modifications

some service

incomingrequest

outgoingrequests

?

Page 30: Monitoring Microservices

Can this be done without application modifications?

Page 31: Monitoring Microservices

By intercepting systems calls, build up a data structure of:• which threads are reading

to / writing from which FDs• which FDs are talking to

which IPs & ports

Use this to infer causality between incoming and outgoing connections.

some service

kernel

?

System calls

Page 32: Monitoring Microservices

Demo Time

Page 33: Monitoring Microservices

Try it out!

https://github.com/weaveworks/scope/tree/master/experimental/tracer

Page 34: Monitoring Microservices

Tracing

Page 35: Monitoring Microservices

VisualisationMonitoring Tracing

0255075

100

Page 36: Monitoring Microservices

We’re hiring!

Page 37: Monitoring Microservices

@weaveworks github.com/weaveworks

[email protected]

@tom_wilkie

Page 38: Monitoring Microservices

https://weave-scope-slack.herokuapp.com

https://github.com/weaveworks/scope

https://github.com/weaveworks/flux

https://scope.weave.works

Links

https://github.com/weaveworks/scope/tree/master/experimental/tracer

Page 39: Monitoring Microservices

Workshops and Conference: May 9-11, 2016

2016

Stockholm

Please remember torate this session

...Thank You!