Use Saltstack to deploy a full monitoring and supervision ...slides.logilab.fr/2018/cfgmgmtcamp_saltstack_monitoring.pdfSaltStack - huge toolkit to manage infrastructure Netdata -

Use Saltstack to deploy a fullmonitoring and supervisionstack#cfgmgmtcamp18Arthur Lutz ‐ Logilab

#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

(part of) who am IArthur Lutz

Logilab

Python dev / Sysadmin / Debian / Devops

SaltStack Certi�ed Engineer

Paris Salt Meetup

Python Nantes Meetup / Nantes MonitoringMeetup


https://www.logilab.fr/

Big picture ‐ scien�fic cloudapplica�on

Complex scaling parameters

salt-based auto-scaling, scale up and down basedon application logic

scaling is based on metrics collected by salt

automatically deploy application-orientedmonitoring


dashboards from diversedata

grafana provides data source diversity

we're looking for data origin diversity, and agileways to deploy them

system (internal) data

cloud (external) data

application-speci�c data

autoscaling state, etc...


agility by design

consistency by designeasily de�ne and add metrics

compare and test metrics from different points ofview


Overview of componentsSaltStack - huge toolkit to manage infrastructure

Netdata - collect metrics, visualise "realtime"status of server

Sensu - devops oriented monitoring system

Graphite / Carbon - to collect and query timeseries data

Grafana - to build and display dashboards on thecollected data


Components usedsalt pillars

salt scheduler

salt mine

salt returners

salt runners

salt reactor

salt-cloud

... "which ones did we not use?"



Salt formulassalt-formula (what else?)

nginx-formula

nfs-formula

nagios-formula

sensu-formula

rabbitmq-formula

redis-formula


Role based approach

Pillars (with pillar_merge_list: True ) :

base: '*': - role.sensu-client 'saltmaster.local': - role.sensu-server

States :

base: 'I@role:sensu-client': - sensu.client 'I@role:sensu-server': - sensu.server


Salt minesend unique ID used by Netdata to stream clientsto central instance states :

{{ salt['mine.get']('role:netdata-master', 'network.interface_ip' expr_form='pillar').values()[0] }}

use to retrieve location of Rabbitmq server used by Sensu . pillars :

{% set rabbitmq = salt.saltutil.runner("mine.get", tgt='role:rabbitmq', fun="network.interface_ip", tgt_type='pillar').values() %}


Salt returnersCarbon returner to push metrics from applicationand infrastructure

returner address is con�gured in pillars (cfextraction from mine)


Salt custom modulesCustom modules are shipped by salt master from _modules folder

celery statistics (use app.inspect.stats )

core application metrics using HTTP API

sentry (error collector) metrics using the sentrypython binding (which uses the sentry HTTP API)


Custom module ‐ celery inspectdef stats(): ''' Provide stats from celery tasks via app.inspect.stats() [snip] ''' ret = __salt__['redis.sentinel_get_master_ip']('name') r_url = 'redis://{}:{}/0' app = Celery(broker=r_url.format(ret['master_host'], ret['master_port'])) stats = app.control.inspect().stats() _clean(stats) return stats


Salt‐cloudenables custom auto-scaling system

portable between cloud infrastructures

enables to run infrastructure on local openstack(or even devstack)


Salt reactor ‐ event driven!react to new VM connecting

get info about VM (for example kernelversion )

upgrade kernel to linux-backports

reboot system

deploy monitoring

deploy application

Ended up being replaced by Packer generated AMI.


https://www.packer.io/

Let's zoom in


Visualising complexity


monitoring ‐ howlaunch a salt runner to pool the metrics

salt.execute runner is used to get infrastructureview of a given metric

cloud.profile runner is used to provision new VMs

cloud.destroy runner is used to scale down unusedVMs


monitoring ‐ howNetdata sends system metrics

central Netdata sends data to Graphite

Salt sends custom metrics to Graphite

Sensu executes checks (subscriber based model)

Sensu collects metrics and sends them to Graphite

Grafana uses the metrics stored in Graphite todisplay dashboards


MASTER

sensuserversensuapi

web

minion

storage

minion

client

com

pute

nod

e

minion

client

com

pute

nod

e


Demo �me!install netdata

role based architecture

auto con�gure streaming by pushing the API id tothe mine

install graphite

deploy netdata graphite con�guration

install grafana

con�gure grafana datasource with salt

con�gure grafana dashboard with salt#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

Demo �me!install rabbitmq with rabbitmq-formula for sensu

install sensu with sensu-formula

use roles to con�gure subscribed sensu checks

con�gure uchiwa for web ui

con�gure sensu metrics to send data to graphite

update grafana dashboard with salt


What's next / Roadmapconvert schedulers to beacons

various optimisations

use reactor instead of polling

look into salt's thorium (complex reactor system)


https://docs.saltstack.com/en/latest/topics/thorium/index.html

EndThanks for your attention

Questions ?

Comments ?

Improvements ?


sourceshttp://slides.logilab.fr/2018/cfgmgmtcamp_saltstack_monitoring.pdf

Demo : https://github.com/arthurlogilab/salt-vagrant-demo

Contact / Follow us : https://www.logilab.fr

@logilab

@arthurlutz

[email protected]#cfgmgmtcamp18 | January 2018 | Arthur Lutz @arthurlutz | Logilab

http://slides.logilab.fr/2018/cfgmgmtcamp_saltstack_monitoring.pdf

https://github.com/arthurlogilab/salt-vagrant-demo

https://www.logilab.fr/

https://twittter.com/logilab

https://twitter.com/arthurlutz

mailto:[email protected]

Documents

Use Saltstack to deploy a full monitoring and supervision ...slides.logilab.fr/2018/cfgmgmtcamp_saltstack_monitoring.pdfSaltStack - huge toolkit to manage infrastructure Netdata -