Logging @ Scale on AWS

Visualization > Plaintext

Who am I?

• ChrisRiddell• TechCo-Founderofsomestartups,SeniorSoftwareEngineer• BigDataguy– Redshift,S3,EC2,EMR,Hive,Spark,Dynamoandmanyothers….• ImanagebigAWSinfrastructurefromsoftwaretoarchitecturetoDevOps (startuplife...)• Java(+others)/AWS/geek• AWSProfessionallyCertifiedSolutionArchitect(askmehow!)• Notaloggingexpert- liketothinkI’mgettingcloseJ

Logging @ Scale on AWS

• Usecases•Whattolog• Commercialoptions• Commontools• Plausiblearchitectures• Demo!EC2w/Fluentd ->KinesisFirehose ->Elasticsearch

Why log?

• Getvisibilityintoapphealth,centrallyaccessibleandsearchable• Alerts&fasterroreventresponse– Agility!• Nothavingtologintoindividual instancesandtail• Contextualtracking e.g.userbehaviour• Findingcodeoptimisations• …andmanymanyotherreasons!

First have common language

•WhatisDEBUG,INFO,WARNandERRORusedforinyourorganisation?• Havecommonlanguageforwhatshouldbeloggedwhere• Badlevelingmessesupyourstorage• E.g.DEBUGlogsgoingtoyourexpensiveElasticsearch store,whentheyneverneedtobesearched

• Oneguy’sopinion:http://stackoverflow.com/a/8021604/3843660

Centralising the logs

• Let’sgetthemoffthehost• BasicDIY:• Syslog-ng,rsyslogd,nxlog

• AdvancedDIY:• Splunk forwarder,Logstash,Flume,Fluentd

• Thirdparty• SaaS

Commercial options: SaaS

• LogEntries,SumoLogic, Loggly,Splunk Cloud,PaperTrails,AWSCloudWatch Logs…• TypicallyRESTful JSONlogdumpAPIs• Search&visualizationsarecorefeatures•Mosthaveafreetier•Manylibrariesavailableforvariouslanguagesand/orpackagedversions• Costsgoupwithdatasize,retentionperiodandusercount• Nicetohave:Userdefinedalerts;S3archival…..

Self-hosted solutions

• Splunk – Enterprisesolutionconsistingofforwarders,searchheads,andindexers.Licensed.• ELK/EFKstack=Elasticsearch,Logstash orFluentd, Kibana• Today=Fluentd (forwarder),KinesisFirehose,Elasticsearch,Kibana• FKFEKstack?

Elasticsearch?

• Forsearch!• Indexes• Shards- Distributed&scalesout• Replicas• JSONRESTAPI• ApacheLucene• Kibana isanElasticsearch pluginthatprovidesaniceinterfacetothesearchdata*withvisualizations*

Logstash & Fluentd agents

• Packagedinstall• Inputandoutputlogs• Centraliseyourinstancelogs• Oftenusedasasyslogtail’er orasalocalHTTPlogendpoint• Parse/transform/filter/tag• StoreorForward• “Logstash emphasizesflexibilityandinteroperability whereas Fluentd prioritizessimplicityandrobustness” - http://goo.gl/f5I4cL

Lo

Architecture – Pre October 2015

Today’s Solution

Lo

Syslog/FluentdonEC2

KinesisFirehose Elasticsearch

Demo: Set up EC2 and Fluentd

•WespinupadefaultAWSAMIEC2instancewithrolepermissiontopushdatatoFirehose,accessviaSSH&HTTP)•WeSSHinandinstallFluentd• curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh• /usr/sbin/td-agent-gem install fluent-plugin-kinesis #installAWSFHplugin• Thenconfigurefluentd topushoursyslog’s

Demo: Fluentd config (/etc/td-agent/td-agent.conf)## Syslog reader. Configure port 42185 to send events to in rsyslog config<source>

type syslogport 42185bind 0.0.0.0tag system

</source>

## Filters to transform records and add metadata<filter **>

type record_transformerenable_ruby<record>

@timestamp ${require 'time'; Time.now.utc.iso8601}</record>

</filter>

## Output to Firehose using the instance role<match **>

@type kinesis_firehoseregion us-west-2 delivery_stream_name logsflush_interval 2s

</match>

Demo: restart log agents and serve HTTP

# After /etc/td-agent/td-agent.conf has been setup# Send syslog to fluentd listenerecho "*.* @127.0.0.1:42185" | sudo tee /etc/rsyslog.d/22-fluent.conf

sudo service td-agent restartsudo service rsyslog restart

# Let’s make a web server for you to push your own logs!mkdir webcd webecho 'Hello!' > index.htmlsudo python -m SimpleHTTPServer 80 |& logger -t httpsvr &

Demo: Setting up Elasticsearch

•WesetupAWSElasticsearch Service:Somenotes:• Dedicatedmaster- performsclustermanagementtasks,doesn’tholddata• Metrics:Theusualstuff.NotetheJVMMemoryPressuremetric.Amazonrecommendsscaleup/outif>85%• Clusterstatusisyellowonsinglenodebecausereplicascannotbeassigned.Addanodeorchangethesetting• Clusterhasit’sownaccesspolicy.Ifyouchooseinstanceroleaccesscontrol,youmustsignallrequeststoES(useAWSSDKs).YouwillnotbeabletoaccessKibana onthissetting• Checkwhatsizetheinstancestoreisonyourselectedinstancetype,oruseEBS

Demo: Setting up Kinesis Firehose

•WecreatealogsdeliverystreamwithElasticsearch asthetarget• Firehose:Somenotes:• Apipelinetopushdatainathighscale• Dumpdatain,anditbuffersrecrods (logsinourcase)thembeforepushingtoElasticsearch andoptionallyS3(Redshiftalsosupported)• PayperGBofingestion$0.035USD(eachrecordroundedtonearest5kb)• Differentdestinations(e.g.WARN/ERRORtoElasticsearch butrestonlytoS3)wouldneeddifferentFirehose deliverystreams

Limitations? Further features?

• AWS’sElasticsearch islimitedonplugins,butverygoodoutoftheboxsettings• Fastwaytogethighthroughput,highscalelogsintoastoredindexwithS3backups,andvisualisation!• FurtherfeaturesusingAWSLambdaasglue:

• Alerts• CloudTrial/S3logsingestiontoFH/Elasticsearch,• Deletionofoldindexesataspecifiedperiod(wewantthelastXdaysonly)• andmore…

• S3bucketpolicyforstorageoptimisation (eg oldstufftoglacier)• Yourcustomapplications:PushdirectlytoFluentd’s HTTPendpoint,notviasyslog(moreflexibilityandtagging)• Furthercustomisations toFluentd config

The end!

• Thanks!•@ChrisJRiddell•@ParrotAnalytics• HiringIntermediate/SeniorJavaEngineersJ• Upcoming:Webdev &moreengineers• https://parrot-analytics.workable.com/ toapply