Upload
amazon-web-services
View
1.314
Download
0
Embed Size (px)
Citation preview
Gaining Operational Insights out of your logs
Kobi Biton, Solutions Architect
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Common Log sources on AWS• Loading Streaming data into Amazon
Elasticsearch Service• Demo: Real world scenario
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
We want to turn this…
{"eventTime": "2016-06-06T09:59:55Z","eventSource": "signin.amazonaws.com","eventName": "ConsoleLogin","awsRegion": "us-east-1","sourceIPAddress": ”169.254.169.254","userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0)
Gecko/20100101 Firefox/38.0",
"errorMessage": "Failed authentication","requestParameters": null,"responseElements": {
"ConsoleLogin": "Failure"},"additionalEventData": {
"LoginTo": "SomeInfo""MFAUsed": "Yes"
},{...}
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Into this…
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
And then optionally this…
Amazon SNS
Topic
AWS Lambda
CloudwatchLogs
Alarms
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Common Log Sources
• Amazon Cloudfront access logs , S3 server access logs
• Elastic Load Balancer access logs• AWS Cloudtrail , VPC Flow Logs• Amazon SNS notifications• Amazon EC2 OS Logs / Custom application
logs
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Loading Streaming Data Into Amazon Elasticsearch Service
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
From Amazon S3
Amazon ESDomain
AWS Lambda
ELB Logs
Cloudfront Access Logs
S3 Access Logs
S3
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Cloudtrail
VPC Flow Logs
CloudwatchLogs
Amazon ESDomain
AWS Lambda
From Amazon Cloudwatch Logs
Cloudwatch Alarms
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis FirehoseAgent
Amazon Kinesis
Firehose
Amazon ESDomain
Via Amazon Kinesis Firehose
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LogstashAgent
Log Objects
Amazon ESDomain
Via Logstash Agent
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DEMO
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Appendix
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Useful Links:
• https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html
• https://github.com/awslabs/cloudwatch-logs-subscription-consumer/tree/master/configuration/kibana
• https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html• https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Su
bscriptions.html• https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/M
onitoringLogData.html