Upload
amazon-web-services
View
365
Download
0
Embed Size (px)
Citation preview
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
David Elliott, Solutions Architecture Manager, AWS
Anvar Karimson, Senior Application Developer, ICAP
7th July 2016
Building your First Big Data App on
AWS
AccessLog - Common Log Format (CLF)
75.35.230.210 - - [20/Jul/2009:22:22:42 -0700]
"GET /images/pigtrihawk.jpg HTTP/1.1" 200 29236
Your First Big Data Application on AWS
PROCESS: Amazon EMR with Spark & Hive
STORE
ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight
COLLECT: Amazon Kinesis Firehose
ICAP Global Broking – Front Office Technology
• ICAP, leading markets operator and provider of post trade risk
mitigation and information services
• FTSE 250 company founded in 1986 – 30 years
• Operates in more than 60 locations in 32 countries
• More than 4000 employees
• Revenues over £1.2B
• Active investor in Fintech companies
The Challenge - Analytics prior to AWS
• Bespoke implementation per application
• Difficult to run analytics across applications
• Scalability-issues restricting number of
consumers
• Batch-oriented – No streaming
• On-premise solution restrictive in terms of events
that could be captured
The Solution – The Event Stream
• Amazon Kinesis combined with Amazon Redshift
• Standardised way of collecting, transforming, and analysing data across our estate
• Encrypted by default
• Amazon Kinesis allows for tapping into data as it is being streamed• Empirical evidence suggests ~30 seconds between data being ingested and it being available for
queries in Amazon Redshift
• Fine-grained access/resource controls allow us to grant access to a large number of consumers in a controlled and secure manner
• Much higher fidelity of events• Authentication/Authorization Events
• Connection Quality Events
• Business Events
• Metrics
• Quality-of-Service
• End-to-end view of every action performed by a user across the whole estate over the complete user lifecycle
Using AWS enabled
ICAP to deliver an
industrial-grade
analytics platform in
weeks, dramatically
decreasing costs
compared to an on-
premise solution
Greatest benefit of AWS
What’s next with AWS
• Integrate with Amazon Elastic MapReduce and
Spark Streaming for real-time analytics and
anomaly detection
• Expand the use of the Event Stream throughout
the organization
• Evaluate Amazon QuickSight as a complement
to existing business intelligence tools
Your First Big Data Application on AWS
PROCESS: Amazon EMR with Spark & Hive
STORE
ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight
COLLECT: Amazon Kinesis Firehose
Estimated pricing to run this demo is as follows:
Service Est. Cost*
Amazon EC2 $0.00
Amazon Kinesis Firehose $0.04
Amazon S3 $0.51
Amazon EMR $2.28
Amazon Redshift $0.00
Amazon QuickSight $12.00
Est. Total $14.83
Pricing estimate
*Estimated costs assumes: use of free tier where available, lower cost instances, dataset no bigger than 100MB and instances running
for 3 hours. Costs may vary depending on options selected, size of dataset, and usage.