Embed Size (px)
DESCRIPTION
Woot, an Amazon subsidiary, specializes in offering great new product deals every day. Woot's deeply discounted deals; and signature events like the 'Woot Off 'and 'Bag of Crap' sales launch at specific times throughout the day, and the resulting spiky traffic patterns are highly correlated to revenue. In this session, we offer an unvarnished perspective into how Woot uses services such as Amazon DynamoDB, EC2, ELB, CloudSearch, CloudFront, and SES. Learn how to architect for security and PCI for a retail website running on AWS. Dig into the technical details of a data-store comparison between DynamoDB, Mongo, Oracle, and SQLServer, to find the right solution for unique workloads. Join us as we share our musings and real-lessons learned from using a cocktail of AWS services. We encourage you to attend even if none of this makes sense or is interesting. Don't miss the opportunity to hang out with Mortimer the Woot monkey and his crew and to walk away with one of our legendary flying monkeys.
Citation preview
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Drinking Our Own Champagne: How Woot, an Amazon subsidiary, uses AWS technologies
Vivek Sagi, CTO, Woot - [email protected]
Dan Pinkard, Systems Manager, Woot – [email protected]
November 14, 2013
Introduction to Woot
Woot Trivia (Question 1)
11/15/2013 Amazon Confidential 3
Q: What is Woot?
A. A rabid monkey only found in the Amazon jungle
B. A daily deal site, flash site and vibrant deal community
C. The name of a dance performed by hungry SDEs
D. A retail website selling toys
Woot Trivia (Question 2)
11/15/2013 Amazon Confidential 4
Q: Name the event on the Woot website where we sell a
random collection of leftover inventory and “surprises”?
A. Bunch of Carrots
B. Bitter Old Canadians
C. Baldness on Curly
D. Bag of Crap
No Canadians were
harmed in the
making of this
presentation
Woot Trivia (Question 3)
11/15/2013 Amazon Confidential 5
Q: When was Woot acquired by Amazon?
A. 2004
B. 2008
C. 2010
D. None of the above because Woot is only a figment of
a six year old’s imagination
Ye Olde Woot
11/15/2013 Amazon Confidential 6
Modern Day Woot
11/15/2013 Amazon Confidential 7 11/15/2013 Amazon Confidential 7
Growth of
new
categories
Introduction
of Woot!
Plus flash
sales
It’s All About the Benjamins
Woot’s technology stack
• Traditional co-location (since 2003) – 8 servers
– Monolithic application
• Amazon S3 for images from 2006
• Woot Deals launch in 2009 – Amazon EC2 hosted (public)
– SimpleDB MongoDB + SOLR
– Amazon SQS to relay to Woot Retail
S3
EC2
ELB
SQS
Woot (Pre-Acquisition)
• Migrated to Amazon VPC (2011) – 2 months to prep VPC
– VPN to colo (for secure data migration)
– P2V for SQL servers
– Migrate MongoDB replicas
– 18 hours to move production services
• Went from 8 servers to 20
S3
EC2
Route 53
CloudFront
SES
VPC
ELB
SQS
RDS
Woot (Post-Acquisition)
• Business Goals – Support multiple deal types (Daily, Flash, VIP)
– Changes must be visible quickly
• Technology Goals – Services-oriented architecture
– Improve site reliability
– Optimize for cloud deployments
– Language and platform neutral
Application Rewrite in 2012
• Core Technologies – C#, ASP.Net, MVC4, WCF,
– MongoDB (All Content), SQL Server (Transactional), DynamoDB (Shopping Cart)
– Memcached
– RabbitMQ
• Reporting & Operations Technologies – Node.JS, Perl, Python, Ruby, PowerShell
– Durango, Rails
– Oracle, MySQL
– Redis
– Bind9, Postfix
Woot Current Technologies
Woot’s AWS footprint
S3 EC2
CloudFront
SES
CloudSearch
ELB
SQS RDS
VPC
Route 53
AWS Footprint
DynamoDB
IAM
• Multi-node VPN
• 3 accounts
• 3 environments
Production
Development
Staging
Network Infrastructure
• System Center + AWS Management Pack – .Net aware application monitoring
– Ties to external monitors
– Correlates security and system events
Monitoring
• System Center + AWS Management Pack
• OpenNMS – Service discovery
– SNMP integration
– Multi-AZ perspective
Monitoring
• System Center + AWS Management Pack
• OpenNMS
• StatsD + Graphite + Skyline – Great dashboards – Measure everything
– High resolution
– Flexible graphing options
Monitoring
• System Center + AWS Management Pack
• OpenNMS
• StatsD + Graphite + Skyline
• Amazon CloudWatch – Most direct source of truth
– Systems performance and usage metrics
Monitoring
Monitoring
• System Center + AWS Management Pack
• OpenNMS
• StatsD + Graphite + Skyline
• Amazon CloudWatch
• Secret Sauce(s) – Graphite as graphing platform – Integrate Amazon CloudWatch
– Track AMPQ stats
SG – Retail
Public
SG – Retail
Private
SG – Content
Public
SG - Content
Private
SG - Catalog
Public
SG - Catalog
Private
SG – Users
Public
SG – Users
Private
CloudFront
Public
Woot SOA Overview
• Scalability
• Organizational flexibility
• Security
• Faster time to market
• Improved reliability
SOA Benefits for Woot
How do we choose a technology?
• Don’t make assumptions; all options on the table
• AWS is not the default choice
• Simulate real-world behavior
• Compare as many facets as possible
- Driver support
- Performance
- Scalability
- Resiliency
- Support
- Cost
Woot Throwdown Tenets
Contenders Pros Cons
Varnish Flexibility No SSL
NginX Lightweight and fast No HTTP1/1 to origin
Apache HTTP1/1 in/out
Hard to tune
Elastic Load Balancing Cloud Aware
Limited logging
Throw Down 1: Load Balancers
Woot “Cart Service”
– Simple
– Fast
– Resilient
– Lots of small data facts
– Secure
– Host a bag of crap
– Scalable to follow growth
patterns
Throw Down 2: Cart Persistence Storage
Contenders Document Databases Relational Databases
DynamoDB MongoDB Oracle SQL Server
Client API 4 5 2 5
High availability 5 5 2 5
Performance 5 4 -- --
Ramp-up time 3 5 1 2
Tools 4 4 4 5
Cost 5 1 3 3
Operational management 5 4 3 3
Debugging 4 5 5 5
Ad-hoc queries (analytics) 3 4 5 5
Support 5 5 4 4
Documentation 5 5 5 5
Total 48 47 -- --
Cart Persistent Storage
Criteria Document Databases Relational Databases
DynamoDB MongoDB Oracle SQL Server
Client API 4 5 2 5
High availability 5 5 2 5
Performance 5 4 -- --
Ramp-up time 3 5 1 2
Tools 4 4 4 5
Cost 5 1 3 3
Operational management 5 4 3 3
Debugging 4 5 5 5
Ad-hoc queries (analytics) 3 4 5 5
Support 5 5 4 4
Documentation 5 5 5 5
Total 48 47 -- --
Cart Persistent Storage
Document Databases Relational Databases
DynamoDB MongoDB Oracle SQL Server
Client API 4 5 2 5
High availability 5 5 2 5
Performance 5 4 -- --
Ramp-up time 3 5 1 2
Tools 4 4 4 5
Cost 5 1 3 3
Operational management 5 4 3 3
Debugging 4 5 5 5
Ad-hoc queries (analytics) 3 4 5 5
Support 5 5 4 4
Documentation 5 5 5 5
Total 48 47 -- --
Cart Persistent Storage
- MongoDB and DynamoDB were our finalists
- DynamoDB outperformed MongoDB in our
performance tests
- We could launch and scale DynamoDB with a
button push
- We could host a bag of crap
Cart Throw Down Results
0
500
1000
1500
2000
2500
3000
3500
DynamoDB MongoDB Oracle RDS SQL Server
Cost ($/Month)
Cart Persistent Storage: Cost Analysis
64
2060
107
42
1348
70
1
10
100
1000
10000
Add Items Get Update Items
Operations / second
DynamoDB MongoDB
Cart Persistent Storage: Performance
• Only basic index support (primary, secondary) – Use searching technologies such as Amazon CloudSearch to
index data
• Ad-hoc query support not so simple – Export data we want to query into a data warehouse
DynamoDB – Rough Edges
• 20,000,000,000 requests served
• Sub 10ms response times
• 0 errors reported since launch
• No maintenance
• No SLA misses
Cart in Production
Recovering from a cloud outage
• Massive Storm + Maintenance Problem =
No Power in an Availability Zone
• Availability Zone hosted Woot’s primary
SQL and AD role masters
• Amazon EBS failures prevented recovery
• AD DNS collapsed
Triage of an outage: July 2, 2012
• Data integrity means more Availability Zones – Mirror/replicate to other Availability Zones
– Lots of hosts for MongoDB
– Dedicated hosts for irregular patterns
• Backups
• Long-running queries
• Specialized indexes
Lessons Learnt
• Data integrity means more Availability Zones
• Queue everything – Decouple systems
– Save state information
– Inspection of service operation
– Recoverable
Lessons Learnt
• Data integrity means more Availability Zones
• Queue everything
• Hosts are disposable – Don’t store unique data
– Push-button launches
– Centralize configuration
– Dynamic role assignment
Lessons Learnt
• Data integrity means more Availability Zones
• Queue everything
• Hosts are disposable
• Unique solutions – Different failure modes than traditional data center
– Combine off-the shelf items for resiliency
Lessons Learnt
Final thoughts on the cloud
• Same problems as anywhere else – Centralize logging
– Centralize accounts
• Application-level logging – Gain visibility
– Follow state
• Use application tiers – SOA separation
Security
• CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
Best Practices
• CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
• EC2+EBS Best Practices – Deploy EBS-optimized hosts for required bandwidth
– Provisioned IOPs to ensure lower latency for disks
Best Practices
Best Practices
• CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls.
– Perform updates in batches
• EC2+EBS Best Practices – Deploy EBS Optimized hosts for required bandwidth
– Provisioned IOPs to ensure lower latency for disks
• CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
• EC2+EBS Best Practices – EBS-optimized hosts for more bandwidth
– Provisioned IOPs to ensure lower latency
• Scaling Best Practices – No reward for using too much or too little
Best Practices
• Tools continue to improve - AWS Dashboard - Expanded CloudWatch
- AWS Trusted Advisor - Premium Support
• AWS advocates for users
• Capacity really is elastic
• The more we see, the better it looks
Woot’s AWS Experience
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
ARC212
![[AWS re:invent 2013 Report] AWS New EC2 Instance Types](https://img.dokumen.tips/doc/110x75/5565fe5cd8b42a2a4d8b4cc5/aws-reinvent-2013-report-aws-new-ec2-instance-types.jpg)
![[AWS re:invent 2013 Report] AWS CloudTrail](https://img.dokumen.tips/doc/110x75/5463ce9baf79596f338b4620/aws-reinvent-2013-report-aws-cloudtrail.jpg)
