Embed Size (px)
Citation preview
Amazon Web Services:Building Highly Scalable Web Applications
Institutional Web Management WorkshopJuly 2007
Jeff Barr
Senior Web Services Evangelist
Today
Introduction
Amazon Web Services overview
Amazon EC2
Amazon S3
Amazon SQS
Application Architecture
Questions
Who am I?
Software development background
5 Years with Amazon:
Senior Developer
Web Services Evangelist
Senior Web Services Evangelist
Veteran of several startups
Microsoft .Net Team
Visix Software co-founder
Amazon?
Consumers (Buy)
Associates, Sellers & Merchants (Sell)
Developers (Innovate)
Issues Facing Developers
70% of Web Development Effort is “Muck”:
Data Centers
Bandwidth / Power / Cooling
Operations
Staffing
Scaling is Difficult and Expensive:
Large Up-Front Investment
Invest Ahead of Demand
Load is Unpredictable
Dream or Nightmare?
Slashdot/Digg/TechCrunch Effect
Rapid, unexpected customer demand/growth
Seasonal Spikes
More Seasonal Spikes
“Every year, we take the busiest minute of the busiest hour of the busiest day and build capacity on that, we built our systems to (handle that load) and we went above and beyond that.” *
-- Scott Gulbransen
Intuit Spokesman
* http://news.com.com/2100-1038_3-6177341.html
Solution: Web-Scale Computing
Scale capacity on demand
Turn fixed costs into variable costs
Always available
Rock-solid reliability
Simple APIs and conceptual models
Cost-effective
Reduced time to market
Focus on product & core competencies
10
Amazon Web Services Are...
A set of APIs and business models which give developer-level access to Amazon’s infrastructure and content:
Data As A ServiceAmazon E-Commerce Service
Amazon Historical Pricing
Search As A ServiceAlexa Web Information Service
Alexa Top Sites
Alexa Site Thumbnail
Alexa Web Search Platform
Infrastructure As A ServiceAmazon Simple Queue Service
Amazon Simple Storage Service
Amazon Elastic Compute Cloud
People As A ServiceAmazon Mechanical Turk
The Commercial Side
Standard licensing terms
Commercially usable
Aggressive pricing
Monthly credit card billing
Self-serve model:
Sign up as developer
Choose services
Agree to service licenses
Enter payment info
Start coding
Web Services Billing
Infrastructure Services
Compute
Store Message
Elastic ComputeCloud
Simple Storage Service
Simple QueueService
Amazon Simple Storage Service
S3
Amazon Simple Storage ServiceAmazon Simple Storage Service
$.15 per GB per month
storage
• Object-Based Storage• 1 B – 5 GB / object• Fast, Reliable, Scalable• Redundant, Dispersed• 99.99% Availability Goal• Private or Public• Per-object URLs & ACLs• BitTorrent Support $.10 - $.18 per
GB data transfer
$.01 for 1000 to 10000 requests
Amazon S3 Concepts
Objects:Opaque data to be stored (1 byte … 5 Gigabytes)Authentication and access controls
Buckets:Object container – any number of objects100 buckets per account / buckets are “owned”
Keys:Unique object identifier within bucketUp to 1024 bytes longFlat object storage model
Standards-Based Interfaces:REST and SOAPURL-Addressability – every object has a URL
S3 SOAP/Query APIService:
ListAllMyBuckets
Buckets:CreateBucketDeleteBucketListBucketGetBucketAccessControlPolicySetBucketAccessControlPolicyGetBucketLoggingStatusSetBucketLoggingStatus
Objects:PutObjectPutObjectInlineGetObjectGetObjectExtendedDeleteObjectGetObjectAccessControlPolicySetObjectAccessControlPolicy
Amazon Simple Queue Service
SQS
Amazon Simple Queue ServiceAmazon Simple Queue Service
$.10 per 1000 messages
• Scalable Queuing• Elastic Capacity• Reliable, Simple, Secure
Inter-process messaging, data buffering, architecture component
$.10 - $.18 per GB data transfer
Amazon SQS Concepts
Queues:Named message containerPersistent
Messages:Up to 256KB of data per messagePeek / Lock access model
Scalable:Unlimited number of queues per accountUnlimited number of messages per queue
Amazon SQS At Work
SQS SOAP/Query APIQueues:
ListQueues DeleteQueueSetVisibilityTimeoutGetVisibilityTimeout
Messages: SendMessage ReceiveMessage DeleteMessage PeekMessage
Security:AddGrantListGrantsRemoveGrant
Amazon Elastic Compute Cloud
EC2
Amazon Elastic Compute CloudAmazon Elastic Compute Cloud
$.10 per server hour
• Virtual Compute Cloud• Elastic Capacity• 1.7 GHz x86• 1.7 GB RAM• 160 GB Disk• 250 MB/Second Network• Network Security Model
Time or Traffic-based Scaling, Load testing, Simulation and Analysis, Rendering, Software as a Service Platform, Hosting
$.10 - $.18 per GB data transfer
Amazon EC2 Concepts
Amazon Machine Image (AMI):Bootable root diskPre-defined or user-builtCatalog of user-built AMIsOS: Fedora, Centos, Gentoo, Debian, Ubuntu, Windows ServerApp Stack: LAMP, mpiBLAST, Hadoop
Instance:Running copy of an AMILaunch in less than 2 minutesStart/stop programmatically
Network Security Model:Explicit access controlSecurity groups
Inter-service bandwidth is free
Amazon EC2 At Work
StartupsCruxy – Media transcodingGigaVox Media – Podcast Management
Fortune 500 clients:High-Impact, S hort-Term ProjectsDevelopment Host
Science / Research:Hadoop / MapReducempiBLAST
Load-Management and Load Balancing Tools:Pound WeogeoRightscale
EC2 SOAP/Query API
Images:RegisterImageDescribeImagesDeregisterImage
Instances:RunInstancesDescribeInstancesTerminateInstancesGetConsoleOutputRebootInstances
Keypairs:CreateKeyPairDescribeKeyPairsDeleteKeyPair
Image Attributes:
ModifyImageAttribute
DescribeImageAttribute
ResetImageAttribute
Security Groups:
CreateSecurityGroup
DescribeSecurityGroups
DeleteSecurityGroup
AuthorizeSecurityGroupIngress
RevokeSecurityGroupIngress
Sample Web-Scale Architecture
GigaVox Media
Web-Scale Architecture
GigaVox Economics
Implemented Amazon S3, Amazon EC2 and Amazon SQS in November 2006
Created an infinitely scalable infrastructure for less than $100 - building the same infrastructure themselves would have cost thousands of dollars
Reduced staffing requirements - far less responsibility for 24x7 operations
Q & A
Thank You!
PointersPortalhttp://aws.amazon.com
Bloghttp://aws.typepad.com
EC2http://aws.amazon.com/ec2
S3http://aws.amazon.com/s3
Resource Centerhttp://aws.amazon.com/resources
Forumshttp://aws.amazon.com/forums
Jeff BarrSenior Web Services Evangelist
