Embed Size (px)
Citation preview
Building a Massively Scalable
Cloud Service
from the Grounds Up
About your speaker
linkd.in/jbaruch
stackoverflow.com/users/402053/jbaruch
What Frog?
What Frog?
What Frog?
What Frog?
So…
Some Numbers
liftoff + 26 months
Some Numbers
liftoff + 26 monthsRepositories 36K
Some Numbers
liftoff + 26 monthsRepositories 36KPackages 120K
Some Numbers
liftoff + 26 monthsRepositories 36KPackages 120KRequests 1.7 B/Month
Requirements
Requirements
– Download binaries
Requirements
– Download binaries– Web Front
Requirements
– Download binaries– Web FRONT– REST API
Requirements
– Download binaries– Web FRONt– REST API– Backend services
We know developers
%new_sexy_lang% community
Not our fault! AWS failed again!
Downloads must…
Web application must…
Backend Services must…
Choose your battles...
Non-Func. Requirements
Non-Func. Requirements
Requirement RPS Availability
Non-Func. Requirements
Requirement RPS AvailabilityDownload 10K Always
Non-Func. Requirements
Requirement RPS AvailabilityDownload 10K AlwaysInteraction 200 Almost always
Non-Func. Requirements
Requirement RPS AvailabilityDownload 10K AlwaysInteraction 200 Almost alwaysServices 10 Most of the time
DOWNLOAD SERVICE
No Servlets here
Deduplication by Checksum
File A: 46b34
File B: a64ff7
/user-a/repo-z/package-y/file-x
/org-c/repo-m/package-n/file-k
/user-m/repo-w/package-t/file-f
Flat blobs storage
File A: 46b34
File B: a64ff7
/user-m/repo-w/package-t/file-f
Mapping
WEB FRONT
WEB FRONT
UI
Requirements
Requirements
– Rapid Application Development
Requirements
– Rapid Application Development– Flexible schema
Requirements
– Rapid Application Development– Flexible schema– Java Background
Requirements
– Rapid Application Development– Flexible schema– Java Background– Stateless
Why don’t you just use...?
Framework Why not?
Why don’t you just use...?
Framework Why not?
Angular.js Ember.js ж.js Maturity
Why don’t you just use...?
Framework Why not?
Angular.js Ember.js ж.js MaturityWicket State
Why don’t you just use...?
Framework Why not?
Angular.js Ember.js ж.js MaturityWicket StateJSF Model
Why don’t you just use...?
Framework Why not?
Angular.js Ember.js ж.js MaturityWicket StateJSF ModelNon-java No java bg
And look what happened:
And look what happened:
Grails minor update
WEB FRONT
Data Model
Remember?
Grails means
Gorm!
Gorm MongoDB plugin
WEB FRONT
Search
Search
2 types of search
Full Text Search Structured Search
2 types of search
Full Text Search Structured Search
Executive summary
Framework Why not?
Executive summary
Framework Why not?Lucene/compass Only embedded, resource
guzzler
Executive summary
Framework Why not?Lucene/compass Only embedded, resource
guzzlersolr Bad grails integration
Executive summary
Framework Why not?Lucene/compass Only embedded, resource
guzzlersolr Bad grails integrationsphynx Painful incremental index
vs.
vs.
You ask
ElasticSearch answers
WEB FRONT
Additional Services: Indexing, Stats, Logs
Redis to the resque
Did they just add a 4th nosql?!
WEB FRONT
Additional Services: Documentation
DEVOPS
DEVOPS
IaaS vs. SaaS
Leave it to the Pros
Or?
Physical vs. Virtual
Remember this?
Virtualization
VirtualizationPros
VirtualizationPros– Cheap
VirtualizationPros– Cheap– elastic
VirtualizationPros– Cheap– elastic– Volatile
VirtualizationPros– Cheap– elastic– Volatilecons
VirtualizationPros– Cheap– elastic– Volatilecons– Overhead
VirtualizationPros– Cheap– elastic– Volatilecons– Overhead– Tenant, not owner
DEVELOPMENT ENVIRONMENT
Remember?
We are liberal
We are liberal
We are liberal
We are liberal
The Trend
The Trend
Peter Piper picked a peck of pickled
peppers.
Did Peter Piper pick a peck of pickled
peppers?
If Peter Piper Picked a peck of pickled
peppers,
Where's the peck of pickled peppers Peter
Piper picked?
The Trend
Docker Docker picked a peck of Docker
Docker.
Did Docker Docker pick a peck of Docker?
If Docker Docker Docker a peck of Docker
Docker,
Docker Docker Docker Docker Docker
Docker Docker?
How Docker can help?
Where Docker can’t help?
No problem, developers will
manage virtual machines!
No problem, developers will
manage virtual machines!
Vagrant to the rescue
1. Vagrant boots linux VM on virtualbox
1. Vagrant boots linux VM on virtualbox
2. Docker runs all databases and services from images
1. Vagrant boots linux VM on virtualbox
2. Docker runs all databases and services from images
3. Profit!
HIGH AVAILABILITY
(and Locality)
Cluster everything
Remember?
CDN for Download Server
Or?
Or?
Or?
GTD for Web Application
BACKUP
and Vendor lock-out
Here’s the caveat:
Here’s the caveat:
Snapshots and replicas
MONITORING
State and Logs
Prevent this:
All together now
Conclusions time
Conclusions time
– Define Criticality
Conclusions time
– Define Criticality– Embrace the change
Conclusions time
– Define Criticality– Embrace the change– Plan for scale, but be realistic
Conclusions time
– Define Criticality– Embrace the change– Plan for scale, but be realistic– Backup everything!
No, thank you!
