Big Data Apps, on Any Cloud

Uri Cohen @uri1803 | github.com/uric

Head of Product @ GigaSpaces#cassandra12

Why Cloud Portability

Matters

Cloud Portability Myth #1

No one really needs cloud portability

Cloud Portability

Facts

Zynga moved ~80% of their workload from Amazon to their private zCloud

“own the base, rent the spike”

http://code.zynga.com/2012/02/the-evolution-of-zcloud/

Cloud Portability

Facts Started with Linode, then moved to RackSpace, then to AWS

http://code.mixpanel.com/2010/11/08/amazon-vs-rackspace/

Cloud Portability

Facts

• You want the flexibility to choose what’s right for you, when it’s right for you

• Based on pricing, features, availability, performance, etc.

Cloud Portability Myth #2

Cloud Portability ==

Cloud API Standardization

Cloud APIs, Today

Standard APIs (?)OCCIVCloud

OSS FrameworksOpenStackCloudStackEucalyptus

Abstraction frameworksJCloudsDeltacloudFogLibvirt

Cloud APIs, Today

Standard APIsNot practical in the foreseeable future

OSS Projects Need a couple more years to converge &

mature

Abstraction FrameworksProbably the only

practical (near-term) option

Realization:

What You Really Care

about Is App

Portability

OS is the same on any cloud

Most clouds have compute & storage

Elasticity & scaling have same effects on the app, regardless of the cloud

Cloud Portability Myth #3 All infrastructure

clouds were born equal

Food for Thought

Offerings can vary quite a bit:

• Amazon guarantees only 99.5% uptime

• RackSpace will give you $$$ every time they crash

• Joyent claims to be significantly faster than both

And Some Features Are

Unique…

Amazon the only major vendor to offer SSD storage. Netflix says it’s:

• ½ the price for the same throughput

• ⅕ the latency on avg.

• Even slowest requests are 6x faster

http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html

Let’s Talk Big Data on the Cloud

A Typical Big Data App…

Managing All of That on the Cloud

• Auto start VMs• Install and configure

app components • Monitor • Repair • (Auto) Scale• Burst…

Configuration Management

Can Help Quite a Bit

Configuration Management

Tools Are Best at:

• Auto start VMs• Orchestrate• Install and configure• Monitor • Repair (partially) • (Auto) Scale• Burst…

Big Data Apps, on Any Cloud, Your Way

Open source (Apache2)

Your App, Your Recipe

Recipes

Recipes

Recipes

Recipes

How It Works

CommandsResource Allocation

Invokes IaaSAPI

Bootstrap Java & Cloudify Agent

Install, Configure, Start, Monitor, &

Ensures H/A

Process Application

Recipe

ComputeServices

Demo Time!

Thank You!

References: http://www.cloudifysource.org http://github.com/CloudifySource http://code.mixpanel.com/2010/11/08/amazon-vs-rackspace/http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html http://natishalom.typepad.com/nati_shaloms_blog/2011/11/five-misconceptions-on-cloud-portability.html http://code.zynga.com/2012/02/the-evolution-of-zcloud/