Inuit: a System for Inuit: a System for Waste Free ComputingWaste Free Computing

Stefan SavageStefan SavageMichael Frederick, Diwaker Gupta, Marvin McNett, Alex Snoeren, Michael Frederick, Diwaker Gupta, Marvin McNett, Alex Snoeren,

Amin Vahdat, Geoff Voelker, Amin Vahdat, Geoff Voelker, Michael Vrable, Ken YocumMichael Vrable, Ken Yocum

University of California, San DiegoUniversity of California, San Diego

MSR Cambridge Colloquium, 10/27/2005MSR Cambridge Colloquium, 10/27/2005

> From: Rebecca Isaacs [mailto:risaacs@microsoft.com]> Sent: Friday, September 16, 2005 11:53 AM> To: Stefan Savage> Cc: jon.crowcroft@cl.cam.ac.uk; > Subject: RE: Invitation - Cambridge Systems Colloquium 27 Oct> Hi Stefan,> Actually wild gesticulations could be quite entertaining...

First: a warning

Thus, this talk is wild speculation I am unconstrained by

• thinking my ideas through• evaluating if they might be practical• or, any real experience with the problem domain

> From: Stefan Savage [mailto:savage@cs.ucsd.edu]

> Sent: 03 August 2005 09:37

> To: Rebecca Isaacs

> Cc: jon.crowcroft@cl.cam.ac.uk; mvrable@cs.ucsd.edu

> Subject: RE: Invitation - Cambridge Systems Colloquium 27 Oct

> If you'd like to get a talk out of me I can certainly do one, but I'm

> not sure the talks I have on tap will be a perfect fit for the topics

> below (unless you're amenable to a wildly gesticulating rant)

American Culture

The US has developed, by far, the most consumption-oriented, wasteful culture in the history of mankind 400 million tons of waste/year (almost 25% of world) 100 quads energy/year (almost 30% of world usage)

Inuit Culture

Indigenous people of North America and Eastern Russia Subsistence hunters -> Bowhead Whale

“Nothing is wasted for to waste would be disrespectful to the whale” – Abvibich lglaunifat Niginmun

Meat -> protein Jaw bones -> sled runners Ribs -> Fences and arrow points Shoulder blades -> housing structures Baleen -> Rope and baskets Lungs/Liver -> Drums Blubber -> Oil lamps

Modern computing is very American

Most of the time they do nothing of value void cpu_idle (void) (200 Trillion BTUs?)

Yet, we keep buying more (250M last year!) And, we throw out the old ones (150M last year)

Bottom line: computers are treated as disposable items to be acquired to excess, lightly used and then discarded

A Case Study:Systems Computing at UCSD

100+ servers ~2-5 Trillion Instructions per Second ~150GB memory ~22kW power consumption (peak) ~ 100kBTU/hr heat load (peak)

We have ~30 students sharing these machines

The waste paradox Students insist that resources are scarce and we need to buy

more machines They can’t get machines when they need them, and They can’t run the large-scale experiments they want

Yet…. resource utilization is << %5

Why is this?

We all like the abstraction of a machine as a unit of computing allocation This is deep in the DNA of the PC computing model Provides isolation (software, performance)

Our direct implementation of this abstraction is where we get into trouble High preemption cost -> long term allocation

“Get off my machines you @%*#-head” Poor resource sharing

Your disk storage is not unique [Bolosky et al] Your memory is not unique Your computing is not unique

The Inuit system

Unit of allocation is the machine cluster (can be 1 machine) Implementation via virtual machines (modified Xen) Key idea: Try to eliminate all unintended resource redundancy

Extreme multiplexing Storage sharing System continuations Maximize cluster-level parallelism

Aggressive migration Place to maximize resource utilization (sharing AND use)

Cluster activations Allow cluster applications to adapt to changes in resource availability

Potemkin + Xenoservers + Parallax + The Collective + GMS + Single Instance Store + Pastiche + Rx + Transparent Result Caching + judicious amounts of grad student effort

Extreme multiplexing

Share storage between “similar” VMs CoW memory/disk (ala Potemkin) Discover newly shared data (ala ESX server) Throw away “idle” storage remotely available Allows 10’s or 100’s of VM’s per physical machine Many VMs -> likely that a VM is available when CPU goes idle

System continuations: share execution traces across “similar” VMs Pre-cache state changes from common (input, CFG) blocks Roll-forward VMs to cached result state (opposite of ReVirt) Note: leverage controlled non-determinism (Rx) -> Plausibility is

sufficient

Aggressive migration

Cluster “similar” VMs on the same physical machines To maximize cheap sharing opportunities

Migrate VM/storage to maximize global utilization Balance CPU/memory load E.g., Large working set VMs can disrupt multiplexing

Migrate VM away, or migrate memory away (depends on workload)

Cluster activations

For those who need multiple simultaneous machines (virtual clusters)

Gang scheduling of VC VM’s Can specify hard/soft resource requirements Violate utilization maximization for hard requirements

Application notification on soft resource violation Provide feedback about resources backing VMs

Where does this make sense?

Obvious places UCSD Datacenters The “Grid” Application servers

Less obvious Desktops Huh?

Inuit on the desktop?

Two opportunities VMs make Networks of Workstations palatable

Idle CPU/Memory up for grabs

Manageability Encapsulating desktop in VM makes deployment easy Checkpoint/restore of whole system image Cheap migration of whole workstation

(Internet Suspend/Resume/Collective)

Where are we now?

At the social engineering stage Get everyone to eat the dog food and fix problems until its at least

as good as what we had Almost all of our servers now run Xen by default

Students forced to use Xen as their only environment Lots of support infrastructure

Address allocation/routing + DNS (IP addresses & DNS names correspond to VMs not PMs)

Control interface (which images onto which VMs, VM auth,etc) Half-done with virtual reboot/tty interface

Work to improve Potemkin stability/overhead

The rest is a hand wave

Questions

?