SEDA:AnArchitectureforWell-Conditioned,ScalableInternetServices

MattWelsh,DavidCuller,andEricBrewerComputerScienceDivision

UniversityofCalifornia,Berkeley

OperatingSystemsPrinciples(SOSP-18),ChateauLakeLouise,Canada,October21-24,2001.

Motivation

• MillionsofInternetusers• DemandforInternetservicesgrows• Highvariationsinserviceload.Loadspikesareexpected• Webservicesgettingmorecomplex• Notstaticcontentanymore,dynamiccontentthatrequireextensivecomputationandI/O

Problemstatement

• Servicesthatsupportmillionsofusers(massiveconcurrency)• Responsive• Robust• Highlyavailable

• Wellconditionedservice:• Requestratescaleswiththeresponserate• Excessivedemanddoesnotdegradethroughputandallclientsexperienceanequalresponsetimepenaltylineartothelengthofthequeue(gracefuldegradation).

• Anotionoffairness

Thread-basedconcurrency

• Contentionforresourcesandcontextswitchescausehighoverhead• Highnumberofthreadsdegradesthethroughputandresponsetimeisgreatlyincreased• PartialSolution:Boundnumberofthreads

• Throughputmaintained• Butwhataboutmaxresponsetime?Someclientsexperiencelongwaitingtimes

• Overcommittingresources• Transparentresourcevirtualizationpreventsapplicationfromadaptingtoloadchangesandspotting

bottlenecks

Event-drivenconcurrency

• Efficientandscalableconcurrency• Butdifficulttoengineerandtune• Howordertheprocessingofevents.Schedulingchallenges• Difficultytofollowtheflowofevents• LittlesupportfromOS

StagedEvent-DrivenArchitecture(SEDA)

• Hybridapproach• Thread-basedconcurrencymodelsforeaseofprogramming• Event-basedmodelsforextensiveconcurrency

• MainIdea:Decomposeserviceintostagesseparatedbyqueues• Eachstageperformsasubsetofrequestprocessing

SEDA- Stage

• Eventqueuescanposevariouscontrolpolicies

• Modularity.Eachstageimplementedandmanagedindependently

• Expliciteventdeliveryfacilitatestracingflowofeventsandthusspottingbottlenecksanddebugging

Controllers

• Threadpoolcontroller:idealdegreeofconcurrencyforastage• Adjustnumberofthreadsbyobservingtheincomingqueuelength• Idlethreadsareremoved

• Batchingcontroller:aimsatlowresponsetimeandhighthroughput• Batchingfactor:numberofeventsconsumedateachiterationoftheeventhandler

• Largebatchingfactor:morelocality,higherthroughput• Smallbatchingfactor:lowerresponsetime

SEDAPrototype:Sandstorm

• ImplementedinJava• Javaprovidessoftwareengineeringbenefits

• Built-inthreading,automaticmemorymanagement

• APIsareprovidedfornaming,creatinganddestroyingstages,performingqueueoperations,controllingqueuethresholdsandprofilinganddebugging• AsynchronousI/OprimitivesareimplementedusingexistingOSprimitives.• Thesocketsinterfaceconsistsofthreestages:read,writeandlisten• AsynchronousI/Ofileoperations.

Evaluation

• UsedthestaticfileloadfromSpecWEB99benchmark,arealistic,industry-standardbenchmark• 1to1024clientsmakingrepeatedrequests• Filessizesrangefrom102to921600Bytes• Totalfilesetsizeis3.31GB• MemoryCacheof200Mb• Serverrunningon4-waySMP500MHzPentiumIIIsystemwith2GBofRAM• 32machinesofasimilarconfigurationwereusedforloadgeneration

EvaluatedHaboobahigh-performanceSEDA-basedHTTPserver

Otherdesigns• ApacheWebserver• Thread-basedconcurrency• Fixed-sizeprocesspoolof150processes

• FlashWebserver• Event-basedconcurrency• Singleprocesshandlingmostrequest-processingtasks.• Upto506simultaneousconnections(duetolimitationsofselectsystemcall).

• Haboob• Hybridapproach.Event-basedandthread-basedconcurrency• Upto1024concurrentrequests

Evaluation

Evaluation

• SEDAprovidessomefairness

• TheothertechniquessufferfromlongTCPretransmitbackoff times.Requestsrejectedandre-submitted

Evaluation

• Underoverload• Requestswithhighcomputation

andI/Oneeds from1024clients

• Admissioncontrolpolicybyqueues• Canperformprioritizationorfilteringduringheavyload

• Adjustsizeofqueueaccordingtotheresponsetime

• Maintainlowresponsetime

Summary

• Newdesignsneededfortheeverincreasingdemandsofwebservices• SEDAisthusproposedtoreachthedesiredperformance• Combinesthread-basedandevent-basedconcurrencymodels• Splitsanapplicationintoanetworkofstageswitheventqueuesinbetween• Dynamicresourcecontrollersforeachstage• Simplifiedbuildinghigh-concurrentservicesbydecouplingloadmanagementfromcoreapplicationlogic

Strengths

• Highconcurrency.Abilitytoscaletolargenumbersofconcurrentrequests• Easeofengineering.Simplifyconstructionofwell-conditionedservices• Modularity.Eachstageimplementedandmanagedindependently• Adaptiontoloadvariations.Resourcemanagementadjusteddynamically.• Lowvarianceinresponsetime

Weaknesses

• Increasedlatency.Arequesttraversemanystagesandexperiencesmultiplecontextswitchesandadditionaldelaysduetoqueuing.• Onalightlyloadedserver,theworstcasecontextswitchingoverheadcandominate

• Whatabouttheaveragecaseperformance?Istheworstresponsetimethemostimportantmetrictoconsider?

• Programmingstillharderthanthread-basedconcurrencymodels

Thankyou