-
8/9/2019 Highload++ - myYearbook.com Architecture
1/30
myYearbook.comArchitecture
LessonsLearnedfromtheTrialsof
ScalingaHighTrafficWebsite
-
8/9/2019 Highload++ - myYearbook.com Architecture
2/30
Foundedin2005 3rdLargestSocial
NetworkinUnited
States
TeenageDemographic 60+Employees
-
8/9/2019 Highload++ - myYearbook.com Architecture
3/30
January2007
100MPageviews 1DatabaseServer 1WebApplicaOonServer
Dailyissueswithloadandsiteavailability
-
8/9/2019 Highload++ - myYearbook.com Architecture
4/30
September2008
2.5BPageviews 30DatabaseServers 120WebApplicaOonServers
99.94%UpOmeasmeasuredbypingdom.com
-
8/9/2019 Highload++ - myYearbook.com Architecture
5/30
KeyArchitectureComponents PHP5,APC ApachehYpd PostgreSQL
Memcached ApacheAcOveMQ
LighYpd IsilonIQClusteredNAS MessageSystems
eCelerity
Subversion
-
8/9/2019 Highload++ - myYearbook.com Architecture
6/30
WebApplicaOonArchitecture 2005-2007:MonolithicCodeBase
2008:MigraOngtoaServicesrientedArchitecture
ApplicaOonsgetownresourcesLooselyCoupledarchitecture
MVCApplicaOonusingXSLT
-
8/9/2019 Highload++ - myYearbook.com Architecture
7/30
WebApplicaOonArchitecture WhySA?
Monolithicappwasteshardware
CrossData-CenterperaOons
SelecOveMaintenance
-
8/9/2019 Highload++ - myYearbook.com Architecture
8/30
ScalingPostgres
RulesforScaling1.PlanforGrowth2.Knowtheinternals3.BiggerHardwareis
BeYer
-
8/9/2019 Highload++ - myYearbook.com Architecture
9/30
urPostgresScalingHistory
Quarter1,2007Monolithicdatabasewithoneschema,manycomplexjoinsandpooropOmizaOon
NoplanforgrowthNoDBA
-
8/9/2019 Highload++ - myYearbook.com Architecture
10/30
urPostgresScalingHistory
Quarter3,2008HorizontalShardedDataVerOcalParOOoning5000ConnecOons/secAvg
-
8/9/2019 Highload++ - myYearbook.com Architecture
11/30
ScalingPostgres:LessonsLearned
ScalingwebserversmeansmanydatabaseconnecOons,neededpooling
StartedwithpgPoolmovedtopgBouncer
StartedwithSlonyreplicaOngread-onlyslaves
HighI/CPUverhead
-
8/9/2019 Highload++ - myYearbook.com Architecture
12/30
ScalingPostgres:LessonsLearned
BeganscalingverOcallybyseparaOngapplicaOondatabydatabaseserversand
removedreadonlyslaves
Neededfewsmalltablesreplicatedthatcouldbeslightlyinaccurateandeventuallyconsistent(BASE)
-
8/9/2019 Highload++ - myYearbook.com Architecture
13/30
ScalingPostgres:LessonsLearned
EnterplProxyDatabaseparOOoninglanguagebySkypeuOlizingPostgreSQLfuncOons
TriggerbasedplProxyfuncOonsreplicateneededtableswithouttheQueueoverhead
NTTRANSACTINSAFE
-
8/9/2019 Highload++ - myYearbook.com Architecture
14/30
ScalingPostgres:LessonsLearned
StandardUseofplProxyHorizontalparOOoningofdatabyIDacrossmulOpleservers
Example:MessagingSystem 8ServersstoreactualparOOonedmessagedata
Rule#1PlanforGrowth
-
8/9/2019 Highload++ - myYearbook.com Architecture
15/30
ScalingPostgres:LessonsLearned
Knowinginternalspg_catalog
pg_stat_user_tables pg_stat_user_indexes
-
8/9/2019 Highload++ - myYearbook.com Architecture
16/30
ScalingPostgres:KnowingInternals
-
8/9/2019 Highload++ - myYearbook.com Architecture
17/30
ScalingPostgres:LessonsLearned DatabaseEcosystem
PerformanceFactors Indexbloat Usagechanges
Abuse CacheuOlizaOoncontenOon
-
8/9/2019 Highload++ - myYearbook.com Architecture
18/30
ScalingPostgres:LessonsLearned BiggerisBeYer
MoreRAM MoreDisks FasterandMoreCPU
-
8/9/2019 Highload++ - myYearbook.com Architecture
19/30
ScalingPostgres:LessonsLearnedScalingAcrossCPUCores
PostgreSQLScalesto32Cores
ExtensiveBenchmarking@MB
BeforeandA=erUpgade
-
8/9/2019 Highload++ - myYearbook.com Architecture
20/30
ScalingPostgres:FuturePlans MoreParOOoning SADataDistribuOon
Golconde PythonBased ApacheAcOveMQ
-
8/9/2019 Highload++ - myYearbook.com Architecture
21/30
ApacheAcOveMQ JavabasedMessage
Brokersoqware
Clientlanguageneutral ImplementsJMS1.1,
Stomp,XMPP,RESTand
thers
-
8/9/2019 Highload++ - myYearbook.com Architecture
22/30
[email protected]
Uploadedcontentprocessing ImageResize Contentanalysis(R&D)
AnO-VirusScans
CommentandMessageprocessing SpamProcessing
EmailspoolingfromwebapplicaOon
Anywherewecanthatmakessense
TargetedWorkload
MessageQueuesallowfortherightserverforthejob
BeYerdistribuOonofCPUintensivetaskswithoutnegaOvelyimpacOngtheuserexperience
Clusterable,Scalable
-
8/9/2019 Highload++ - myYearbook.com Architecture
23/30
Memcached:KeyforSuccess ValuableScalingTool
ver250kgetreuestssecondduringpeakver750GBofcacheddataEasytoDeployThemoredistributedthecachebecomestheless
impacOngcachefailuresbecome-moreboxesarebeYerthanfewer
-
8/9/2019 Highload++ - myYearbook.com Architecture
24/30
Memcached:PotenOalProblems
LargescaleimplementaOonscanhavesomehidden
problems Lotsofnetworktraffic
Non-parOOonorevenlydistributeddata
Whattodofordatathatisnotevenlydistributed?
Implementedaround-robinclusterofmemcacheserversthatcontainthesamedata
-
8/9/2019 Highload++ - myYearbook.com Architecture
25/30
ResearchandDevelopment Copyr
Copy-on-WriteFilesystemReplicaOon Framewerk
PHP5DevelopmentFramework Golconde
QueueBasedDataDistribuOonforPostgreSQL Lightr
PHP5XMPPClassLibrary mod_xsltd
LighYpdXSLTransformaOonmodule Playr
PostgreSQLLogReplay Staplr
STAOsicalPackageLogicallyengineeredRight
-
8/9/2019 Highload++ - myYearbook.com Architecture
26/30
ToolsforSuccess
peraOonsPortalExecuOveLevelverviewofperaOonalStatusandProducOonChangeLog
StaplrTrending&AnalyOsSystem
-
8/9/2019 Highload++ - myYearbook.com Architecture
27/30
peraOonsPortal
-
8/9/2019 Highload++ - myYearbook.com Architecture
28/30
TrendingandAnalysis:Staplr Version0.6
PHPBased
Processforking ShelledRRDCommands
Version2.0 PythonBased Threaded Pythonwrapperstolibrrd
-
8/9/2019 Highload++ - myYearbook.com Architecture
29/30
TrendingandAnalysis:Staplr Pollsfor:
ApachehYpd ApacheAcOveMQ lighYpd memcached MySQL pgBouncer
PostgreSQL SNMPData
APC,Isilon,F5,Xiotech,thers SysStat
-
8/9/2019 Highload++ - myYearbook.com Architecture
30/30
QuesOons?
