View
779
Download
0
Category
Preview:
Citation preview
@aviranm
Aviran MordoHead of Engineering
@aviranm
linkedin.com/in/aviran
aviransplace.com
Scaling with Microservices Architecture & Multi-cloud platforms
@aviranm
@aviranm
Wix in NumbersOver 72M users (website builders)
Static storage is >2PB of data
3 data centers + 3 clouds (Google, Amazon, Azure)
2B HTTP requests/day
1000 people work at Wix
@aviranm
Initial Architecture
Built for fast developmentStateful login (Tomcat session), Ehcache, file uploadsNo consideration for performance, scalability and testingIntended for short-term use
Tomcat, Hibernate, custom web framework
Lighttpd(file
serving) MySQLDB
Wix(Tomcat)
@aviranm
The Monolithic GiantOne monolithic server that handled everything
Dependency between features
Changes in unrelated areas of the system caused deployment of the whole system
Failure in unrelated areas will cause system wide downtime
@aviranm
Breaking the System Apart
@aviranm
Concerns and SLA
Data Validation
Security / Authentication
Data consistency
Lots of data
Edit websites
High availability
High performance
Lots of static files
Very high traffic volume
Viewport optimization
Long tail (immutable)
Serving Media
High availability
High performance
High traffic volume
Long tail (mutable)
View sites, created by Wix editor
@aviranm
Wix Segmentation
1. Editor Segment 3. Public Segment2. Media Segment
Networking
@aviranm
HTML Editor
Flash Editor
MSM
Private Media
Public Media
Editor Segment Public Segment
Premium Services
eCommerse
List DB
App Builder
App Store
App Market
Dashboard
Statics/media
Mailer
TimeZone
Public HTML API
Public API (Flash)
MSP
Public Server
HTML Renderer
HTML SEO Renderer
Flash Renderer
Flash SEO Renderer
Sitemap Renderer
Robots.txt Renderer
User Server
Template Viewer
ContactsHUBActivit
y
Site Members
Provided Mailing Service
Comments
Snapshoter
User Pref
Feed Me
Shout-out Hotels
PETRI
Site Pref
Dist LoggerSlicer
eCom Renderer
eCom Cart
eCom Checkout
eCom Catalog
eCom Orders
Payment Facade
Account Info
HTML API
HTML Embeder
BlogMobile
@aviranm
TRADE-OFFIt is all about
@aviranm
Microservices GuidelinesEach service has its own DB schema (if one is needed)
Only one service should write to a specific DB table(s)
There may be additional read-only services that directly accesses the DB (for performance reasons)
Services are stateless
No DB transactions
Cache is not a building block, but an optimization
@aviranm
Microservices TradeoffsEach service has its own DB schema (if one is needed)
Gain - Easy to scale microservices based on service level concerns Tradeoff – system complexity, performance
Only one service should write to a specific DB table(s)Gain - Decoupling architecture – faster development Tradeoff – system complexity / performance
May have additional read-only services that accesses the DBGain - Performance gainTradeoff - coupling
Services are statelessGain - Easy to scale out (just add more servers)Tradeoff - performance / consistency
No DB transactionsGain - Better DB performance, easier to scaleTradeoff - system complexity
@aviranm
1. Editor Segment
@aviranm
Editor ServerImmutable JSON pages (~3M / day)
Site revisions
Active – standby MySQL cross datacenters
Editor Server
MySQL Active Sites
MySQL Archiv
e
@aviranm
Find Your Critical Path
@aviranm
Protect The DataDB outage with fast recovery = replication
Data poisoning/corruption = revisions / backup
Make the data available at all times = data distribution to multiple locations / providers
@aviranm
Browser
Editor Server
GCS
MySQL Active Sites
MySQL Archiv
e
Saving Editor DataWixMedia(Amazon)
WixMedia(Google)
Save Page(s)
200 OKUpload
Save Page
DC replication
Notify
MySQL Archiv
e
MySQL Active Sites
S3
WixMedia(DC-1)
@aviranm
Browser
Editor Server
GCS
MySQL Active Sites
MySQL Archiv
e
WixMedia(Amazon)
WixMedia(Google)
Save Page(s)
200 OKUpload
Save Page
DC replication
Notify
MySQL Archiv
e
MySQL Active Sites
S3
WixMedia(DC-1)
Self Healing Process
@aviranm
No DB TransactionsSave each page (JSON) as an atomic operation
Page ID is a content based hash (immutable/idempotent)
Finalize transaction by sending site header (list of pages)
Can generate orphaned pages, not a problem in practice
@aviranm
2. Media Segment (WixMP)
@aviranm
Wix Media Platform (WixMP)
Eventual consistent distributed file system(2PB user media files)
Dynamic media processing
Multi datacenter aware
Automatic fallback cross DC
Run on commodity servers & cloud
@aviranm
T
Google Cloud
Prospero – Wix Media Manager
get image.jpg
First fallback
Second
fallback
If not in
CDN
Amazon
x36Tx36
Tx32
Austin
CDN
@aviranm
3. Public Segment
@aviranm
Public Segment RolesRouting (resolve URLs)
Dispatching (to a renderer)
Rendering (HTML,XML,TXT) Public
Server
HTML Rendere
r
HTML SEO
Renderer
Flash Renderer
Sitemap Rendere
r
Robots.txt
Renderer
www.example.com
Flash SEO
Renderer
@aviranm
Public SLAOur goal: 99% response time <100ms at peak traffic
@aviranm
Publish SitePublish site header (a map of pages for a site)
Publish routing table
Publish site header / routes (CQRS)Editor Segment Public Segment
@aviranm
Built For SpeedMinimize out-of-process hops (2 DB, 1 RPC)
Lookup tables are cached in memory, updated every few minutes
Denormalized data – optimize for read by primary key (MySQL)
Minimize business logic
@aviranm
How a Page Gets RenderedBootstrap HTML template that contains only data
Only JavaScript imports
JSON data (site-header + dynamic data)
No “real” HTML view
@aviranm
Offload rendering work to the browser
@aviranm
The average Intel Core i750 can push up to 7 GFLOPS without overclocking
@aviranm
Why JSON?Easy to parse in JavaScript and Java/Scala
Fairly compact text format
Highly compressible (5:1 even for small payloads)
Easy to fix rendering bugs and cross browsers issues (just deploy a new code)
@aviranm
Minimum Number of Public Servers Needed to Serve 66M Sites
4
@aviranm
Public SLABe Available 99.999%
@aviranm
Serving a Site – Sunny Day
Archive
CDN WixMP
Browserhttp://
example.wix.com
Store HTML to cache
HTTP Request
Notify site view
LB
Public
Renderer
HTML
Resources / Media
HTTP Request
Failure Points
@aviranm
Serving a Site – DC Lost
Archive
CDN WixMP
Browserhttp://
example.wix.com
LB
Public
Renderer
LB
Public
RendererChange DNS
HTTP Request
@aviranm
Serving a Site – Public Lost
Archive
Browserhttp://
example.wix.com
LB
Public
Renderer
Get Cached HTML Version
HTMLHTTP Request
LB
Public
Renderer
Fallback to 2nd DC
@aviranm
Living in the BrowserCDN WixM
P
Browserhttp://
example.wix.com
LB
Public
Renderer
Editor Pages
Fallback
JSON / Media
HTMLHTTP Request Fallbac
k
@aviranm
SummaryIdentify concerns and SLA for different parts of the system
Build redundancy in critical path (for availability)
De-normalize data (for performance)
Minimize out-of-process hops (for performance)
Take advantage of client’s CPU power
@aviranm
@aviranm
http://engineering.wix.com
http://goo.gl/THbU2K@WixEng
We’re hiringwww.wix.com/jobs
Q&A
Recommended