Scaling Wanelo.com 100x in Six Months

Proprietary and Confidential

Scaling

100x in six months

by Eric Saxby & Konstantin GredeskoulApril 2013

1Thursday, April 18, 13

What is Wanelo?

■ Wanelo (“Wah-nee-lo” from Want, Need Love) is a global platform for shopping.

What is Wanelo?

■ Wanelo (“Wah-nee-lo” from Want, Need Love) is a global platform for shopping.

■ It’s marketing-free shopping across 100s of thousands of unique stores

Personal Activity Feed...4Thursday, April 18, 13

iOS + Android

Early Decisions

■ Optimize for iteration speed, not performance

Early Decisions

■ Keep scalability in mind, track metrics, and fix as needed

Early Decisions

■ Keep scalability in mind, track metrics, and fix as needed

■ Introduce many levels of caching early

Early Decisions

Technology Timeline

Proprietary and Confidential 7Thursday, April 18, 13

Technology Timeline

■ 2010 - 2011 Wanelo v1 stack is Java, JSP, MySQL, Hibernate90K lines of code, 53+ DB tables, no tests

Technology Timeline

■ May 2012 - June 2012Rewrite from scratch to RoR on PostgreSQL (v2)

Technology Timeline

■ May 2012 - June 2012Rewrite from scratch to RoR on PostgreSQL (v2)

■ Ruby app is 10K LOC, full test coverage, 8 database tables, less features

The “Big” Rewrite

More info here....

http://building.wanelo.com/

More info here....

http://building.wanelo.com/

Growth Timeline

■ 06/2012 - RoR App Relaunches

Growth Timeline

■ 2-3K requests per minute (RPM) peak ■ 06/2012 - RoR App Relaunches

Growth Timeline

■ 08/2012 - iOS App is launched

■ 2-3K requests per minute (RPM) peak ■ 06/2012 - RoR App Relaunches

Growth Timeline

■ 2-3K requests per minute (RPM) peak

■ 10-40K RPM peak

Growth Timeline

■ 12/2012 - Android app launched

■ 10-40K RPM peak

Growth Timeline

■ 40-120K RPM peak

■ 10-40K RPM peak

Growth Timeline

■ 03/2013 - #24 top free apps iTunes

■ 10-40K RPM peak

Growth Timeline

■ 03/2013 - #24 top free apps iTunes

■ 10-40K RPM peak

Requests Per Minute (RPM)

Current Numbers...

■ 4M active monthly users

■ 5M products saved 700M times

■ 8M products saved per day

■ 200k stores

Backend Stack & Key Vendors

■ MRI Ruby 1.9.3 & Rails 3.2

■ PostgreSQL 9.2.4, Solr 3.6

■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace

■ Circonus, Chef + OpscodeMonitoring, graphing, alerting, automation

■ Amazon S3 + Fastly CDN

■ NewRelic, statsd, Graphite, nagios

Wanelo Web Architecture

haproxy

unicorn x 14

haproxy pgbouncer twemproxy

PostgreSQLSolr Redis MemCached

sidekiq

20 x 8GB 4 x 8GB

6 x 2GB

This talk is about:

1. How much traffic can your database handle?

This talk is about:

2. Special report on counters

This talk is about:

3. Scaling database reads

This talk is about:

3. Scaling database reads

4. Scaling database writes

1.How much traffic can your database handle?

PostgreSQL is Awesome!

■ Does a fantastic job of not corrupting your data

■ Streaming replication in 9.2 is extremely reliable

■ Won’t write to a read-only replica

■ But... No master/master replication

■ But... No master/master replication(good!)

Is the database healthy?

What’s healthy?

■ Able to respond quickly to queries from application (< 4ms disk seek time)

What’s healthy?

■ Has enough room to grow

What’s healthy?

■ Has enough room to grow

■ How do we know when we’re approaching a dangerous threshold?

NewRelic Latency (yellow = database)

pg_stat_statements

■ Maybe your app is to blame for performance... select query, calls, total_time from pg_stat_statements order by total_time desc limit 12;

pg_stat_statements

■ Maybe your app is to blame for performance... select query, calls, total_time from pg_stat_statements order by total_time desc limit 12;

Similar to Percona Toolkit, but runs all the time collecting stats.

pg_stat_statements

pg_stat_user_indexes

■ Using indexes as much as you think you are?

■ Using indexes at all?

pg_stat_user_indexes

■ Using indexes as much as you think you are?

■ Using indexes at all?

pg_stat_user_tables

■ Full table scans? (seq_scan)

pg_stat_user_tables

■ Full table scans? (seq_scan)

Throw that in a graph

Reads/second for one large table, daily

Non-linear changes

Suspicious spike!

Correlate different data

Deployments! Aha!

Utilization vs Saturation

# of Active PostgreSQL connections

Red line: % of max connections establishedPurple: % of connections in query

Disk reads/writes

green: reads, red: writes

Disk reads/writes

Usage increases, but are the disks saturated?

green: reads, red: writes

[How much are you waiting on disk?

File system cache (ARC)

Watch the right things

Hit ratio of the file system cache (ARC)

Watch the right things

Hit ratio of the file system cache (ARC)

Room to grow...

Size (including indexes) of a key table

Working set in RAM?

Adding index increases the size

Working set in RAM?

Adding index increases the size

Collect all the data you can

Once we knew where to look, graphs addedlater could explain behavior we could

only guess at earlier

Collect all the data you can

Once we knew where to look, graphs addedlater could explain behavior we could

only guess at earlier

2.Special report on Counters and Pagination

Problem #1: DB Latency Up...

■ iostat shows 100% disk busy

device r/s w/s Mr/s Mw/s wait actv svc_t %w %b

sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100

Problem #1: Diagnostics

■ Database is running very very hot. Initial investigation shows large number of counts.

■ Turns out anytime you page with Kaminari, it always does a count(*)!

SELECT "stores".* FROM "stores" WHERE (state = 'approved') LIMIT 20 OFFSET 0

SELECT COUNT(*) FROM "stores" WHERE (state = 'approved')

Problem #1: Pagination

■ Doing count(*) is pretty expensive, as DB must scan many rows (either the actual table or an index)

■ We are paginating everything! Even infinite scroll is a paged view behind the scenes.

■ But we really DON’T want to run count(*) for every paged view.

■ We are showing most popular stores

■ Maybe it’s OK to hard-code the total number to, say, 1000?

■ We are showing most popular stores

■ Maybe it’s OK to hard-code the total number to, say, 1000?

■ How do we tell Kaminari NOT to issue a count query in this case?

Problem #1: Pagination (ctd)

Solution #1: Monkey Patch!!

Solution #1: Pass in the counter

SELECT "stores".* FROM "stores" WHERE (state = 'approved') LIMIT 20 OFFSET 0

Solution #1: Pass in the counter

■ AKA: We still are doing too many counts!

Problem #2: Count Draculas

■ Rails makes it so easy to do it the lazy way.

■ But it just doesn’t scale well

Problem #2: Too Many Counts!

■ Fortunately, Rails has just a feature for this...

■ Unfortunately, it has one massive issue:

Counter Caches

■ It causes database deadlocks at high volume

Counter Caches

■ Because many ruby processes are creating child records concurrently

Counter Caches

■ Each is executing a callback, trying to update counter_cache column on the parent, requiring row-level lock

Counter Caches

■ Each is executing a callback, trying to update counter_cache column on the parent, requiring row-level lock

■ Deadlocks ensue

Possible Solution:Use Background Jobs

■ It works like this:

■ As the record is created, we enqueue a request to recalculate counter_cache on the parent

■ The job performs a complete recalculation of the counter cache and is idempotent

Solution #2: Explained

■ Sidekiq with UniqueJob extension

■ Short wait for “buffering”

■ Serialize updates via small number of workers

■ Can temporarily stop workers (in an emergency) to alleviate DB load

Solution #2: Code

Things are better. BUT...

Things are better. BUT...Still too many fucking counts!

■ Even doing count(*) from workers is too much on the databases

Still too many fucking counts!

■ We need to stop doing count(*) in DB. But keep counter_caches. How?

■ We could use Redis for this.

Solution #3: Counts Deltas unicorn

PostgreSQL

sidekiq

save product product_id

RedisCounters

RedisSidekiq

3. Dequeue

2. ProductCountWorker.enqueue product_id

4. GET5. RESET

1. INCR product_id

5. SQL Update INCR by N

counter_cache column

Solution #3: Counts Deltas unicorn

PostgreSQL

sidekiq

save product product_id

RedisCounters

RedisSidekiq

3. Dequeue

2. ProductCountWorker.enqueue product_id

4. GET5. RESET

1. INCR product_id

5. SQL Update INCR by N

counter_cache column

■ Web request increments counter value in Redis

■ Enqueues request to update counter_cache

■ Background Job picks up a few minutes later, reads Redis delta value, and removes it.

■ Updates counter_cache column by incrementing it by delta.

Define counter_cache_on...

■ Internal GEM, will open source soon!

Can now use counter caches in pagination!

3.Scaling reads

Multiple optimization cycles

■ Cachingaction caching, fragment, CDN

■ Personalization via AJAXCache the entire page, then add personalized details

■ 25ms/req memcached time is cheaper than 12ms/req of database time

Cache optimization

40% hit ratio! Woo!Wait... is that even good?

Cache optimization

Increasing your hit ratio means lessqueries against your database

Cache optimization

Caveat: even low hit ratio cachescan save your ass. You’re removing

load from the DB, remember?

Cache saturation

How long before your cachesstart evicting data?

Blue: cache writesRed: automatic evictions

Cache saturation

Ajax personalization

■ Rails Action CachingRuns before_filters, so A/B experiments can still run

■ Extremely fast pages4ms application time for some of our computationally heaviest pages

■ Could be served via CDN in the future

Sad trombone...

■ Are you actually logged in?Pages don’t know until Ajax successfully runs

■ Selenium AND Jasmine tests!

Read/write splitting

■ Sometime in December 2012...

■ Database reaching 100% saturation

■ Latency starting to increase non-linearly

■ We need to distribute database load

■ We need to use read replicas!

DB adapters for read/write

■ Looked at several, including DbCharmer

DB adapters for read/write

■ Features / Configurability / Stability

■ Thread safety? This may be Ruby, but some people do actually use threads.

■ If I tell you it’s a read-only replica, DON’T ISSUE WRITES

■ Failover on errors?

■ Looked at several, including DbCharmer

Chose Makara, by TaskRabbit

■ Used in production

■ We extended it to work with PostgreSQL

■ Works with Sidekiqs (thread-safe!)

■ Failover code is very simple. Simple is sometimes better.

https://github.com/taskrabbit/makara

We rolled out Makara and...

■ 1 master, 3 read-only async replicas

We rolled out Makara and...

Wait, what?

■ 1 master, 3 read-only async replicas

A note about graphs

■ NewRelic is great!

■ Not easy to predict when your systems are about to fall over

■ Use something else to visualize Database and disk saturation

3 days later, in production

■ 3 read replicas distributing load from master

■ app servers and sidekiqs create lots of connections to DB backends

■ Mysterious spikes in errors at high traffic

Replication! Doh!

Replication lag (yellow) correlates with application errors (red)

Replication lag! Doh!

■ Track latency sending xlog to slavesselect client_addr, pg_xlog_location_diff(sent_location, write_location) from pg_stat_replication;

■ Track latency applying xlogs on slavesselect pg_xlog_location_diff( pg_last_xlog_receive_location(), pg_last_xlog_replay_location()), extract(epoch from now()) - extract(epoch from pg_last_xact_replay_timestamp());

Eventual Consistency

■ Some code paths should always go to master for reads (ie, after signup)

■ Application should be resilient to getting RecordNotFound to tolerate replication delays

■ Not enough to scale reads. Writes become the bottleneck.

■ Application should be resilient to getting RecordNotFound to tolerate replication delays

Write load delays replication

Replicas are busy trying to apply XLOGsand serve heavy read traffic

4.Scaling database writes

First, No-Brainers:

■ Move stuff out of the DB. Easiest first.

First, No-Brainers:

■ Tracking user activity is very easy to do with a database table. But slow.

First, No-Brainers:

■ 2000 inserts/sec while also handling site critical data? Not a good idea.

First, No-Brainers:

■ 2000 inserts/sec while also handling site critical data? Not a good idea.

■ Solution:UDP packets to rsyslog, ASCII delimited files, log-rotate, analyze them later

Next: Async Commits

■ PostgreSQL supports delayed (batched) commits

Next: Async Commits

■ Delays fsync for some # of microseconds

Next: Async Commits

■ Delays fsync for some # of microseconds

■ At high volume helps disk IO

PostgreSQL Async Commits

ZFS Block Size

■ Default ZFS block size is 128Kb

ZFS Block Size

■ PostgreSQL block size is 8Kb

ZFS Block Size

■ Small writes require lots of bandwidth

ZFS Block Size

sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100

■ Small writes require lots of bandwidth

ZFS Block Size (ctd.)

■ Solution: change ZFS block size to 8K:

device r/s w/s Mr/s Mw/s wait actv svc_t %w %b sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100

device r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd1 130.3 219.9 1.0 4.4 0.0 0.7 2.1 0 37sd1 329.3 384.1 2.6 11.3 0.0 1.9 2.6 1 78sd1 335.3 357.0 2.6 8.7 0.0 1.8 2.6 1 80sd1 354.0 200.3 2.8 4.9 0.0 1.6 3.0 0 84sd1 465.3 100.7 3.6 1.7 0.0 2.1 3.7 0 91

Next: Vertical Sharding

■ Move out largest table into its own master database (150 inserts/sec)

■ Remove any SQL joins, do them in application, drop foreign keys

■ Switch model to establish_connection to another DB. Fix many broken tests.

Vertical Sharding

unicorns

PostgreSQLmain master

PostgreSQLsaves masterPostgreSQL

main replicaPostgreSQLmain replica

streaming replication

Vertical Sharding: Results

■ Deploy All Things!

Future: Services Approach

unicorns

PostgreSQLmain master

Shard1

PostgreSQLmain replica

streaming replication

sinatra services app

Shard2 Shard3

http / json

In Conclusion. Tasty gems :)

https://github.com/wanelo/pause

https://github.com/wanelo/spanx

https://github.com/wanelo/redis_with_failover

https://github.com/kigster/ventable

■ distributed rate limiting using redis

■ rate-limit-based IP blocker for nginx

■ attempt another redis server if available

■ observable pattern with a twist

Thanks.

Comments? Questions?

https://github.com/wanelo

https://github.com/wanelo-chef

@kig & @sax

@kig & @ecdysone

@kigster & @sax

Scaling Wanelo.com 100x in Six Months

Technology

SOLUCIONARIO - adistanciaginer.files.wordpress.com · 2 unidad 1. nÚmeros reales solucionario. 310 16. x = 0,1818…. 100x = 18,1818…, restando 100x = 18,1818 x = 0,1818….

Data Dissemination Best Practices and Challenges …...Big Data Project Lessons Learned 1.2 PBs of climate and weather data accessed through Google BigQuery, in 4 months 30-100x of

blood!100x! blood!400x! glandular!epithelium!40x ... 218...artery!and!nerve!100x! basophil!400x! basophil!and!neutrophil!400x!!!!! !!!! ! eosinophil!400x! lymph!node!cortex!100x! lymph!node!medulla!100x!!!!!

2013 Perforce Collaboration Tour - 100x and Insights

100x Organisasi Unit Maintenance.pdf

Common Gulf of Maine Phytoplankton v12 · 2019-02-26 · Ceratium fusus (CF) (100x) Ceratium lineatum (CL) (100x) Ceratium longipes (CP) (100x) Phaeocystis sp (PC) (100x) Chaetoceros

th19 Annual Wealth Creation Study 2009-2014 100x · th19 Annual Wealth Creation Study 2009-2014 By Raamdeo Agrawal 12 December 2014 . 1 100x . 2 Theme Discussion Points 100x: What

Using a USB would've been 100x easier

ISSN 1309-100X Pressure Transmission with Cuspal

The Best-Kept Secret of the Most Effective CISOs: 100x Cybersecurity … · 2020-04-21 · Best kept secret: 100x cybersecurity posture visibility Great cybersecurity posture visibility

(ARC346) Scaling To 25 Billion Daily Requests Within 3 Months On AWS

Accu Vision Ct12-100x

SEH-100X - KOSHIN LTD No.012867603 MODEL:SEH-100X (GX240) FORM:SEH-100X-BAE-0 SEH-100X PARTS CODE PARTS NAME QTY MATERIAL SIZE REMARKS 1 CE OPERATION MANUAL (5LANGS) 011476909 1,100

Swingline Stack-and-Shred 100X Shredder Auto+ Technology

SEM-100X - KOSHIN LTD No.012877602 MODEL:SEM-100X (GM231) FORM:SEM-100X-BAA-0 SEM-100X PARTS CODE PARTS NAME QTY MATERIAL SIZE REMARKS OPERATION MANUAL (E) 011577103 1 160 DWG No SE

Vertical Dry Adhesive Climbing with a 100x …bdml.stanford.edu/uploads/Main/MicroTugs/ClimbingMicro...Vertical Dry Adhesive Climbing with a 100x Bodyweight Payload Elliot W. Hawkes

LRAD 100X Datasheet

DLRO100E, 100EB, 100X, 100XB, 100H & 100HB 100A High

7414 100gr 100x REF

REGISTRO FOTOGRÁFICO HALLAZGOS CITOLÓGICOS (4x-40x-100x) · 2020. 12. 15. · REGISTRO FOTOGRÁFICO HALLAZGOS CITOLÓGICOS (4x-40x-100x) Paciente: Hector- Bulldog Ingles- Clinica