data tieringSqueezing scale out of MySQL
Julien, VP Engineering at HouseTrip
github.com/mezis
disclaimer
IANADBA*I’m not a database administrator
!3
the scene
!4
!5
Load balancer(ELB)
30x10web workers
(Passenger/Rack)
6x20job workers
(DJ)
Memcache4x7GB
MySQL400GB
5x replica
MondoDB120GB
2x replica
S3TBs
85k qpm
3k rpm
predictable traffic ~25% searches
!6
search = [destination, start date, end date]
↓[ [property, price], … ]
!7
property!
destination_id
availability!
property_id rate_id
start_date end_date
rate!
price
search → big bulky join
booking → long transaction+ business logic
+ many small R/W
!8
property!
destination_id
availability!
property_id rate_id
start_date end_date
rate!
price
the crisis
!9
peak traffic 7pm - 10pm write queries !
read queries "
write IO ⛅️
cpu load ⛅️
memory ☀️
!10
contention
noun (kənˈtɛnʃən)
1. a struggling between opponents
2. competition for limited resources
!11
slow reads ? poor use of indices
during large write transactions
!12
http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-transaction-management.html http://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_covering_index
slow writes ? load+locking on rollback segments
!13
http://dev.mysql.com/doc/refman/5.1/en/innodb-multi-versioning.htmlhttp://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_rollback_segment
digging & deeper SHOW ENGINE INNODB STATUS '
!14
---TRANSACTION 72C, ACTIVE 755 sec 4 lock struct(s), …, 3 row lock(s), undo log entries 12 TABLE LOCK …RECORD LOCKS …RECORD LOCKS … locks rec but not gap RECORD LOCKS … lock_mode X locks gap before rec
http://www.mysqlperformanceblog.com/2012/03/27/innodbs-gap-locks/http://dev.mysql.com/doc/refman/5.1/en/innodb-monitors.html
!15
(not) fixing it
!16
horizontal scaling “throw money at it”
→ does not work
→ ops + maintenance cost
!17
fine-tunethe DB engine
“bring in the experts”
→ only works short-term
!18
vertical scaling “throw more money at it”
→ bigger database instances~ +4k$/mo
→ only 2 cartridgesin that gun
!19
put it in a service de-normalise the data
use that noSQL thing
!20
webscalewebscale
webscale
good solution ? → live in 1-2 weeks
→ buys 6-12 months
!21http://xkcd.com/1205/
( intermission (
!22
frame tearing (not tiering)
!23
frame tearing caused by “simple buffering”
!24
render scene in buffert
draw to screen from buffer t
frame(buffer) tiering aka “double buffering”
!25
render scene in buffert
draw to screen from buffer t
data tiering separate read and write tables
!26
availabilities_back
http://en.wikipedia.org/wiki/File:Comparison_double_triple_buffering.svg
availabilities_frontread
availabilitiesread/write
� swap
copy
how it works
!27
tables availabilitiesavailabilities_0availabilities_1
data_tiering_switchesdata_tiering_sync_logs
!28
using tiered tables DataTiering::Switch.new.active_scope_for(Availability)
equivalent one of
Availability.scoped(:from => 'properties_0')Availability.scoped(:from => 'properties_1')
!29
availabilities_back
availabilities_frontread
availabilitiesread/write
� swap
copy
syncing DataTiering::Sync.sync_and_switch!
regularly* scheduled task(every 5 min for us, takes ~ 60s)
*depend on acceptable staleness
!30
availabilities_back
availabilities_frontread
availabilitiesread/write
� swap
copy
syncing: schema lazily (no migrations) :- create missing /.*_[01]/ tables
- compare schemas with SHOW CREATE TABLE
!31
availabilities_back
availabilities_frontread
availabilitiesread/write
� swap
copy
syncing: bulk - run TRUNCATE TABLE then INSERT INTO … SELECT FROM
- too slow at runtime (only for setup / after migrations)
!32
availabilities_back
availabilities_frontread
availabilitiesread/write
� swap
copy
syncing: deltas - deletions : SELECT id … LEFT JOIN … DELETE … WHERE id IN …
- insertions & updates : REPLACE INTO … row_touched_at > X
- remember last sync in data_tiering_sync_logs
- row_touched_at “magic” TIMESTAMP column
!33
availabilities_back
availabilities_frontread
availabilitiesread/write
� swap
copy
swapping - renaming tables not transactional
- atomically change a pointer instead (and cache it)
→ data_tiering_switches
!34
availabilities_back
availabilities_frontread
availabilitiesread/write
� swap
copy
gem by
@kratob@hubb@mconnell@danielgrieve
!35
outcome
!36
outcome averageDB time unchanged
nominal at peak traffic
deadlocks timeoutsfaster reads
!37
epilogue
!38
so long, data tiering ! you served us well
!39
keep calm ♡
refactor
!40http://dev.housetrip.com/
@mezis_fr