Corpus collapsum: Partition tolerance testing of Galera with Docker and NetEm

Corpus collapsumPartition tolerance testing of Galera with

Docker and NetEm

Raghavendra Prabhu [email protected]

Percona [email protected] randomsurfer wnohang.net rdprabhu ronin13

mailto: [email protected]

mailto: [email protected]

http://twitter.com/randomsurfer

http://wnohang.net

https://linkedin.com/in/rdprabhu

http://https://github.com/ronin13

The Title

Split Brain?

Split brain

Introduction

Seed quotes..

“ ’Network is reliable’ - a fallacy of the distributedsystem. ”

“ A distributed system is one in which the failure of acomputer you didn’t even know existed can render your owncomputer unusable. ” - Leslie Lamport

“ Never attribute to malice that which is adequatelyexplained by stupidity. ” - Hanlon’s Razor

“ Never attribute to Byzantine failure which can beexplained by an ill node(s) ” - Me

Raghavendra Prabhu (Percona) Corpus collapsum 20 February, 2015 5 / 68

https://blogs.oracle.com/jag/resource/Fallacies.html

Introduction

Seed quotes..







Introduction

Seed quotes..







Introduction

Seed quotes..







20000 feet view

Introduction

Actors

▶ Database - WSREP/PXC▶ Plugin - Galera▶ Traffic control

♦ Traffic Control - tc♦ NetEm


Introduction

Actors




Introduction

Actors




Introduction

Actors

▶ Containers - Docker▶ Load

♦ Generators - Sysbench, RQG▶ Network

♦ Dnsmasq♦ nsenter


Introduction

Actors

▶ Containers - Docker▶ Load

♦ Generators - Sysbench, RQG▶ Network

♦ Dnsmasq♦ nsenter


Introduction

Actors

▶ Jenkins♦ Build flow and CI

▶ Storage♦ Why


Distributed Systems TestingA Kobayashi Maru

Cheat on CAP!

Details

Rationale

▶ The ‘P’ in CAP▶ WAN scalability▶ Real Reason - fun!▶ Tolerance to latency variance


Details

Rationale



Details

Rationale



Details

Rationale



Details

Rationale

▶ Failures in warehouses.▶ Not quorum, but consensus.▶ Real world networks and synchronous replication

- Delay- Partition- Non-graceful exits


https://queue.acm.org/detail.cfm?id=2655736

Galera

Details

Galera

▶ Data-centric approach▶ Extended Virtual Synchrony▶ Causality and Synchronous▶ Flow control and temporal

Synchrony


Details

Galera

▶ Latency- Global ordering- Certification and not apply- Communication overhead

▶ Layers- Replication- Certification- Group communication

▶ Isolation- REPEATABLE-READ- SNAPSHOT-ISOLATION


Where did it start

Details

Where did it start

▶ Bug! https://bugs.launchpad.net/galera/+bug/1274192▶ Loss of PC▶ Crash▶ HAT


lp:1274192

One can bring the wholedown

Details

Tests

▶ Chaos testing▶ Flow control with sysbench▶ Network Loss▶ Future


There is no higher menace thandistributed systems testing

Details

NetEm

▶ Initial setup- Bridge- Egress only- IFB- Present state

▶ NetEm- tc qdisc buckets- packet loss, delay, corruption, duplication, reordering- nsenter

▶ Future- Docker exec- Rocket ACI


Details

Tests: Chaos testing

▶ Nodes killed at random around sysbench▶ Less than half of nodes are chosen▶ docker inspect && SIGKILL▶ Configurable sleep && retry

♦ Snapshot/Incremental State Transfer- Composability of transactional databases

▶ docker restart && repeat


Details

Tests: Network Loss

▶ Loss nodes▶ Detach/Keep qdisc▶ Reconciliation▶ Sanity checks▶ Formation of PC || time to recover


The Flow

Details

Basic Flow

Jenkins Build images Start Dnsmasq Bootstrap

Load/SysbenchSST/OthersPre-sanitynsenter/netem


Details

Basic Flow




Details

Basic Flow




Details

Basic Flow




Details

Basic Flow




Details

Basic Flow




Details

Basic Flow




Details

Basic Flow




Details

Basic Flow

RR sysbench

Detach/Keep

Sanity check Reconciliation

Post sanity Core trace

Cleanup Collect logs


Details

Basic Flow

RR sysbench

Detach/Keep





Details

Basic Flow

RR sysbench

Detach/Keep





Details

Basic Flow

RR sysbench

Detach/Keep





Details

Basic Flow

RR sysbench

Detach/Keep





Details

Basic Flow

RR sysbench

Detach/Keep





Details

Basic Flow

RR sysbench

Detach/Keep





Details

Basic Flow

RR sysbench

Detach/Keep





Details

Parameters

▶ Sysbench▶ Segment▶ Reconciliation period▶ Loss nodes


Details

Parameters



Details

Parameters



Details

Parameters



Plumbing the pressure

Details

Parameters

▶ NetEm▶ Qdisc detach▶ fsync▶ Shutdown


Details

Parameters



Details

Parameters



Details

Parameters



Containers!

Details

Docker

▶ Why not virtualizeOccamNamespaces

▶ Simplicity♦ Network

Logical scalability♦ One application per node


Details

Docker

▶ Portability- Qualitative behavior.

▶ Reproducibility- Makes it determinstic

▶ Configurable and CI- Byproducts


Details

Docker

▶ QEMU vis-à-vis Docker▶ Scalability

♦ Performance♦ Feature

▶ Abstraction of channels


Details

Container Networking

▶ Linking didn’t help▶ Dnsmasq to rescue!

♦ Hosts file and volumes♦ SIGHUP and refresh

▶ Potential issues


Testing methods

Details

Overview

▶ Transient noise▶ Lasting ’sickness’▶ Sick nodes▶ Dead members


Details

Method I

▶ Qdisc is detached after load▶ Objective

- Time to recover of full cluster▶ Done with a larger subset


Details

Method II

▶ Qdisc is kept till the end▶ Objective

- Formation of primary component▶ Comparatively smaller set


Details

Observations

▶ Post sanity types- Why

▶ Which method is more pertinent▶ State transfer issues

- Beginning- During re-emergence


Details

Observations

▶ Direct load to affected nodes▶ Partition external to system▶ Logs

- journalctl- Streaming?


Details

Other noises

▶ Aim▶ Fsync

- libeatmydata- Variance

▶ Correlation with network▶ How with Docker

- LD_PRELOAD


System Load

Details

Load generation

▶ Sysbench- Generation- Reconnect on partition

▶ Sockets chosen- Load on affected nodes

▶ Distribution of Load- RR with socat- Native sysbench support- HAProxy?


Details

Load generation

▶ Nature of data/load- DDL

▶ RQG in future- Fuzz testing


The Fix

Strike Out!

Details

Eviction

▶ STONITH▶ Permanent eviction▶ ’N’ strikes & out!

- Timers - evs parameters- wsrep_evs_delayed and wsrep_evs_evict_list


https://github.com/codership/documentation/issues/27

Details

Eviction

▶ Aim▶ Quorum required

- Why? - Not shoot each other- Non-PC nodes also.


Details

Eviction

▶ Aim▶ Quorum required

- Why? - Not shoot each other- Non-PC nodes also.


Details

Coredumps with Docker

▶ Breakdown of abstraction▶ Lack of isolation▶ What was done

- Volumes- core_pattern & sysctl- suid and ulimit


Details

WAN Segments

▶ How they work▶ Simulates data center▶ Random allocation - latency multiplier▶ Joiner starvation▶ Donor selection


Epilogue

The code

▶ Github:- https://github.com/percona/pxc-docker-https://github.com/percona/percona-xtradb-cluster/- https://github.com/percona/galera

▶ Jenkins:- http://jenkins.percona.com/job/PXC-5.6-netem/- http://jenkins.percona.com/job/PXC-5.6-bench/- http://jenkins.percona.com/job/PXC-5.6-chaos/

▶ Contributions/testing/bugs welcome!


https://github.com/percona/pxc-docker

https://github.com/percona/percona-xtradb-cluster/

https://github.com/percona/galera

http://jenkins.percona.com/job/PXC-5.6-netem/

http://jenkins.percona.com/job/PXC-5.6-bench/

http://jenkins.percona.com/job/PXC-5.6-chaos/

Epilogue

Code: todo

▶ Docker automated builds▶ Orchestration▶ Docker

♦ Injection♦ Signal proxying


Epilogue

Code: todo

▶ => Proof of concept to a framework =>▶ Run it bare - CoreOS, Atomic▶ Overlay with etcd/fleet/libswarm


Future work

Epilogue

Future work

▶ Fault injection♦ Memory

- Poisoned memory♦ Disk

- libeatmydata- Opposite- ENOSPC


Epilogue

Fault injection

▶ CPU- NUMA?- Hotplug

▶ More network- corruption, duplication, reordering, rate-limit- Better distribution- Other shaping


Worst case improves Averagecase

Epilogue

Future work

▶ Disturb cluster more!- Membership changes* Manual eviction* Pull the cord!- Corrupt nodes

▶ Introduce inconsistencies- Consistency voting- Silent corruptions


Epilogue

Eventual consistency

▶ CAP▶ Latency factor▶ Is Galera EC? No!

- ACIDs only, No BASE▶ Bounded Staleness

- PBS▶ ACID and CAP▶ Instrumentation▶ Lambda architecture


Epilogue

Further Reading

▶ Resources▶ Byzantine fault tolerance

- Reaching agreement in presence of faults▶ The Network is Reliable▶ NetEm▶ Latency: The New Web Performance Bottleneck▶ Galera Cluster Documentation▶ Auto eviction code▶ Don’t Settle for Eventual Consistency▶ Extended Virtual Synchrony▶ Galera Flow Control


http://galeracluster.com/2014/05/all-the-galera-cluster-resources-you-need-from-the-makers-of-galera-cluster/

https://en.wikipedia.org/wiki/Byzantine_fault_tolerance

http://research.microsoft.com/en-us/um/people/lamport/pubs/reaching.pdf


http://www.linuxfoundation.org/collaborate/workgroups/networking/netem

https://www.igvita.com/2012/07/19/latency-the-new-web-performance-bottleneck/

http://galeracluster.com/documentation-webpages/

https://github.com/codership/galera/blob/3.x/gcomm/src/evs_proto.cpp


www.cs.jhu.edu/~yairamir/dcs-94.ps

http://www.percona.com/blog/2013/05/02/galera-flow-control-in-percona-xtradb-cluster-for-mysql/

Epilogue

Further Reading

▶ Worst-Case Distributed Systems Design▶ HAT, not CAP: Introducing Highly Available Transactions▶ Bridging the Gap: Opportunities in Coordination-Avoiding

Databases▶ Linearizability versus Serializability


http://www.bailis.org/blog/worst-case-distributed-systems-design/

http://www.bailis.org/blog/hat-not-cap-introducing-highly-available-transactions/

http://www.bailis.org/blog/bridging-the-gap-opportunities-in-coordination-avoiding-databases/

http://www.bailis.org/blog/bridging-the-gap-opportunities-in-coordination-avoiding-databases/

http://www.bailis.org/blog/linearizability-versus-serializability/

Epilogue

We are Hiring Too!

▶ Looking for build engineer - Packaging and Jenkins/CI are yourstrengths and you are a linux geek.bonus points if you are a linux distrouser/contributor/maintainer.

▶ Senior C/C++ developer - if linux userspace development anddatabases (and distributed systems) is your thing.

▶ Apply here: http://percona.theresumator.com/.


http://percona.theresumator.com/

Conference for Databasegeeks!

My Talk: Securing databases withsystemd for containers and

services

https://www.percona.com/live/mysql-conference-2015/sessions/securing-databases-systemd-containers-and-services



Epilogue

About/Contact - HA compliant

▶ /me: Raghavendra Prabhu, Product Lead, Percona XtraDBCluster, Percona.

▶ Slides will be at slideshare.net/slidunder.▶ About.me: raghavendra.prabhu▶ Keybase.io: rdprabhu▶ Presentation under CC BY-SA 4.0


http://www.percona.com/about-us/our-team/raghavendra-prabhu

http://www.percona.com/about-us/our-team/raghavendra-prabhu

http://www.slideshare.net/slidunder

http://about.me/raghavendra.prabhu

https://keybase.io/rdprabhu

https://creativecommons.org/licenses/by-sa/4.0/

Epilogue

Image Credits▶ http://galeracluster.com/documentation-webpages/▶ https://en.wikipedia.org/wiki/Network_theory▶ https://upload.wikimedia.org/wikipedia/commons/6/60/Corpus_callosum.png▶ http://www.thebarrow.org/Neurological_Services/Epilepsy/204354▶ https://flic.kr/p/9J6GNu▶ http://schauerte.me/data.html▶ https://secure.flickr.com/photos/brewbooks/7780990192▶ https://www.flickr.com/photos/kwerfeldein/2649294869▶ https://secure.flickr.com/photos/mindmob/51951632▶ https://secure.flickr.com/photos/arenamontanus/2227769907▶ https://www.flickr.com/photos/markop/477199204▶ https://www.flickr.com/photos/gcwest/281385801▶ https://www.flickr.com/photos/29233640@N07/13466208953▶ https://www.flickr.com/photos/bob_in_thailand/9782777742/▶ http://ok-panic.net/art/jeff/dennis.jpg▶ https://www.facebook.com/sciencedump/photos/a.296290153732762.90161.

111815475513565/985102638184840/?type=1▶ http://upload.wikimedia.org/wikipedia/commons/0/05/Sna_large.png▶ http://background-kid.com/background-images-light-blue-color.html


http://galeracluster.com/documentation-webpages/

https://en.wikipedia.org/wiki/Network_theory

https://upload.wikimedia.org/wikipedia/commons/6/60/Corpus_callosum.png

http://www.thebarrow.org/Neurological_Services/Epilepsy/204354

https://flic.kr/p/9J6GNu

http://schauerte.me/data.html

https://secure.flickr.com/photos/brewbooks/7780990192

https://www.flickr.com/photos/kwerfeldein/2649294869

https://secure.flickr.com/photos/mindmob/51951632

https://secure.flickr.com/photos/arenamontanus/2227769907

https://www.flickr.com/photos/markop/477199204

https://www.flickr.com/photos/gcwest/281385801

https://www.flickr.com/photos/29233640@N07/13466208953

https://www.flickr.com/photos/bob_in_thailand/9782777742/

http://ok-panic.net/art/jeff/dennis.jpg

https://www.facebook.com/sciencedump/photos/a.296290153732762.90161.111815475513565/985102638184840/?type=1

https://www.facebook.com/sciencedump/photos/a.296290153732762.90161.111815475513565/985102638184840/?type=1

http://upload.wikimedia.org/wikipedia/commons/0/05/Sna_large.png

http://background-kid.com/background-images-light-blue-color.html

Software

Corpus collapsum: Partition tolerance testing of Galera with Docker and NetEm