Scaling mysql with python (and Docker)

Scaling MySQL with pythonEuroPython 2015, Bilbao

Roberto Polli - roberto.polli@par-tec.it

Par-Tec Spa - Rome Operation UnitP.zza S. Benedetto da Norcia, 33

00040, Pomezia RM - www.par-tec.it

20 July 2015

Agenda

IntroMySQLPython UtilitiesConnectors

ReplicationFailoverFabricProvisioningFabric in the Cloud

Who? What? Why?

Roberto Polli - Solutions Architect @ par-tec.it. Loves writing in C, Java andPython. Red Hat Certified Engineer and Virtualization Administrator.Par-Tec – Proud sponsor of this talk ;) Contributes to various FLOSS.Provides:

• IT Infrastructure & Services expertise;• Business Intelligence solutions;• Vertical Applications for the financial market.

Manage, replicate, scale MySQL is easy with python (and Docker).

Intro Roberto Polli - roberto.polli@par-tec.it

• Single-process, multi-thread• 2-tier architecture

• Connection + SQL• Storage Backend

• Greatly improved in 5.6• More to come in 5.7: Dynamic Buffers & Multi-source replication

MySQL Roberto Polli - roberto.polli@par-tec.it

Many Backends aka Storage Engines:• InnoDB1: ACID, MVCC2, Redo|Undo logs;• Memory: fast & volatile;• NDB: MySQL Cluster, network distributed, auto-sharded;

Replication based on changelog files (binary logs).

1default2Multi Versioning Concurrency Control

Manage & Use a database

Monitor:• Database size: Tables, Indexes,

Binary Logs• Replication inconsistencies• Failover

Connect:• Native Drivers

• for Python 2.6, 2.7 and 3.3• SSL & compression

• OrchestrateDownload scripts from some blog, add random queries from docs.mysql.com,Shake.

MySQL Utilities

A python package with utilities, drivers, fabric orchestrator

$ wget http://bit.ly/1CxNuZe$ tar xf mysql-utilities-1.6.1.tar.gz$ cd mysql-utilities-1.6.1$ python setup.py install

Python Utilities Roberto Polli - roberto.polli@par-tec.it

MySQL Utilities:

Start with mysqluc for a single entrypoint• entrypoint for all utilities• contextual help• TAB completion

Or access each utility separately.Specify server URIs as user:pass@host[:port].

MySQL Utilities

mysqluc> help utilities

Utility Description----------------- --------------------------------------------------------mysqlauditadmin audit log maintenance utilitymysqlauditgrep audit log search utility...mysqldbcompare compare databases for consistency...mysqldiskusage show disk usage for databasesmysqlfailover automatic replication health monitoring and failover

Disk Usage

A single command to show all disk usage infos (excluded system logs)$ mysqldiskusage --all --server=$SERVER...| db_name | total || akonadi | 969150919 |...Total database disk usage = 971233529 bytes or 926,24 MB...| ib_logfile0 | 67108864 || ib_logfile1 | 67108864 || ibdata1 | 44040192 |Total size of InnoDB files = 178257920 bytes or 170,00 MB

Connectors

from django.db.backends import BaseDatabaseValidation

if django.VERSION < (1, 7):from django.db import models

else:from django.core import checksfrom django.db import connection

class DatabaseValidation(BaseDatabaseValidation):if django.VERSION < (1, 7):

Connectors Roberto Polli - roberto.polli@par-tec.it

Log-based Replication: Master

Replication is asynchronous or semi − synchronous.A Master with a Unique Server ID

• produces a changelog named binary log;• assign each transaction a Global Transaction ID;• grants access to a replica user;

Optionally:• stores replication infos in ACID tables;• may track slave-updates;• waits for one slave ack on semi − synchronous replication;

Replication Roberto Polli - roberto.polli@par-tec.it

Log-based Replication: Slave

Slave• connects to the master as the replica user;• retrieves the binlog;• applies the changes;• ∀ slave ∃! master;

Optionally• forbids local changes from non-root users;

If binlogs have been purged, you need to import the master database first!Replication Roberto Polli - roberto.polli@par-tec.it

Benefits of replication

• Availability.• Scaling reads with R/W split.• Scaling reads differencing indexes.• Backup and data warehouse

strategies.

# mysqlrplshow output## Replication Topology Graphs-1.docker:3306 (MASTER)

|+--- s-3.docker:3306 - (SLAVE)|+--- s-4.docker:3306 - (SLAVE)

mysqlreplicate

Configuring replication with one command:mysqlreplicate -b --pedantic \

--master=$MASTER --slave=$SLAVE --rpl-user=repl:rpass

# master uuid = 2cda700...# slave uuid = 7520cf0...# Checking for binary logging on master...# Granting replication access to# replication user...# Connecting slave to master...# CHANGE MASTER TO MASTER_HOST = ’172.17.0.5’,...

• preliminary checks;• create replica user on the master;• point the slave to the master;• start loading the first available

transaction in binlogs;

Initialize new slavesBeware!

1. Avoid replicating user provisioning and other host-related setup!@SESSION.SQL_LOG_BIN=0;

2. Binlogs are usually retained for a fixed period - eg 15days.To provision new slave after that period you must:

• start from a backup or a snapshot;• apply the binlogs.

mysqldbexport >> data.sql --server=root:pass@master --rpl-user=repl:rpass \--export=both --rpl=master --all

mysqldbimport --server=root:pass@slave data.sql

Failover BasicsFrom replication to high availability, automagically discovering slaves.

• promote the most updated slave• reconfigure the others• disable the master• eventually fix ip-address

# Read carefully the docs of# mysqlfailover and of all# the supported parameters!mysqlfailover --master=$MASTER \--discover-slaves-login=root:password \--candidates=$SLAVE1,$SLAVE2 \--exec-before=/pre-fail.sh \--exec-after=/post-fail.sh

Failover Roberto Polli - roberto.polli@par-tec.it

Fabric Orchestrator

You can try the content of those slides downloading code and baked dockerimages from here:

git clone https://github.com/ioggstream/mysql-communitycd mysql-community/fabricdocker-compose scale fabric=1 slave=4

Let me know after the talk!

Fabric Roberto Polli - roberto.polli@par-tec.it

Fabric - HLA

A python framework for managing, replicating and scaling mysql servers.• aggregate servers in high availability groups• configure single-master replication topologies• monitor and heal failures• director for rw/split• fabric connectors (caching db topology)

Fabric - HLA

mysqlfabric

mysqlfabric commands:• manage - setup and manage fabric service• group - administer high availability groups• server - manage, clone, provision and decommission servers• provider - manage server providers (eg. openstack, ...)

Further functionalities:• statistics, dump, role, user,• threat, snapshot, event

ManageSetup and administration is done viamysqlfabric manage [setup|start|ping|stop|teardown]

Configure in /etc/fabric.cfg

• listening ports for xmlrpc service• admin user (eg. fabric to manage servers)• applicative user credentials• db to store fabric data

Setup and start the servicemysqlfabric manage setupmysqlfabric manage start --daemon

Replication Groups

Groups are managed viamysqlfabric group [create|add|promote|activate] $GROUP

Crete an High Availability group and add a bunch of servers.

mysqlfabric group create $GROUPmysqlfabric group add $GROUP my-host-1:3306mysqlfabric group add $GROUP my-host-2:3306mysqlfabric group add $GROUP my-host-3:3306

Replication Groups

Promote one server as a master, eventually picking one.mysqlfabric group promote $GROUP [--slave_id=UUID]

Show the server groupmysqlfabric group lookup_servers haFabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249eTime-To-Live: 1server_uuid address status mode weight------------ ----------- --------- --------- ------2cda..110015 172.17.0.21 PRIMARY READ_WRITE 1.02cda..110016 172.17.0.22 SECONDARY READ_ONLY 1.02cda..110017 172.17.0.23 SECONDARY READ_ONLY 1.0

Replication GroupsFabric support spare servers.mysqlfabric server set_status $UUID spare

Monitoring failover and deactivating the master.mysqlfabric group activate $GROUPmysqladmin -h $MASTER shutdown

Checking group health we’ll see that...mysqlfabric group health ha

uuid is_alive status ..error status..io_error sql_error----------- -------- --------- --- --- --- --- -------- ---------2cd..110015 0 FAULTY 0 0 0 0 False False2cd..110016 1 SECONDARY 0 0 0 0 False False2cd..110017 1 PRIMARY 0 0 0 0 False False

Connecting programmatically

from mysql.connector import connect, fabric# Ask for a connection to the fabric server.c = connect(fabric={host: .., port: .., user: ..},

autocommit=True,database=’sample’, **sql_user)

# Fabric will point you to a suitable host# - in this case the master - of the given# groupc.set_property(mode=fabric.MODE_READWRITE, group="my-cluster-1")

Provisioning a new slave

Fabric uses mysqldump + mysql internally to clone a new slave.# Always reset TARGET configuration# before reinitializingmysql -e ’SET @SESSION.SQL_LOG_BIN=0;STOP SLAVE;RESET MASTER;’

Cloning doesn’t attach the server to a group, nor starts the replica.mysqlfabric server clone $GROUP $TARGET

Provisioning Roberto Polli - roberto.polli@par-tec.it

Reingesting a failed master

Reingesting a failed master is not trivial.• non-replicated transactions;• corrupt data;

Always reset it!mysqlfabric server set_status f484...110039 sparemysqlfabric server set_status f484...110039 secondary

Provisioning Roberto Polli - roberto.polli@par-tec.it

Fabric in the CloudFabric can provision new machines via eg. Openstack API.We implemented a DockerProvider: deploy containers, not machines.

# mysql.fabric.providers.dockerproviderclass MachineManager(AbstractMachineManager):

"""Manage Docker Containers."""def create(self, parameters, wait_spawning):

...def search(self, generic_filters, meta_filters):

...def destroy(self, machine_uuid):

Fabric in the Cloud Roberto Polli - roberto.polli@par-tec.it

Docker Provisioning# Register a provider (requires docker in http)mysqlfabric provider register mydocker

user passwordhttp://172.17.42.1:2375--provider_type=DOCKER

# Create or remove Containersmysqlfabric server create mydocker

--image=name=ioggstream/mysql-community--flavor=name=v1--meta=command="mysqld --log-bin=foo"

# List serversmysqlfabric server list docker

Next steps

Openstack interface supports machine snapshots via NovaClient.Docker volumes doesn’t support snapshots, but something is moving (in theexperimental tree).VM Snapshot alternatives require root access to docker host and a FLUSHTABLES on the source container:

• rsync on volumes• implement snapshot via docker volume plugins

Wrap Up

• Replication is easier with Fabric and MySQL 5.6• You can clone servers• Failover is just one command ahead• Don’t re-ingest failed masters (luckily;)• Try Fabric with Docker!• Play with docker volumes

That’s all folks!

Thank you for the attention!Roberto Polli - roberto.polli@par-tec.it

Scaling mysql with python (and Docker)

Internet

High Performance PHP & MySQL Scaling Techniques

Scaling PHP/MySQL Presentation from Flickr

Scaling Jenkins with Docker and Kubernetes

Running and Scaling Docker Containers with Kontena

Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?

MySql Scaling

Scaling Your First 1000 Containers with Docker

Scaling With Sun Systems For MySQL Jan09

Scaling VividCortex's Big Data Systems on MySQL

MySQL Scaling for SCaLE 12x conference

Scaling docker with kubernetes

Webinar slides: MySQL on Docker: Understanding the Basics

Scaling MySQL -- NY Code Camp

Scaling MySQL: Benefits of Automatic Data Distribution

Scaling your web app with MySQL replication

Scaling a Web Application (mostly PHP/MySQL and …assets.en.oreilly.com/1/event/27/Scaling Your Web MySQL Application... · Scaling a Web Application (mostly PHP/MySQL and almost

Scaling MySQL Strategies for Developers

Scaling MongoDB with Docker and cgroups

Scaling your jenkins master with docker

Scaling MySQL in Amazon Web Services