MySQL Fabric Tutorial, October 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL FabricTutorial

Lars Thalmann, Development DirectorNarayanan Venkateswaran, Principal Software EngineerMats Kindahl, Sr. Principal Software Engineer

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2014-10-02

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |2014-10-02

Tutorial Outline

What is MySQL Fabric?1

2

3

4

5

Downloading and Installing

High Availability

Sharding

Closing Remarks


What is MySQL Fabric?


What is MySQL Fabric?

An extensible and easy-to-use framework for managing a farm of MySQL servers supporting high-availability and sharding


What does all that mean?

● Management System– Manages a MySQL Farm– Distributed Framework

● Framework– Procedure execution– State store– Transaction Routing

● Extensible– Extensions are first-class– High-availability groups– “Semi-automatic” sharding

● Written in Python● Latest Release 1.5.2 GA● Open Source

– You can participate– Suggest features– Report bugs– Contribute patches

● MySQL 5.6 is focus


Birds-eye View

High Availability Groups

Application

SQL

SQL

Connector

Connector

Connector

Operator

MySQLFabricNode

DatabaseServers


Downloading & Installing


Download & Install● Download MySQL Connector Python

– http://dev.mysql.com/downloads/connector/python/

– This connector is needed for MySQL Fabric

● Download MySQL Utilities– http://dev.mysql.com/downloads/utilities/

– MySQL Fabric is part of the MySQL Utilities release

● Install MySQL Connector Python– For Ubuntu, simply click on the downloaded deb package

● Install MySQL Utilities– For Ubuntu, simply click on the downloaded deb package

– Note that MySQL Connector Python must be installed before this step


Download & Install● Install MySQL Server 5.6

– For Ubuntu 14.04, just two commands:● sudo apt-get install mysql-client-core-5.6● sudo apt-get install mysql-server-5.6

– MySQL 5.6 server is needed for the MySQL Fabric backing store

– For the rest of the tutorial, we will assume that you have this server running


Starting MySQL Fabric


Starting Fabric

$ mysqlfabricUsage: mysqlfabric [param, config] <grp> <cmd> [arg, ...].

MySQL Fabric 1.5.2 MySQL server farm management framework

Options: version show program's version number and exit h, help show this help message and exit param=CONFIG_PARAMS Override a configuration parameter. config=FILE Read configuration from FILE.

Basic commands: help <grp> <cmd> Show help for command help commands List all commands help groups List all groups


Getting Help


Getting Help$ mysqlfabric help groupsAvailable groups: statistics, group, dump, manage, server, role, user, threat, sharding, event

$ mysqlfabric help groupCommands available in group 'group' are: group activate group_id [synchronous] group description group_id [description=NONE] [synchronous] group deactivate group_id [synchronous] group create group_id [description=NONE] [synchronous] group remove group_id server_id [synchronous] group add group_id address [timeout=5] [update_only] [synchronous] group health group_id group lookup_servers group_id [server_id=NONE] [status=NONE] [mode=NONE] group destroy group_id [force] [synchronous] group demote group_id [update_only] [synchronous] group promote group_id [slave_id=NONE] [update_only] [synchronous] group lookup_groups [group_id=NONE]


Getting Help

$ mysqlfabric help group demotegroup demote group_id [update_only] [synchronous]

Demote the current master if there is one.

If users just want to update the state store and skip provisioningsteps such as configuring replication, the update_only parametermust be set to true. Otherwise any write access to the master isblocked, slaves are synchronized with the master, stopped andtheir replication configuration reset. Note that no slave ispromoted as master.


Configuring


MySQL Fabric Users● Backing Store User

– The user identity that Fabric will use to modify the backing store

– Can be configured in /etc/mysql/fabric.cfg [storage]

● Server User– The user identity used to connect to managed MySQL Servers

– Can be configured in /etc/mysql/fabric.cfg [servers]

● Admin user– The user identity used to control MySQL Fabric

– Can be configured in /etc/mysql/fabric.cfg [protocol.xmlrpc]

● Invoked client tools user– The user identity used to connect to managed MySQL Servers via Fabric-invoked command-line tools

– Can be configured in /etc/mysql/fabric.cfg [protocol.xmlrpc]


Configuring the backing store● On the MySQL Backing Store Server

– Create a Backing Store user on the backing store MySQL Server

– Grant access for the user to the “fabric” database

lars@silver3:~$ mysql u root

mysql> CREATE USER 'fabric'@'localhost' IDENTIFIED BY 'SECRET';

mysql> GRANT ALL ON fabric.* TO 'fabric'@'localhost';


Originallars@silver3:~$ sudo emacs /etc/mysql/fabric.cfg

[DEFAULT]prefix = sysconfdir = /etclogdir = /var/log

[storage]address = localhost:3306user = fabricpassword = database = fabricauth_plugin = mysql_native_passwordconnection_timeout = 6connection_attempts = 6connection_delay = 1

[servers]user = fabricpassword =

[protocol.xmlrpc]address = localhost:32274threads = 5user = adminpassword = disable_authentication = norealm = MySQL Fabricssl_ca = ssl_cert = ssl_key =

[executor]executors = 5

[logging]level = INFOurl = file:///var/log/fabric.log

[sharding]mysqldump_program = /usr/bin/mysqldumpmysqlclient_program = /usr/bin/mysql

[statistics]prune_time = 3600

[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0detections = 3detection_interval = 6detection_timeout = 1prune_time = 3600

[connector]ttl = 1

[client]password =


Editedlars@silver3:~$ sudo emacs /etc/mysql/fabric.cfg

[DEFAULT]prefix = sysconfdir = /etclogdir = /var/log

[storage]address = localhost:3306user = fabricpassword = SECRETdatabase = fabricauth_plugin = mysql_native_passwordconnection_timeout = 6connection_attempts = 6connection_delay = 1

[servers]user = fabricpassword =

[protocol.xmlrpc]address = localhost:32274threads = 5user = adminpassword = disable_authentication = yesrealm = MySQL Fabricssl_ca = ssl_cert = ssl_key =

[executor]executors = 5

[logging]level = INFOurl = file:///var/log/fabric.log

[sharding]mysqldump_program = /usr/bin/mysqldumpmysqlclient_program = /usr/bin/mysql

[statistics]prune_time = 3600

[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0detections = 3detection_interval = 6detection_timeout = 1prune_time = 3600

[connector]ttl = 1

[client]password =


Creating the backing store


Creating the backing store● Create the database storing info for MySQL Fabric

– Since we have disabled authentication for this tutorial, admin/xmlrpc password will actually not be used for anything

$ mysqlfabric manage setup [INFO] 1411584079.046989 MainThread Initializing persister:user (fabric), server (localhost:3306), database (fabric).Finishing initial setup=======================Password for admin user is not yet set.Password for admin/xmlrpc: Repeat Password: Password set.


A deeper look into the architecture


Birds-eye View


Application

SQL

SQL

Connector

Connector

Connector

Operator

MySQLFabricNode

DatabaseServers


High-Level Components

● Fabric-aware Connectors– Python, PHP, Java, .NET, C– Enhanced Connector API

● MySQL Fabric Node– Manage information about farm– Provide status information– Execute procedures

● MySQL Servers– Organized in high-availability

groups– Handling application data

High AvailabilityGroup

ApplicationConnector

Connector

Connector

MySQLFabricNode


MySQL Fabric Node Architecture

MySQL

MySQL FabricFramework

Executor State Store(Persister)

Sh

?HA

MySQL-RPCAMQP XML-RPC

?Connector

Connector

Connector

Protocols

Extensions

BackingStore


Starting the Fabric server● MySQL Fabric will be run in the foreground

– You can also run as deamon in the backgroud, using --deamonize

$ mysqlfabric manage start[INFO] 1411822940.181810 MainThread Initializing persister: user (root), server (localhost:13000), database (fabric).[INFO] 1411822940.188654 MainThread Loading Services.[WARNING] 1411822940.198591 MainThread Authentication disabled[INFO] 1411822940.198668 MainThread MySQLRPC protocol server started, listening on localhost:32275[WARNING] 1411822940.198773 MainThread Authentication disabled[INFO] 1411822940.212181 MainThread Executor started.[INFO] 1411822940.215672 MainThread Starting failure detector.[INFO] 1411822940.216747 XMLRPCServer XMLRPC protocol server ('127.0.0.1', 32274) started.[INFO] 1411822940.217083 XMLRPCServer Setting 1 XMLRPC session(s).[INFO] 1411822940.217331 XMLRPCSession0 Started XMLRPCSession.


High Availability


High-Availability Concepts

● Redundancy– Duplicate components

● Monitoring– Detecting failing components– Monitor load

● Procedures– Activate replacements– Distribute load


Using the Fabric client● Create a High Availability group

– The Fabric client will connect to the Fabric server that is configured in the Fabric.cfg file

$ mysqlfabric group create "group1"Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result 254f8b628ea94350a46e77fefafc0f41 1 1 1

state success when description 3 2 1411822993.11 Triggered by <mysql.fabric.events.Event object at 0x7fbdef77aa50>. 4 2 1411822993.12 Executing action (_create_group). 5 2 1411822993.12 Executed action (_create_group).


Add a MySQL server to the group● The MySQL server is now managed by MySQL Fabric

$ mysqlfabric group add group1 localhost:13001Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result bb73e411070547dd86f9b99798c97c9c 1 1 1

state success when description 3 2 1411823050.61 Triggered by <mysql.fabric.events.Event object at 0x7fbdef77ae10>. 4 2 1411823050.61 Executing action (_add_server). 5 2 1411823050.62 Executed action (_add_server).


Add a second MySQL server to the group● The MySQL servers are now managed by MySQL

Fabric

$ mysqlfabric group add group1 localhost:13002Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result feba445d47bd4681a8b88ae88d1d12a3 1 1 1

state success when description 3 2 1411823077.84 Triggered by <mysql.fabric.events.Event object at 0x7fbdef77ae10>. 4 2 1411823077.84 Executing action (_add_server). 5 2 1411823077.85 Executed action(_add_server).


Listing the MySQL servers of the group● Two servers in the high availability group

$ mysqlfabric group lookup_servers group1Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

server_uuid address status mode weight 1906aa45464611e4926fe82aea9348c9 localhost:13001 SECONDARY READ_ONLY 1.01937e215464611e4926fe82aea9348c9 localhost:13002 SECONDARY READ_ONLY 1.0


Promote a server to master ● Will pick the “best” server of the group

– Since the servers are empty, this will be a random choice

$ mysqlfabric group promote group1Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result 4295faefe6e14df194931b9b9cdc3a48 1 1 1

state success when description 3 2 1411823267.15 Triggered by <mysql.fabric.events.Event object at 0x7fbdef4615d0>. 4 2 1411823267.15 Executing action (_define_ha_operation). 5 2 1411823267.16 Executed action (_define_ha_operation).


Promote a specific server to become the master ● The server specified with --slave-id will be the

master

$ mysqlfabric group promote group1 slave_id=1906aa45464611e4926fe82aea9348c9Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result 4295faefe6e14df194931b9b9cdc3a48 1 1 1

state success when description 3 2 1411823267.15 Triggered by <mysql.fabric.events.Event object at 0x7fbdef4615d0>. 4 2 1411823267.15 Executing action (_define_ha_operation). 5 2 1411823267.16 Executed action (_define_ha_operation).


Using the High Availability Groupof the farm


Birds-eye View


Application

XML-RPCMySQL-RPC

SQL

SQL

Connector

Connector

Connector

Operator

MySQLFabricNode

DatabaseServers


MySQL Connector Python● There is also Java, PHP, Connector/C (labs), .Net

– And support for frameworks Doctrine, Hibernate

import mysql.connectorfrom mysql.connector import fabric


Create a table in the group● Connect to Fabric, not to MySQL servers directly

conn = mysql.connector.connect( fabric={"host" : "localhost", "port" : 32274, "username": "admin", "password" : "", 'report_errors': True }, user="root", password="", autocommit=True)conn.set_property(group="group1", mode=fabric.MODE_READWRITE)cur = conn.cursor()cur.execute("CREATE DATABASE IF NOT EXISTS employees")cur.execute("USE employees")cur.execute("DROP TABLE IF EXISTS employees")cur.execute( "CREATE TABLE employees (" " emp_no INT, first_name CHAR(40), last_name CHAR(40)" ")")


Insert data into the group● Specifying it is for group-1, read-write transaction

def add_employee(conn, emp_no, first_name, last_name): conn.set_property(group="group1", mode=fabric.MODE_READWRITE) cur = conn.cursor() cur.execute("USE employees") cur.execute( "INSERT INTO employees VALUES (%s, %s, %s)", (emp_no, first_name, last_name) ) # We need to keep track of what we have executed in order to, # at least, read our own updates from a slave. cur.execute("SELECT @@global.gtid_executed") for row in cur: print "Transactions executed on the master", row[0] return row[0]


Select data from the group● Specifying that it is group-1, and that this is a

read-only transactiondef find_employee(conn, emp_no, gtid_executed): conn.set_property(group="group1", mode=fabric.MODE_READONLY) cur = conn.cursor() # Guarantee that a slave has applied our own updates before # reading anything. cur.execute( "SELECT WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS('%s', 0)" % (gtid_executed, )) for row in cur: print "Had to synchronize", row, "transactions." cur.execute("USE employees") cur.execute( "SELECT first_name, last_name FROM employees " "WHERE emp_no = %s", (emp_no, )) for row in cur: print "Retrieved", row


Failure Detection


Failure DetectorFabric Monitoring

Jan 2010

● Monitors servers within each group

● On master failure,– mark server as faulty, and– trigger fail-over

● On slave failure,– mark as faulty



Connector

Connector

MySQLFabricNode


Failure Detector (fabric.cfg)Fabric Monitoring

Jan 2010 Sept 2014

2x Engineering Staf3x QA Staf2x Support Staf

[failure_tracking]detections = 3detection_interval = 6detection_timeout = 1

detectionsThe number of times a server fails alive checks, before being marked as FAULTY.

detection_intervalTime between alive checks.

detection_timeoutThe time after which a server alive ping timeouts.


Distributed Failure Reporting

Jan 2010 Sept 2014

● Connector reports error to Fabric

● Error used to update backing store

● Trigger fail-over

– Error count reaches threshold



Connector

Connector

MySQLFabricNode


Distributed Failure Reporting (fabric.cfg)

Jan 2010 Sept 2014


[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0prune_time = 3600

notificationsThe number of error notifications required to mark a server FAULTY.

notification_clientsThe number of clients that need to report an error to mark a server FAULTY.


Distributed Failure Reporting (fabric.cfg)

Jan 2010 Sept 2014


notification_intervalThe interval of interest in which we want to verify error notifications.

failover_intervalThe interval at which we can safely do a failover in a group without causing system instability.

prune_timeInterval at which we need to prune the error logs.


Distributed Failure Reporting (Connector)Enabling Error Reporting

Jan 2010 Sept 2014


● The option to report errors is part of the Fabric configuration

● “report_errors” can be turned on as follows

● Causes a connector-server connection error to be reported to Fabric

fabric_config = { 'host': .., 'report_errors': True, }

cnx = mysql.connector.connect(fabric=fabric_config)


Distributed Failure Reporting (Connector)Error Reporting

Jan 2010 Sept 2014

● The following errors are reported by default

● The above can be extended by setting extra_failure_report

REPORT_ERRORS = ( errorcode.CR_SERVER_LOST, errorcode.CR_SERVER_GONE_ERROR, errorcode.CR_CONN_HOST_ERROR, errorcode.CR_CONNECTION_ERROR, errorcode.CR_IPSOCK_ERROR,)

from mysql.connector.fabric import extra_failure_report extra_failure_report([error_code_0, error_code_1, ...])


Distributed Failure Reporting (Connectors)Cache Invalidation

Jan 2010 Sept 2014


● Define which errors cause invalidation of the connector cache

● The following errors cause a cache invalidation by default

● The above can be extended by

RESET_CACHE_ON_ERROR = ( errorcode.CR_SERVER_LOST, errorcode.ER_OPTION_PREVENTS_STATEMENT,)

from mysql.connector.fabric import RESET_CACHE_ON_ERROR RESET_CACHE_ON_ERROR.append(error_code_0)


Sharding


Birds-eye View


Application

XML-RPCMySQL-RPC

SQL

SQL

Connector

Connector

Connector

Operator

MySQLFabricNode

DatabaseServers


Creating a sharding definition● In this example, we use Hash sharding

– Fabric also supports Range sharding

– This will define Group-1 as the “global group” of the shard definition

– Group-1 is not a shard

– All shard groups that are added, will be slave groups to the global group

$ mysqlfabric sharding create_definition HASH group1Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result 90bbd7a789964a5aa97132d10e3591c4 1 1 1


List the sharding definitions● The created sharding definition has identifier 1

$ mysqlfabric dump shard_mapsFabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

mapping_id type_name global_group_id 1 HASH group1


Register a table to the sharding definition● Table “db1.t1” to be sharded according to

definition “1”– The table will be created later

– The column “emp_no” will be used as the sharding key

$ mysqlfabric sharding add_table 1 db1.t1 emp_noFabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result 57aa21ef13154da78295aa2a0f4339e0 1 1 1


Add shards (groups) to the sharding definition● Add two groups as shards

– Global Group: group-1

– Shards: group-2 and group-3

– --state=ENABLED will start replication

$ mysqlfabric sharding add_shard 1 group2,group3 state=ENABLEDFabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result f1e2d8841d674a3ab5d25b29b2a6d451 1 1 1


What do we have now?● The groups

– One global group

– Two shards


Using the Global Groupof the farm


Create a table (on all shards)● Use the global group (SCOPE_GLOBAL)

– Afterwards set back to the default, i.e. SCOPE_LOCAL

– The properties are for the connection, not only for a single transaction

conn.set_property(tables=["db1.t1"], scope=fabric.SCOPE_GLOBAL, mode=fabric.MODE_READWRITE)cur = conn.cursor()cur.execute("CREATE DATABASE IF NOT EXISTS db1")cur.execute("USE db1")cur.execute("DROP TABLE IF EXISTS t1")cur.execute( "CREATE TABLE t1 (" " emp_no INT, " " first_name CHAR(40), " " last_name CHAR(40)" ")")conn.set_property(scope=fabric.SCOPE_LOCAL)


Table propagated to all groups● The table

– First created on the global group

– Then replicated to the two shards


Using the Sharding Groupsof the farm


Inserting into the shards● Using the table name and the sharding key

def add_employee(conn, emp_no, first_name, last_name, gtid_executed): conn.set_property(tables=["db1.t1"], key=emp_no, mode=fabric.MODE_READWRITE, scope=fabric.SCOPE_LOCAL) cur = conn.cursor() synchronize(cur, gtid_executed) cur.execute("USE db1") cur.execute( "INSERT INTO t1 VALUES (%s, %s, %s)", (emp_no, first_name, last_name) ) conn.commit()


Select data from the shards● Using the table name and the sharding key

– Will only execute this on the correct shard

def find_employee(conn, emp_no, gtid_executed): conn.set_property(tables=["db1.t1"], key=emp_no, mode=fabric.MODE_READONLY) cur = conn.cursor() synchronize(cur, gtid_executed) cur.execute("USE db1") for row in cur: print "Had to synchronize", row, "transactions." cur.execute( "SELECT first_name, last_name FROM t1 " "WHERE empno = %s", (emp_no, ) ) for row in cur: print row


Shard split


Create a new group with MySQL servers● Creating the 4th group● Adding two servers● Promote one of them to master

$ mysqlfabric group create "group4"

$ mysqlfabric group add group4 localhost:13007


$ mysqlfabric group promote group4


Sharding information● Two shards 1, 2● Hosted on two groups “group-2” and “group-3”

$ mysqlfabric dump sharding_informationFabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

schema_name table_name column_name lower_bound shard_id type_name group_id global_group db1 t1 empno E6416... 1 HASH group2 group1 db1 t1 empno DC3AD... 2 HASH group3 group1


Check how much data we have● Checking on a MySQL server (shard 1 on “group-2”)

directly

mysql> use db1;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with A

Database changedmysql> select count(empno) from t1;++| count(empno) |++| 1009 |++1 row in set (0.00 sec)


Split the shard into two shards

$ mysqlfabric sharding split_shard 1 group4Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

uuid finished success result 80788bebbe2f4b2684f520bb2964ff70 1 1 1

state success when description 4 2 1411863534.16 Executing action (_check_shard_information). 4 2 1411863534.17 Executing action (_backup_source_shard). 4 2 1411863534.25 Executing action (_restore_shard_backup). 4 2 1411863534.37 Executing action (_setup_replication). 4 2 1411863534.39 Executing action (_setup_sync). 4 2 1411863534.4 Executing action (_setup_resharding_switch). 4 2 1411863535.45 Executing action (_prune_shard_tables_after_split).

● Part of shard 1 (“group-2”) moved to “group-4”


How much data after the split?● Checking on a MySQL server (shard 1 on “group-2”)

directly

mysql> use db1;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with A

Database changedmysql> select count(empno) from t1;++| count(empno) |++| 532 |++1 row in set (0.00 sec)


Looking up servers responsible for sharding key● For any sharding key, you can check which servers

are responsible

$ mysqlfabric sharding lookup_servers db1.t1 42000Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1

server_uuid address status mode weight 83f2dd4f46a111e494c4e82aea9348c9 localhost:13007 SECONDARY READ_ONLY 1.08426dd9246a111e494c4e82aea9348c9 localhost:13008 PRIMARY READ_WRITE 1.0


Shard move


Create a new group with MySQL servers● Creating the 5th group● Adding two servers● Promote one of them to master

$ mysqlfabric group create "group5"



$ mysqlfabric group promote group5


Looking up servers for a sharding key● For any sharding key, you can check which servers

are responsible (Ports 13007, 13008 = “group-4”)


server_uuid address status mode weight 83f2dd4f46a111e494c4e82aea9348c9 localhost:13007 SECONDARY READ_ONLY 1.08426dd9246a111e494c4e82aea9348c9 localhost:13008 PRIMARY READ_WRITE 1.0


Perform the shard move● Move shard “4” to “group-5”

$ mysqlfabric sharding move_shard 4 group5Fabric UUID: 5ca1ab1ea007feedf00dcab3fe13249eTimeToLive: 1…


Verify the move● Afterwards, check which servers are responsible

(Ports 13009, 13010 = “group-5”)


server_uuid address status mode weight 8489b2ff46a111e494c4e82aea9348c9 localhost:13009 SECONDARY READ_ONLY 1.084bc4d5946a111e494c4e82aea9348c9 localhost:13010 PRIMARY READ_WRITE 1.0


Fabric Executor


MySQL Fabric executor● Event driven

– Events will trigger execution of procedures

– Procedures can trigger events themselves

– Each step of a procedure is called a job

● Procedures– Written in Python

– Interacts with servers

– Write state changes into backing store

– Lock manager for conflict resolution● Conservative two-phase locking strategy● Avoid deadlocks

Queue

BackingStore

Events


Automatically adding a server to a group

● Register procedure for event– @on_event decorator

– Accept event to register for

● Fetch the group the server belonged to

● Fetch a new server from the provider

● Add the server to the group

@on_event(SERVER_LOST)def _add_server(group_id, server_uuid): group = Group.fetch(group_id) machines = PROVIDER.create_machines( parameters ) server = MySQLServer( server_uuid, address ) MySQLServer.add(server) group.add(server) _configure_as_slave(server)


MySQL Fabric procedure execution● Before starting a job:

– Acquire the necessary locks

– Checkpoint execution state in backing store

– Start a transaction on the backing store

● When executing job:– Make updates to backing store inside transaction

– Interact with servers

● After executing a job:– Mark job completed in internal log

– Commit transaction on backing store

Queue

BackingStore

Events

What about crashes?


MySQL Fabric executor recovery● Two types of jobs:

– Idempotent: Restart the job

– Not idempotent: Execute compensation

● Recovery procedure– Start the executor

– Collect unfinished checkpoints

– Execute compensation activity for each job associated with the checkpoint

… if there is one

– Re-schedule each job

Queue

BackingStore

Events


Defining an compensation procedure

● Use the @undo decorator● Compensating action

– Search for the machine by UUID

– Give it back to provider

● Compensating actions shall succeed

@_add_server.undodef _undo_add_srv(group_id, server_uuid):

meta_filters = { 'mysqlfabricmachinegroupuuid': server_uuid }

machines = MANAGER.search_machines( {}, meta_filters )

for machine in machines: MANAGER.destroy_machine( machine.uuid )


Going Forward


What do we have now?● High-availability

– Group Concept– Slave promotion

● Sharding– Range and hash sharding– Shard move and shard split

● Connector APIs– Transaction properties– “Virtual” connections

● Cloud integration– “Server providers”– OpenStack Nova

● Enhanced connectors– Connector/Python– Connector/PHP– Connector/J– Connector/NET

● Interfaces– Command-line– XML-RPC– MySQL-RPC

● Distributed failure detector– Connectors report failures– Custom failure detectors

● Credentials– RFC 2617– SSL support


Reading for the Interested

● MySQL Forum: Fabric, Sharding, HA, Utilitieshttp://forums.mysql.com/list.php?144

● MySQL Fabric Documentationhttp://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html

● Migrating From an Unsharded to a Sharded Setuphttp://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-migration.html

● Configuring and running MySQL Fabrichttp://alfranio-distributed.blogspot.com/2014/03/mysqlfabric-installation.html

http://forums.mysql.com/list.php?144

http://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html

http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-migration.html

http://alfranio-distributed.blogspot.com/2014/03/mysqlfabric-installation.html


Want to contribute?

● Check it– … and send us use-case and feature suggestions

● Test it– … and send comments to the forum

● Break it– … and send in bugs to http://bugs.mysql.com


Lars ThalmannTwitter: @larsthalmann

http://larsthalmann.blogspot.com

Mats KindahlTwitter: @mkindahlhttp://mysqlmusings.blogspot.com

Alfranio CorreiaTwitter: @alfraniohttp://alfranio-distributed.blogspot.com

Keeping in Touch

Narayanan VenkateswaranTwitter: @vn_tweetshttp://vnwrites.blogspot.com

Geert VanderkelenTwitter: @geertjanvdkhttp://geert.vanderkelen.org

http://larsthalmann.blogspot.com/

http://mysqlmusings.blogspot.com/

http://alfranio-distributed.blogspot.com/

http://vnwrites.blogspot.com/

http://geert.vanderkelen.org/


MySQL Central Sessions

● Elastic Scalability in MySQL Fabric with OpenStack Thursday, Oct 2, 1:15 PM-2:00 PMMoscone South, 252

● MySQL Fabric: High Availability at Different Levels Wednesday, Oct 1, 2:00 PM-2:45 PMMoscone South, 250


Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2014-10-02