View
157
Download
0
Category
Preview:
Citation preview
OSS4B: Installing & managing MySQL like a real devops
Automation in production
Frédéric Descamps20 Sep 2013
Who am I ?
• Frédéric Descamps “lefred”• @lefred• http://about.me/lefred• Percona Consultant since 2011• Managing MySQL since 3.23 (as far as
I remember)• devops believer
2
Agenda
• devops ?• Installation automation• Test automation• Schema management automation
3
devops ?
devopsis a movement willing to help
aligning IT with company needs
(starting by aligning devs and ops)
@endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011
We can say that devops is bringing together people
and practices from development and
operations
devops ?
We can say that devops is bringing together people
and practices from development and
operations
devops ? I am a happy dev !I am a happy op !
devops ?
devops is about CAMS
devops ?
CULTUREAUTOMATION
MEASUREMENTSHARING
@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/
devops ?
What techniques can I use ?
MonitoringInfrastructure as Code
Continuous Improvement
devops & DBAs ?
What about DBAs ?
devops & DBAs !
Join the team, follow the
movement !
devdbaops
You are already between devs
and ops
devdbaops
You are already between devs
and ops
I am a happy DBA !
CAM
CULTUREAUTOMATION
MEASUREMENTSHARING
CAMS 1st principle: CULTURE
Communication !
Usually DEVS communicate with managers (all kinds of!)
CAMS 1st principle: CULTURE
Communication !
OPS communicate with support, ISP's and... their servers
CAMS 1st principle: CULTURE
Communication !
DBAs just don't communicate
CAMS 1st principle: CULTURE
The interaction between developers, sysadmins, dbas and all the actors
should begin from the start of any projects !
and do itin team !
CAMS 1st principle: CULTURE
Discuss everything database related with developers and sysadmins:
disks and raid design
schema design & partitions
additional columns & indexes
tables repair, ...
CAMS 1st principle: CULTURE
This is just a culture change
CAM
CULTUREAUTOMATIONMEASUREMENT
SHARING
CAMS 2nd principle: AUTOMATION
Can we automate DBAs job ?
Should we ??
Even if we have only one database ?
CAMS 2nd principle: AUTOMATION
CAMS 2nd principle: AUTOMATION
YES !YES !
CAMS 2nd principle: AUTOMATION
A lot of things to take care
binaries installation
configs
cron jobs
monitoring
grants
databases
replication
CAMS 2nd principle: AUTOMATION
Documentation is needed and must be up to date
CAMS 2nd principle: AUTOMATION
Everything needs to be reproducible
quickly without missing anything !
CAMS 2nd principle: AUTOMATION
The “easy” part to automate is the part
close to the dark side of the force: sysadmins
CAMS 2nd principle: AUTOMATION
The “easy” part to automate is the part
close to the dark side of the force: sysadmins
CAMS 2nd principle: AUTOMATION
Installation, configuration, cron jobs, monitoring and even grants can be automated using a configuration management system
Consider your infrastructure as code
CAMS 2nd principle: AUTOMATION
Example
node percona1 { include percona::repository include percona::toolkit
Class['percona::repository'] > Class['percona::server']
class { 'percona::server': mysql_version => "5.6", perconaserverid => "1", enable => "true", ensure => "running" }}
CAMS 2nd principle: AUTOMATION
But how do you manage schemas ?
Is it automated ?
And in production ?
V 1 V 2
Database Schema Migration
There are some existing tools:– Liquibase (probably the best right now)
– Flyway
– c5-db-migration
– dbdeploy
– MyBatis
– Autopatch
– Sqitch
Database Schema Migration
Most of them are ...
Integrated with a framework or an ORM
Database Schema Migration
Most of them are ...
for JAVA applications
Database Schema Migration
Most of them use ...
XML definitions
What is easier ? This:
<?xml version="1.0" encoding="UTF8"?> <databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog"xmlns:xsi="http://www.w3.org/2001/XMLSchemainstance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangeloghttp://www.liquibase.org/xml/ns/dbchangelog/dbchangelog2.0.xsd"> <changeSet id="1" author="lefred"><createTable tableName="department"><column name="id" type="int"><constraints primaryKey="true" nullable="false"/></column><column name="name" type="varchar(50)"><constraints nullable="false"/></column><column name="active" type="boolean" defaultValueBoolean="true"/></createTable></changeSet> </databaseChangeLog>
Or this ?
CREATE TABLE department (id INT NOT NULL PRIMARY KEY,name VARCHAR(50) NOT NULL
);
Database Schema Migration
also they are...
blocking
schema changes
Database Schema Migration
and finally...
not made to be used
via a configuration management system
Is there an alternative ?
Yes ;-)
online-migration
Online-migration: what is it ?
standalone Python script
command line only
Uses Oracle mysql-utilities
Uses pt-online-schema-change
Uses normal SQL statements
Can be easily integrated with puppet as a type and a provider are available
Online-migration: features
schema versioning and status
$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | none | pending | none || 0002 | none | pending | none || 0003 | none | pending | add nice column to cit |+++++
Online-migration: features
schema migration up$ ./onlinemigration.py up world to 3INFO : You want to migrate up to version 0003INFO : Ok this version is pendingINFO : Preparing migration to version 0001INFO : Applied changes match the requested schemaINFO : Preparing migration to version 0002INFO : Applied changes match the requested schemaINFO : Preparing migration to version 0003INFO : Applied changes match the requested schema
$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | 20130222 09:17:46 | ok | none || 0002 | 20130222 09:18:24 | ok | none || 0003 | 20130222 09:18:36 | ok | add nice column to cit |+++++
Online-migration: features
schema checksum
$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | 20130222 09:17:46 | checksum problem | none |+++++
$ ./onlinemigration.py checksum worldworld's current schema checksum = 3239e7245d98d27681020e36d0085676
Online-migration: features
schema migration down (rollback)$ ./onlinemigration.py down world 2INFO : You want to migrate down 2 version(s)INFO : rollback from 0003 to 0002INFO : rollback from 0002 to 0001
$ ./onlinemigration.py status worldMigration of schema world :+++++| VERSION | APPLIED | STATUS | COMMENT |+++++| 0000 | 20130222 09:17:16 | ok | Initial file || 0001 | 20130222 09:17:46 | ok | none || 0002 | 20130222 09:18:24 | rollback | none || 0003 | 20130222 09:18:36 | rollback | add nice column to cit || 0002 | none | pending | none || 0003 | none | pending | add nice column to cit |+++++
Online-migration: features
schema dif$ ./onlinemigration.py diff worldWARNING : Schema of world doesn't have expected checksum(4478d85870969436400bac023f2b2b7c)
TABLE `City` `CountryCode` varchar(10) DEFAULT NULL,+ `CountryCode` char(3) NOT NULL DEFAULT '',
Online-migration with puppet
Example
node percona1 { include percona::repository include percona::toolkit include onlinemigration
Class['percona::repository'] > Class['percona::server']
mysql_schema { "world": ensure => present, require => [ Package["onlinemigration"],
Service['mysql'] ], version => 1, cwd => "/root/om", } ...}
CAM
CULTUREAUTOMATION
MEASUREMENTSHARING
CAMS 3rd & 4th principles: MEASUREMENT & SHARING
This can be summarized in two other words :
TEST&
VISIBILITY
TEST
TEST
What should you test as a
devops DBA ?
TEST as a devops DBA
Test if you can deploy everything and you didn't forget anything
Test every changes (even small configuration change)
Test also the performance ! Is it faster with this change ?
Automate your tests !
TEST as a devops DBA
To test performance you can use several tools from Percona like:– pt-query-digest
– pt-upgrade
– Percona-playback
Visibility
Share the following metrics with everybody in
your company
Visibility: tests results
Share the results of the tests:
jenkins dashboard
Visibility: monitoring
Monitor your database with an alerting system
(nagios)
Visibility: monitoring
But don't alert too much !
Visibility: monitoring
Your alerting system should answer those questions:
Is the database server running ?
Does it allow connections
Is replication lagging on production slaves ?
Visibility: trending
The most interesting facts for a DBA are the trending results !
Use cacti with Percona Monitoring Plugins, they provide very good metrics
http://www.percona.com/doc/perconamonitoringplugins/cacti/mysqltemplates.html
Visibility: trending
Correlate changes in the trending results with your
configuration changes...
everything should be tracked
Time for a demo ?
Installation of a development and a production environment (using vagrant)
You can view this part of demo at http://tinyurl.com/kafboua
demo
Let's create a schema for our application in development
And push it to production
demo
Now let's modify the schema
And deploy it to production
demo
Now let's modify the schema
And deploy it to production
V 0 empty database “world”
demo
v1 (release 1 of the schema)
SQL can be downloaded at http://tinyurl.com/n3upcyq
Demo available at http://tinyurl.com/k7lm2yf
demo
v2
SQL can be downloaded at http://tinyurl.com/ld7pwf7
alter table City add PolutionLevel int;alter table City add index pol_idx(PolutionLevel);
demo
v3
SQL can be downloaded at http://tinyurl.com/knqx4ql
alter table City add nice tinyint default 1;
Warning !
A current bug in pt-online-schema-change prevents the program to work as expected when a table has constraints
https://bugs.launchpad.net/percona-toolkit/+bug/1207186
demo
Now let's upgrade this in production
Preparation
– copy all meta files into our puppet module
– specify the version in our manifest
Deployment
– run puppet
Demo at http://tinyurl.com/mq6sw33
demo
mysql_schema { "world": ensure => present, require => [
Package["onlinemigration"], Service['mysql']
], version => 1, cwd => "/root/om", }
...info: Mysql_schema[world](provider=mysql_schema): performing the check, last_version is 0info: Mysql_schema[world](provider=mysql_schema): we need to migrate to another version (1)info: Mysql_schema[world](provider=mysql_schema): performing the migration...
Resources
Percona – http://www.percona.com
Percona playback - http://www.percona.com/doc/percona-playback/
online-migration - https://github.com/lefred/online-migration
pt-online-schema-change - http://www.percona.com/doc/percona-toolkit/pt-online-schema-change.html
mysql-utilities - http://dev.mysql.com/downloads/tools/utilities/1.3.html
Vagrant schema versioning environment - https://github.com/lefred/schemas-versioning
shelr.tv – http://shelr.tv/users/4f85fb1b9660807969000005
Percona provides
24 x 7 Support Services Quick and Easy Access to Consultants Same Day Emergency Data Recovery Remote DBA Services QUOTE OSS4B for Special Consulting Offer – Free Mini Health Audit with Gold Support and Free Performance Audit with Platinum Support Valid September-October 2013
sales@percona.com or 00442081330309
Thank you !
Questions?
Recommended