High Availability?Setting Up MySQL

Our ApproachDBI Framework

Questions

High Availability DBI & MySQLYAPC::NA 2005

High AvailabilityDBI & MySQL

Steve Purkis

High Availability?Setting Up MySQL

Our ApproachDBI Framework

Questions

High Availability DBI & MySQL

Slide 3

High Availability?

• What is High Availability?• Ever have an application fail because a

mission critical server crashed?

• High Availability is an attempt to provide zero application downtime.

Slide 4

High Availability?

• How do I achieve HA?:• Analyze your apps• Most important components?

• Remove single points of failure (SPOFs):• Add redundant servers• Replicate data (backups, realtime)• Load balancing• Failover when things go pear-shaped

Slide 5

High Availability?

• This talk…• Introduces one approach to building HA Perl

apps that rely on MySQL.

• We’ve used similar techniques with Oracle

• They work for us…• They may not be right for you!

High Availability?Setting Up MySQL

Our ApproachDBI Framework

Questions

High Availability DBI & MySQL

Slide 7

Setting Up MySQL

• Some typical architectures…

Slide 8

Setting Up MySQL

• Ye olde basic setup• Not HA.

Application Server

MySQLServer

Slide 9

Setting Up MySQL

• One master, one read-only slave• Good for load balancing• Still not really HA!

Application Server

MySQLMaster

MySQL Slave Data

Replication

reads writes

Load Balancer

Slide 10

Setting Up MySQL

• Two masters• It’s HA, but…• Flaky, tricky to work with!

Application Server

MySQLMasterTwo-way

Replication

MySQLMaster

Load Balancer

Slide 11

MySQL Slave

Setting Up MySQL

• Two masters, multiple slaves• Only one master active at a time• HA

Application Server

MySQL Slaves Data

Replication

reads writes

Primary Master

2nd aryMaster

Master Failover

Load Balancer

• HA (unless your hosting center goes down)

Slide 12

Setting Up MySQL

• Your HA MySQL architecture will depend on:• SLAs• Application requirements• Load• Physical spread of servers• Budget

• Caveats:• Are your apps read or write heavy?• Beware of replication lag time

• Common in high-latency networks• Do your apps need the latest data?

• Transactions?

Slide 13

Setting Up MySQL

• Load balancing• Hardware?• Software?

• Master Failover• On Slaves:

CHANGE MASTER TO …• Manual?• Automate?

• Cron? Application? SLB?

• On App servers:• Connect to Secondary

• Not required if using a hot spare w/same ip

Slide 14

Setting Up MySQL

• Recommended reading:• High Performance MySQL

- Jeremey Zawodny & Derek Balling

• MySQL Replication docs• MySQL Cluster white paper

• For Linux:• Linux Virtual Server Project• Linux-HA

Slide 15

Setting Up MySQL

• If you have any trouble:• MySQL mailing list

High Availability?Setting Up MySQL

Our ApproachDBI Framework

Questions

High Availability DBI & MySQL

Slide 17

Our Approach

• Our requirements:• Read-heavy apps• Over 750 clients, many with SLAs• Reliability:

• Read availability most important• Must work if a server farm goes down

• Speed is of essence:• Millions of requests a day

• Don’t need transactions• Minimize costs

Slide 18

Our Approach

Application Server

Application Server

Application Server

MySQL Slave

MySQL Slaves

MySQL Slaves

Application Server

MySQL Master

MySQL2nd ary

Master F

ailover

Data Replication

95% 5%

readswrites

Slide 19

Farm 2

Farm 3Farm 1

Our Approach

Application Server

Application Server

MySQL Slave

Application Server

MySQL Master

MySQL 2nd ary

Master F

ailoverIn-Farm Data Replication

Application Server

MySQL SlaveMySQL Slaves

MySQL SlaveMySQL Slaves

Query caching

Slide 20

Our Approach

• DBI Framework:• Load balancing• Server selection• Read / Write query?

• Failover

Application

DBI Framework

----------------------------------- snip: how to tell reads from writes? ---------------------------------

my ($action, $dbh) = $query =~ /\Aselect|\A\(select|\Ashow|\Adesc/i ? ("read", $dbh_read) : ("write", $dbh_write);

----------------------------------------------------------------------------------------------------------

Slide 21

Our Approach

• Load balancing• On connect• Weighted server selection

• Availability, number of processes, user weights, which farm

• MySQL idle timeout

• Failover• Connect to next slave• Automatic query retry

Application

DBI Framework

MySQL Slave

MySQL Slave

reads

Slide 22

Our Approach

• Failover• Persistent connection to

all masters• Automatic query retry• Automatic fallback

Application Server

DBI Framework

Primary Master

2nd ary Master

Master F

ailover

writes

High Availability?Setting Up MySQL

Our ApproachDBI Framework

Questions

High Availability DBI & MySQL

Slide 24

DBI Framework

• Two wrapper functions:• dbconnect( 'read' | 'write' )• Read: Select slave to connect to• Write: Connect to all masters

• $sth = sql( $query )• Read / write dbh selection• Failover & fallback

• Pseudo code?

Slide 25

DBI Framework

• dbconnect( 'read' | 'write' )• Read:

For each slaveweight = 0, next if can’t pingcheck number of processes with mysqladminweight = no. processes / user weighting

Connect to slave with lowest weightSanity check: run a simple queryTry next slave if that failed

• Write:For each master

ConnectSanity check: run a simple query

Set $write_dbh to primary master

Slide 26

DBI Framework

• $sth = sql( $query )• Determine query type…• Read:

Execute query.Failover to next slave on error, and retry query.

• Write:Fallback?

If not using primary, and we failed over X seconds ago, try reconnecting to master.

Execute query.Failover to next master on error, and retry query.

Slide 27

DBI Framework

• Major Drawbacks:• Using DBI-based CPAN modules is hard!

Slide 28

DBI Framework

• Looking Ahead…

• Push HA logic into the DBI layer• Write DBD::MysqlHA ?• DBIx::HA ?• DBD::Multiplex ?• DBIx::DBCluster ?• SQL Relay ?

• MySQL Cluster?

High Availability?Setting Up MySQL

Our ApproachDBI Framework

Questions

High Availability DBI & MySQL

Slide 30

DBI Frameworks on CPAN

• If there’s time…

Slide 31

DBI Frameworks on CPAN

• The ones I know a bit about:• DBIx::HA• DBD::Multiplex• DBIx::DBCluster

Slide 32

DBI Frameworks on CPAN

• DBIx::HA• Generic HA solution

(written & tested for Sybase)• Configurable by db name (%DATABASE::conf)• Looks well thought out

Slide 33

DBIx::HA

Pros• Does Failover

• On query failure & dbh disconnected

• Does timeouts:• connect, query execute• Safe signals

• Connect all dsns on init• Supports Apache::DBI

Cons• No read / write distinction• No way to choose which

dbh to use on failover• (want to use a mysql-

specific algorithm)

• Timeouts with SIGALRM• (but how else, really?)

• No ping checks for non-Apache::DBI• (uses $dbh->ping anyways -

we prefer ICMP ping)

Slide 34

DBIx::HA

• Also: written & tested for Sybase!

• Potential DBD::Mysql problems:• auto_reconnect

we may reconnect to a db when we should be failing over

Slide 35

DBI Frameworks on CPAN

• DBD::Multiplex• send requests to multiple dsn's• Configure servers to use in $dsn

(pipe-separated list)

Slide 36

DBD::Multiplex

Pros• Supports master/slave

setup:• differentiates between

reads / writes• Connects to all dsns

(good for fast master failover)

• Does failover:• default behaviour

reads: first_successwrites: first_error

Cons• Can only specify one

master• (though if you specify none,

writes can go to all with 'first_error' exit mode)

• Connects to all dsns• (don't want to connect to all

slaves)

• No customizable slave failover algorithm

• No reconnects• No fallback!

Slide 37

DBI Frameworks on CPAN

• How could we re-use CPAN modules?

• DBIx::HA• Sub-class to introduce MySQL specific functionality?• Introduce a callback for server selection on failover?• Use in conjunction with DBD::Multiplex for read/write dbh

selection?• There could be problems though…

• In general:• Backwards compat• sql() wrapper (for backwards compat)• Custom logging?• Our db wrappers do other things too…

Slide 38

DBI Frameworks on CPAN

• Maybe the best way to reuse them is to nick their ideas?

• DBD::MysqlHA ?• Combination of DBIx::HA and DBD::Multiplex• MySQL specific:

• Customizeable server selection algorithm• Persistent connections