Online Solutions with MySQL · Hands on Replication - Single, Multiple, Circular, Multimaster 3. 2nd May - MySQL Enterprise To Control Mission Critical Online Services 4. 23rd May

Copyright 2007 MySQL AB The World’s Most Popular Open Source Database

High Availability and Scalability for Online

Applications with MySQL

April 2007

Part 1I - Advanced Replication

Ivan ZorattiSales Engineering Manager EMEA

[email protected]

mailto:[email protected]



Agenda

Welcome back! and Welcome to the new attendees!

Introduction to MySQL - Usual 2 slides!

Understanding MySQL Replication

Hands-on Replication

Tips and Tricks

Replication Examples

Q & A

2


High Availability and Scalability for Online Applications with MySQL27th March - High Availability and Scalability Architectures

http://www.mysql.com/news-and-events/on-demand-webinars/part1-architectures-200703.php

2. 19th April - Advanced Scalability Solutions

Hands on Replication - Single, Multiple, Circular, Multimaster

3. 2nd May - MySQL Enterprise To Control Mission Critical Online Services

4. 23rd May - 99.999% High Availability solutions

5 solutions to explore - Hands on MySQL Cluster

5. 13th June - MySQL Enterprise performance and benchmarking

6. 27th June - Advanced HA solutions

3






Overview of MySQL AB

• Founded in 1995• Operations in 26 countries• 10 million product installations• 50,000 downloads each day• Over 9 million downloads of 5.0 to date• Dramatically reduces TCO of database management• Bundled by hundreds of SW and HW companies• Sold by partners such as HP, Dell, Novell, & others

4


Chosen by Successful, Modern Companies

• Embedded‣ ”Batteries included” database in

software applications‣ Network elements

• High Volume Web Sites‣ Dynamic content‣ eCommerce‣ Gaming & entertainment‣ Scale Out

• Enterprise‣ Data Warehousing‣ High-Volume Transactions‣ Departmental‣ Intranet/Extranet‣ Scale Out

5


Pluggable Storage Engine Architecture

6


Replication

Application Slave Master

•Better response time by splitting the load for processing queries between the master and slave

•SELECT queries may be sent to the slave to reduce the query processing load of the master

•Enabled for all storage engines•Achieve higher availability with a master/slave setup

•In the event of problems with the master, you can switch to the slave as a backup

•Replicate data from one MySQL server/storage engine combination to a different MySQL Server/storage engine combination

•Perform backups using a slave server and continue to process updates on the master while the backup is being made on the slave

•Extremely simple configuration to enable very robust replication

•Multiple replication topologies supported

7

9 0. 0 0 0 %


Replication - How it works

8

‣ Data written on the master is also written into the binary log

‣ The I/O thread on the slave collects logs from the master binary log and writes a relay log on the slave

‣ The SQL thread on the slave reads teh relay log and apply the writes on the slave‣ Slave writes are optionally

added to the binary log on the slave


Replication Topologies

Single

Multiple

Chain Circular

Multi - CircularMulti - Master

9


Why Replication?• Mirroring & High Availability

• Same Storage Engines• Backup

• Different Storage Engines

• Data Warehouse/Data Mart

• Web Search/Surfing• Intensive Read/Scale-out

• Application Integration

• Web Logging/Blogging• Intensive Write

10


Replication - How to

11

‣ On the Master‣ Turn on the binary log‣ Set the server-id‣ Create a new user with

REPLICATION SLAVE privilege

‣ On the slave‣ Set the server-id‣ Load the backup from the

master‣ Set up the master coordinates‣ Start Replication


Advanced Replication

12

‣ Circular and Multi-master‣ apply log-slave-update to add

replicated statements to the binary log

‣ Application logic, different storage engines and delayed writes may increase write performance

‣ Multi-master on a single table‣ PKs using AUTOINCREMENT‣ Define offsets for each

server


Hands onReplication

13


High Availability with ReplicationManual Fail-over

14

‣ The availability of the master must be checked by the client‣ Timeouts are somewhat

unreliable‣ Switchover is handled at a

connection level within the application or manually

‣ The master must be isolated in order to avoid inconsistencies

Applications

Master Slave

Replication


High Availability with ReplicationAutomatic Fail-over

15

‣ Third party products such as Linux-HA Heartbeat

‣ Applications refer to the VIP to access the Master server

‣ Switchover is handled at Cluster level

‣ The HA Slave server can be used as an ordinary slave during the normal operations

‣ Scale-out slaves are used for read-only operations

Applications

Virtual IPMaster HA Slave

Replication

Scale-out Slave Scale-out Slave Scale-out Slave

Replication


Replication in 5.1• Row Based Replication (RBR)

• Effective row values are sent to the slave

• Statement Based Replication (SBR) is still available• SQL Statements are sent to the slave

• Mixed Based Replication (MBR)• SBR is used by default, but the system switches

automatically to RBR when:• When a DML statement updates an NDB table• When a function contains UUID()• When 2 or more tables with AUTO_INCREMENT are

updated• When any INSERT DELAYED is executed• When a call to a UDF is involved

16


Row Based vs. Statement Based

17

• The master logs statements. Since a statement can affect multiple rows, SBR log is usually smaller than RBR log

• Logs can be used for auditing• Logs can be used for point in time

recovery• Master and slave tables can have a

different row structure• Some non-deterministic operations are

automatically handled• RAND(), NOW(), timestamps etc.

Row Based Statement Based• The master logs rows. It is the safest

replication to avoid inconsistencies• Some DDL operations are

automatically converted to SBR• Write operations apply fewer locks• RBR is open to multi-threading

extensions

• Log files are generally larger compared to SBR

• Rollback of transactions that affect a large number of rows will be logged anyway

• The binary log cannot be examined for auditing

• Bulk writes are considered as ordinary writes on the slave, bringing to a different locking behaviour between master and slave

• Non-deterministic writes cannot be replicated


Controlling Replication withMySQL Enterprise

18

‣ Replication Advisors‣ Slave Has Been Stopped ‣ Slave Has Login Accounts

With Inappropriate Privileges ‣ Slave Has Stopped Replicating ‣ Slave I/O Thread Not Running ‣ Slave Not Configured As Read

Only ‣ Slave SQL Thread Not

Running ‣ Slave Too Far Behind Master ‣ Slave Without REPLICATION

SLAVE Accounts‣ Replication Delay Graphs


Replication - Points to Consider• Asynchronous Replication

• Good for geographical redundancy and write performance

• Less good for high availability

• Platform and Storage Engine independence• Different storage engines can be applied on master and

slaves• BLACKHOLE can be used to discard unnecessary tables

and save space on the slave

• HA considerations• I/O Thread and SQL Thread improve performance and

High Availability• Use sync_binlog = 1 for maximum safety

19


Replication - Points to Consider• Read performance

• Linear and unlimited scalability provided by multiple slaves

• Multilevel configurations reduce network traffic on the master

• INSERT DELAYED can help when the master uses InnoDB and the slaves use MyISAM (must be SBR)

• Write performance• DELAY_KEY_WRITE and

innodb_flush_log_at_trx_commit = 0 improve performance on slaves

• Concurrency on MyISAM tables on slaves has less impact than on masters

20


Replication - Points to Consider• Read performance

• Linear and unlimited scalability provided by multiple slaves

• Multilevel configurations reduce network traffic on the master

• INSERT DELAYED can help when the master uses InnoDB and the slaves use MyISAM (must be SBR)

• Write performance• DELAY_KEY_WRITE and

innodb_flush_log_at_trx_commit = 0 improve performance on slaves

• Concurrency on MyISAM tables on slaves has less impact than on masters

20

Applications

Master

Slave/Master

Scale-out Slave

Slave/Master Slave/Master

Scale-out Slave Scale-out Slave Scale-out Slave Scale-out Slave Scale-out Slave Scale-out Slave

Write

Read


Replication - Points to Consider• Multimaster and Circular Replication

• It provides positive effects on performance only if a write delay is acceptable

• The Application must have the logic to identify the proper connections for reading and writing

• Multimaster on different tables schemas is more effective‣ Reduces concurrency

• Updates on the same table from different masters may bring to unresolved conflicts

• Use two phase commit with FEDERATED engine in 5.1 or XA Transactions in 5.0 to guarantee integrity on multiple masters

• The use of the FEDERATED on an InnoDB on a slave server can produce the “multi-source effect”

21


Replication - Points to Consider• Multimaster and Circular Replication

• It provides positive effects on performance only if a write delay is acceptable

• The Application must have the logic to identify the proper connections for reading and writing

• Multimaster on different tables schemas is more effective‣ Reduces concurrency

• Updates on the same table from different masters may bring to unresolved conflicts

• Use two phase commit with FEDERATED engine in 5.1 or XA Transactions in 5.0 to guarantee integrity on multiple masters

• The use of the FEDERATED on an InnoDB on a slave server can produce the “multi-source effect”

21

Applications

Scale-out Slave

Master FEDERATED


MasterMasterMaster

Write

Read

Slave Slave Slave


Application-side Aspects• Master Connections and Slave Connections

• Define Master Connections for write operations• Use Virtual IP for automatic failover• Use multiple connections for multimaster replication

• Define Slave Connections for read operations• Load balance vs Sticky vs Round Robin‣ Use Sticky (perhaps with HTTP load balance) if possible

‣ Preferred and alternative servers

• Connector/J provides Replication-ready features• jdbc:mysql://[host][,failoverhost...][:port]/[database]

[?propertyName1][=propertyValue1][&propertyName2][=propertyValue2]...

• autoReconnect• roundRobinLoadBalance• Connection.setReadOnly( false ) for masters and

Connection.setReadOnly( true ) for slaves

22


Application-side Aspects• Master Connections and Slave Connections

• Define Master Connections for write operations• Use Virtual IP for automatic failover• Use multiple connections for multimaster replication

• Define Slave Connections for read operations• Load balance vs Sticky vs Round Robin‣ Use Sticky (perhaps with HTTP load balance) if possible

‣ Preferred and alternative servers

• Connector/J provides Replication-ready features• jdbc:mysql://[host][,failoverhost...][:port]/[database]

[?propertyName1][=propertyValue1][&propertyName2][=propertyValue2]...

• autoReconnect• roundRobinLoadBalance• Connection.setReadOnly( false ) for masters and

Connection.setReadOnly( true ) for slaves

22


Hardware Considerations• Use twin servers for HA

• Same HW, same storage engines• Use the mirror machine as slave for read during normal

operations• Apply rolling upgrades of schemas, applications and base

software on the mirrored slave• Do not apply failback whenever possible

• Similar slave servers can help with balancing• Storage

• Use RAID 10 for Masters and RAID 0 for Slaves• 15k SCSI disks have a significant impact on performance

• Place data and logs (binary and relay logs) on different drives

23


And when things get really tough...• Apply Application partitioning• Partition Masters and Slaves accordingly• Build queries handling multiple connections to

groups of masters and servers

24


Replication - Heavy Web Site Example

25


Write

Read

Web/Application Servers

Master HA Slave

Virtual IP

Master HA Slave

Virtual IP

Master HA Slave

Virtual IP


*Ideas* for the Future

• Multi-source Replication• Conflict Resolution

• Synchronous Replication

• Multithreaded Replication

26


YahooApplication Personalized content and search options.

Chatrooms, free e-mail, clubs, and pager - #1 in Alexa

More than 50 MySQL ServersLAMP stack

MySQL Key Points MySQL used for many services (more

than 200 applications), such as Yahoo! Finance

"Our needs vary considerably, and that is one reason that MySQL works well with us. It handles the high-volume, 260 million record tables just as well as the small, low-volume ones. One of our servers handled over a quarter of a billion queries in a month-and-a-half, and it still has capacity to spare."

Jeremy ZawodnyDatabase Expert

Yahoo! Finance

27


WikipediaApplication An online collaborative encyclopaedia -

#11 in AlexaLAMP stackRAID 0 disks12 DB servers

Language based server splitting (application partioning)

Performance 3,000 pageviews per second

8,000+ http requests per second25,000+ SQL requests per second

MySQL Key Points 90% of DB Server Memory is InnoDB

buffer poolGeneral availability and scalability using replication

28


FlickrApplication Picture galleries available with social

networking, chat, groups, and photo ratings

LAMP stackPHP, Java and XML w/XPATHInnoDB Storage Engine

Language based server splitting (application partioning)

Performance 1,000 pageviews per second

25,000+ SQL requests per second

MySQL Key Points• Great session manager, to handle 100%

stateless PHP application on client side• Replication and vertical partitioning• Fulltext indexing within MySQL• Partial replication on multiple servers

29


Mixi.jp

Application Largest & fastest growing social networking site in

Japan. More than 3M users, with 15k new users every day.Ranked 30th on Alexa worldwideLAMP stack application

Key Business Benefit By using more than 100 servers (with 10 new

servers added every month) and horizontal partitioning strategy, combined with replication, mixi.jp can provide 2bn pageviews/day with a page load < 0.02sec

Problems with level 2

NODE 1

member tables

NODE 2

member tables

NODE 3

member tables

• Too many connections to different DBs

• Fortunately, on mixi, the majority are small data sets

• Cache them all by using distributed memory caching

• We rarely hit the DB

NODE 1

community tables

NODE 2

community tables

• Average page load time is about 0.02 sec*

* depending on data sets average load time may vary

30


More Resources Available• Consulting• MySQL Architecture & Design• MySQL Performance tuning• MySQL HA Jumpstarthttp://www.mysql.com/consulting/

• Training• MySQL 5.0 Performance Tuning• MySQL 5.0 for DBAs• MySQL Cluster for High Availabilityhttp://www.mysql.com/training/

• White Papers• A Guide to Database HA• What’s new in Cluster 5.1...and many morehttp://www.mysql.com/why-mysql/white-papers/

31

http://www.mysql.com/consulting/

http://www.mysql.com/consulting/

http://www.mysql.com/training/

http://www.mysql.com/training/

http://www.mysql.com/why-mysql/white-papers/

http://www.mysql.com/why-mysql/white-papers/


Special Offers!Valid until 30th June 2007

32

• Purchase any performance tuning consulting offering and attend a 4 day performance tuning class for only 795 EUR (1,595 EUR value)!

• Register for one of the following classes :• MySQL 5.0 for Developers• MySQL 5.0 for DBAs

And win an iPod shuffle !


Thank You!

Ivan ZorattiSales Engineering Manager EMEA

[email protected]

High Availability and Scalability for Online applications with MySQL

Don’t Miss Part III:MySQL Enterprise in Mission Critical Applications

Wednesday, 2nd May

33



Documents

Online Solutions with MySQL · Hands on Replication - Single, Multiple, Circular, Multimaster 3. 2nd May - MySQL Enterprise To Control Mission Critical Online Services 4. 23rd May