Scaling MySQL Deployments With Percona Server and Virident tachIOn DrivesA Percona White Paper

By Vadim Tkachenko (Percona), Shridar Subramanian (Virident), and Baron Schwartz (Percona)

Abstract

Until recently, MySQL and InnoDB were unable to take advantage of high-performance hardware, espe-cially high-performance storage systems. Large-scale deployments relied on so-called horizontal scalingwith a technique called sharding—partitioning data across many small-to-midsize servers. Sharded archi-tectures arose from the different nature of read and write workloads. If the database’s write workload wasnot too heavy, then a MySQL deployment could be scaled horizontally with replication, using read repli-cas. Reads could be scaled because replication creates multiple copies of the data, which can serve queriesindependently of each other. However, writes must be repeated on every replica’s copy of the data, soreplication does not help scale write traffic. The traditional way to do that with MySQL was throughsharding.

Today, MySQL—especially Percona’s enhanced version of MySQL, Percona Server with XtraDB—iscapable of exercising much more powerful servers and storage systems. As a result, server consolidationand vertical scaling is a viable—and often much more economical—path to high performance with MySQL.This white paper explains how to configure servers for high performance with MySQL and the ViridenttachIOn PCIe solid-state storage device. Key benefits of this approach are lower power consumption,decreased rack space for lower CapEx, and decreased architectural complexity to reduce operational andadministrative costs.

1 Traditional MySQL Scaling Strategy

The prevailing scaling strategy with MySQL is to“scale out” by making many copies of the data withMySQL replication and distributing them to multi-ple servers. This scales the read workload. How-ever, as the demand on the database grows, this ap-proach can result in a large number of replicas, lead-ing to a management and operational nightmare—not only requiring maintenance of many machinesbut also managing several copies of data. In ad-dition, replication requires application modificationto accommodate its asynchronous nature. More im-portantly, users often find that this approach doesnot scale very well as the associated write workloadgrows. The number of replicas that can be added toaddress the read workload is limited by the numberof writes that have to be replicated and the single-threaded nature of MySQL replication. The net re-sult is that scaling of the read workload is not inde-pendent of the write workload.

Scaling for writes is much more complex. Some ap-proaches include increasing DRAM sizes, using aSAN or external storage, and sharding. Increasingthe server’s DRAM capacity looks like an easy fix,

but works only up to a certain point. As the writeworkload and dataset size grows, this solution be-comes prohibitively expensive due to the non-linearprice-to-density relationship for DRAM, and there isa physical limitation on the amount of memory thatcan be added per server. Adding memory also doesnot solve the ultimate need for durable, persistentstorage. That requires MySQL to periodically flushthe data from memory to the backing store. As aresult, the steady-state performance of such a solu-tion is limited by the performance of backing storedrives, and hence performance can be poor. An-other limitation of this approach is the significantlyhigh “warm-up time,” which is the length of time be-tween server startup and the time it can accept highloads.

Although SANs can deliver good I/O throughput,the latency remains a problem. Latency is critical forMySQL performance, especially for replication, longqueries, and batch jobs. These operations are run se-rially; hence their response time is directly impactedby the large latencies of a SAN device. SAN-basedsolutions are also very expensive.

Sharding is the last option. But this is never an easy

Scaling MySQL Deployments With Percona Server and Virident tachIOn Drives 2

solution, and should be considered only as a last re-sort. It requires complex data management alongwith changes to applications. The architectural de-sign must be thought out in advance in detail, andit often leads to limitations on applications. For ex-ample, it becomes difficult or impractical to executesome types of queries because data is placed in dif-ferent locations. Joining data across different serversrequires emulated joins in the application code. Thedata becomes married to one form of architectureand a subsequent change to the architecture, to meetnew business needs, could be very difficult.

Sharding is also complex—the application devel-oper has to write more code to be able to han-dle sharding logic, and operational issues becomemore difficult (backing up, adding indexes, chang-ing schema). Thus, scaling for read and write work-loads using traditional methods leads to a morecomplex infrastructure and higher costs.

2 Scaling Up with Flash Storage

Flash-based storage, such as solid-state drives(SSDs), has created a paradigm shift in the way inwhich data is stored, managed, and accessed. Bothread and write performance issues can be signifi-cantly alleviated, and many applications can see in-stant improvement in I/O performance. Of the var-ious types of SSDs, the highest performance (high-est bandwidth and lowest latencies) is delivered byPCIe-based SSDs. SATA-based SSDs are inherentlylimited by bandwidth and have higher latency. Thisis not due to the underlying storage technology, butrather to the physical interconnect technology andtopology.

Virident’s tachIOn drive is a flash storage device thatworks well for scaling MySQL servers without theneed for sharding or replication. Virident tachIOndrives can deliver over 330,000 4kB read IOPS (I/Ooperations per second) and 200,000 IOPS in a mixed,random, read-write OLTP-like workload at 100%capacity utilization. This is orders of magnitudehigher than spindle-based disk drives. The tachIOndrive’s I/O performance is alse exremely consistentand predictable, helping produce high quality of ser-vice to the database’s end user. The following aresome of the characteristics of the tachIOn drive:

Performance

Benchmarking and customer deploymentshave shown that, depending on the server’sworkload and working set of data, the tachIOndrive can provide up to 15x performance gaincompared to HDDs. The Virident device letscompanies scale their MySQL infrastructuremore effectively and less expensively thanbuying more DRAM, purchasing a SAN, orhaving to shard the database, with no furtherchanges to the system (e.g. no need for a pro-prietary variant of MySQL or a closed-sourcestorage engine).

Latency

The tachIOn drive connects directly with theCPU using PCIe, giving latencies in the tens ofof µs (microseconds). This is better, by ordersof magnitude, than a SAN or even SATA- orSAS-based SSDs.

Capacity

The tachIOn drive is available in usable capac-ities ranging from 300GB to 800 GB.

Form Factor

The Virident tachIOn low profile PCIe SSDs canbe installed into any server chassis without theneed for additional power.

Modularity and Fault Tolerance

The tachIOn drive design is modular, con-sisting of a base card with field replaceableflash modules. It has an onboard flash-awareRAID5, which ensures high data availability atall times, even in the event of a flash modulefailure. This is in addition to the ECC imple-mented at the flash level.

3 Scaling Read Workloads

We benchmarked a read-only workload on bothtachIOn and traditional RAID hard drives. The re-sults show that the traditional phenomenon of per-formance dropping dramatically as data size ex-ceeds RAM size is significantly less of a problemwith the tachIOn drive. With fast storage such as thetachIOn , it is possible to get very good performance

Copyright c© 2011 Percona Inc. http://www.percona.com/

Scaling MySQL Deployments With Percona Server and Virident tachIOn Drives 3

on datasets that are much larger than memory. Thispermits consolidating many servers into one.

4 Scaling Write Workloads

The tachIOn drive is not only good at handlingreads—it is very efficient in handling write trafficas well. The following benchmark is a read-writebenchmark called tpcc-mysql, which is designed tomimic the industry-standard TPCC-C benchmark.

As the benchmark shows, a single tachIOn could in-crease a server’s capacity by a factor of almost 10.This is a significant improvement. This enables thetachIOn drives to improve MySQL replication per-formance, alleviating the single-threaded replicationbottleneck and making it possible to scale reads andwrites more independently. More importantly, usingtachIOn drives on a master can eliminate the need toshard the database. Many sharded environments we

have worked with could be consolidated to a singleserver with this performance increase.

The preceding benchmark illustrates the situationwhen the RAM is limited, and the dataset is muchlarger than memory. However, even when the entiredata set can fit into DRAM, using a tachIOn drivecan deliver a 5X improvement in the ability of theMySQL master server to handle write workloads:

5 Scaling When Data Exceeds Memory

If the entire data set fits on a single card (up to 800GBusable capacity), the previously discussed perfor-mance improvements can be achieved by movingall the data into the tachIOn card. No additionalchanges are required. If the dataset size is too large,there are three ways to arrange the data.

1. Stripe: Stripe the data across multiple tachIOncards on a single server

2. Tier: Locate the most IO-intensive files on thetachIOn drive

3. Cache: Implement a flash-memory-friendlycaching solution such as FlashCache

You may also consider separating the files in the fol-lowing manner: place the transaction logs and bi-nary logs on a RAID 10 SAS HDD and the entire re-maining index and data files on the tachIOn drive.This conserves valuable space on the PCIe SSD. A

Copyright c© 2011 Percona Inc. http://www.percona.com/

Scaling MySQL Deployments With Percona Server and Virident tachIOn Drives 4

RAID 10 HDD array with a battery-backup unit is agood choice. The RAID array can also be used forother logs such as the slow query log and error log.

Performance can be further improved by puttingsystem tablespace (ibdata1) on separate HDD stor-age, because I/O patterns for this tablespace are dif-ferent from I/O patterns of data and index files. Theadditional improvement can be as much as 1.45x.This improvement is over and above all the per-formance improvement obtained by using tachIOndrives for write intensive workloads where the sizeof the hot data is significantly larger than the DRAMsize.

6 Using Percona Server with XtraDB

An important additional measure for improving theoverall performance is to use Percona Server withXtraDB, which has much higher performance thanthe standard MySQL server, due to its more scalabledesign on modern hardware. Percona Server withXtraDB provides significant throughput increases,reduced response times, and much more consistentperformance. It is particularly important on ex-tremely fast I/O devices such as the tachIOn drive,because the standard MySQL server simply cannottake advantage of all the I/O capacity available to it.The standard MySQL server will reach its through-put ceiling while there is still idle CPU and I/O ca-pacity. In addition, the standard InnoDB’s check-point algorithm must be tuned precisely for a par-ticular workload an I/O device in order to avoid pe-

riodic server stalls—and this is possible only in anartificial setting such as a laboratory benchmark. Areal workload has variations, and no single combi-nation of server settings will produce good perfor-mance all the time under a varying workload withstandard MySQL and InnoDB.

0

20000

40000

60000

80000

100000

120000

140000

New

-Ord

er T

rans

actio

ns P

er M

inut

eTime

Percona Server 5.5.7MySQL 5.5.8

7 Conclusion

The sharding approach that has been advocated forthe last five years or so is becoming increasinglyquestionable advice in some environments. Tra-ditionally, IT departments have adopted a multi-pronged strategy to address the scaling problem.They typically separate out the read and write traf-fic. Masters absorb the write traffic, while the ma-jority of the read traffic is directed to the replicas.Today’s solid-state PCIe hardware offers extremelyhigh-bandwidth, low-latency I/O performance, ex-emplified by the Virident tachIOn drive. And today’sMySQL server, especially MySQL 5.5 and even moreso Percona Server with XtraDB, is capable of utiliz-ing much more of that hardware’s available capacityeffectively. “Scaling up” is once again a viable andeconomical strategy for MySQL, and “scaling out”need no longer be the default database architecture.

Copyright c© 2011 Percona Inc. http://www.percona.com/

Scaling MySQL Deployments With Percona Server and Virident tachIOn Drives 5

About Percona

Percona is the oldest and largest independentprovider of commercial support, consulting, train-ing, and engineering services for MySQL databasesand the LAMP stack. You can contact us throughour website at http://www.percona.com/, or to callus. In the USA, you can reach us during businesshours in Pacific (California) Time, toll-free at 1-888-316-9775. Outside the USA, please dial +1-208-473-2904. You can reach us during business hours in theUK at +44-208-133-0309.

About Virident Systems

Virident Systems builds enterprise-class solutionsbased on Flash and other storage-class memories(SCM). These disruptive technologies will revolu-tionize the data center and cloud computing bysolving performance, reliability, and serviceabilityproblems that further compound in large-scale de-ployment of SSDs in current environments. Visithttp://www.virident.com for more information, orcall us at (408) 503-0100 during business hours in Pa-cific (California) Time.

About Percona Server

Percona Server is an enhanced, high-performance version of the world’s most popular open-source database,MySQL. MySQL is used by many of the world’s largest websites, including Facebook, Flickr, and YouTube.MySQL is also deployed widely in industries such as financial services, government, education, pharma-ceuticals, and telecommunications. Its simplicity, reliability, and ease of use make it cost-effective to man-age, and because it is open-source, it can be used without license fees. Percona Server is derived fromthe MySQL database, to which it adds features such as enhanced monitoring and configurability. PerconaServer offers much faster and more consistent performance than the standard MySQL server. Percona alsoprovides a free hot-backup program, Percona XtraBackup.

Percona, XtraDB, and XtraBackup are trademarks of Percona Inc. InnoDB and MySQL are trademarks of Oracle Corp. Virident and tachIOn aretrademarks of Virident Systems Inc.

Copyright c© 2011 Percona Inc. http://www.percona.com/