MySQL Performance Tuning at COSCUP 2014

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1

MySQL Performance Tuning

MySQL Global Business UnitSales Consulting Senior Manager, JAPAC梶山隆輔 / Ryusuke Kajiyama

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

MAKE MySQLa Better MySQL


MySQL 5.5MySQL Cluster 7.3

MySQL Enterprise Monitor 2.3MySQL Enterprise Backup

SecurityScalabilityHAAudit

MySQL 5.6MySQL Workbench 6.0

MySQL Utilities

MySQL Applier for Hadoop

MySQL Workbench 5.2

MySQL Enterprise Oracle Certifications

Over 3,5 Years of MySQL Innovation

MySQL Cluster ManagerWindows installer & Tools

MySQL Cluster 7.2MySQL Cluster 7.1

MySQL Migration Wizard


MySQL: Everywhere and GrowingPlatform of Choice for Next Generation Web, Cloud & Embedded Applications

World’s Most Popular Open

Source Database

Database Platform for

Next Gen Web Apps

Leading Database for

Web Applications

Leading Database in the Cloud

Integrated with Hadoop in

majority of Big Data platforms

#1 Linux Career IT skill: MySQL *

* Source: Linux Career IT Skills Watch update July 2013

Downloads & Sales Growth

Best of both Worlds SQL+NoSQL

LAMP + Python +Node.js + Go +...

Awards & Social Media Momentum


MySQL 5.6: Best Release Ever!


MySQL 5.6: Scalability

Users can fully utilize latest generations of hardware and OS Scales as data volumes and users grow


MySQL 5.5

MySQL 5.6

MySQL 5.6 SysBench Benchmarks

Up to 234% Performance Gain

Replication Binary Log Group Commit • Days of losing 50%+ master

performance are gone!• 10% overhead when configuring

sync_binlog=0• Additional 5% when configuring

sync_binlog=1

https://blogs.oracle.com/MySQL/entry/mysql_5_6_replication_performance


Same app can leverage: Key-value access to InnoDB via

familiar Memcached API SQL for rich queries, JOINs,

FKs, etc. Fully transactional

MySQL 5.6: InnoDBNoSQL Key Value Access to InnoDB

Up to 9x performance boost for updates

Great for fast data ingestion in Big Data pipeline


MySQL 5.6: Optimizer

• Simple word but many meanings• Main objective:

• Users (direct or indirect) should be satisfied• Most typical performance metrics

• Throughput• Latency / Response time• Scalability • Combined metrics

Starvation

transactions/time

Defining Performance

• Multi User applications • Request waits in queue before being processed • User response time = queueing delay + service time

• Non high tech example – support call center.• “Hockey Stick” - queuing delay grows rapidly when

system getting close to saturation• Need to improve queueing delay or service time

to improve performance• Improving service time reduces queuing delay

Connections

Response T

ime

Queuing Theory

• Main Question – where does service time comes from ?• network, cpu, disk, locks...

• Direct Measurements• Sum up all query times from web pages

• Indirect measurements• CPU usage• Disk IO latency• Network traffic• Load Average• Number of running queries• etc.

Service Time: Key to the hotspot

Benchmark Tests

• Great tool to:• Quantify application performance• Measure performance effect of the changes• Validate Scalability• Plan deployment

• But• Can be very misleading if done wrong• Need to be able to read results correctly

• Typical Errors• Testing with 1GB size with 100G in production• Using uniform distribution

• “Harry Potter” ordered as frequent as Zulu dictionary• Testing in single user scenario

Benchmark Tools for MySQL• Creating own benchmark tool

– Generating from SQL statements in General Log– Fetching SQL statements with MySQL Proxy or TCP Dump

• DBT2– http://osdldbt.sourceforge.net/ – http://nippondanji.blogspot.com/2009/03/dbt-2.html

• mysqlslap MySQL 5.1 +– http://dev.mysql.com/doc/refman/5.6/en/mysqlslap.html

• SysBench– http://sysbench.sourceforge.net/

• Performance costs money, whatever road you take• Investigate different possibilities

• Better hardware could be cheaper than a major rewrite• How much performance/scalability/reliability do you need ?

• 99.999% could be a lot more expensive than 99.9% • Is peak traffic requirements 100x average or just 3x ?

• Take a look at whole picture• Is this the largest risk/bottleneck ?

• Identify which optimizations are critical for business• Optimization of “everything” is often waste of resources• What is the cost of suboptimal performance ?

Business side of optimization

Basic1: Checking Server Configurations

• The MySQL server is controlled by “System Variables”

• Set via:• Option File: 　 my.cnf / my.ini• Temporary change: SET [GLOBAL] <vriable>=<value>

• Can be per connection (LOCAL) or server wide(GLOBAL)

Basic2: Checking Status of Server Monitor system performance using “Status Variables”

• Knowing internal commands of one queryMysql>　FLUSH　STATUS;　<run　query>;　SHOW　STATUS;

• Checking status periodically shell>　mysqladmin　-u　-p　...　ex　-i　15　-r　|　grep　-v　‘　0　‘

http://dev.mysql.com/doc/refman/5.6/en/server-status-variables.html

Client2 ClientN

Connection Thread Pool

Parser Query 101101

Client1

Optimizer

Storage Engines InnoDB MyISAM MERGE MEMORY ARCHIVE

Query Cache

MySQL Server

MySQL Server Architecture

• max_connections (151)• number of connections server will

allow. May run out of memory if too high, because of per connections memory usage

• thread_cache_size (8)• Keep up to this amount of threads

“cached” after disconnect• Typical setting

max_connections/3

Client2 ClientN


Client1

mysql> show status;• Max_used_connections

• check if it matches max_connections, too low value or sign of overload

• Threads_created• thread_cache misses• should be low.

Server Connections & Threads

• sort_buffer_size (2M)• Memory to allocate for sort. Will use

disk based sort for larger data sets. Often fine at 512K or 1M

• other buffers, read, read_rnd, etc… smaller defaults often OK

• You can change dynamically if large sort operation is needed in batch operation etc

Client2 ClientN


Client1

mysql> show status;• Sort_merge_passes -

• number of passes made during file merge sort.

• check if file sort needs to be done at all

• use index if possible

Connection Thread Work Buffers

• query_cache_size (0)• Amount of memory to use for query

cache• Typically 32M is fine, some databases

need 128M• query_cache_type (ON)

• Worst case performance overhead is about 15%-20%

• Favors servers with higher SELECT/WRITE ratios

• Best Practice• Set to DEMAND• Add SQL_CACHE to appropriate

queries


Parser Query 101101

Query Cache

mysql> show status;• Qcache_hits, Qcache_inserts

• hits/inserts cache hit ratio, if small, maybe disable query cache

• Qcache_lowmem_prunes• times older queries were removed due to

low memory, need to increase query_cache_size if high

Server Query Cache

• innodb_buffer_pool_size • 80% of memory on Innodb only system• caches data & indexes unlike MyISAM

• innodb_log_file_size• A key parameter for write performance• Recovery time is no more an issue.• Bigger is better for write QPS stability

• innodb_flush_log_at_trx_commit• 1 (slow) will flush (fsync) log at each commit. Truly

ACID• 2 (fast) will only flush log buffer to OS cache on commit,

sync to disk once/sec.• 0 (fastest) will flush (fsync) log every second or so

• innodb_file_per_table• Always good choice to distribute i/o• Default ON from 5.6

Storage Engines InnoDB MyISAM MERGE MEMORYARCHIVE

mysql> SHOW ENGINE INNODB STATUS;

Great way to see what is going on inside InnoDB, hard to parse

• File IO• Buffer Pool • Log activity • Row activity

InnoDB Performance Tips

• innodb_flush_method = O_DIRECT• Not to consume OS cache

• innodb_buffer_pool_instances (5.5+)• To avoid mutex contention• 2 or more in can

• innodb_io_capacity (5.5+)• Enlarge if you have fast disks• Default 200 is good for 2 disks striped

• innodb_read_io_threads (5.5+)• innodb_write_io_threads (5.5+)

• Enlarge if you have fast disks• Default 4 is usually good enough

Storage Engines InnoDB MyISAM MERGE MEMORYARCHIVE

mysql> SHOW ENGINE INNODB STATUS;

Great way to see what is going on inside InnoDB, hard to parse

• File IO• Buffer Pool • Log activity • Row activity

InnoDB Performance Tips (Cont.)

DBT-2 (W200) Transactions per Minute %user %iowait

Buffer pool 1G 1125.44 2% 30%

Buffer pool 2G 1863.19 3% 28%

Buffer pool 5G 4385.18 5.5% 33%

Buffer pool 30G(All data in cache) 36784.76 36% 8%

Semi-in-memory Database with InnoDB

• DBT-2 benchmark (write intensive)• 20-25GB hot data (200 warehouses, running 1 hour)• Nehalem 2.93GHz x 8 cores, MySQL 5.5.2, 4 RAID1+0 HDDs• RAM size affects everything. Not only for SELECT,

but also for INSERT/UPDATE/DELETE• INSERT: Random reads/writes happen when inserting into indexes in random order• UPDATE/DELETE: Random reads/writes happen when modifying records

• Normalization• good for OLTP, writes• data redundancies eliminated• join performance penalty• smaller total data set• E/R diagram clean translation

• Denormalization• good for OLAP, reporting• data redundancies across tables for

better indexing• maybe eliminate joins

• Data types– use tinyint, smallint, mediumint,

save space!– join columns same data type– varchar(64) instead of char(64)– declare NOT NULL if true– varchar(64) instead of varchar(255)– INT not DECIMAL(9)

• Indexing– multi-column– ordered BTREE– index prefixes– covering

Schema Design

#Time: 08073101 16:25:24#User@Host: root[root] @ localhost [127.0.0.1]#Query_time: 8 Lock_time: 0 Rows_sent: 20 Rows_examined: 243661SELECT part_num FROM `inventory`.`parts` WHERE(`ven` = "foo") ORDER BY `delivery_datetime` DESC LIMIT 100;

Monitoring Queries - Slow Query Log

Cons• Requires MySQL restart (5.0 and lower)• Growth must be managed using FLUSH LOGS• Entries must be parsed/sorted for relevance• mysqldumpslow helps, but still tedious, takes time

Pros• Logs queries that took longer than X (user defined)• Logs queries that do not use indexes (5.0 and higher)• Includes data needed to trace offending queries

Monitoring Queries – SHOW PROCESSLIST;

mysql> SHOW FULL PROCESSLIST\G ******** 1. row *****************Id: 1User: MyUserHost: localhostdb: inventoryCommand: QueryTime: 1030455State: Sending Data Info: SELECT part_num from ‘inv’;

…..2 rows in set (0.00 sec)

Pros• Shows current processes• Shows status of executing

queries• Includes data needed to trace

offending queries

ConScripting needed to:• automate, • integrate with Slow Query Log,• Aggregate/parse results for

analysis,• notify DBA of problem

Fixing Problem Queries – EXPLAIN; EXPLAIN SELECT part_num FROM `inventory`.`parts` WHERE (`ven` = "foo") ORDER BY `delivery_datetime` DESC LIMIT 100;\G ******** 1. row ************* ID: 1 select_type: SIMPLE table: parts type: refpossible_keys: ven, part# key: ven key_len: 3 ref: null rows: 872 Extra: Using WHERE1 row in set (0.00 sec)

Analyze• How indexes are being used (or

not…)• required filesorts• What tables, columns are being

queried

Fix/Tune - involves iterations of:• Add/alter indexes• Alter tables, columns, datatypes• Alter query structure• Test, 10 GOTO 10 until done

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1230

MySQL Enterprise Monitor

Real-time MySQL performance and availability monitoring

Visually find & fix problem queries

Disk monitoring for capacity planning

Cloud friendly architecture (no agents)

Start monitoring MySQL in 10 minutes

Remote agent option provides OS monitoring

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1231

MySQL Query Analyzer

Real-time query performance

Visual correlation graphs

Find & fix expensive queries

Detailed query statistics

Query Response Time index (QRTi)

“With the MySQL Query Analyzer, we were able to identify and analyze problematic SQL code, and triple our database performance. More importantly, we were able to accomplish this in three days, rather than taking weeks.”

Keith Souhrada Software Development Engineer Big Fish Games

MySQL Enterprise ScalabilityMySQL Thread Pool

• MySQL default thread-handling – excellent performance, can limit scalability as connections grow

• MySQL Thread Pool improves sustained performance/scale as user connections grow

• Thread Pool API

New!

MySQL Enterprise Scalability

ConfigurationMySQL 5.6.11Oracle Linux 6.3, Unbreakable Kernel 2.6.324 sockets, 24 cores, 48 ThreadsIntel(R) Xeon® E7540 2GHz CPUs512GB DDR RAM


Performance Schema in MySQL 5.5 & MySQL 5.6


Performance　Schema　in　MySQL　5.5

Laid out the framework for instrumentationFocused early on low level instruments

– Prove it could handle load, build from ground up

Not turned on by default, had some overhead issues



17 Tables222 Instruments

Instrument Event　ClassFile IO wait/io/file/%Mutexes wait/synch/mutex/%Read/Write Locks wait/synch/rwlock/%

Conditions wait/synch/cond/%





Fixed lots of performance issuesFocused on bringing the instrumentation to the DBA / Developer

Statements / Statement DigestsExecution StagesObject tracking (table / index IO, table locks)Network IO

Turned on by default, with a subset of instrumentation disabled



52 Tables (+35)545 Instruments (+323)

Instrument　Type Event　Class

Statements statement/%Stages stage/%Table IO wait/io/table/%Table Locks wait/lock/table/%Network IO wait/io/socket/%Idle Timing idle




Improvements made to date in MySQL 5.7

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12

Insert Picture Here

42

Memory UsageMetadata LockingReplication Configuration & StatusPrepared StatementsTransactionsStored Programs


Improvements　made　to　date　in　MySQL　5.7

75 Tables (+23)784 Instruments (+239)

Instrument　Type Event　Class

Transactions transaction

Memory memory/%

Metadata Locks wait/lock/metadata/%


Improvements　made　to　date　in　MySQL　5.7


DMRs: Development Milestone Releases

Fully Functional Release Candidate Quality

Two to Four per Year

Early Community Testing, Use, & Feedback

Accelerates Rate of Enhancements

Improves Quality


MySQL 5.7

InnoDB for better transactional throughput, availability Replication for better scalability and availability Utilities for dev/ops automation Performance Schema for better performance metrics Optimizer for better EXPLAINing query performance

Available Now! Get it here:

MySQL 5.7.2 builds on MySQL 5.6 by improving:

dev.mysql.com/downloads/mysql/

New!


MySQL 5.7 Sysbench BenchmarksSysbench Point Select

Intel(R) Xeon(R) CPU X7560 x86_644 sockets x 8 cores-HT (64 CPU threads)2.27GHz, 256G RAMOracle Linux 6.2

95% Faster than MySQL 5.6 172% Faster than MySQL 5.5

500,000 QPS


RO In-Memory @MySQL 5.7

Sysbench OLTP_RO 8-tables, 32cores-HT :


RO In-Memory @MySQL 5.7



第一次台灣MySQL社群聚會

我們將在 7月 22日下午 6:00到 8:00借用台北市忠孝西路一段 35樓甲骨文的會議室，舉辦第一次台灣MySQL社群聚會，歡迎您抽空參加。在聚會中我們將會中提供簡單的晚餐，同時分享MySQL的最新動態，包括：

– 展示如何用新發表的MySQL Fabric支持上 PB級資料的應用系統，做資料庫分表分庫，高可用和讀寫分離

– MySQL 5.7.4 DMR的新功能和産品發展路線也非常歡迎您分享您的心得和意見如果您打算參加 22日的聚會，或打算分享您的心得

1st TWMUG meeting

Documents

MySQL Performance Tuning at COSCUP 2014