Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Basic MySQL Troubleshootingfor Oracle DBAs
Sveta SmirnovaSenior Principal Technical Support EngineerMySQL SupportSeptember 29, 2014
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Introduction
Basic troubleshooting techniques
High concurrency issues
High availability solutions
More information
1
2
3
4
5
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Introduction
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Click icon to add picture
MySQL architecture
Base - Installation layout - Log filesConnectors - Clients - APIsOptimizerCaches & buffersStorage enginesManagement
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Typical installation layout
• datadir
– Schema directory– Table files: *frm, *ibd, *MYD, *MYI, *par, etc.
– Trigger files
– Schema
– …
– InnoDB shared tablespace
– Log files– InnoDB log files
– Binary, relay logs
– Error log
– Slow query log
– General query log
Configurable
You can setup custom path for each component
Including custom path for tables
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Connectors
• Clients–MySQL CLI
–MySQL Workbench
–MySQL Enterprise Monitor (MEM)
–Oracle Enterprise Manager with MEM plugin
• APIs
–Exist for most popular programming languages
–C, C++, JDBC, PHP, Python, Net, ODBC
The way you connect to MySQL Server
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Plugins
• Storage engines
• Full-text parsers
• Daemon
• INFORMATION_SCHEMA
• Semisynchronous Replication
• Audit
• Authentication
• Password-validation
• Protocol Trace
MySQL Server is highly configurable via plugins
5.7
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Storage engines
• Own data
• Own index format
• Own locking model
• Own diagnostic
• Own log files
• CHECK TABLE
From troubleshooting point of view
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Basic troubleshooting techniques
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
• Privileged client cannot connect
• Unprivileged client can connect
• Privileged user cannot perform operation
• Unprivileged user has undesired access
Typical issues
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
• No roles by default, limited user limits
• All records are in the mysql database (schema)
• Pluggable authentication since version 5.5
• Connections● TCP/IP with login-password
● Socket (Unix)
● Named pipe (Windows)
Overview
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
select user, host from mysql.user order by user desc, host desc;
• Most descriptive host first, then wildcard
• Socket connection by default on Unix
Where to find out why you have connection issues?
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
mysql> select user, host from mysql.user order by user desc, host desc;
+------+------------+
| user | host |
+------+------------+
| root | localhost |
| root | delly |
| root | ::1 |
| root | 127.0.0.1 |
| foo | % |
| | localhost |
+------+------------+
6 rows in set (0.00 sec)
Where to find out why you have connection issues?
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
SHOW GRANTS [FOR user@host]
• Grant tables● mysql.db● mysql.tables_priv● mysql.columns_priv● mysql.procs_priv● mysql.proxies_priv
SELECT USER(), CURRENT_USER()
Wrong access
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
mysql> show grants;
+--------------------------------------------------------------------+
| Grants for root@localhost |
+--------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION|
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION |
+--------------------------------------------------------------------+
2 rows in set (0.02 sec)
Wrong access
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
mysql> show grants for foo@'%';
+-----------------------------------------------------+
| Grants for foo@% |
+-----------------------------------------------------+
| GRANT USAGE ON *.* TO 'foo'@'%' |
| GRANT ALL PRIVILEGES ON `test`.* TO 'foo'@'%' |
+-----------------------------------------------------+
2 rows in set (0.02 sec)
Wrong access
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Error handling
mysql> select max (f1) from t1;
ERROR 1630 (42000): FUNCTION test.max does not exist. Check the
'Function Name Parsing and Resolution' section in the Reference Manual
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
+-----------+--------+----------------------------------------------------+
| Level | Code | Message |
+-----------+--------+----------------------------------------------------+
| Warning | 1292 | Truncated incorrect DOUBLE value: 'f1' |
+-----------+--------+----------------------------------------------------+
1 row in set (0.00 sec)
Errors vs warnings
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Access Privilege System
• Error information● mysql_errno● mysql_error
• Warnings● mysql_info● mysql_sqlstate● mysql_warning_count
Application (C API)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Error handling
[sveta@delly ~]$ perror 1630
MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):
FUNCTION %s does not exist. Check the 'Function Name Parsing and
Resolution' section in the Reference Manual
[sveta@delly ~]$ perror 1292
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated
incorrect %.32s value: '%.128s'
[sveta@delly ~]$ perror 2
OS error code 2: No such file or directory
[sveta@delly ~]$ perror 150
MySQL error code 150: Foreign key constraint is incorrectly formed
perror
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Error handling
● GET DIAGNOSTICS● GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;● GET DIAGNOSTICS CONDITION 1 code = RETURNED_SQLSTATE, msg = MESSAGE_TEXT;
● http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html
In stored routines
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
What can affect query execution?
• Startup options or system variables
• How optimizer creates query plan
• Storage engine used
• Parallel execution next section
You run a query, it does not return an error, but still behaves not as expected
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
• Global● Control parameters, necessary for all server processes
– Location of server files: datadir etc.
– Shared buffers
– More
• Session● Control connection-specific parameters
• http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html
Scope
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
• SET [GLOBAL] var_name = NEW_VAL
• Command line option
• Configuration file● In default location
– http://dev.mysql.com/doc/refman/5.6/en/option-files.html
● Specified by option --defaults-file
How to set
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
• Global options and few session options● A user with privilege SUPER
• Session options● Anybody
• There is no limits!
Who can change
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
• Those which control behavior of whole server● Once at server startup
● Can start with low values, then grow to specified
• Connection options● For every connection when connection opens
• Operation-specific● For every operation when needed
When allocated
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
• SHOW [GLOBAL] STATUS
• GLOBAL● Since server start
• SESSION● For operations in current session
● Can be reset
– FLUSH STATUS
How to control
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
mysql> show global status like 'Handler_read_rnd_next'\G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 27
1 row in set (0.00 sec)
mysql> show status like 'Handler_read_rnd_next'\G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 7
1 row in set (0.00 sec)
How to control
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
• Record currently used variables● SHOW [GLOBAL] VARIABLES
• Make change dynamically if possible● SET [GLOBAL] var_name=NEW_VAL
• Test in one session first
• Then change global variable
• Change configuration file after you are happy with results
Troubleshooting best practices
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
System variables and startup options
• Record currently used variables● SHOW [GLOBAL] VARIABLES
• Start mysqld with option --no-defaults● This option must be first one!
• Check if problem is solved
• Change variable values one-by-one until you find one which leads to the problem
When affecting option is not known
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Diagnostic tool: INFORMATION_SCHEMA
• Contain metadata information● Tables
● Indexes
● Other
• Allows to create plugins● InnoDB plugins
– We will discuss them later
• Oracle analog● Data Dictionary Views
When affecting option is not known
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Optimizer
• EXPLAIN is less powerful if compare with Oracle● It is improved in version 5.7.3
● Graphic EXPLAIN in MySQL Workbench 6.0
• EXPLAIN EXTENDED• EXPLAIN PARTITIONS• EXPLAIN FORMAT=JSON• INFORMATION_SCHEMA.TRACE
• Status variables 'Handler_%'
Overview
Default in 5.7
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
EXPLAIN in Oracle
EXPLAIN PLAN SET statement_id = 'example_plan4' FOR
SELECT h.order_number, l.revenue_amount, l.ordered_quantity
FROM so_headers_all h, so_lines_all l
WHERE h.customer_id = :b1
AND h.date_ordered > SYSDATE30
AND l.header_id = h.header_id ;
Plan
--------------------------------------------------
SELECT STATEMENT
NESTED LOOPS
TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL
INDEX RANGE SCAN SO_HEADERS_N1
TABLE ACCESS BY INDEX ROWID SO_LINES_ALL
INDEX RANGE SCAN SO_LINES_N1
http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
EXPLAIN in MySQL
mysql> EXPLAIN SELECT user, host FROM mysql.user\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
Extra: Using index
1 row in set (0.13 sec)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
EXPLAIN in MySQL
mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.user\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
filtered: 100.00
Extra: Using index
1 row in set, 1 warning (0.00 sec)
Variant: EXTENDED
mysql> SHOW WARNINGS\G******* 1. row ******* Level: Note Code: 1003Message: /* select#1 */ select `mysql`.`user`.`User` AS `user`,`mysql`.`user`.`Host` AS `host` from `mysql`.`user`1 row in set (0.00 sec)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
EXPLAIN in MySQL
mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.user\G
************** 1. row **************
EXPLAIN: {
"query_block": {
"select_id": 1,
"table": {
"table_name": "user",
"access_type": "index",
"key": "PRIMARY",
...
Variant: FORMAT=JSON
...
"used_key_parts": [
"Host",
"User"
],
"key_length": "228",
"rows": 4,
"filtered": 100,
"using_index": true
}
}
}
1 row in set, 1 warning (0.03 sec)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
EXPLAIN in MySQL
• http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen
• http://dev.mysql.com/doc/refman/5.6/en/explain-output.html
• MySQL EXPLAIN in Practice [HOL9232] (past)
• MySQL 5.7: What’s New in the Parser and the Optimizer? [CON2830] (past)
• How to Analyze and Tune MySQL Queries for Better Performance [TUT3157] (past)
• Using MySQL Workbench Performance Tools [HOL9237] (past)
More information
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Optimizer
mysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW STATUS LIKE 'Handler_read_%';
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Handler_read_first | 0 |
| Handler_read_key | 0 |
| Handler_read_last | 0 |
| Handler_read_next | 0 |
| Handler_read_prev | 0 |
| Handler_read_rnd | 0 |
| Handler_read_rnd_next | 0 |
+----------------------------+-------+
7 rows in set (0.00 sec)
Handler_% status variables
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Optimizer
mysql> select count(*) from employees join titles using(emp_no) where title='Senior Engineer'\G
*************************** 1. row ***************************
count(*): 97750
1 row in set (3.24 sec)
mysql> SHOW STATUS LIKE 'Handler_read_%';
+----------------------------+--------+
| Variable_name | Value |
+----------------------------+--------+
| Handler_read_first | 1 |
| Handler_read_key | 300027 |
| Handler_read_last | 0 |
| Handler_read_next | 397774 |
...
Handler_% status variables
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Optimizer
• INFORMATION_SCHEMA.TRACE● http://dev.mysql.com/doc/internals/en/optimizer-tracing.html
● join_preparation, join_optimization, join_execution– considered_execution_plans, refine_plan, more
• Query Analyzer in MEM● http://dev.mysql.com/doc/mysql-monitor/3.0/en/mem-quan-using.html
• Graphic EXPLAIN in MySQL Workbench 6.0
Other tools
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Storage Engines
• Own way to handle● Corruption
● Index statistics
● CHECK TABLE to check for errors
• They care about physical data, so all data information are on their level
• Options usually start from engine name● myisam_*, innodb_*, custom_*
Specifics
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Storage Engines
• Transactional storage engine
• Physical layout● *frm file – table definition
● Shared tablespace
● *ibd file – tablespace for individual table
– Optional: --innodb_file_per_table– Recommended and default since 5.6
● Redo log files
• OPTIMIZE TABLE = ALTER + ANALYZE• Automatic startup check
InnoDB
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
High concurrency issues
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
High concurrency issues
• Query or transaction waits a lock, held by another one
• Fight for system resources
• Resource overload
• Resource underload
Common problems
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Transactions and their relation to locks
• Lock types
• Levels
–MDL
– Table-level
– Row-level
• What do they lock–Read locks
– Block writes
–Write locks
– Block both reads and writes
● Transactions● Server-level
– MDL locks● Engine level
– Table locks
– Row locks● AUTOCOMMIT
● Supported
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Locking issues
• SHOW [FULL] PROCESSLIST● Universal tool which lists all currently running connections
• SHOW ENGINE INNODB STATUS• INFORMATION SCHEMA
● PROCESSLIST● InnoDB tables
• PERFORMANCE SCHEMA● MDL locks
● Table locks
● Server-level transactions
Diagnostic tools
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Storage engine specifics
• Transactions and engine-level locks are done at the engine level
• Storage engines have own diagnostic tools
• Use them when hit an issue with such a lock
Diagnostic tools
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Storage engine specifics
• SHOW ENGINE INNODB STATUS• InnoDB monitors
● Pseudo-tables
– Turn periodical logging into error log● CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB;
– Lock monitor● Changes output format of InnoDB Monitor● CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB;
● Options
– innodb_status_output– innodb_status_output_locks
InnoDB
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Storage engine specifics
• Tables in INFORMATION_SCHEMA● INNODB_TRX● INNODB_LOCKS● INNODB_LOCK_WAITS● INNODB_METRICS
– Options innodb_monitor_*● Option innodb_print_all_deadlocks
InnoDB
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Diagnostic tool: performance_schema
• Monitors internal operations● Events
● Waits
● Mutexes
● Statements
● Stages
● MDL, table-level locks, transactions
● Memory
● Replication
• Similar to Oracle wait interface
For whole server
5.7+
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Diagnosting locks
• Table-level locks● PROCESSLIST: “Waiting for table lock”
● P_S.TABLE_HANDLES
• Row-level locks● InnoDB monitors, SHOW ENGINE INNODB STATUS● Tables in INFORMATION_SCHEMA● Option --innodb_print_all_deadlocks
• MDL locks● PROCESSLIST: “Waiting for metadata lock”
● P_S.METADATA_LOCKS
Summary
5.7+
5.7+
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
High availability solutions
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Cluster
• Special storage engine: NDB
• Stores data on two or more physical machines
• Two or more copies
• General troubleshooting techniques● Applicable
• Specific NDB storage engine techniques
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
MySQL Replication
• Always available, but you must setup it before use
• Asynchronous master-slave
• Master● Keeps all updates in separate file: binary log
• Slave● IO thread read updates from master and stores in relay log file
● SQL thread executes updates
Overview
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Replication IO thread
• Access error● Check slave error log
● SHOW SLAVE STATUS● P_S.replication_connection_status● Try to connect using normal MySQL client using slave credentials
– SHOW GRANTS● Fix privileges on master
● Restart slave
Communication issues
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Replication SQL thread
• Simple master-slave● Data is different on master and slave
– Replication event can not be applied● Different errors on master and slave
● Slave lags far behind the master
• Circular replication or other writes in addition to slave SQL thread● Data is different on master and slave
Typical issues
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Replication SQL thread
• Was the table modified besides the SQL thread?● How?
● Can it affect content of the table in the wrong way?
• Are the table definitions same on master and slave?● MySQL Utilities
– mysqlrplsync, mysqldbcompare, mysqldiff
• Is it possible that master events was applied in wrong order?● Use mysqlbinlog to find queries which caused the issue
● Check master's application to find what is wrong
Data is different on master and slave
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Replication SQL thread
• Lock issues● InnoDB
• Triggers● SET GLOBAL slave_skip_counter● Synchronize tables!
• Different options● Start slave with master's options, then check
Events from master were applied in wrong order
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Replication SQL thread
• Threads● Master runs in multiple update threads
● Slave uses single
• Seconds_behind_master is growing
• Tune slave performance● Buffers
● Indexes (for statement-based replication)
Slave lags far behind the master
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
More information
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
OpenWorld sessions
• Practical MySQL Optimization [CON5522]● Peter Zaitsev, 3:30 PM, Moscone South - 262
• What’s New in MySQL 5.7 Security [CON1985] ● Georgi Kodinov, 4:45 PM, Moscone South - 262
• MySQL Cost Model [CON3163]● Olav Sandstå, 5:30 PM, Moscone South - 250
October, 01
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
OpenWorld sessions
• MySQL Troubleshooting with the Performance Schema [HOL9215]● Sveta Smirnova, Lig Isler-turmelle
● 8:30 AM, Hotel Nikko - Monterey
• MySQL Enterprise Edition Features in Practice [HOL9216]● Matt Lord, 10:00 AM, Hotel Nikko - Monterey
• MySQL Performance: Demystified Tuning and Best Practices [CON5097] ● Dimitri Kravtchuk, 10:45 AM, Moscone South - 252
October, 02
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Further reading
• MySQL Troubleshooting book● http://shop.oreilly.com/product/0636920021964.do
• Marc Alff's Performance Schema blog● http://marcalff.blogspot.ru/
• Planet MySQL● http://planet.mysql.com/
● http://dev.mysql.com/support/blogs/
● http://mysqlserverteam.com/
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
References
• MySQL User Reference Manual● http://dev.mysql.com/doc/refman/5.6/en/index.html
• Knowledge Management Database
• Forums● http://forums.mysql.com
• Bug trackers● http://bugs.mysql.com
● Oracle Internal Bugs database
• My Oracle Support● https://support.oracle.com
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
You will find me at
• https://twitter.com/svetsmirnova
• https://blogs.oracle.com/svetasmirnova/
• http://www.slideshare.net/SvetaSmirnova/
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
?
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Thank you!
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |