1.

2. MySQL Monitoring Mechanisms: How the different pieces fit together Mark Leith (@MarkLeith) 3. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle. 4. Program Agenda

Overview of Past MySQL Monitoring

5. The New MySQL Monitoring Vision 6. Performance Schema 7. New InnoDB Instrumentation 8. Optimizer Trace 9. Enhanced EXPLAIN 10. Other Goodies 11. Summary 12. A long time ago in a galaxy far, far away.... MySQL Episode IV.IThe SHOW statement continues to rule supreme...USE THE FORCE LUKE!! 13. Overview of Past MySQL Monitoring Prior to 5.5 instrumentation within MySQL was limited

Various SHOW statements allowed some global visibility

14. INFORMATION_SCHEMA for (mostly) object metadata 15. Slow / General query logs for statement history 16. EXPLAIN / SHOW PROFILE for statement profiling 17. SHOW Statements

Great for simple commands to get metadata, i.e:

• SHOW CREATE TABLE ...;

18. SHOW TABLES;

Not so great for scripting or tooling, i.e:

• SHOW ENGINE INNODB STATUS parsinghurts

19. Can not filter, or re-order columns

20. Can not use dynamically (prepared statements etc.) 21. INFORMATION_SCHEMA

Added within 5.0+

It's just SQL and (virtual) tables..

Can be used dynamically (prepared statements etc.)

22. Order output independently (columns, rows, etc.) Plugin interface available

Optimization path is unique, and disk accesses are high with some tables (I_S.TABLES etc.)

23. Slow / General Query Logging

Available in all versions of MySQL

24. Allow analyzing problems after the fact

Not nearly enough information within them for serious diagnosis (even with patches)

25. Overhead can be excessive

General query log bottleneck at start of statements

26. CSV based log tables in 5.1+ are not ideal 27. EXPLAIN / SHOW PROFILE

Good high-level overviews of how a query runs

28. Allows some response time break downs

Table based outputs belie true execution flows

29. Not enough statistics are reported

EXPLAIN doesn't show all optimizer decisions

30. EXPLAIN doesn't support all statement types 31. SHOW PROFILE often isn't fine grained enough 32. MySQL Episode V.V A NEW HOPE 33. 5.5+ Instrumentation

5.5 saw a huge investment in instrumentation

34. ThePerformance Schemaframework was introduced

Many finer detail instrumentation points added

35. Focused on IO and internal synchronization pointsNew INFORMATION_SCHEMA tables for InnoDB

Current transaction and locking statistics

36. Compression statistics 37. 5.6 And Beyond Accelerating investment in instrumentation and tooling

Fill more gaps in Performance Schema statistics

38. Expose even more detailed InnoDB statistics 39. Take the guess work out of optimizer workings 40. Increase flexibility of instrumentation configuration 41. Provide more structured data for richer tooling 42. 5.6 And Beyond Instrumentation Vision

SHOW - Object Metdata, limited extension otherwise

43. INFORMATION_SCHEMA Standard Object Metadata 44. INFORMATION_SCHEMA Plugin Specific Extensions 45. PERFORMANCE_SCHEMA Consolidated Statistics 46. Performance Schema 47. Performance Schema in 5.5

Foundation for a powerful instrumentation framework

48. Similar in goals to the Oracle Wait Interface 49. TracksEventsby time/latency (to picosecond precision) 50. Entirely non-locking, and in fixed memory 51. Implemented as a real storage engine

No unique optimization paths

52. Not persistent 53. Performance Schema in 5.5 What is an Event ?

At it's most basic, a measurement of a unit of work

In all cases this includes time taken to complete

54. In some we record other things, like bytes requested Naming roughly follows Linnean Taxinomy

/Class/Order/Family/Genus/Species

55. Performance Schema in 5.5

First instrumentation is more MySQL developer focused

56. Started at some of the smallest internal units of work

Mutexes -wait/synch/mutex/

57. Read/Write locks -wait/synch/rwlock/ 58. Conditions -wait/synch/cond/ File IO instrumented as well, across all file types

• wait/io/file/

• Tracks both time waited and bytes per request

59. Performance Schema in 5.5 THREAD_ID:17 EVENT_ID:57265474 EVENT_NAME:wait/synch/mutex/innodb/kernel_mutex SOURCE:lock0lock.c:5489 TIMER_START:1716417836794180 TIMER_END:1716417836828110 TIMER_WAIT:33930 SPINS:NULL OBJECT_SCHEMA:NULL OBJECT_NAME:NULL OBJECT_TYPE:NULL OBJECT_INSTANCE_BEGIN:4307564056 NESTING_EVENT_ID:NULL OPERATION:lock NUMBER_OF_BYTES:NULL FLAGS: 0 Mutex Event ~33 nanosecs Toacquirelock 60. Performance Schema in 5.5 SELECT event_name, ROUND(SUM(sum_timer_wait) / 1000000000000, 2) total_sec, ROUND(MIN(min_timer_wait) / 1000000, 2) min_usec, ROUND((SUM(sum_timer_wait) / SUM(COUNT_STAR))/1000000, 2) avg_usec, ROUND(MAX(max_timer_wait) / 1000000, 2) max_usec FROMperformance_schema.events_waits_summary_global_by_event_name WHERE sum_timer_wait > 0GROUP BY event_name ORDER BY total_sec DESC limit 10; Top waits bylatency 61. Performance Schema in 5.5 THREAD_ID:6 EVENT_ID:1036 EVENT_NAME:wait/io/file/innodb/innodb_data_file SOURCE:os0file.c:4891 TIMER_START:338825682207740 TIMER_END:338825703463000 TIMER_WAIT:21255260 SPINS:NULL OBJECT_SCHEMA:NULL OBJECT_NAME:/Users/mark/sandboxes/msb_5_5_16/data/ibdata1 OBJECT_TYPE:FILE OBJECT_INSTANCE_BEGIN:4683519488 NESTING_EVENT_ID:NULL OPERATION:write NUMBER_OF_BYTES:32768 FLAGS:0 File IO Event ~21 microsecs To write 32K On this file 62. Performance Schema in 5.5 SELECT substring_index(file_name, '/', -1) AS file,count_read,count_write,ROUND(sum_number_of_bytes_read / (1024*1024)) read_mb,ROUND(sum_number_of_bytes_write / (1024*1024)) write_mb, ROUND((sum_number_of_bytes_read +sum_number_of_bytes_write) / (1024*1024*1024)) total_gb FROMperformance_schema.file_summary_by_instanceORDER BY total_gb DESC LIMIT 10; Top filesby IOconsumption 63. Performance Schema in 5.5

Proved the model, can handle the flood

64. Already aided finding bottlenecks 65. But..

Needed more DBA/Developer focused statistics

66. Needed more control on who/what is traced 67. Needed a little tuning for high load systems 68. Performance Schema in 5.6

Adding DBA / Developer focused instrumentation

69. With the ability to drill down between each level 70. Statements Latency and key statistics

Stages Statement execution states (i.e. Sending data )

Network IO / Idle time- at the TCP / socket layers

71. Table IO- at the storage engine handler layer 72. Table Locks- at the storage engine handler layer 73. .. as well asfile IO ,mutexes ,conditions ,rwlocks 74. Performance Schema Statement Statistics 75. Performance Schema Stages The stage (state) Within statement execution When it started,ended, and How long it took The statement ID it was related to 76. Performance Schema in 5.6 Event Nesting

Events are related in a hierarchy

77. Currently uses the Adjacency List model 78. EVENT_ID (child) -> NESTING_EVENT_ID (parent) 79. Investigating the Nested Set model 80. Performance Schema in 5.6 Event Nesting 81. Performance Schema in 5.6 Graph Events http://www.markleith.co.uk/?p=471 The Statement The Stages The individual wait events by type 82. Performance Schema in 5.6 Graph Events 83. Performance Schema in 5.6 Summaries Summarize (GROUP BY) each class of event byuser, host, account ( [email_address] ), thread, and globally New Object, Network IO Table IO and Table Lock Wait summaries 84. Performance Schema Table IO Summary The table and it's high level aggregated stats A breakdown By table IO type 85. Not everything that can be counted counts,and not everything that counts can be counted. Albert EinsteinYet 60 nanoseconds of time could mean SO MUCH! 86. Performance Schema Configuration

There is always some overhead with instrumentation

87. Whether, and how, we time events can be important 88. Set expectations up front on both:

What you want to monitor

89. How selectively you want to monitor it 90. Performance Schema Overhead http://marcalff.blogspot.com/2011/06/performance-schema-overhead-tuning.html June 2011 Lots of tuning done since this. New benchmarksshould be out soon 91. Performance Schema Setup 92. Performance Schema What's Missing?

Storage engine level lock latency i.e. InnoDB row locks

93. User lock latency i.e. SELECT GET_LOCK(); 94. A normalized statement level digest 95. Transactions level statistics (tied to global transaction ID) 96. Memory allocations 97. Distribution statistics (histograms) 98. Background thread stages, server cache statistics 99. ...and...and...and... -We're listening, tell us your needs 100. New InnoDB Instrumentation 101. New InnoDB Instrumentation

MySQL 5.6:11 New INFORMATION_SCHEMA tables

7Data Dictionaryrelated

102. 3Buffer Poolrelated 103. 1General Statisticsgold mine

MySQL 5.5:7 New INFORMATION_SCHEMA tables

3Transaction / Lockingrelated

104. 4Compressionrelated 105. InnoDB System Tables http://dev.mysql.com/doc/refman/5.6/en/innodb-sys-tablestats-table.html View InnoDB's internal Data Dictionary and table stats 106. INNODB_SYS_TABLES[TATS] SELECT innodb_sys_tables.name,CASEflag >> 5WHEN 0 THEN 'Antelope' WHEN 1 THEN 'Barracuda' WHEN 2 THEN 'Cheetah' END AS file_format, @page_flags :=(flag >> 1) & 15AS page_flags,@page_size:=IF(@page_flags = 0,16384,512 0 ORDER BY total_mb DESC; 109. INNODB_METRICS

Consolidated InnoDB statistics

110. 199 metrics currently collected ( suggest just one more! ) 111. More flexible statistics collection than SHOW STATUS

Can enable/disable individually

112. Track when / how long each has been enabled 113. Provide extra statistics inline per metric 114. Provide extra metadata inline per metric 115. INNODB_METRICS Metric Details http://dev.mysql.com/doc/refman/5.6/en/innodb-metrics-table.html The variable Stats since start Stats since reset How long enabled Stat Metadata 116. INNODB_METRICS Configuration Enable named metric Disable named metric Enable metric module Reset metric module 117. INNODB_METRICS Looking at Counters http://dev.mysql.com/doc/refman/5.6/en/innodb-metrics-table.html SELECT concat(subsystem,'/', name)AS event,count,ROUND(count/ time_elapsed) AS avg_per_secondFROMinformation_schema.innodb_metrics WHERE status = 'enabled' AND type LIKE '%counter%' OR type = 'set_owner' ORDER BY avg_per_second DESC; 118. Optimizer Trace 119. Optimizer Trace

EXPLAIN tells youhow a query will execute

120. Optimizer Trace tells youwhy it will execute that way 121. New INFORMATION_SCHEMA.OPTIMIZER_TRACE 122. Contains trace of optimizer decisions, in JSON format 123. Enabled per thread, on demand 124. Targeted for advanced users, or in use with bug reporting 125. Optimizer Trace Enable Tracing Run Statement Select the trace 126. Optimizer Trace Expose querytransformations 127. Optimizer Trace Consideredplans With costs Estimations Of rows And costs WhetherIndexes are Usable ornot 128. Enhanced EXPLAIN 129. Enhanced EXPLAIN Plans

In the past EXPLAIN was only supported with SELECT

130. To EXPLAIN an UPDATE - Re-write equivalent SELECT

However optimization paths may be different

Now you can EXPLAIN:

UPDATE | DELETE | REPLACE | INSERT

131. Enhanced EXPLAIN Plans

Structured EXPLAIN is in development

132. A properly structured EXPLAIN plan (again, JSON) 133. Sorry no examples yet, it's very early in implementation 134. Other Goodies 135. Instance UUIDs / SHOW SLAVE STATUS

Automatically generated unique instance identifiers

136. Persisted in $datadir/auto.cnf 137. Can be used for reliable replication topology discovery

Hard in past given IP/Host matching (virtual IPs?)

More configuration and thread statistics 138. Q&A 139. 140.