- 1. Monitoring IO performanceusing iostat &
pt-diskstatsMySQL Conference & Expo 2013Ben MildrenMySQL Team
Technical Lead
2. About Me Ben Mildren Team Technical Lead at Pythian Over 10
years experience as a productionDBA. Experience of MySQL, SQL
Server, andMongoDB.Email: [email protected]:
benmildrenTwitter: @productiondbaSlideshare:
www.slideshare.net/benmildren2 3. Recognized Leader: Global
industry-leader in remote database administration services
andconsulting for Oracle, Oracle Applications, MySQL and Microsoft
SQL Server Work with over 250 multinational companies such as
Forbes.com, Fox Sports,Nordion and Western Union to help manage
their complex IT deployments Expertise: Pythians data experts are
the elite in their field. We have the highestconcentration of
Oracle ACEs on staff10 including 2 ACE Directorsand 2Microsoft
MVPs. Pythian holds 7 Specializations under Oracle Platinum Partner
program,including Oracle Exadata, Oracle GoldenGate & Oracle
RAC Global Reach & Scalability: Around the clock global remote
support for DBA and consulting, systemsadministration, special
projects or emergency responseWhy Pythian?3 4.
LatencyComparisonNumbersL1cachereference0.5nsBranchmispredict5nsL2cachereference7ns14xL1cacheMutexlock/unlock25nsMainmemoryreference100ns20xL2cache,200xL1cacheCompress1KbyteswithZippy3,000nsSend1Kbytesover1Gbpsnetwork10,000ns0.01msRead4KrandomlyfromSSD*150,000ns0.15msRead1MBsequentiallyfrommemory250,000ns0.25msRoundtripwithinsamedatacenter500,000ns0.5msRead1MBsequentiallyfromSSD*1,000,000ns1ms4XmemoryDiskseek10,000,000ns10ms20xdatacenterroundtripRead1MBsequentiallyfromdisk20,000,000ns20ms80xmemory,20XSSDSendpacketCA>Netherlands>CA150,000,000ns150ms*Assuming~1GB/secSSDCreditByJeffDean:http://research.google.com/people/jeff/OriginallybyPeterNorvig:http://norvig.com/21days.html#answers(https://gist.github.com/jboner/2841832)Why
the interest in IO monitoring?4 5. Monitoring IO on Linux There are
a number of tools available to monitor IO on Linux.
Thispresentation looks at two tools, iostat and pt-diskstats. These
toolslook to provide an overview of block devices iops, throughput
andlatency. You can dive deeper: Tools such as iotop and atop can
be used to expose processlevel IO performance by gathering data
from /proc/[process]/io. blktrace can be used with blkparse, either
independently or byusing btrace. Output can also be analysed using
seekwatcher. Tools such as bonnie/bonnie++, iometer, iozone, and
ORIONcan be used to benchmark a block device.5 6. iostat iostat is
part of the sysstat utilities which is maintained by
SebastienGodard. sysstat is open source software written in C, and
is available underthe GNU General Public License, version 2. Other
utilities in the sysstat package include mpstat, pidstat,
sar,nfsiostat, and cifsiostat. sysstat is likely available from
your favourite repo, but the latestversion can be found
here:http://sebastien.godard.pagesperso-orange.fr/download.html If
you have an old (or very old) version installed, it is
recommendedyou install the latest stable version.6 7. iostat At the
time of writing the current stable version is 10.0.5. Note version
10 removes support for kernels older than 2.6. The version you have
installed can be found with -V
option.[ben@lab~]$iostatVsysstatversion10.0.3(C)SebastienGodard(sysstatorange.fr)7
8. iostat By default iostat produces two reports; the CPU
Utilization reportand the Device Utilization report. The default
invocation shows the statistics since system start
up.[ben@lab~]$iostatLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)avgcpu:%user%nice%system%iowait%steal%idle17.562.933.622.420.0073.47Device:tpskB_read/skB_wrtn/skB_readkB_wrtnsda8.51100.1744.62890987396917dm04.8778.766.8070056560488dm17.4020.7437.45184505333128dm20.140.190.37169333008
9. iostat Inclusion of the CPU and Device Utilization reports can
becontrolled with the -c and -d options. The Network Filesystem
report (-n) was deprecated in version 10and replaced with the
nfsiostat and cifsiostat
utilities.[ben@lab~]$iostatcLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)avgcpu:%user%nice%system%iowait%steal%idle17.012.663.692.350.0074.29[ben@lab~]$iostatdLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:tpskB_read/skB_wrtn/skB_readkB_wrtnsda8.2894.6045.60926915446773dm04.6575.006.3473486162104dm17.2819.0038.75186129379644dm20.150.170.51170150249
10. iostat The default Device Utilization report can be replaced
with extendedstatistics using the -x
option.[ben@lab~]$iostatcx|[ben@lab~]$iostatxLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)avgcpu:%user%nice%system%iowait%steal%idle16.232.273.672.200.0075.63Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.342.673.334.3981.1545.4432.810.5166.4129.5994.307.776.00dm00.000.003.240.8264.105.7234.380.3791.2935.84309.174.942.01dm10.000.001.345.7716.5439.1415.660.3143.2032.9545.596.424.56dm20.000.000.050.100.150.589.560.0168.8126.9689.1416.180.25[ben@lab~]$iostatdxLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.342.673.324.3981.0945.4232.800.5166.4029.5994.277.776.00dm00.000.003.240.8264.055.7134.380.3791.2935.84309.174.942.00dm10.000.001.345.7716.5339.1215.660.3143.2032.9545.586.424.56dm20.000.000.050.100.150.589.560.0168.7526.9689.0016.210.2510
11. iostat Whilst reviewing the stats since since system start up
can be useful,more often you will want to review current activity.
Current activity can be displayed by specifying an interval
measuredin seconds. Output will continue until interrupted or a
specified count has
beenreached.[ben@lab~]$iostatdx[interval][count][ben@lab~]$iostatdx3The
above command will display extended device statistics since system
start up in the firstreport, and the deltas for the last 3 seconds
in subsequent reports until interrupted.[ben@lab~]$iostatdx33The
above command will display extended device statistics since system
start up in the firstreport, and the deltas for the last 3 seconds
in two further reports.11 12.
iostat[ben@lab~]$iostatdx33Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm20.000.000.000.000.000.000.000.000.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.7712
13. iostat Since version 7.1.3 the report can be made more readable
byincluding the registered device mapper names using the -N option.
Using the -N can skew the columns on each line, so it can be
usefulto also specify the -h option to keep the report easily
readable. Adding -k or -m will specify kB / mB per second
respectively.(If the POSIXLY_CORRECT environment variable is NULL
the dataread / written is displayed in kB by default) Adding the -t
option will include a timestamp with each report. Adding the -z
option will exclude any inactive devices from theindividual
reports. In version 10.1.3 (currently development version), adding
the -yoption suppresses the first report showing statistics since
systemstart up.13 14.
iostat[ben@lab~]$iostatdxNhtz32Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)04/10/201311:37:08AMDevice:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.072.632.674.1765.0443.6031.760.4363.1729.5484.728.065.52vg_proddbalv_root0.000.002.590.7451.364.9433.800.3089.5135.81276.285.091.70vg_proddbalv_home0.000.001.085.5813.2737.9315.360.2841.3232.8242.976.394.26vg_proddbalv_tmp0.000.000.040.110.120.7311.040.0160.8426.9672.7015.280.2304/10/201311:37:11AMDevice:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.003.330.003.330.0034.6720.800.0928.100.0028.1016.705.57vg_proddbalv_home0.000.000.006.000.0034.6711.560.1016.000.0016.009.285.5714-N
(registered device mapper name)-h (columns stay aligned even with
long device names)-t (timestamp reported with sample)-z (report
only includes active devices for this sample) 15. iostat In version
10.0.5, the devices in the Device Utilization can begrouped using
the -g option. It is possible to specify multiple groups by
supplying the -g optionmultiple times. Its probably easier not to
use the -N option, as the group devicesnamed would have to
reference the registered device mappernames. Using the -T option
will display the group totals only.15 16.
iostat[ben@lab~]$iostatVsysstatversion10.0.5(C)SebastienGodard(sysstatorange.fr)[ben@lab~]$iostatdxgMyLVMdm0dm1dm2gOthersdaLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utildm00.000.002.280.7645.534.7333.150.2480.1232.12224.365.081.54dm10.000.000.865.4010.5737.4515.360.2539.9632.6041.136.404.00dm20.000.000.030.160.091.0211.630.0149.6426.9654.0014.570.28MyLVM0.000.003.166.3156.1943.2020.970.5053.0132.2063.446.151.94sda0.832.592.364.0756.4243.2030.960.3757.7326.6775.738.145.24Other0.832.592.364.0756.4243.2030.960.3757.7326.6775.738.145.24[ben@lab~]$iostatdxTgMyLVMdm0dm1dm2gOthersdaLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilMyLVM0.000.003.166.3156.1843.2020.970.5053.0132.2063.446.141.94Other0.832.592.364.0856.4143.2030.960.3757.7326.6775.728.145.2416
17. iostat The Device Utilization report can be produced on a
partition levelusing the -p option. Historically partition
statistics were restricted when moving from the2.4 kernel to 2.6
kernel, however the enhanced statistics were madeavailable again
from the 2.6.25 kernel. The -p option was mutually exclusive to the
-x option up untoversion 8.1.8 The partition focused report, can be
limited to specific devices butnot specific partitions. Currently
the group options (-g and -T) dont work well with thepartition
focused report.17 18.
iostat[ben@lab~]$iostatpdxLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda5.404.9411.359.14398.2687.9947.462.33113.7556.49184.876.2312.76sda10.030.000.230.000.930.008.080.0015.8815.7660.0015.870.36sda20.020.000.170.000.780.008.930.0010.7810.780.0010.780.19sda35.364.9410.848.19396.1287.9950.902.29120.1158.50201.695.219.92dm00.000.009.892.11311.619.3753.512.86238.35119.45796.494.745.69dm10.000.006.2111.3483.3178.0918.390.9051.5624.0466.645.199.10dm20.000.000.250.130.940.537.680.0136.0727.5351.9911.560.44[ben@lab~]$iostatpsdadxLinux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda2.684.746.529.47235.6090.1140.731.4590.4850.74117.866.9211.07sda10.010.000.110.000.450.008.080.0015.8815.7660.0015.870.18sda20.010.000.080.000.370.008.930.0010.7810.780.0010.780.09sda32.664.746.288.40234.5690.1144.231.4095.4552.19127.775.227.6718
19. pt-diskstats pt-diskstats is part of the percona toolkit.
pt-diskstats is open source software written in Perl, and is
availableunder the GNU General Public License, version 2. Other
utilities in the percona toolkit include pt-stalk,
pt-table-checksum, pt-table-sync, pt-query-digest, and pt-summary.
Percona toolkit can be downloaded
fromhttp://www.percona.com/software/percona-toolkit/or more simply
from the command line:wgetpercona.com/get/perconatoolkit.tar.gz
Tools can also be downloaded
individually:wgetpercona.com/get/TOOLe.g.
wgetpercona.com/get/ptdiskstats19 20. pt-diskstats At the time of
writing the current stable version of percona toolkit isversion
2.1.1. Full documentation of pt-diskstats can be found
here:http://www.percona.com/doc/percona-toolkit/2.1/pt-diskstats.html
The version of pt-diskstats you have installed can be found
with--version
option.[ben@lab~]$./ptdiskstatsversionptdiskstats2.2.120 21.
pt-diskstats By default pt-diskstats produces useful stats
reporting currentdevice activity. Inactive devices are hidden, but
subsequently addedif they become active. The default invocation
shows the statistics for the last interval (whichby default is 1
second), and will continue until interrupted. Similar to: iostat
-dxy
1[ben@lab~]$./ptdiskstats#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime1.0sda0.00.00.00%0.00.01.08.00.050%0.00.04%11.05.817.51.0sda30.00.00.00%0.00.01.08.00.050%0.00.00%01.00.00.01.0dm20.00.00.00%0.00.03.04.00.00%0.00.04%13.02.911.71.0sda1.04.00.00%0.172.013.04.00.132%0.211.125%014.00.712.31.0sda31.04.00.00%0.172.07.07.40.146%0.01.49%08.00.06.41.0dm11.04.00.00%0.172.016.03.00.00%0.18.821%017.00.412.11.0dm20.00.00.00%0.00.00.00.00.00%0.10.04%00.00.00.01.0sda0.00.00.00%0.00.01.028.00.086%0.00.04%11.00.65.01.0sda30.00.00.00%0.00.01.028.00.086%0.00.00%01.00.00.01.0dm10.00.00.00%0.00.00.00.00.00%0.00.00%00.00.00.01.0dm20.00.00.00%0.00.00.00.00.00%0.00.00%00.00.00.021
22. pt-diskstats The interval can be adjusted using the --interval
option The number of samples can be limited using the --iterations
option. Similar to: iostat -dxy 3
3[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.522
23. pt-diskstats Displayed columns can be restricted using perl
regex with the--columns-regex option. Displayed devices can be
restricted using perl regex with the--devices-regex option. Adding
the --show-timestamps option will include a timestamp witheach
report. The output can be grouped by sample or by disk using the
--group-by option. Samples (of /proc/diskstats) can be saved for
later analysis usingthe --save-samples option. Contrary to the
documentation --version-check is enabled bydefault, if you wear a
tin foil hat, disable using --noversion-check.23 24. The interlude:
A primer in Linux IO24Read http://kernel.dk/blk-mq.pdf for recent
development 25. A primer in Linux IO Applications in the user space
make requests to the VFS. The request is passed to the block layer
if the page is not in thepage cache or the request is made using
direct IO. Requests in the block layer can be split if the request
is acrossmultiple devices, or remapped from a device partition to
theunderlying block device. Requests are handled by the IO
scheduler on a per block devicebasis. Dependent on the scheduler,
the request could be merged tothe front or back of existing
requests (all schedulers), or sorted inthe request queue (cfq &
deadline). The anticipatory scheduler wasremoved from the kernel in
version 2.6.33. Statistics are calculated at the block layer.25 26.
A primer in Linux IO To understand what is acceptable performance
theres no substituteto understanding the block device hardware. A
block device is an abstraction, potentially it could be a 10
diskarray with a hardware RAID controller, it could be a LUN
exposedfrom a SAN, even if its a single disk, the expected disk
performancecould vary greatly dependent on its specification.
Expected IOPs, latency and throughput can be gathered viabenchmarks
or estimated using calculations:
http://www.techish.net/hardware/iops-calculator-and-raid-calculators-estimators/
http://www.wmarow.com/strcalc/ As a rough estimate you can
expect:75-100 iops from a 7200 rpm disk125-150 iops from 10k rpm
disk175-200 iops from 15k rpm disk1000s iops from SSD
(++++++++++)26 27. Block layer disk statistics From the 2.6 kernel,
statistics are held for all block devices andpartitions in
/proc/diskstats. /proc/diskstats lists the block devices major
number, minor number,and name as well as a statistic set of 11
counters. Prior to 2.6.25 the statistic set of partitions was only
made up of 4counters and the counters werent consistent with the
underlyingblock device statistics. Statistics are also held for
individual devices and partitions in sysfs. /sys/block/[dev]/stat
holds the statistic set for the device.
/sys/block/[dev]/[partition]/stat holds the statistic set for the
devicepartition.27 28. /proc/diskstats
(2.6.25+)[ben@lab~]$cat/proc/diskstats70loop00000000000071loop10000000000072loop20000000000073loop30000000000074loop40000000000075loop50000000000076loop60000000000077loop70000000000080sda44783154702257302121071185999542241808924608767501087763729834981sda146316341766464204106215646582sda22673121364146000004053414683sda343885152762249646119736973520542241808920557562006545526772954110sr0000000000002530dm0427360179622613913251541401876563199366030400145906972531dm1164760449218530482113707015720324549033083821750795242532dm25740341015473374704923218556006139920103428
29.
/sys/block/[dev]/stat[ben@lab~]$cat/sys/block/sda/stat450001547022622941214926903535675019062766249938011331197464821[ben@lab~]$cat/sys/block/sda/sda1/stat463163417664642041062156465[ben@lab~]$cat/sys/block/sda/sda2/stat26731213641460000040534146[ben@lab~]$cat/sys/block/sda/sda3/stat4410215276225463812015847738256796190729657153140679274691685729
30. Block layer disk statisticsField 1 read_IOs: Total number of
reads completed (requests)Field 2 read_merges: Total number of
reads merged (requests)Field 3 read_sectors: Total number of
sectors read (sectors)Field 4 read_ticks: Total time spent reading
(milliseconds)Field 5 write_IOs: Total number of writes completed
(requests)Field 6 write_merges: Total number of writes merged
(requests)Field 7 write_sectors: Total number of sectors written
(sectors)Field 8 write_ticks: Total time spent writing
(milliseconds)Field 9 in_flight: The number of I/Os currently in
flight. It does not include I/Orequests that are in the queue but
not yet issued to the device driver. (requests)Field 10 io_ticks:
This value counts the time during which the device has had
I/Orequests queued. (milliseconds)Field 11 time_in_queue: The
number of I/Os in progress (field 9) times the numberof
milliseconds spent doing I/O since the last update of this field.
(milliseconds)30 31. iostat extended
statistics[ben@lab~]$iostatdx33Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm20.000.000.000.000.000.000.000.000.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.7731
32. iostat extended
statistics[ben@lab~]$iostatdx33Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm20.000.000.000.000.000.000.000.000.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.7732
33. iostat - extended statistics
rrqm/s(requests)delta[read_merges(f2)]/interval
wrqm/s(requests)delta[write_merges(f6)]/interval
r/s(requests)delta[read_IOs(f1)]/interval
w/s(requests)delta[write_IOs(f5)]/interval33 34. iostat extended
statistics[ben@lab~]$iostatdx33Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm20.000.000.000.000.000.000.000.000.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.7734
35. iostat - extended statistics
rkB/s(sectors|kB|mB)(delta[read_sectors(f3)]/interval)/conversionfactor
wkB/s(sectors|kB|mB)(delta[write_sectors(f7)]/interval)/conversionfactor
avgrqsz(sectors)delta[read_sectors(f3)+write_sectors(f7)]/delta[read_IOs(f1)+write_IOs(f5)](or0.0ifnoIO)35
36. iostat extended
statistics[ben@lab~]$iostatdx33Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm20.000.000.000.000.000.000.000.000.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.7736
37. iostat - extended statistics
avgqusz(requests)(delta[time_in_queue(f11)]/interval)/1000.0
await(milliseconds)delta[read_ticks(f4)+write_ticks(f8)]/delta[read_IOs(f1)+write_IOs(f5)](or0.0ifnoIO)37
38. iostat extended
statistics[ben@lab~]$iostatdx33Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm20.000.000.000.000.000.000.000.000.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.7738
39. iostat - extended statistics
r_await(milliseconds)delta[read_ticks(f4)]/delta[read_IOs(f1)](or0.0ifnoreadIOs)
w_await(milliseconds)delta[write_ticks(f8)]/delta[write_IOs(f5)](or0.0ifnowriteIOs)39
40. iostat extended
statistics[ben@lab~]$iostatdx33Linux3.8.4102.fc17.x86_64(lab.mysqlhome)
04/10/2013 _x86_64_
(2CPU)Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda1.162.662.894.2770.3744.6632.100.4664.3729.5587.967.945.69dm00.000.002.800.7855.535.2333.940.3290.1235.83286.385.031.80dm10.000.001.175.6814.3938.7315.500.2942.1232.8144.056.384.37dm20.000.000.040.110.130.7010.680.0163.3026.9677.3415.470.24Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.670.001.000.0021.3342.670.0223.330.0023.3323.332.33dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.005.330.0021.338.000.035.250.005.254.382.33dm20.000.000.000.000.000.000.000.000.000.000.000.000.00Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.001.670.009.000.0041.339.190.4550.440.0050.446.525.87dm00.000.000.000.000.000.000.000.000.000.000.000.000.00dm10.000.000.008.330.0033.338.000.4654.920.0054.922.522.10dm20.000.000.002.000.008.008.000.0945.500.0045.5018.833.7740
41. iostat - extended statistics
svctm(milliseconds)((delta[read_IOs(f1)+write_IOs(f5)]*HZ)/interval)/(delta[IO_ticks(f10)]/interval)(or0.0iftput=0)*HZ=tickspersecond,(1000onmostsystems).**Thisfieldwillberemovedinafuturesysstatversion.
%util(percent)((delta[IO_ticks(f10)]/interval)/10)/devices*devices=1orthenumberofdevicesinthegroup(goption).41
42.
pt-diskstats[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.542
43.
pt-diskstats[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.543
44. pt-diskstats
rd_s(requests)Comparabletoiostatr/sdelta[read_IOs(f1)]/interval
rd_avkb(kB)Similartoiostatavgrqszbutisolatesreadsconversionfactor*delta[read_sectors(f3)]/delta[read_IOs(f1)](Conversionfactor=2=documentationbug)
rd_mb_s(mB)ComparabletoiostatrmB/sconversionfactor*delta[read_sectors(f3)]/interval(Conversionfactor=2=documentationbug)
rd_mrg(percent)Similartoiostatrrqm/sexpressedas%100*delta[read_merges(f2)]/(delta[read_merges(f2)]+delta[read_IOs(f1)])44
45.
pt-diskstats[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.545
46. pt-diskstats
rd_cnc(Littleslaw)iostathasnoequivalentdelta[read_ticks(f4)]/interval/1000/devicesingroup
rd_rt(milliseconds)Differstoiostatr_awaitdelta[read_ticks(f4)]/(delta[read_IOs(f1)]+delta[read_merges(f2)])46
47.
pt-diskstats[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.547
48. pt-diskstats
wr_s(requests)Comparabletoiostatw/sdelta[write_IOs(f5)]/interval
wr_avkb(kB)Similartoiostatavgrqszbutisolateswritesconversionfactor*delta[write_sectors(f7)]/delta[write_IOs(f5)](Conversionfactor=2=documentationbug)
wr_mb_s(mB)ComparabletoiostatwmB/sconversionfactor*delta[write_sectors(f7)]/interval(Conversionfactor=2=documentationbug)
wr_mrg(percent)Similartoiostatwrqm/sexpressedas%100*delta[write_merges(f6)]/(delta[write_merges(f6)]+delta[write_IOs(f5)])48
49.
pt-diskstats[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.549
50. pt-diskstats
wr_cnc(Littleslaw)iostathasnoequivalentdelta[write_ticks(f8)]/interval/1000/devicesingroup
wr_rt(milliseconds)Differstoiostatw_awaitdelta[write_ticks(f8)]/(delta[write_IOs(f5)]+delta[write_merges(f6)])50
51.
pt-diskstats[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.551
52. pt-diskstats
busy(percent)Comparabletoiostat%util100*delta[io_ticks(f10)]/(1000*interval)
in_prg(requests)iostathasnoequivalent#BIOsin_flight(f9)
ios_s(requests)Comparabletoiostatr/s+w/s(delta[readIOs(f1)]+delta[write_IOs(f5)])/interval52
53.
pt-diskstats[ben@lab~]$./ptdiskstatsinterval3iterations3#tsdevicerd_srd_avkbrd_mb_srd_mrgrd_cncrd_rtwr_swr_avkbwr_mb_swr_mrgwr_cncwr_rtbusyin_prgio_sqtimestime3.0sda0.00.00.00%0.00.01.33.00.00%0.037.24%01.39.428.23.0sda30.00.00.00%0.00.01.04.00.00%0.022.72%01.00.022.73.0dm10.00.00.00%0.00.00.00.00.00%0.00.02%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.030.02%00.70.030.03.0sda0.00.00.00%0.00.01.36.00.050%0.010.14%21.30.714.63.0sda30.00.00.00%0.00.01.08.00.057%0.01.11%11.00.86.33.0dm00.00.00.00%0.00.00.74.00.00%0.00.01%20.70.018.03.0dm10.00.00.00%0.00.01.34.00.00%0.00.01%11.31.89.03.0dm20.00.00.00%0.00.01.04.00.00%0.027.03%01.00.027.03.0sda0.00.00.00%0.00.02.33.40.00%0.143.45%02.326.420.03.0sda30.00.00.00%0.00.01.74.80.00%0.133.43%01.712.120.63.0dm00.00.00.00%0.00.00.34.00.00%0.1191.03%00.3211.092.03.0dm10.00.00.00%0.00.00.00.00.00%0.00.03%00.00.00.03.0dm20.00.00.00%0.00.00.74.00.00%0.023.52%00.70.023.553
54. pt-diskstats
qtime(milliseconds)!iostatavgquszdelta[time_in_queue(f11)]/(delta[read_IOs(f1)+read_merges(f2)+write_IOs(f5)+write_merges(f6)]+delta[in_flight(f9)])delta[io_ticks(f10)]/(delta[read_IOs(f1)+read_merges(f2)+write_IOs(f5)+write_merges(f6)])
stime(milliseconds)!iostatsvctmdelta[io_ticks(f10)]/(delta[read_IOs(f1)+read_merges(f2)+write_IOs(f5)+write_merges(f6)])54
55. An example using
iostat[ben.mildren@316403db7~]$iostatdx5Linux2.6.18194.17.1.el5(xxxx)
04/25/2013....Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.600.0016.800.0085.6010.190.063.290.003.290.060.10sdb7607.005.20253.6023.4031467.20114.40228.030.863.092.934.862.6874.10dm00.000.000.000.400.001.608.000.004.000.004.002.000.08dm10.000.000.0028.200.00112.808.000.114.070.004.070.040.12dm20.000.00374.800.0031441.600.00167.780.792.112.110.001.9874.18dm30.000.007860.400.4031441.601.608.0023.212.952.954.000.0974.16dm40.000.000.000.000.000.000.000.000.000.000.000.000.00....55
56. RAID Controller CLIs Adaptecarcconf? Dellomreport?
HPSmartArrayhpacucliType"help"foralistofsupportedcommands.Type"exit"toclosetheconsole.
LSIMegaCli? Oracleraidconfig?56 57. RAID Controller CLIs
InthiscaseitsaDellserverandcontrolleromreportstoragecontrollerControllerPERC6/iIntegrated(Embedded)ControllersID:0Status:OkName:PERC6/iIntegrated...State:ReadyFirmwareVersion:6.2.00013...DriverVersion:00.00.04.17RH1...NumberofConnectors:2...CacheMemorySize:256MB...57
58. RAID Controller CLIs
/dev/sdbisRAID5SASHDDs[root@316402db6srvadmin]#omreportstoragevdiskListofVirtualDisksintheSystem...ID:1Status:Ok...Layout:RAID5Size:836.63GB(898319253504bytes)DeviceName:/dev/sdbBusProtocol:SASMedia:HDDReadPolicy:NoReadAheadWritePolicy:WriteBack...StripeElementSize:64KBDiskCachePolicy:Disabled58
59. RAID Controller CLIs
/dev/sdbisRAID5SASHDDs[root@316402db6srvadmin]#omreportstoragepdiskcontroller=0vdisk=1ListofPhysicalDisksonControllerPERC6/iIntegrated(Embedded)ControllerPERC6/iIntegrated(Embedded)ID:0:0:2...State:OnlineFailurePredicted:No...BusProtocol:SASMedia:HDD...Capacity:136.13GB(146163105792bytes)...VendorID:DELLProductID:ST3146356SS...59
60. Calculating IOPS ST3146356SS - Cheetah 15K.6 SAS(Googled -
http://www.datasheets.pl/hard_drives/ST3450856SS.pdf)Disk rotation
speed 15k rpmAvg rotational latency 2.0ms = (1 / (15000/60)) * 0.5
* 1000Avg read access time 3.4msAvg write access time 3.9ms Looking
at omreport storage vdiskLayout RAID-5 Counting disks in omreport
storage pdisk controller=0 vdisk=1# Disks 460 61. Calculating IOPS
http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182To
calculate the IOPS range, use this formula: Average IOPS: Divide 1
by the sumof the average latency in ms and the average seek time in
ms (1 / (average latencyin ms + average seek time in ms). Using
gathered data:1 / (2ms + ((3.4ms +3.9ms)/2))1 / (0.002 +
((0.0034+0.0039)/2)) = 177 iops (Single disk) This is inline with
the earlier rough estimate of 175-200 iops from 15k rpm disk!Avg
read/write access time are the only measures that should differ per
model,rotational latency should be constant dependent on the disk
rotation speed, so theestimates should be reasonable.61 62.
Calculating multi disk array IOPS
http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182(Total
Workload IOPS * Percentage of workload that is read operations) +
(TotalWorkload IOPS * Percentage of workload that is write
operations * RAID IOPenalty Using gathered data:((4 * 177) * 0.9) +
(((4 * 177) * 0.1) / 4) = 654.9 iops This is doesnt account for the
controller cache, so should be a worst case limit.Picture credit
also
http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/218262
63. Calculating throughput Formula to calculate maximum
throughputMB/s = IOPS * KB per IO / 1024 Using gathered data for
IOPs, and estimates for KB per IO:654.9 * 8 / 1024 = 5.1 mb/s654.9
* 16 / 1024 = 10.2 mb/s654.9 * 64 / 1024 = 40.9 mb/s654.9 * 128 /
1024 = 81.9 mb/s Where KB per IO fits on this scale depends on how
random vs sequential the IO is.63 64. An example using
iostat[ben.mildren@316403db7~]$iostatdx5Linux2.6.18194.17.1.el5(xxxx)
04/25/2013....Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.600.0016.800.0085.6010.190.063.290.003.290.060.10sdb7607.005.20253.6023.4031467.20114.40228.030.863.092.934.862.6874.10dm00.000.000.000.400.001.608.000.004.000.004.002.000.08dm10.000.000.0028.200.00112.808.000.114.070.004.070.040.12dm20.000.00374.800.0031441.600.00167.780.792.112.110.001.9874.18dm30.000.007860.400.4031441.601.608.0023.212.952.954.000.0974.16dm40.000.000.000.000.000.000.000.000.000.000.000.000.00....Observations:
~275 IOPs well below estimated limit of ~650 IOPs Average queue
size (avgqu-sz) is low. At ~30mb/s (rkB/s + wkB/s), throughput will
not be saturating the controller,rearranging the throughput formula
we can see IOPs are about 114kb per IO whichleans to a more
sequential workload. Latency (read/write average wait time) is
inline with drive manufacturerexpectations.64 65. An example using
iostat[ben.mildren@316403db7~]$iostatdx5Linux2.6.18194.17.1.el5(xxxx)
04/25/2013....Device:rrqm/swrqm/sr/sw/srkB/swkB/savgrqszavgquszawaitr_awaitw_awaitsvctm%utilsda0.004.600.0016.800.0085.6010.190.063.290.003.290.060.10sdb7607.005.20253.6023.4031467.20114.40228.030.863.092.934.862.6874.10dm00.000.000.000.400.001.608.000.004.000.004.002.000.08dm10.000.000.0028.200.00112.808.000.114.070.004.070.040.12dm20.000.00374.800.0031441.600.00167.780.792.112.110.001.9874.18dm30.000.007860.400.4031441.601.608.0023.212.952.954.000.0974.16dm40.000.000.000.000.000.000.000.000.000.000.000.000.00....Observations:
Average request size (avgrq-sz) is ~114k (matches throughput calc),
so the load isprobably not being generated by MySQL. High number of
merges, not a bad thing, but might want to investigate why..(In
this case they were caused by an LVM Snapshot being read during a
backup).65 66. Thank you Q&ATo contact
[email protected] follow
ushttp://www.pythian.com/bloghttp://www.facebook.com/pages/The-Pythian-Group/163902527671@pythianhttp://www.linkedin.com/company/pythian6666
