Embed Size (px)
Citation preview
Linux Performance Analysis
Will Sterling
Linux & UNIX Consultant
@
PARSEC Group
Agenda
•Performance Analysis Tools
•Performance Profiling Tools
•Identifying Common Performance Issues
Linux Performance Analysis
Performance Analysis Tools
Linux Performance Analysis
CPUtopmpstatpssar/procnmon
MemorytopvmstatpssarFree/procnmon
ProcesspsnmonDisknmonSarIostatvmstat
Networknettopsarethtoolnmon
Toptop 01:45:25 up 12 min, 1 user, load average: 12.33, 7.84, 3.42Tasks: 141 total, 14 running, 117 sleeping, 0 stopped, 10 zombieCpu(s): 93.4%us, 6.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%stMem: 8191996k total, 267248k used, 7924748k free, 10532k buffersSwap: 0k total, 0k used, 0k free, 106008k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23202 root 20 0 13060 992 620 R 12.3 0.0 0:00.37 bc 23210 root 20 0 13060 992 620 R 12.3 0.0 0:00.37 bc 23218 root 20 0 13060 992 620 R 9.6 0.0 0:00.29 bc 23226 root 20 0 13060 992 620 R 9.6 0.0 0:00.29 bc 23234 root 20 0 13060 992 620 R 8.0 0.0 0:00.24 bc 23242 root 20 0 13060 944 620 R 6.3 0.0 0:00.19 bc 23250 root 20 0 13060 928 620 R 4.3 0.0 0:00.13 bc 23258 root 20 0 13060 892 620 R 3.3 0.0 0:00.10 bc 2519 root 20 0 104m 1652 1272 S 2.0 0.0 0:05.85 bc.ksh 23266 root 20 0 12928 860 620 R 1.7 0.0 0:00.05 bc 23274 root 20 0 12928 852 620 R 1.3 0.0 0:00.04 bc 1 root 20 0 19400 1508 1192 S 0.0 0.0 0:01.20 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd 3 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/0 4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 5 root RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
Linux Performance Analysis
Top Cont.top 01:59:43 up 26 min, 1 user, load average: 0.63, 2.83, 4.08Tasks: 121 total, 3 running, 118 sleeping, 0 stopped, 0 zombieCpu(s): 43.3%us, 8.7%sy, 0.0%ni, 48.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%stMem: 8191996k total, 266016k used, 7925980k free, 11944k buffersSwap: 0k total, 0k used, 0k free, 108088k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24222 root 20 0 103m 928 780 R 98.8 0.0 0:03.16 awk 17758 root 20 0 104m 1572 1264 S 7.9 0.0 0:40.30 bc.ksh 1 root 20 0 19400 1508 1192 S 0.0 0.0 0:01.54 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd 3 root RT 0 0 0 0 S 0.0 0.0 0:00.83 migration/0 4 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/0 5 root RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.76 migration/1 8 root RT 0 0 0 0 S 0.0 0.0 0:00.06 migration/1 9 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/1 10 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 11 root RT 0 0 0 0 S 0.0 0.0 0:00.80 migration/2 12 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/2 13 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/2 14 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 15 root RT 0 0 0 0 S 0.0 0.0 0:00.80 migration/3
Linux Performance Analysis
mpstat[root@server6201204290832190000 ~]# mpstat P ALL 5 2Linux 2.6.32220.2.1.el6.x86_64 (server6201204290832190000.localdomain) 04/29/2012 _x86_64_ (4 CPU)
02:24:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle02:24:27 AM all 26.32 0.00 0.05 0.00 0.00 0.00 0.00 0.00 73.6302:24:27 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.0002:24:27 AM 1 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 99.8002:24:27 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.0002:24:27 AM 3 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Linux Performance Analysis
mpstat[root@server6201204290832190000 ~]# mpstat P ALL 5 2Linux 2.6.32220.2.1.el6.x86_64 (server6201204290832190000.localdomain) 04/29/2012 _x86_64_ (4 CPU)
02:24:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle02:24:27 AM all 26.32 0.00 0.05 0.00 0.00 0.00 0.00 0.00 73.6302:24:27 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.0002:24:27 AM 1 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 99.8002:24:27 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.0002:24:27 AM 3 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Linux Performance Analysis
ps aux
[root@server6201204290832190000 ~]# ps auxUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.0 19400 1508 ? Ss 01:33 0:01 /sbin/initroot 2 0.0 0.0 0 0 ? S 01:33 0:00 [kthreadd]root 3 0.0 0.0 0 0 ? S 01:33 0:01 [migration/0]root 4 0.0 0.0 0 0 ? S 01:33 0:00 [ksoftirqd/0]root 5 0.0 0.0 0 0 ? S 01:33 0:00 [migration/0]root 6 0.0 0.0 0 0 ? S 01:33 0:00 [watchdog/0]sniproot 1295 0.0 0.0 97816 3836 ? S 01:34 0:00 sshd: root@pts/root 1298 0.0 0.0 108356 1908 pts/0 Ss 01:34 0:00 bashroot 7420 0.0 0.0 17336 588 ? Ss 02:01 0:00 /usr/sbin/anacrroot 24143 0.0 0.0 110396 1148 pts/0 R+ 02:33 0:00 ps auxroot 24138 88.5 0.0 105996 928 pts/0 R 02:32 0:12 awk BEGIN {for(
Linux Performance Analysis
Sar: CPU[root@server6201204290832190000 ~]# sar uLinux 2.6.32220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM CPU %user %nice %system %iowait %steal %idle01:50:01 PM all 83.58 0.00 5.92 0.00 0.00 10.5002:00:01 PM all 30.32 0.00 18.20 0.01 0.00 51.4802:10:01 PM all 7.97 0.00 15.11 0.02 0.00 76.9002:20:01 PM all 1.55 0.00 0.02 0.00 0.00 98.4202:30:01 PM all 1.61 0.00 0.03 0.00 0.00 98.3602:40:01 PM all 1.83 0.00 0.03 0.00 0.00 98.14Average: all 21.24 0.00 6.19 0.00 0.00 72.56
Linux Performance Analysis
SAR: CPU[root@server6201204290832190000 ~]# sar uLinux 2.6.32220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM CPU %user %nice %system %iowait %steal %idle01:50:01 PM all 83.58 0.00 5.92 0.00 0.00 10.5002:00:01 PM all 30.32 0.00 18.20 0.01 0.00 51.4802:10:01 PM all 7.97 0.00 15.11 0.02 0.00 76.9002:20:01 PM all 1.55 0.00 0.02 0.00 0.00 98.4202:30:01 PM all 1.61 0.00 0.03 0.00 0.00 98.3602:40:01 PM all 1.83 0.00 0.03 0.00 0.00 98.14Average: all 21.24 0.00 6.19 0.00 0.00 72.56
Linux Performance Analysis
SAR: Memory[root@server6201204290832190000 ~]# sar rLinux 2.6.32220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit01:50:01 PM 7930632 261364 3.19 10940 106008 64096 0.7802:00:01 PM 7926756 265240 3.24 11976 108088 66280 0.8102:10:01 PM 7926956 265040 3.24 12696 108488 64652 0.7902:20:01 PM 7926700 265296 3.24 12824 108580 64652 0.7902:30:01 PM 7925568 266428 3.25 12980 108624 65824 0.8002:40:01 PM 7926460 265536 3.24 13076 108628 64652 0.7902:50:01 PM 7925964 266032 3.25 13196 108776 64652 0.7903:00:01 PM 7925964 266032 3.25 13344 108788 64652 0.7903:10:01 PM 7925732 266264 3.25 13480 108832 64660 0.7903:20:01 PM 7913020 278976 3.41 13608 109444 80496 0.98Average: 7925375 266621 3.25 12812 108426 66462 0.81
Linux Performance Analysis
SAR: Disk[root@server6201204290832190000 ~]# sar dLinux 2.6.32220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM DEV tps rd_sec/s wr_sec/s avgrqsz avgqusz await svctm %util01:50:01 PM dev2520 0.47 0.75 12.86 28.91 0.00 1.01 0.94 0.0401:50:01 PM dev25216 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0002:00:01 PM dev2520 0.74 7.79 16.17 32.30 0.00 0.73 0.50 0.04snip
[root@server6201204290832190000 ~]# ls l /dev/vd*brwrw 1 root disk 252, 0 Apr 29 01:33 /dev/vdabrwrw 1 root disk 252, 16 Apr 29 01:33 /dev/vdb
Linux Performance Analysis
Sar: Network[root@server6201204290832190000 ~]# sar n DEVLinux 2.6.32220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s01:50:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.0001:50:01 PM eth0 1.19 1.02 0.08 0.58 0.00 0.00 0.0002:00:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.0002:00:01 PM eth0 1.30 0.83 0.10 0.27 0.00 0.00 0.0002:10:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.0002:10:01 PM eth0 1.47 1.16 0.11 0.18 0.00 0.00 0.0002:20:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.0002:20:01 PM eth0 0.40 0.20 0.03 0.04 0.00 0.00 0.0002:30:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.0002:30:01 PM eth0 0.80 0.53 0.07 0.16 0.00 0.00 0.0002:40:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.0002:40:01 PM eth0 0.64 0.42 0.05 0.27 0.00 0.00 0.00
Linux Performance Analysis
/proc: CPU[root@server6201204290832190000 proc]# cat cpuinfoprocessor : 0snip processor : 3vendor_id : GenuineIntelcpu family : 6model : 2model name : QEMU Virtual CPU version 0.14.0stepping : 3cpu MHz : 2666.760cache size : 4096 KBfpu : yesfpu_exception : yescpuid level: 4wp : yesflags : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good unfair_spinlock pni cx16 popcnt hypervisor lahf_lmbogomips : 5333.52clflush size : 64cache_alignment : 64address sizes : 40 bits physical, 48 bits virtualpower management:
Linux Performance Analysis
/proc: /CPU
[root@server6201204290832190000 proc]# cat statcpu 295757 0 86819 1581480 160 49 190 0 0cpu0 70616 0 22135 398368 47 40 55 0 0cpu1 74539 0 21982 394194 23 0 47 0 0cpu2 70228 0 21854 399389 42 6 46 0 0Cpu3 80372 0 20846 389528 46 2 40 0 0
Total cycles spent: USER NICE SYSTEM IDLE IOWAIT
Linux Performance Analysis
nmon: CPU
http://nmon.sourceforge.net/pmwiki.php
Linux Performance Analysis
Nmon: Memory/Disk/Network
Linux Performance Analysis
ksar
Linux Performance Analysis
Common Performance Issues•Network Bottle Neck to NAS
•Slow IO to Virtual Drives
•Memory Paging
•CPU
•Processes Waiting on Upstream Applications
•Miss-configured Storage
Linux Performance Analysis
