64
IBM Systems Group © 2010 IBM Corporation Author: Dan Braden [email protected] Presenter: Steve Nasypany AIX Advanced Technical Skills http://w3.ibm.com/support/americas/pseries Disk IO Tuning in AIX 6.1 Session ID: PE23 IBM Power Systems Technical University October 18–22, 2010 — Las Vegas, NV © 2010 IBM Corporation

Disk IO Tuning in AIX 6.1

  • Upload
    jlmer

  • View
    326

  • Download
    12

Embed Size (px)

Citation preview

Page 1: Disk IO Tuning in AIX 6.1

IBM Systems Group

© 2010 IBM Corporation

Author: Dan Braden [email protected]: Steve NasypanyAIX Advanced Technical Skillshttp://w3.ibm.com/support/americas/pseries

Disk IO Tuning in AIX 6.1

Session ID: PE23

IBM Power Systems Technical University

October 18–22, 2010 — Las Vegas, NV

© 2010 IBM Corporation

Page 2: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Agenda

� The importance of IO tuning� Disk basics and performance overview� AIX IO stack� Data layout� Characterizing application IO� Disk performance monitoring tools� Testing disk subsystem performance� Tuning

Page 3: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� Moore's law

� Processors double in price performance every 18 months

� Disk growth

� Disk densities are doubling every 12 months

� Customers are doubling storage capacities every 12-18 months

� Actuator and rotational speed increasing relatively slowly

� Network bandwidth - doubling every 6 months

Why is disk IO tuning important?

Approximate CPU cycle time 0.0000000005 secondsApproximate memory access time 0.000000270 secondsApproximate disk access time 0.010000000 seconds

� Memory access takes 540 CPU cycles� Disk access takes 20 million CPU cycles, or 37,037 memory accesses

� System bottlenecks are being pushed to the disk

� Disk subsystems are using cache to improve IO service times

� Customers now spend more on storage than on servers

Page 4: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Why is disk IO tuning important?Seagate 15k RPM/3.5" Drive Specifications

73

450

171

75

3.43.6

2002 2008

Capacity (GB)

Max SustainedDR (MB/s)Read Seek (ms)

+35%

+15%

-1%

� Disk IO service time not improving compared to processors

Page 5: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� Disk metrics

� MB/s

� IOPS

� With a reasonable service time

� Application metrics

� Response time

� Batch job run time

� System metrics

� CPU, memory and IO

� Size for your peak workloads

� Size based on maximum sustainable thruputs

� Bandwidth and thruput sometimes mean the same thing, sometimes not

� For tuning - it's good to have a short running job that's representative of your workload

Performance metrics

Page 6: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Performance metrics� Use a relevant metric for testing

� Should be tied to business costs, benefits or requirements

� Batch job run time

� Maximum or sufficient application transactions/second

� Query run time

� Metrics that typically are not so relevant

� Application transaction time if < a few seconds

� Metrics indicating bottlenecks

� CPU, memory, network, disk

� Important if the application metric goal isn’t met

� Be aware of IO from other systems affecting disk performance to shared disk

� If benchmarking two systems, be sure the disk performance is apples to apples and you’re not really comparing disk subsystem performance

Page 7: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Disk performance

� Interface type� ATA� SATA� SCSI� FC� SAS

“ZBR” Geometry

� IO service times are predominately

seek + rotational latency + queueing time

Page 8: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Disk performance � When do you have a disk bottleneck?

� Random workloads� Reads average > 15 ms� With write cache, writes average > 2.5 ms

� Sequential workloads� Two sequential IO streams on one disk� You need more thruput

IOPS vs IO service time - 15,000 RPM disk

0

100

200

300

400

500

25 50 75 100 125 150 175 200 225 250 275 300 325

IOPS

IO s

ervi

ce t

ime

(ms)

Page 9: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

How to improve disk performance� Reduce the number of IOs

� Bigger caches

� Application, file system, disk subsystem

� Use caches more efficiently

� No file system logging

� No access time updates

� Improve average IO service times

� Better data layout

� Reduce locking for IOs

� Buffer/queue tuning

� Use SSDs or RAM disk

� Faster disks/interfaces, more disks

� Short stroke the disks and use the outer edge

� Smooth the IOs out over time

� Reduce the overhead to handle IOs

Page 10: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� A misleading indicator of disk performance

� A type of CPU idle

� Percent of time the CPU is idle and waiting on an IO so it can do some more work

� High %iowait does not necessarily indicate a disk bottleneck

� Your application could be IO intensive, e.g. a backup

� You can make %iowait go to 0 by adding CPU intensive jobs

� Low %iowait does not necessarily mean you don't have a disk bottleneck

� The CPUs can be busy while IOs are taking unreasonably long times

� If disk IO service times are good, you aren’t getting the performance you need, and you have significant %iowait – consider using SSDs or RAM disk

� Improve performance by potentially reducing %iowait to 0

What is %iowait?

Page 11: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Solid State Disk (SSD)

� High performance electronic disk

� From 14,000 – 27,000 IOPS possible for a single SSD

� SSD IO bandwidth varies across Power and disk subsystems

� Typically small (69-177 GB) and expensive compared to HDDs

� Read or write IOs typically < 1 ms

� About the same IO service time as compared to writes to disk subsystem cache

� About 5-15X faster than reads from disk

� Positioned for high access density (IOPS/GB) random read data

� Implementation involves finding the best data to place on the SSDs

� SSDs can save disk costs by reducing the number of spindles needed

� When high access density data exists

� A mix of SSDs and HDDs is often best

Page 12: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

SSD vs. HDD performance

SSD offers up to 33x – 125x more IOPS

HDD SSD

HDD IO service time typically 5X to 40X

slower* 125X

33X

HDD SSDAccess time is drive-to-drive, ignoring any caching by SAS controller

5X

40X

1X1X

Page 13: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

RAM disk� Use system RAM to create a virtual disk

� Data is lost in the event of a reboot or system crash

� IOs complete with RAM latencies

� For file systems, it takes away from file system cache

� Taking from one pocket and putting it into another

� A raw disk or file system only – no LVM support

# mkramdisk 16M/dev/rramdisk0# mkfs -V jfs2 /dev/ramdisk0mkfs: destroy /dev/ramdisk0 (yes)? yFile system created successfully.16176 kilobytes total disk space.Device /dev/ramdisk0:

Standard empty filesystemSize: 32352 512-byte (DEVBLKSIZE) blocks

# mkdir /ramdiskfs# mount -V jfs2 -o log=NULL /dev/ramdisk0 /ramdiskf s# df -m /ramdiskfsFilesystem MB blocks Free %Used Iused %Iused M ounted on/dev/ramdisk0 16.00 15.67 3% 4 1% /ramdiskfs

Page 14: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Application memory area caches data to avoid IO

NFS caches file attributes NFS has a cached filesystem for NFS clients

The AIX IO stack

JFS and JFS2 cache use extra system RAMJFS uses persistent pages for cacheJFS2 uses client pages for cache

Queues exist for both adapters and disksAdapter device drivers use DMA for IODisk subsystems have read and write cacheDisks have memory to store commands/data

IOs can be coalesced (good) or split up (bad) as they go thru the IO stackIOs adjacent in a file/LV/disk can be coalescedIOs greater than the maximum IO size supported will be split up

Write cache Read cache or memory area used for IODisk

Disk subsystem (optional)Adapter Device Drivers

Disk Device Drivers

LVM (LVM device drivers)

Ra

w disks

Ra

w L

Vs

Application

Logical file system

JFS JFS2 NFS Other

VMM

Multi-path IO driver (optional)

Page 15: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Synchronous vs Asynchronous IOs� Definition depends on the frame of reference� Programmers/application

� When an application issues a synchronous IO, it waits until the IO is complete

� Asynchronous IOs are handed off to the kernel, and the application continues, and uses the AIO facilities in AIX�When a group of asynchronous IOs complete, a signal is sent to the

application�Allows IO and processing to run simultaneously

� Filesystem IO� Synchronous write IOs to a file system must get to disk� Asynchronous IOs only need to get to file system cache

� GLVM or disk subsystem mirroring� Synchronous mirroring requires that writes to both mirrors complete

before returing an acknowledgement to the application� Asynchronous mirroring returns an acknowledgement when the write

completes at the local storage�Writes to remote storage are done in the same order as locally

Page 16: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Data layout� Data layout affects IO performance more than any tunable IO parameter� Good data layout avoids dealing with disk hot spots

� An ongoing management issue and cost� Data layout must be planned in advance

� Changes are often painful� iostat and filemon can show unbalanced IO� Best practice: evenly balance IOs across all physical disks

Random IO best practice: � Spread IOs evenly across all physical disks� For disk subsystems

� Create RAID arrays of equal size and RAID level� Create VGs with one LUN from every array� Spread all LVs across all PVs in the VG

� The SVC can, and XIV does do this automatically

Page 17: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Random IO data layout

RAID array

LUN or logical disk

1

2

3

4

5

1

PV

2 3 4 5

datavg

# mklv lv1 –e x hdisk1 hdisk2 … hdisk5# mklv lv2 –e x hdisk3 hdisk1 …. hdisk4…..Use a random order for the hdisks for each LV

Disk subsystem

Page 18: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Data layout for sequential IO� Many factors affect sequential thruput

� RAID setup, number of threads, IO size, reads vs. writes� Create RAID arrays with data stripes a power of 2

� RAID 5 arrays of 5 or 9 disks� RAID 10 arrays of 2, 4, 8, or 16 disks

� Do application IOs equal to, or a multiple of, a full stripe on the RAID array

� Or use multiple threads to submit many IOs� N disk RAID 5 arrays can handle no more than N-1 sequential IO streams before the IO becomes randomized� N disk RAID 10 arrays can do N sequential read IO streams and N/2 sequential write IO streams before the IO becomes randomized� Sometimes smaller strip sizes (around 64 KB) perform better� Test your setup if the bandwidth needed is high

Page 19: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Data layoutBest practice for VGs and LVs

�Use Big or Scalable VGs�Both support no LVCB header on LVs (only important for raw LVs)

� These can lead to issues with IOs split across physical disks� Big VGs require using mklv –T O option to eliminate LVCB� Scalable VGs have no LVCB

�Only Scalable VGs support mirror pools (AIX 6100-02)

�For JFS2, use inline logs�For JFS, one log per file system provides the best performance

� If using LVM mirroring, use active MWC� Passive MWC creates more IOs than active MWC

�Use RAID in preference to LVM mirroring� Reduces IOs as there’s no additional writes for MWC

�Use PP striping in preference to LV striping

Page 20: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

LVM limits

�Max PPs per VG and max LPs per LV restrict your PP size� Use a PP size that allows for growth of the VG� Use a PP size that allows your LVs to be spread across all PVs

�Unless your disk subsystem ensures your LVs are spread across all physical disks

�Valid LV strip sizes range from 4 KB to 128 MB in powers of 2 for striped LVs

2,097,152130,04832,512Max LPs/LV

2,097,152130,04832,512Max PPs/VG

4096512256Max LVs/VG

102412832Max PVs/VG

Scalable VGBig VGStandard VG

Page 21: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

LUN size and how many?�Does LUN size matter? It depends.� Fewer larger LUNs are easier to manage� We can do many IOs in parallel to a LUN – depends on its queue_depth

�Typically limited by:� Backend physical disks � Other disk subsystem bottlenecks

�Theoretical bandwidth is:� Queue_depth/average IO service time IOPS� # physical disks x physical disk IOPS taking into account use of RAID� Assumes no other disk subsystem bottlenecks

�More LUNs mean more hdisk driver threads �Very high IOPS rates will require more LUNs

Page 22: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Application IO characteristics� Random IO

� Typically small (4-32 KB)� Measure and size with IOPS� Usually disk actuator limited

� Sequential IO� Typically large (32KB and up)� Measure and size with MB/s� Usually limited on the interconnect to the disk actuators

� To determine application IO characteristics� Use filemon

# filemon –o /tmp/filemon.out –O lf,lv,pv, detailed –T 500000; sleep 90; trcstop

� Check for trace buffer wraparounds which may invalidate the data, run filemon with a larger –T value or shorter sleep

Page 23: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� Summary reports at PV and LV layers

Most Active Logical Volumes--------------------------------------------------- ---------------------

util #rblk #wblk KB/s volume description--------------------------------------------------- ---------------------

1.00 10551264 5600 17600.8 /dev/rms09_lv /RMS/bormspr0/oradata071.00 6226928 7584 10394.4 /dev/rms06_lv /RMS/bormspr0/oradata041.00 128544 3315168 5741.5 /dev/rms04_lv /RMS/bormspr0/oracletemp1.00 13684704 38208 22879.4 /dev/rms02_lv /RMS/bormspr0/oradata010.99 11798800 16480 19698.9 /dev/rms03_lv /RMS/bormspr0/oradata020.99 600736 7760 1014.5 /dev/rms13_lv /RMS/bormspr0/oradata110.98 6237648 128 10399.9 /dev/oraloblv01 /RMS/bormspr0/oralob010.96 0 3120 5.2 /dev/hd8 jfslog0.55 38056 104448 237.6 /dev/rms041_lv /RMS/bormspr0/oraredo0.48 2344656 3328 3914.6 /dev/rms11_lv /RMS/bormspr0/oradata09

Most Active Physical Volumes--------------------------------------------------- ---------------------

util #rblk #wblk KB/s volume description--------------------------------------------------- ---------------------

1.00 3313059 4520 5531.2 /dev/hdisk66 SAN Volume Controller Device1.00 7563668 22312 12647.6 /dev/hdisk59 SAN Volume Controller Device1.00 53691 1868096 3204.1 /dev/hdisk61 SAN Volume Controller Device1.00 11669 6478 30.3 /dev/hdisk0 N/A1.00 6247484 4256 10423.1 /dev/hdisk77 SAN Volume Controller Device1.00 6401393 10016 10689.3 /dev/hdisk60 SAN Volume Controller Device1.00 5438693 3128 9072.8 /dev/hdisk69 SAN Volume Controller Device

filemon summary reports

Page 24: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� Detailed reports at PV and LV layers (only for one LV shown)� Similar reports for each PV

VOLUME: /dev/rms09_lv description: /RMS/bormspr0/o radata07

reads: 23999 (0 errs)

read sizes (blks): avg 439.7 min 16 max 2 048 sdev 814.8

read times (msec): avg 85.609 min 0.139 max 1113.574 sdev 140.417

read sequences: 19478

read seq. lengths: avg 541.7 min 16 max 122 88 sdev 1111.6

writes: 350 (0 errs)

write sizes (blks): avg 16.0 min 16 max 16 sdev 0.0

write times (msec): avg 42.959 min 0.340 max 289.907 sdev 60.348

write sequences: 348

write seq. lengths: avg 16.1 min 16 max 32 sdev 1.2

seeks: 19826 (81.4%)

seek dist (blks): init 18262432, avg 24974715.3 min 16

max 157270944 sdev 44289553.4

time to next req(msec): avg 12.316 min 0.000 max 537.792 sdev 31.794

throughput: 17600.8 KB/sec

utilization: 1.00

filemon detailed reports

Page 25: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� Look at PV summary report� Look for balanced IO across the disks� Lack of balance may be a data layout problem

� Depends upon PV to physical disk mapping � LVM mirroring scheduling policy also affects balance for reads� IO service times in the detailed report is more definitive on data layout

issues�Dissimilar IO service times across PVs indicates IOs are not

balanced across physical disks

� Look at most active LVs report� Look for busy file system logs

� Look for file system logs serving more than one file system

�At 6.1, filemon also has reports showing the processes/threads doing IO to files

Using filemon

Page 26: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Using iostat� Use a meaningful interval, 30 seconds to 15 minutes

� The first report is since system boot (if sys0’s attribute iostat=true)

� Examine IO balance among hdisks

� Look for bursty IO (based on syncd interval)

� Useful flags:

� -T Puts a time stamp on the data

� -a Adapter report (IOs for an adapter) for both physical and virtual

� -m Disk path report (IOs down each disk path)

� -s System report (overall IO)

� -A or –P For standard AIO or POSIX AIO

� -D for hdisk queues and IO service times

� -R to reset min and max values for each interval

� -l puts data on one line (better for scripts)

� -p for tape statistics

� -f/-F for file system statistics (AIX 6.1 TL1)

Page 27: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Using iostat

# iostat <interval> <count> For individual disk and system statisticstty: tin tout avg-cpu: % user % sys % id le % iowait

24.7 71.3 8.3 2.4 85.6 3 .6

Disks: % tm_act Kbps tps Kb_read Kb_wrtn

hdisk0 2.2 19.4 2.6 268 894

hdisk1 5.0 231.8 28.1 1944 11964

hdisk2 5.4 227.8 26.9 2144 11524

hdisk3 4.0 215.9 24.8 2040 10916

...

# iostat –ts <interval> <count> For total system statisticsSystem configuration: lcpu=4 drives=2 ent=1.50 path s=2 vdisks=2tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc

0.0 8062.0 0.0 0.4 99.6 0 .0 0.0 0.7Kbps tps Kb_read Kb_wrtn82.7 20.7 248 0

0.0 13086.5 0.0 0.4 99.5 0 .0 0.0 0.7Kbps tps Kb_read Kb_wrtn80.7 20.2 244 0

0.0 16526.0 0.0 0.5 99.5 0 .0 0.0 0.8

Page 28: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Using iostat

# iostat -f <interval> <count>

FS Name: % tm_act Kbps tps Kb_read Kb _wrtn/ - 85.7 113.3 257 0/usr - 961.1 274.1 2892 0/var - 0.0 0.0 0 0/tmp - 0.0 0.0 0 0/home - 0.0 0.0 0 0/admin - 0.0 0.0 0 0/proc - 7.6 17.3 22 0/opt - 0.0 0.0 0 0/var/adm/ras/livedum - 0.0 0.0 0 0/oracle - 2.2 22.9 0 6/staging - 0.0 0.0 0 0/ggs - 0.0 0.0 0 0

Page 29: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Using iostat

# iostat -DRTl <interval> <count>Disks: xfers read write queue time

-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------

%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv

act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull

hdisk41 4.6 89.8K 5.7 24.8K 65.0K 3.0 8.5 0.2 28.9 0 0 2.6 9.4 0.4 233.2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk44 21.6 450.2K 52.0 421.5K 28.7K 51.5 4.3 0.2 39.0 0 0 0.6 5.9 0.5 30.9 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk42 6.6 57.3K 6.8 42.3K 15.0K 5.2 10.9 0.2 32.7 0 0 1.6 7.0 0.3 22.4 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk43 37.2 845.5K 101.4 818.2K 27.3K 99.9 4.0 0.2 47.6 0 0 1.5 17.2 0.4 230.2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk37 94.4 700.0K 2.2 0.0 700.0K 0.0 0.0 0.0 0.0 0 0 2.2 1.1S 117.9 4.1S 0 0 0.0 0.0 0.1 0.0 0.1 0.0 04:52:25

hdisk53 23.5 296.2K 35.5 269.5K 26.8K 32.9 7.7 0.2 47.0 0 0 2.6 2.5 0.4 27.7 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk51 32.5 471.2K 55.6 445.5K 25.7K 54.4 6.7 0.2 58.8 0 0 1.2 3.1 0.4 13.0 0 0 0.0 0.0 0.1 0.0 0.0 0.0 04:52:25

hdisk56 19.5 178.0K 20.7 122.3K 55.7K 14.9 9.8 0.2 55.0 0 0 5.7 55.8 0.4 318.9 0 0 2.8 0.0 194.4 0.0 0.0 0.6 04:52:25

hdisk48 18.0 149.6K 18.0 101.0K 48.6K 12.3 10.6 0.2 38.5 0 0 5.7 19.0 0.4 250.2 0 0 0.0 0.0 3.7 0.0 0.0 0.3 04:52:25

hdisk46 12.9 167.4K 19.8 156.7K 10.6K 19.1 6.8 0.2 37.5 0 0 0.7 4.4 0.4 17.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk57 55.2 608.8K 71.1 574.4K 34.4K 69.5 8.9 0.2 118.3 0 0 1.6 10.1 0.4 216.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk55 13.4 244.9K 29.8 234.0K 10.9K 28.6 4.8 0.2 36.9 0 0 1.3 2.6 0.4 22.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk50 48.6 616.7K 73.3 575.5K 41.2K 70.3 7.9 0.2 84.5 0 0 3.1 5.7 0.4 40.1 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk52 14.5 174.2K 20.6 116.0K 58.1K 14.2 7.7 0.2 36.9 0 0 6.5 10.7 0.4 270.1 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk49 4.9 46.1K 5.6 33.6K 12.6K 4.1 10.7 1.0 33.6 0 0 1.5 3.6 0.4 23.1 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk58 11.4 233.4K 26.2 190.6K 42.9K 23.3 4.3 0.2 47.2 0 0 3.0 9.4 0.4 259.8 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk59 21.2 219.8K 26.8 219.8K 0.0 26.8 8.4 0.2 44.4 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.1 0.0 0.0 0.0 04:52:25

hdisk38 99.8 387.5K 1.2 1.5K 386.0K 0.4 1.0 0.2 8.0 0 0 0.8 1.3S 0.4 3.9S 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk40 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 04:52:25

hdisk47 12.9 229.0K 20.1 146.3K 82.7K 17.9 7.1 0.2 43.0 0 0 2.2 7.3 0.4 167.3 0 0 0.0 0.0 0.1 0.0 0.0 0.0 04:52:25

�Shows average IO service times for reads and writes, IO rates, IOPS (tps) and time in the hdisk driver queue�One can calculate R/W ratio and average IO size�Time spent in the queue indicates increasing queue_depth may improve performance

Page 30: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Using lvmstat� Provides IO statistics for LVs, VGs and PPs

� Useful for SSD data placement

� You must enable data collection first for a VG:

# lvmstat –e –v <vgname>

� Useful to find busy LVs and PPs root/ # lvmstat -sv rootvg <interval length> <number of intervals>Logical Volume iocnt Kb_read Kb_wrtn Kbps

hd8 212 0 848 24 .00hd4 11 0 44 0 .23hd2 3 12 0 0 .01hd9var 2 0 8 0 .01

..hd8 3 0 12 8 .00

.hd8 12 0 48 32 .00hd4 1 0 4 2 .67

# lvmstat -l lv00 1 Log_part mirror# iocnt Kb_read Kb_wrtn Kbps

1 1 65536 32768 0 0.02 2 1 53718 26859 0 0.01

Log_part mirror# iocnt Kb_read Kb_wrtn Kbps 2 1 5420 2710 0 14263.16

Log_part mirror# iocnt Kb_read Kb_wrtn Kbps3 1 4449 2224 0 13903.12 2 1 979 489 0 3059.38

Page 31: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Using NMON# nmon - then press “a” for all adapters or “^” for FC adapters

� Easy way to monitor adapter thruput

� NMON can also be used to create Excel graphs showing IO over time

� Plus CPU, memory, and network IO data

Page 32: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Testing thruput� Sequential IO

� Test sequential read thruput from a device:

# timex dd if=<device> of=/dev/null bs=1m count=100

# timex dd if=/dev/rhdisk20 of=/dev/null bs=1m count=1024

1024+0 records in.1024+0 records out.real 3.44user 0.00sys 0.17

� 1024 MB/3.44 s = 297.7 MB/s

� Test sequential write thruput to a device:

# timex dd if=/dev/zero of=<device> bs=1m count=100

� Note that /dev/zero writes the null character, so writing this character to files in a file system will result in sparse files

� For file systems, either create a file, or use the lptest command to generate a file, e.g., # lptest 127 32 > 4kfile

Page 33: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Testing thruput with ndisk

�Use ndisk which is part of the nstress package, or ndisk64 for structures > 2GB

http://www.ibm.com/collaboration/wiki/display/WikiPtype/nstress

�Do IO to a file or raw LV or hdisk

�Do IO to multiple devices or files

�Specify the number of threads doing IO

� You need a lot of threads doing IO to stress a disk subsystem

�Synchronous or asynchronous writes to file system files

�Specify the IO size or a set of IO sizes

�Specify the R/W ratio

Page 34: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Testing thruput with ndisk# ndisk64 -R -f /dev/rhdisk20 -r 100 -t 30 –M 20 –s 716800

Command: ndisk -R -f /dev/rhdisk20 -r 100 -t 60Synchronous Disk test (regular read/write)No. of processes = 20I/O type = RandomBlock size = 4096Read-Write = Read onlySync type: none = just close the fileNumber of files = 1File size = … 716800 MBRun time = 30 secondsSnooze % = 0 percent

----> Running test with block Size=4096 (4KB) .Proc - <-----Disk IO----> | <-----Throughput------> RunTime

Num - TOTAL IO/sec | MB/sec KB/sec Second s

1 - 12577 419.2 | 1.64 1676.94 30.00

…20 - 12577 419.2 | 1.64 1676.98 30.00

TOTALS 251540 8384.8 | 32.75 Rand procs= 20 r ead=100% bs=4KB

Page 35: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Dealing with cache effects�Prime the cache (recommended)

�Run the test twice or more and ignore the first run

�It's recommended to prime the cache, as most applications will be using it and you've paid for it, so you should use it

or

�Flush the cache

�Unmount and remount file systems

�For disk subsystems, use #cat <unused file(s)> > /dev/null

�The unused file(s) must be larger than the disk subsystem read cache

�Write cache

�If we fill up the write cache, IO service times will be at disk speed, not cache speed

�Use a long running job

�Reads from the disk subsystem will also inhibit unloading of the write cache

Page 36: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

AIX 6.1 Restricted Tunables� Some ioo/vmo/schedo/raso/nfso/no tuning parameters are now restricted

�Generally should only be changed per AIX Support

�Display all the tunables using: # <ioo|vmo|schedo|raso|nfso|no> -FL

�Display non-restricted tunables without the –F flag

� smit access via # smitty tuningDev

� Dynamic change will show a warning message

� Permanent changes require a confirmation

� Permanent changes will result in a warning message at boot in the error log

� Some restricted tunables relating to disk IO tuning include:

Most aio tunablesj2_nBufferPerPagerDeviceminpgaheadnumclustpv_min_pbufsync_release_ilock

lru_file_repagelru_poll_intervalmaxpermminpermstrict_maxclientstrict_maxpermpage_steal_method

Page 37: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Tuning IO buffers and queues�General rule – increase buffers or queue depths until either:

� You aren’t running out of buffers or filling the queue

� IO service times indicate a bottleneck at the disk subsystem or SAN

�IOs are delayed due to lack of a buffer or a queue slot

�Disks and disk subsystem have limits to the maximum number of in-flight IOs they can handle

� More than this will result in lost IOs, time outs and resubmission of the IO which severely affects IO performance

Page 38: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

File system buffers at this layer

The AIX IO stack

DiskDisk subsystem (optional)

Adapter Device DriversDisk Device Drivers

LVM (LVM device drivers)

Ra

w disks

Ra

w L

Vs

Application

Logical file system

JFS JFS2 NFS Other

VMM

Write cache Read cache or memory area used for IO

Disk buffers (pbufs) at this layer

Multi-path IO driver (optional)

hdisk queue_depthadapter num_cmd_elems

Page 39: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

# vmstat -v | tail -5 <-- only last 5 lines neede d

0 pending disk I/Os blocked with no pbuf

0 paging space I/Os blocked with no psbuf

8755 filesystem I/Os blocked with no fsbuf

0 client filesystem I/Os blocked with no fsbuf

2365 external pager filesystem I/Os blocked with no fsbuf

� First field is a count of IOs delayed since boot due to lack of the specified buffer� For pbufs, use lvmo to increase pv_pbuf_count (see the next slide)� For psbufs, stop paging or add paging spaces� For filesystem fsbufs, increase numfsbufs with ioo� For external pager fsbufs, increase j2_dynamicBufferPreallocation with ioo� For client filesystem fsbufs, increase nfso's nfs_v3_pdts and nfs_v3_vm_bufs

(or the NFS4 equivalents)� Run # ioo –FL to see defaults, current settings and what’s required to make the

changes go into effect

Tuning IO buffers

Page 40: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Disk pbuf tuning� Disk buffers are configurable for disks in VGs

lvmo -v VgName -o Tunable [ =NewValue ]

lvmo [-v VgName] -a

# lvmo -a

vgname = rootvg

pv_pbuf_count = 512 - Number of pbufs added when one PV is added to the VGtotal_vg_pbufs = 512 - Current pbufs available for the VGmax_vg_pbuf_count = 16384 - max pbufs available for the VG, requires remountpervg_blocked_io_count = 1 - delayed IO count since last varyon for the VGpv_min_pbuf = 512 - Minimum number of pbufs added when PV is added to any VGglobal_blocked_io_count = 1 - System wide delayed IO count

� To increase a VG’s pbufs dynamically:# lvmo –v <vgname> -o pv_pbuf_count=<new value>

� pv_min_pbuf is tuned via ioo or lvmo� Changes to pv_pbuf_count via lvmo are dynamic� Increase value, collect statistics and change again if necessary� See # lvmo –L

Page 41: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Queue depth tuning� If IO service times are reasonably good, but queues are getting filled up, then

� Increase queue depths until:� You aren’t filling the queues or� IO service times start degrading

�You’ve moved the bottleneck to the disk� For hdisks, queue_depth controls the maximum number of in-flight IOs� For FC adapters, num_cmd_elems controls maximum number of in-flight IOs� Drivers for hdisks and adapters have service and wait queues

� When the queue is full and an IO completes, then another is issued

Page 42: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Queue depth tuningIO flow:� Multipath IO code submits IO to hdisk driver

� SDD queues IOs and won’t submit more than queue_depth IOs to a hdisk

� Disable this with # datapath qdepth disable for heavy IO� SDDPCM does not queue IOs

� Hdisk driver has in process and wait queues – in process queue contains up to queue_depth IOs

� Hdisk driver submits IOs to adapter driver� Adapter driver has in process and wait queues – FC adapter in process

queue contains up to num_cmd_elems IOs� Adapter driver uses DMA to do IOs

� Adapter driver submits IOs to disk subsystem

Page 43: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Queue depth tuning� Useful things to know about attributes for all AIX devices:� List device attributes with # lsattr –EHl <device>

# lsattr -EHl hdisk10attribute value de scription user_settable

PCM PCM/friend/vscsi Path Control Module Falsealgorithm fail_over Algorithm Truehcheck_cmd test_unit_rdy Health Check Command T ruehcheck_interval 0 Hea lth Check Interval Truehcheck_mode nonactive Health Check Mode Truemax_transfer 0x40000 Maximu m TRANSFER Size Truepvid 00cee79eaed0872d0000000000000000 Physical volum e identifier Falsequeue_depth 3 Queue D EPTH Truereserve_policy no_reserve Reserve Policy True

� Attributes with user_settable=True can be changed� List allowable values for an attribute with # lsattr –Rl <device> -a <attribute># lsattr -Rl hdisk10 -a queue_depth1...256 (+1)

� To change an attribute use # chdev –l <device> -a <attribute>=<new value> -P� Then reboot, or if the device is not in use, eliminate the “-P” so the change is

immediately effective

Page 44: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Queue depth tuning� Fibre channel adapter attributes:

# lsattr -El fcs0bus_intr_lvl 8355 Bus interrupt level Falsebus_io_addr 0xffc00 Bus I/O address Falsebus_mem_addr 0xf8040000 Bus memory address Falseinit_link al INIT Link flags Trueintr_priority 3 Interrupt priority Falselg_term_dma 0x1000000 Long term DMA Truemax_xfer_size 0x100000 Maximum Transfer Size Truenum_cmd_elems 200 Maximum number of COMMANDS to queue to t he adapter Truepref_alpa 0x1 Preferred AL_PA Truesw_fc_class 2 FC Class for Fabric True

� The max_xfer_size attribute also controls a DMA memory area used to hold data for transfer, and at the default is 16 MB

� Changing to other allowable values increases it to 128 MB and increases the adapter’s bandwidth� Often changed to 0x200000� This can result in a problem if there isn’t enough memory on PHB chips in

the IO drawer with too many adapters/devices on the PHB� Make the change and reboot – check for Defined devices or errors in the

error log, and change back if necessary� For NPIV and virtual FC adapters the DMA memory area is 128 MB at 6.1

TL2 or later

Page 45: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Queue depth tuning� How to determine if queues are being filled

� With SDDPCM: # pcmpath query devstats <interval> <count># pcmpath query adaptstats <interval> <count>

� With SDD: # datapath query devstats <interval> <count># datapath query adaptstats <interval> <count>

� With iostat: # iostat –D for data since system boot and# iostat –D <interval> <count> for interval data

� With fcstat: # fcstat fcs0

Page 46: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Queue depth tuning� Sample output:# pcmpath query devstats…DEV#: 4 DEVICE NAME: hdisk4===============================

Total Read Total Write Active Read Active Write MaximumI/O: 446958 1510783 0 0 128SECTOR: 13006902 25715160 0 0 24536

Transfer Size: <= 512 <= 4k <= 16 K <= 64K > 64K1032 1825589 40591 54946 35 583

� Indicates that we queued 128 IOs to the hdisk driver� If queue_depth is 128, we filled the queue

# iostat -D hdisk10

System configuration: lcpu=4 drives=35 paths=35 vdi sks=2

hdisk10 xfer: %tm_act bps tps bread bwrtn0.1 1.4K 0.2 442.6 940.9

read: rps avgserv minserv maxserv timeouts fa ils0.0 4.6 0.3 67.9 0 0

write: wps avgserv minserv maxserv timeouts f ails0.2 8.2 0.5 106.8 0 0

queue: avgtime mintime maxtime avgwqsz avgsqsz sqfull5.4 0.0 579.5 0.0 0.0 0.4

� The sqfull value represents the number of times we filled the queue per second� Non-zero values indicate we filled the queue

Page 47: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Adapter queue depth tuning

# fcstat fcs0…FC SCSI Adapter Driver Information No DMA Resource Count: 4490 <- Increase max_xfer_siz eNo Adapter Elements Count: 105688No Command Resource Count: 133 <- Increase num_cmd_el ems…

# pcmpath query adaptstatsAdapter #: 0 =============

Total Read Total Write Active Read Active Write Max imum I/O: 1105909 78 3 0 200SECTOR: 8845752 0 24 0 88

�Maximum of 200 with num_cmd_elems=200 means we filled the queue

Page 48: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

VIO � The VIO Server (VIOS) uses multi-path IO code for the attached disk

subsystems� The VIO client (VIOC) always uses SCSI MPIO if accessing storage thru two

VIOSs� In this case only entire LUNs are served to the VIOC

� Set the queue_depth at the VIOC equal to queue_depth at the VIOS for the LUN

� If you increase vFC adapter num_cmd_elems, also do it on the real FC adapter� Preferably treat the real FC adapter num_cmd_elems as a shared resource

� The VSCSI adapter has a queue also� To avoid queuing on the VSCSI adapter:� Max LUNs/VSCSI adapter =INT(510/(Q+3))

� Q is the queue depth of the LUN assuming all are the same� One can monitor adapters with NMON in the oem_setup_env shell

Page 49: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� Read ahead detects that we're reading sequentially and gets the data before the application requests it

� Reduces %iowait

� Too much read ahead means you do IO that you don't need

� Operates at the file system layer - sequentially reading files

� Set maxpgahead for JFS and j2_maxPgReadAhead for JFS2

� Values of 1024 for max page read ahead are not unreasonable

� Disk subsystems read ahead too - when sequentially reading disks

� Improves IO service time and thruput

� Tunable on DS4000, fixed on ESS, DS6000, DS8000 and SVC

Read ahead

Page 50: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� Initiates writes from file system cache before syncd does it

� Write behind tuning for sequential writes to a file

� Tune numclust for JFS

� Tune j2_nPagesPerWriteBehindCluster for JFS2

� These represent 16 KB clusters

� Larger values allow IO to be coalesced

� When the specified number of sequential 16 KB clusters are updated, start the IO to disk rather than wait for syncd

� Write behind tuning for random writes to a file

� Tune maxrandwrt for JFS

� Tune j2_maxRandomWrite and j2_nRandomCluster for JFS2

� Max number of random writes allowed to accumulate to a file before additional IOs are flushed, default is 0 or off

� j2_nRandomCluster specifies the number of clusters apart two

consecutive writes must be in order to be considered random

� If you have bursty IO, consider using write behind to smooth out the IO rate

Write behind

Page 51: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

More on write behind and syncd� syncd - system wide file system cache flushing

� Historical Unix feature to improve performance� Applies to asynchronous IOs (not necessarily aio)� inode is locked when each file is synced

� There is a tradeoff:� Longer intervals allow more IO to be coalesced� Longer intervals can:

� Create bursty IO� Bursty IO can slow down other IO� As IOPS increase, IO service times also increase

Page 52: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

JFS2 Sync Tunables– ioo JFS2 Sync Tunables

The file system sync operation can be problematic in situations where there is very heavy random I/O activity to a large file. When a sync occurs all reads and writes from user programs to the file are blocked. With a large number of dirty pages in the file the time required to complete the writes to disk can be large. New JFS2 tunables are provided to relieve that situation.

• j2_syncPageCountLimits the number of modified pages that are scheduled to be written by sync in one pass for a file. When this tunable is set, the file system will write the specified number of pages without blocking IO to the rest of the file. The sync call will iterate on the write operation until all modified pages have been written.Default: 0 (off), Range: 0-65536, Type: Dynamic, Unit: 4KB pages

• j2_syncPageLimitOverrides j2_syncPageCount when a threshold is reached. This is to guarantee that sync will eventually complete for a given file. Not applied if j2_syncPageCount is off.Default: 16, Range: 1-65536, Type: Dynamic, Unit: Numeric

• If application response times are impacted by syncd, try j2_syncPageCount settings from 256 to 1024. Smaller values improve short term response times, but still result in larger syncs that impact response times over larger intervals.

• These will likely require a lot of experimentation, and detailed analysis of IO behavior.

• Does not apply to mmap() memory mapped files. May not apply to shmat() files (TBD)

Page 53: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Mount options� Release behind: rbr, rbw and rbrw

� Says to throw the data out of file system cache � rbr is release behind on read� rbw is release behind on write� rbrw is both� Applies to sequential IO only

� DIO: Direct IO� Bypasses file system cache� No file system read ahead� No lrud or syncd overhead� No double buffering of data� Half the kernel calls to do the IO� Half the memory transfers to get the data to the application� Requires the application be written to use DIO

� CIO: Concurrent IO� The same as DIO but with no i-node locking

Page 54: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

i-node locking: when 2 or more threads access the same file, and one is a write, the write will block all other read/write threads at this level

The AIX IO stack

Write cache Read cache or memory area used for IO

DiskDisk subsystem (optional)

Adapter Device DriversDisk Device Drivers

LVM (LVM device drivers)

Ra

w disks

Ra

w L

Vs

Application

Logical file system

JFS JFS2 NFS Other

VMM

Multi-path IO driver

Page 55: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Mount options� Direct IO

� IOs must be aligned on file system block boundaries� IOs that don’t adhere to this will dramatically reduce performance

� Avoid large file enabled JFS file systems - block size is 128 KB after 4 MB# mount -o dio# chfs -a options=rw,dio <file system>

� DIO process

1 Application issues IO request to kernel2 Kernel initiates disk IO3 Data transferred to/from application buffer to disk

1 Application issues read request2 Kernel checks FS cache

1 Data found - kernel copies it to the application buffer

2 Data not found - kernel does disk IO 3 Data transferred FS cache4 Kernel copies data to app buffer

1 Application issues write request2 Kernel writes data to FS cache and returns

acknowledgment to app1 Application continues

3 Syncd periodically flushes dirty cache to disk

� Normal file system IO process

File system reads File system writes

Page 56: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Mount options� Concurrent IO for JFS2 (not JFS):

# mount -o cio

# chfs -a options=rw,cio <file system>

� Assumes that the application ensures data integrity for multiple simultaneous IOs to a file

� Changes to meta-data are still serialized

� I-node locking: When two threads (one of which is a write) to do IO to the same file are at the file system layer of the IO stack, reads will be blocked while a write proceeds

� Provides raw LV performance with file system benefits

� Requires an application designed to use CIO

� For file system maintenance (e.g. restore a backup) one usually mounts without cio during the maintenance

� Some applications now make CIO/DIO calls directly without requiring cio/diomounts, in which case don’t use the mount options

� Important for times when alignment requirements aren’t met, or when file system read ahead helps (like during backups)

Page 57: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

No time for atime� Ingo Molnar (Linux kernel developer) said:

� "It's also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: 'For every file that is read from the disk, lets do a ... write to the disk! And, for every file that is already cached and which we read from the cache ... do a write to the disk!'"

� If you have a lot of file activity, you have to update a lot of timestamps� File timestamps� File creation (ctime)� File last modified time (mtime)� File last access time (atime)

� The noatime mount option disables last access time updates for JFS2� File systems with heavy inode access activity due to file opens can have

significant performance improvements� First customer benchmark efix reported 35% improvement with DIO

noatime mount (20K+ files)� Most customers should expect much less for production environments

# mount –a rw,noatime /myfilesystem

Page 58: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Disabling file system journaling

� You may lose data in the event of a system crash!

� Improves performance for meta-data changes to file systems

� When you frequently add, delete or change the size of files

� Eliminates IOs to file system log

� JFS

# mount –o nointegrity <file system>

� JFS2

# mount -o log=NULL <file system>

Page 59: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

� IO pacing - causes the CPU to do something else with a specified amount of IOs to a file in process � Turning it off improves backup times and thruput� Turning it on ensures that no process hogs CPU for IO, and ensures

good keyboard response on systems with heavy IO workloads� Default is on with minpout=4096 and maxpout=8193

� Originally used to avoid HACMP's dead man switch� Old HACMP recommended values of 33 and 24 significantly

inhibit thruput but are reasonable for uniprocessors with non-cached disk

� Changed via:# chgsys –l sys0 –a maxpout=<new value> -a minpout=< new value>

� IO pacing per file system via the mount command# mount -o minpout=256 –o maxpout=513 /myfs

IO Pacing

Page 60: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Asynchronous IO� Asynchronous IO automatically turned on at AIX 6.1� AIO kernel threads automatically exit after aio_server_inactivity seconds

� AIO kernel threads not used for AIO to raw LVs or CIO mounted file systems � Only aio_maxservers and aio_maxreqs need to be changed

� Defaults are 21 and 8K respectively per logical CPU� Set via ioo� Some may want to adjust minservers for heavy AIO use

� maxservers is the maximum number of AIOs that can be processed at any one time

� maxreqs is the maximum number of AIO requests that can be handled at one time and is a total for the system (they are queued to the AIO kernel threads)

� Typical values:

Default OLTP SAPminservers 3 200 400maxservers 10 800 1200maxreqs 4096 16384 16384

Page 61: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

Asynchronous IO tuning� Use iostat –A to monitor AIO (or -P for POSIX AIO)

# iostat -A <interval> <number of intervals>

System configuration: lcpu=4 drives=1 ent=0.50

aio: avgc avfc maxg maxf maxr avg-cpu: %user %sys %idle %iow physc %entc

25 6 29 10 4096 30.7 36.3 15.1 17.9 0.0 81.9

Disks: % tm_act Kbps tps Kb_read Kb_wrtn

hdisk0 100.0 61572.0 484.0 8192 53380

� avgc - Average global non-fastpath AIO request count per second for the specified interval

� avfc - Average AIO fastpath request count per second for the specified interval for IOs to raw LVs (doesn’t include CIO fast path IOs)

� maxg - Maximum non-fastpath AIO request count since the last time this value was fetched

� maxf - Maximum fastpath request count since the last time this value was fetched� maxr - Maximum AIO requests allowed - the AIO device maxreqs attribute� If maxg or maxf gets close to maxr or maxservers then increase maxreqs or

maxservers

Page 62: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

SAN attached FC adapters� Set fscsi dynamic tracking to yes

� Allows dynamic SAN changes� Set FC fabric event error recovery fast_fail to yes if the switches support it

� Switch fails IOs immediately without timeout if a path goes away� Switches without support result in errors in the error log

# lsattr -El fscsi0attach switch How this adapter is CONNE CTED Falsedyntrk no Dynamic Tracking of FC De vices Truefc_err_recov delayed_fail FC Fabric Event Error REC OVERY Policy Truescsi_id 0x1c0d00 Adapter SCSI ID Falsesw_fc_class 3 FC Class for Fabric True

# chdev –l fscsi0 –a dyntrk=yes –a fc_err_recov=fas t_fail –P

# shutdown –Fr

Page 63: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

VMM the Virtual Memory Manager� All IO requires memory

�All input goes to memory, all output comes from memory�File system cache consumes memory�File system cache takes CPU cycles to manage

� Initial tuning recommendations:�minfree = max [ 960, lcpus x 120 / mempools ]�maxfree = minfree + MaxIOSizeIn4KBunits x lcpus / mempools� lcpus = number of logical CPUs�mempools = number of memory pools (from vmstat –v)�MaxIOSizeIn4KBunits is j2_maxPageReadAhead for file system IOs, or what you expect your application’s biggest IO request to be

� The idea is that when we call lrud to free up memory for a large IO, we only call it once and it frees up sufficient memory, but not too much

Page 64: Disk IO Tuning in AIX 6.1

2010 IBM Corporation

QUESTIONS? COMMENTS?