View
251
Download
1
Category
Preview:
Citation preview
Slide 12005 NAS Industry Conference
AIX NFS Client Performance Improvements for
Databases on NASOctober 20, 2005
Sanjay GulabaniSr. Performance Engineer
Network Appliance, Inc.
gulabani@netapp.com
Diane FlemmingAdvisory Software Engineer
IBM Corporation
dianefl@us.ibm.com
Slide 22005 NAS Industry Conference
Agenda• Why Databases on NAS?• AIX NFS improvements - CIO• Simple IO Performance Tests• OLTP Performance Comparison• Future work• Conclusion• Q&A
Slide 32005 NAS Industry Conference
Databases on SAN
• Databases do block based I/O and prefer raw blocks
• But, most admins still put a volume manager and a file system: JFS2, VxFS, UFS over HW RAID storage
Reason: Simplicity -- Easier backups and provisioning
Slide 42005 NAS Industry Conference
NAS Myths
• Myth: NFS consumes more CPU for Databases
Reality: Most NFS client performance problems with DB apps are due to kernel locking and are fixed in good NFS clients
Slide 52005 NAS Industry Conference
NAS Myths• Myth: Ethernet is slower than SAN
Reality: Ethernet is 1Gb, FC is 2Gb. 4Gb FC is almost here but so is 10GbE.Ø Ethernet is likely to takeover bandwidth of FC soon;
cost effectivelyØ Its easy to setup multiple wires to match FC
bandwidth with just 1GbE, todayØ Storage latencies at database layer are measured in
msecs, differences between SAN and NAS latencies are in usecs78usec more for 8K blocks < 0.1msec difference!
Slide 62005 NAS Industry Conference
Why NAS?• Networking is simpler• Networking promotes sharing• Sharing = Higher utilization (~grid
storage)• NAS solutions are cheaper than SAN• No one has won against Ethernet!• Seriously, even blocks based storage is
moving to Ethernet (iSCSI)
Slide 72005 NAS Industry Conference
Why DB on NAS?• Complex storage management is
offloaded from the database servers to storage appliances
• Performance vs. manageabilityØ Database requirements different than
traditional NFS workloads: home directories, distributed source code
Ø single writer lock can be a problem
Slide 82005 NAS Industry Conference
Industry Support for NAS• Oracle On-Demand hosts ERP+CRM
1000+ Linux servers on NetApp NFS • Yahoo runs Database on NetApp• Sun 2003 presentation McDougall +
Colaco promoted NFS• NetApp #1 Leader of NAS $1.59b(FY05)• IBM AIX 5.3 ML-03 improvements now• No one got fired for buying from IBM!
(IBM resells NetApp products)
Slide 92005 NAS Industry Conference
Oracle’s Austin Datacenter
• More than 15,000 x86 servers • 3.1 Petabytes of NetApp storage• 100s of mission-critical hosted Oracle apps
Source: Oracle Magazine Mar/Apr 2005
Slide 102005 NAS Industry Conference
I/O, I/O… It’s off to disk I go• Raw logical volumes
– No memory management overhead
• Buffered I/O– Variety of kernel services
• Direct I/O– No memory management overhead– defaults to synchronous accesses over NFS
• Concurrent I/O– DIO plus no inode/rnode contention
Slide 112005 NAS Industry Conference
NFS File Transfer• Metrics:
– CPU utilization and throughput in KB/sec
• Options:– Default (buffered I/O) and CIO
• AIX Client and Server– p630 1.45GHz 4-way– GbE 1500-byte MTU– AIX 5.3 ML-03
• Demonstrates algorithm behavior– Atomic, Single-Threaded
Slide 122005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, 32KB RPCs
NFS File T ransfer Sequential Reads and Writes
0
20000
40000
60000
80000
100000
120000
Seq Write Seq Read
Th
rou
ghp
ut
(KB
/sec
)
0
10
20
30
40
CP
U U
til (
%)
Buff I/O Throughput CIO ThroughputBuff I/O CPU Util CIO CPU Util
Slide 132005 NAS Industry Conference
Simple I/O Load Generator• Metrics
– CPU utilization, IOPS, throughput, etc.• Options
– NFS client mount options to specify I/O mode: Default (buffered I/O), DIO, CIO
• NetApp Filer FAS880– ONTAP 6.5.1
• AIX Client Model p630 1.45 GHz 4-way– AIX 5.3 ML-03– GbE 1500-byte MTU
Slide 142005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
SIO Throughput 50% Writers, 256MB file
01000020000300004000050000600007 000080000
1 2 4 8 16 32 64 128 256
Number of Threads
KB
/sec Buff I/O
CIO
DIO
Slide 152005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
SIO Throughput100% Writers, 256MB File
0
10000
20000
30000
40000
50000
60000
1 2 4 8 16 32 64 128 256
Number of Threads
KB
/sec Buff I/O
CIO
DIO
Slide 162005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
SIO Scaled Throughput using CIO, 256MB file
0
50000
100000
150000
200000
1 2 4 8 16 32 64 128 256
Number of Threads
KB
ps/
CP
U
100% Writers
50% Writers
Slide 172005 NAS Industry Conference
NFS FT and SIO Summary• CIO not appropriate for single threaded
sequential read/write workloads.• CIO aimed at workloads with higher
concurrency such as databases.• Significant gain in raw throughput
performance using CIO vs. Buffered I/O for workloads with higher concurrency.
• CPU utilization illustrates issue with TCP socket lock contention with increased concurrency.
Slide 182005 NAS Industry Conference
OLTP Performance
• AIX 5.2 ML-5 and AIX 5.3 ML-03 (beta)
• PowerPC pSeries 650 (2 * 1.2 GHz, 4 GB RAM)
• Oracle 10.1
• 1GbE for NFS or iSCSI or 2Gb FCP card
• FAS 940c
• 48 Disk Spindles (4 x DS14) 144GB 10K RPM
• ONTAP 6.5.1
Slide 192005 NAS Industry Conference
Transactions on AIX 5.2, 5.3, iSCSI, FCP
OETs = Order Entry Transactions (an Oracle OLTP benchmark, approx 2:1 read to
write ratio)
Chart 1: Transactions for protocols
4710
11817 12171
14139
0
2000
4000
6000
8000
10000
12000
14000
16000
kernel NFS 5.2@60u
NFS 5.3* 60u iSCSI raw 5.3*60u
FCP raw 5.3*60u
tran
sact
ion
s
OETs
Slide 202005 NAS Industry Conference
Host CPU Utilization
Chart 2: Transactions and cpu utilization for protocols
4710
11817 12171
14139
49
9890
88
0
2000
4000
6000
8000
10000
12000
14000
16000
kernel NFS 5.2@60u
NFS 5.3 60u iSCSI raw 5.360u
FCP raw 5.3 60u
tran
sact
ion
s
0
20
40
60
80
100
cpu
OETscpu
Slide 212005 NAS Industry Conference
Protocol EfficienciesComputed using (%CPU used/ OETs) * K:
where K is a constant such that NFS CPU cost per K transactions is 1.0 and relative cost for iSCSI and FCP is computed using the same constant K transactions.
Chart 3: Relative CPU cost per protocol for fixed transactions
1
0.89
0.75
0
0.2
0.4
0.6
0.8
1
1.2
NFS 5.3 60u iSCSI raw 5.3 60u FCP raw 5.3 60u
protocol
no
rmal
ized
cp
u c
ost
(lo
wer
is
bet
ter)
relative cpu cost
Slide 222005 NAS Industry Conference
Oracle Block Read Response TimeNot typical response time. Random workload used. Filer with more
cache improves block response time or more sequential reads improve avg. block response time.
Chart 4: Oracle Block Read Response Time
29
13 1412
0
5
10
15
20
25
30
35
kernel NFS5.2 @60u
NFS 5.3 60u iSCSI raw 5.360u
FCP raw 5.360u
protocol
Blo
ck r
ead
Res
p Ti
me
(low
er is
bet
ter)
block read resp time
Slide 232005 NAS Industry Conference
Future Work
• Testing and analysis on systems with higher numbers of CPUs.
• Further investigation on socket lock scaling issue on AIX
Slide 242005 NAS Industry Conference
Conclusions• AIX 5.3 ML-03 with new Concurrent
I/O ‘CIO’ mount option delivers an OLTP performance on NFS comparable to that of block-based protocols iSCSI and FCP
• Don’t be afraid to deploy databases on AIX NFS, we will support you.
• Proof of AIX NFS performance up to 4 CPU completed.
Slide 252005 NAS Industry Conference
References• AIX Performance with NFS, iSCSI and FCP
Using an Oracle Database on NetApp White Paper @ http://www.netapp.com/library/tr/3408.pdf
• NetApp Best Practices Paper located at: http://www.ibm.com/servers/storage/support/nas/
• Download SIO toolhttp://now.netapp.com/NOW/download/tools/sio_ntap/index.shtml
• Improving Database Performance with AIX Concurrent I/O: http://www-03.ibm.com/servers/aix/whitepapers/db_perf_aix.pdf
Slide 262005 NAS Industry Conference
Questions/Answers
Recommended