High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,

IBM ATS Deep Computing

© 2007 IBM Corporation

High Performance IO HPC Workshop – University of KentuckyMay 9, 2007 – May 10, 2007

Andrew Komornicki, Ph. D.Balaji Veeraraghavan, Ph. D.



Agenda

Introduction

General IO performance

Results of some small tests.

Modular IO libraries, Linux and AIX



I/O Optimization

Analyze the IO pattern

Determine optimization method

Optimize in user space

Minimize source code changes

Possibly relink with libtkio.so



General I/O Performance

C: Do not use fopen(), fread(), or fwrite(); • These are inefficient due to small (4KB) IO blocks and extra

memory copies.Use instead: • POSIX open(), read(), write() • Direct (raw) IO will eliminate an additional memory copy

FORTRAN: Use unformatted IO



Asynchronous IO, an example

Non Blocking IO aio_read(), aio_write(), aio_return();

Completion Notification Polling with aio_error(); Block until complete with aio_suspend():

Cancellation of IO requests aio_cancel();

Large File enabled Removes the 2GB file size limitation

POSIX conforming



Results of Bonnie IO test

Run on Blade system in San Mateo Lab

System Memory, 5 Gbytes

File systems, ext2, and ext3

All tests done in four stages: Writing with putc()...done

Rewriting...done

Writing intelligently...done

Reading with getc()...done

Reading intelligently... done



Results of Bonnie IO test, Block IO performance

Size (MB) Write (Kbytes/sec) Read(Kbytes/sec)

__________________________________________

2000 841,524 2,233,282

4000 83,237 1,658,013

8000 56,599 50,974

16000 49,656 50,677




Results for ext2 file system, time in seconds

Size (MB) User System Elapsed

_________________________________

2000 48.7 10.8 84.3

4000 97.6 23.9 252.6

8000 194.9 56.4 1009.2

16000 388.4 111.8 2088.3




Results for ext3 file system, time in seconds

Size (MB) User System Elapsed

______________________________________

2000 48.7 19.2 96.3

4000 97.7 45.5 265.8

8000 194.6 90.1 1016.9

16000 396.9 201.8 2058.3



Modular I/O (MIO)Modular I/O (MIO)

Familiar and flexible runtime interfaceMIO modulesmiotracepf

MIO available on both Linux and AIX



MIO user code interface

open MIO_open

read MIO_read

write MIO_write

close MIO_close

lseek MIO_lseek

fcntl MIO_fcntl

ftruncate MIO_ftruncate



MIO run time interfaceMIO run time interface

MIO_STATS="file name"MIO_FILES=" *.dat* [trace|pf ] *.inp [aix]"MIO_DEBUG="ALL"MIO_DEFAULTS="trace/mbytes , pf/cache=10m“



trace moduletrace modulesummary of file activitybinary events filelow cpu overheadtypical options

/stats/mbytes /gbytes /tbytes/events=mio.evt



pf module

User selectable cache size

User selectable page size

User selectable prefetch depth

Direct or system buffered IO

Global or private cache

Usage summary



pf modulepf moduledetects sequential I/Ouser memory bufferingoptions

/global/cache_size=10m/page_size=1m/prefetch=1/stride=1/direct/stats



Relink with libtkio.a

libtkio.a has shared object memberstkio.so 32 bit and 64 bit

Entry points for• open,open64,close,read,write,lseek,lseek64• fcntl,ffinfo,fstat,fstat64,fstatfs,fsync• ftruncate,ftruncate64• unlink,aio_...



Default tkio behavior

Uses dlopen and dlsym for runtime linking

……libc(shr.o) fsyncfsynclibc(shr.o) lseek64lseek64libc(shr.o) writewritelibc(shr.o) readreadlibc(shr.o) closecloselibc(shr.o) open64open64

callstkio entry



tkio runtime interface

setenv TKIO_ALTLIB so_name/print/abort

export TKIO_ALTLIB=so_name/print/abort

so_name is name of shared library• Either name.so or libname.a(name.so)

tkio calls function in so_name that returns a structure filled with I/O entry points to replace default entry points

/print option outputs a print to stderr indicating success of load

/abort issues exit(-1) if load is not successfull



tkio using MIOsetenv TKIO_ALTLIB get_mio_ptrs_64.so

…libmio(mio.o) MIO_fsyncFsynclibmio(mio.o) MIO_lseek64Lseek64libmio(mio.o) MIO_writeWritelibmio(mio.o) MIO_readReadlibmio(mio.o) MIO_closeCloselibmio(mio.o) MIO_open64Open64

Calls tkio entry



kernel

Application libclibtkio

Fortran I/O

Demonstration only

open64writereadlseek64close

->open64->write->read->lseek64->close

stdiofopenfrwritefreadfclose

libmio

->MIO_open64->MIO_write->MIO_read->MIO_lseek64->MIO_close

X



kernel

libclibtkio

open64writereadlseek64close

->open64->write->read->lseek64->close

libmio

->MIO_open64->MIO_write->MIO_read->MIO_lseek64->MIO_close t

race

pf

aix



System buffered Data MovementSystem buffered Data Movement

user space

kernel

256kb

system buffers

MIO space

pf cached Data Movementpf cached Data Movement

user space

kernel

256kb

5 x 2mb

system buffers

MIO space

O_DIRECT Data MovementO_DIRECT Data Movement

user space

kernel

O_DIRECT

256kb

5 x 2mb

system buffers

MIO space

Asynchronous Data MovementAsynchronous Data Movement

user space

kernel

O_DIRECT

256kb

5 x 2mb

system buffers

MIO space



Trace close : program <-> pf : /bmwfs/cdh108.T20536_13.SCR300 :(281946/2162.61)=130.37 mbytes/s

current size=0 max_size=16277mode =0777 sector size=4096oflags =0x302=RDWR CREAT TRUNCopen 1 0.01write 478193 462.10 59774 59774 131072 131072read 1777376 1700.48 222172 222172 131072 131072seek 911572 2.83fcntl 3 0.00trunc 16 0.40close 1 0.03size 127787

MSC.NASTRANMSC.NASTRANtracetrace output from output from program <program <-->pf>pf

Min/MaxRequest sizein bytes

Mbytes requestedand Mbytes delivered

Number ofoccurances



Trace close : pf <-> aix : /bmwfs/cdh108.T20536_13.SCR300 : (276645/1460.73)=189.39 mbytes/s

current size=0 max_size=16276mode =0777 sector size=4096oflags =0x8000302=RDWR CREAT TRUNC DIRECTopen 1 0.01write 4382 154.86 684 684 131072 2097152awrite 33390 1.42 58491 58491 131072 2097152suspend 33390 240.00 242.27 mbytes/sread 5178 272.71 10354 10354 1048576 2097152aread 103560 5.70 207115 207115 524288 2097152suspend 103560 786.04 261.59 mbytes/sseek 136950 0.00fcntl 3 0.00trunc 16 0.40close 1 0.00size 11013pages 138477

MSC.NASTRAN MSC.NASTRAN tracetrace outputoutput



pf close for /bmwfs/cdh108.T20536_13.SCR300global cache 0: 150 pages of 2097152 bytes29739/29749 pages not preread for write138316/139754 prefetches : prefetch=3

29576 write behinds478193 writes1777376 reads

page writes 37772/33124mbytes transferredprogram --> 59774 --> pf --> 59176 --> aixprogram <-- 222172 <-- pf <-- 217469 <-- aix

MSC.NASTRAN MSC.NASTRAN pfpf outputoutput



time ( seconds )

file position ( bytes )

DataView file activity plotDataView file activity plot



time ( seconds )


DataView file activity plotDataView file activity plot



time ( seconds )


suspend time

hidden time

queuing time

Asynchronous I/O plottingAsynchronous I/O plotting



time ( seconds )


cache page activitycache page activity



MSC.Nastran performance gainsMSC.Nastran performance gains

16 cpu 32GB NH2 node 2.2M dof, 767GB I/O, 8 copies2GB memory per copy

114MB/sec 198MB/sec

8 SSA, 16 loops, 4 disk/loop



MIO Summary

Demonstrated performance gains

Simple to implement

Flexible run time interface

Delivered as a shared object library

Contact: [email protected]

Documents

High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,