7/24/2019 Corso GPFS Part 2

1/20

Tutorial for GPFS v.3.4

12/3/12 V.Sapunenko - GPFS tutorial 1

7/24/2019 Corso GPFS Part 2

2/20

12/4/12 V.Sapunenko - GPFS tutorial 2

Outline

Part 2:

! Intelligent Life-circle Management (ILM)! Storage pools and fileset!

Policies! Performance tuning

7/24/2019 Corso GPFS Part 2

3/20

12/4/12 V.Sapunenko - GPFS tutorial 3

Information LifecycleManagement (ILM)

! policy-driven automation and tiered storage management! storage pools

! Storage poolsallow the creation of disk groups within a filesystem (hardware partitioning)

! new feature in v. 3.2 - external storage pools (to interact with anexternal storage management, e.g. TSM)

! filesets! sub-tree of the file system namespace! administrative boundary to set quotas! control initial data placement or data migration

! user-defined policies

7/24/2019 Corso GPFS Part 2

4/20

Information LifecycleManagement in GPFS

! File placement policies

! Define where the data will be created(appropriate storage pool)

! Rules are determined by attributeslike

! File name! User name! Fileset

! File management policies! Possibility to move data from one pool

to another without changing filelocation in the directory structure

! Change replication status!

Prune file system (deleting files asdefined by policy)

! Determined by attributes like

! Access time! Path name! Size of the file

12/4/12 V.Sapunenko - GPFS tutorial 4

7/24/2019 Corso GPFS Part 2

5/20

Storage Pools!

ILM Manages sets of storage called "storage pools! What is a storage pool?

! A named subset of disks and tapes

! Each file is assigned to a storage pool based uponpolicy rules! placement policies (where to place files upon creation)

! migration policies (moving files from one pool to another)

! deletion policies (removing files from the storage system)

!

What are they are good for?! Tiered storage (files aged to slower/cheaper disk)

! Dedicated storage (e.g., per user or per project or per directory subtree)

! Failure containment!

To limit the amount of data lost due to a failure

! To bound the performance impact of RAID rebuild

! Appropriate use of special-purpose storage! Different RAID levels Enterprise grade disk vs. consumer-grade disk

! Multimedia friendly storage

12/4/12 V.Sapunenko - GPFS tutorial 5

7/24/2019 Corso GPFS Part 2

6/20

GPFS Filesets!

What they are:! A named subtree of a GPFS file system

!

Somewhat like a distinct file system, i.e. a fileset can be unlinkedwithout

! deleting it, and it can subsequently be linked using its name as a handle

!

What they are good for:

! Filesets can have quotas associated with them (global; not per-pool).

! - Fileset quotas are independent of user and group quotas

! Filesets can be used to restrict the effect of policies to specific files

!

Side effects:! Unlinked filesets can confuse programs that scan the file system

(e.g.incremental backup programs)

! Moving and linking between filesets is not allowed, in keeping with their

being like little file systems

12/4/12 V.Sapunenko - GPFS tutorial 6

7/24/2019 Corso GPFS Part 2

7/20

GPFS ILM/HSM Integration!

GPFS Integrates its ILM Policies with tape based HSMProducts! GPFS extends its Information Lifecycle Management (ILM)

functionality to integrate with HSM (Hierarchical StorageSystem) products.!

A single set of policies is used to move data between GPFS storagepools and tape storage pools.

! Supported HSM products include!

High Performance Storage System (HPSS)

! Tivoli Storage Manager (TSM)

! Cool Feature: very fast file scans

! 1 million files in 13 seconds!

1 billion files in 75 minutes

12/4/12 V.Sapunenko - GPFS tutorial 7

7/24/2019 Corso GPFS Part 2

8/20

Policies!

Initial placement policy:! Default rule:

! SET POOL system

!

Must be defined!

For a file system with SP system dedicated formetadataOnly

! With default rule cant write a single byte

! Can be changed at any time usingmmchpolicy fs name

!

To list policy usemmlspolicy fs name -L

12/4/12 V.Sapunenko - GPFS tutorial 8

7/24/2019 Corso GPFS Part 2

9/20

Sample GPFS ILM PolicyStatements

12/4/12 V.Sapunenko - GPFS tutorial 9

7/24/2019 Corso GPFS Part 2

10/20

More complex exampleExample of balanced placement policy for a file system with 4 storage pools:

RULE 'DATA3' SET POOL 'data1' LIMIT(99) WHERE INTEGER(RAND()*40)

7/24/2019 Corso GPFS Part 2

11/20

12/4/12 V.Sapunenko - GPFS tutorial 11

Migrate and delete policies! If the storage pool named system has an occupancy percentage above 90%

now, bring the occupancy percentage of system down to 70% by migratingthe largest files to storage pool data:

RULE 'mig1' MIGRATE FROM POOL system' THRESHOLD(90,70)WEIGHT(KB_ALLOCATED) TO POOL data'

! Delete files from the storage pool named system that have not been

accessed in the last 30 days, and are named like temporaryfiles or appearin any directory that is named tmp:

RULE 'del1' DELETE FROM POOL system' WHERE (DAYS(CURRENT_TIMESTAMP) DAYS(ACCESS_TIME) > 30) AND (lower(NAME) LIKE '%.tmp' OR PATH_NAME LIKE '%/tmp/%')

7/24/2019 Corso GPFS Part 2

12/20

Execution of migrate policy!

Manualy via mmapplypolicy:mmapplypolicy fs name P policy file I yes

! Via callback (user exit)mmaddcallback

! cannot be run from a Windows node

mmapplypolicy {Device | Directory} [-A IscanBuckets] [-a IscanThreads] [-BMaxFiles] [-D yyyy-mm-dd [@hh:mm[:ss]]] [-e]

[-f FileListPrefix] [-g GlobalWorkDirectory] [-I {yes | defer |test | prepare}] [-i InputFileList] [-L n] [-M

name=value...] [-m ThreadLevel] [-N {all | mount | Node[,Node...] | NodeFile | NodeClass}] [-n DirThreadLevel] [-P

PolicyFile] [-q] [-r FileListPathname...] [-S SnapshotName] [-sLocalWorkDirectory] [--single-instance] [--sort-buffer-

size Size]

12/4/12 V.Sapunenko - GPFS tutorial 12

7/24/2019 Corso GPFS Part 2

13/20

Performance Tuning!

File system! GPFS supports block sizes from 16 KB to 4MB with a default of

256 KB.

12/4/12 V.Sapunenko - GPFS tutorial 13

IO Type Application Examples Blocksize

Large Sequential IO Scientific Computing, Digital Media 1MB to 4MB

Relational Database DB2, Oracle 512kb

Small I/O SequentialGeneral File Service, File basedAnalytics,Email, Web Applications 256kb

Special* Special 16KB-64KB

*Since GPFS 3.3 there are very few workloads that benefit from a file system blocksize of16KB or 64KB. If you do not have a chance to test your application performance with variousfile system blocksize settings you should use the default of 256KB.

7/24/2019 Corso GPFS Part 2

14/20

File system (cont.)

! Data and Metadata separation! Use small and fast disks for metadata

! If network adapter on NSD servers gets saturateddedicated separate NSD servers for metadata

disks! place disks of different sizes in different

Storage pools

! scatter block allocation map usually performs

better

12/4/12 V.Sapunenko - GPFS tutorial 14

7/24/2019 Corso GPFS Part 2

15/20

Pagepool

! Default pagepool = 64M! generally to small

! If FS blocksize=N*256K (default) usepagepool=N*64M

! 256M good starting point

!

On NSD servers:! 30% of pagepool >

nsdThreadsPerDisk*#LUNS*maxBlockSize.

! On NSD servers running applications (NFS,XrootD, etc,) use pagepool>4G

12/4/12 V.Sapunenko - GPFS tutorial 15

7/24/2019 Corso GPFS Part 2

16/20

Some parameters!

maxMBpS! Default 150

! Recommended value =2*network BW

! maxFilesToCache!

Default 1000! Too high for worker nodes

! Too low for NFS server!

As a rule the total of ((maxFilesToCache + maxStatCache) *nodes) should not exceed (600,000 * (tokenMemLimit/256M) *(The number of manager nodes - 1))

! maxStatCache! Default 4*maxStatCache

12/4/12 V.Sapunenko - GPFS tutorial 16

7/24/2019 Corso GPFS Part 2

17/20

10 GigaBit Ethernet

TCP/IP Proc settings! net.ipv4.conf.all.arp_filter=1

! net.ipv4.conf.all.rp_filter=0

! net.ipv4.ipfrag_low_thresh=196608! net.ipv4.ipfrag_high_thresh=262144

! net.ipv4.ip_no_pmtu_disc=0

! net.ipv4.tcp_syncookies=1

! net.ipv4.tcp_max_syn_backlog=4096

! net.ipv4.tcp_window_scaling=1

! net.ipv4.tcp_sack=1

! net.ipv4.tcp_timestamps=0

! net.ipv4.tcp_rfc1337=1

! net.ipv4.tcp_adv_win_scale=2

! net.ipv4.tcp_rmem=4096 87380 16777216

! net.ipv4.tcp_wmem=4096 87380 16777216

! net.ipv4.tcp_mem=196608 262144 393216

! net.core.rmem_default=524287

!

net.core.wmem_default=524287! net.core.rmem_max=16777216

! net.core.wmem_max=16777216

! net.ipv4.tcp_keepalive_intvl=75

! net.ipv4.tcp_keepalive_probes=9

! net.ipv4.tcp_keepalive_time=7200

! net.core.netdev_max_backlog = 250000

12/4/12 V.Sapunenko - GPFS tutorial 17

Jumbo Frames Myri10GE Driver: options myri10ge myri10ge_initial_mtu=9000Configure 10GigE Interface to 9000MTU (ifcfg mtu 9000)

7/24/2019 Corso GPFS Part 2

18/20

TCP Memory (10Gbit Ethernet)

net.ipv4.tcp_rmem=4096 87380 16777216net.ipv4.tcp_wmem=4096 87380 16777216

net.ipv4.tcp_mem=1310720 2621440 3145728

net.core.rmem_default=524287

net.core.wmem_default=524287

net.core.rmem_max=16777216

net.core.wmem_max=16777216

net.ipv4.tcp_keepalive_intvl=75

net.ipv4.tcp_keepalive_probes=9

net.ipv4.tcp_keepalive_time=7200

net.core.netdev_max_backlog = 250000

12/4/12 V.Sapunenko - GPFS tutorial 18

7/24/2019 Corso GPFS Part 2

19/20

12/4/12 V.Sapunenko - GPFS tutorial 19

GPFS tuning guide

! http://www.ibm.com/developerworks/wikis/display/hpccentral/GPFS+Tuning+Parameters

7/24/2019 Corso GPFS Part 2

20/20

12/4/12 V.Sapunenko - GPFS tutorial 20

Acknowledgements

Materials used in this presentation, along with presenters ownexperience, have been mainly obtained from the following sources:! GPFS Best Practices Programming, Configuration, Environment and

Performance Perspectives by Raymond L. Paden, Deep Computing, IBM,2010

!

An Introduction to GPFS Version 3.2

by Scott Fadden, IBM Corporation, 2007!

IBM Cluster Information Center Website: http://publib.boulder.ibm.com

.