7/26/2019 Adressing Performance

1/3

KNOWLEDGE BASE

Knowledge Base Article: 000173061How to address/approach Unified File and Celerra Performance issues (000173061)

Version:10

Audience:Level 40 = Employees and Partners Original Create Date:Thu Apr 16 10:32:04 GMT 2015 Article Type:How To

Channels:Customer , Internal App First Published:Wed Nov 06 19:52:17 GMT 2013 Validation Status:Final Approved

Originally Created By:Wesley Bush Last Modified:Thu Apr 16 10:40:37 GMT 2015 Last Published:Thu Apr 16 10:40:37 GMT 2015

Summary:This How-to article is the starting point for every NAS performance isssue, be they VNX or Celerra.

Goal: How to address/approach Unified File and Celerra Performance issues

Issue: Every performance latency issue reported against any VNX-File or Celerra NAS system:

VNX performance issueNAS performance issueCelerra performance issueCIFS performance issueNFS performance issueMPFS performance issueNDMP performance issueiSCSI performance issue

Environment:Product:VNX VG10Product:VNX VG2Product:VNX VG8Product:VNX5100Product:VNX5200Product:VNX5300Product:VNX5400

Product:VNX5500Product:VNX5600Product:VNX5700Product:VNX5800Product:VNX7500Product: VNX7600Product: VNX8000Product: Celerra NSProduct: Celerra NS-120Product: Celerra NS-480Product: Celerra NS-960Product: Celerra NS-G2Product: Celerra NS-G8Product: Celerra NS350/NS350SProduct: Celerra NS40Product: Celerra NS40 DC NEBS GatewayProduct: Celerra NS40 DC NEBS IntegratedProduct: Celerra NS40 FC Integrated

Product: Celerra NS42Product: Celerra NS500-AUXProduct: Celerra NS500/NS500GProduct: Celerra NS600/NS600GProduct: Celerra NS600S/NS600GSProduct: Celerra NS700/NS700SProduct: Celerra NS701G/NS701GSProduct: Celerra NS704/NS704GProduct: Celerra NS80Product: Celerra NS80GProduct: Celerra NS20Product: Celerra NS20 FC IntegratedProduct: Celerra NS20 IntegratedProduct: Celerra NS22Product: Celerra NX4

7/26/2019 Adressing Performance

2/3

1.2.3.4.5.6.7.

1.2.3.4.5.

1.

Product: Celerra NX4 Dual Data Mover EnclosureProduct: Celerra NX4 Integrated Storage ArrayProduct: Celerra NX4 Single Data Mover Enclosure

Cause:

Change:

Resolution:

Section 1 : Clarifying the issue ( This is the most important part of the process )Specific protocols involved ? -- NFS (version), CIFS (version), NDMP, iSCSI, FTPIs the issue related to a specific export/spare, filesystem/lun ?What time of day does the issue occur ?What is the active workload at the time ? login, logout, backup, batch jobs What hosts are involved ? Backup, VMs, workstations, Servers get as much detail on the applications as possible.What does the Network config look like ? Specific interface/NIC ?Are network interfaces isolated on separate nics (ie: replication traffic on it's own single nic)Note: There is no such thing as too much information.Section 2 : Enumeration of the environment 1. The Celerra/VNX-FILE component.Gather the collect_support materials./nas/tools/collect_support_materials

Identify features in use; Replication, Dedupe, NDMP, CIFS, NFS, MPFS, etcReview the network port configuration. Review file system layout from the collect support

2.The Backend Array (Clariion/Symm/VNX-Block).Gather SPcollects, see Navisphere (or Unisphere) Help article "Generating and transferring SP log files" or:See EMC Knowledgebase for instructions on how to obtain SPcollects on a CLARiiON running Navisphere.5253See EMC Knowledgebase for instructions on how to obtain SPcollects on a VNX or CLARiiON running Unisphere.13969

3.The Network(Customer).See EMC Knowledgebase for instructions on how to handle the Global Network Services.34570

4.The Client Host(s)/Host Environment.

EMC Supportlink will search for the Grab utilities for different OS types.Download the utility and the README file. Follow the README file to run the utility for the host operating system chosen.Alternatively, many of these grabs can also be downloaded from .ftp://ftp.emc.com/pub/emcgrab/Note: During the initial stages some of the details of the customer network and environment may not be immediately available. If it is necessary tonecessary.Section 3 : Collection guidanceBase NAS, Array, Host and infrastructure CollectionsNetwork statisticsArray statisticsOffline Data Mover Statistics - see section 6Online Data Mover Statistics - see section 5Link to VNX/File Tiage KBA (189439)

Section 4 : What is available for VNX-File/Celerra performance data.

1.Server_stats is the keystone to VNX-File/Celerra analysis run "server_stats server_2 -list > stats.list"Note : running this command generates a list of every statistic that is available a the code thisdata mover is running. The first several are the typical ones to use when monitoring a systemonline. ( run "grep "\-std" stats.list" to get just the standard system stats, the top talkers -- cifs.client and nfs.client -- will not be listed )

2./nas/tools/profile_slotThis command writes its output to a file on the datamover, as specified with the -w flag. The output of this command is a summary of where each con "pcsample" or "function". -- the function method is the most frequently usedSection 5 : Live monitoring

Link to KBA for online perf analysis of VNX-File/CelerraAt a bare minimum you should have a terminal session running "server_stats server_X -m basic-std -i 30 -te no" at all times. This statistic provideda feel for the system activity.

https://emc.my.salesforce.com/articles/How_To/emc60493-How-to-gather-and-retrieve-SPcollect-files-from-a-CLARiiON-Storage-Processor-SPhttps://emc.my.salesforce.com/knowledge/publishing/articleOnlineDetail.apexp?id=kA0700000004JAwhttps://emc--c.na5.visual.force.com/apex/KB_BreakFix_1?id=kA1700000000CPghttps://support.emc.com/search/?text=EMC%20grab%20utilityftp://ftp.emc.com/pub/emcgrab/https://support.emc.com/kb/189439https://support.emc.com/kb/170271https://support.emc.com/kb/170271https://support.emc.com/kb/189439ftp://ftp.emc.com/pub/emcgrab/https://support.emc.com/search/?text=EMC%20grab%20utilityhttps://emc--c.na5.visual.force.com/apex/KB_BreakFix_1?id=kA1700000000CPghttps://emc.my.salesforce.com/knowledge/publishing/articleOnlineDetail.apexp?id=kA0700000004JAwhttps://emc.my.salesforce.com/articles/How_To/emc60493-How-to-gather-and-retrieve-SPcollect-files-from-a-CLARiiON-Storage-Processor-SP

7/26/2019 Adressing Performance

3/3

2.3.4.

1.2.

3.

With intermittant issues the top talker stats ( cifs.client and nfs.client ) should be running.Save the outputs to a file on one of the file systems built on the internal ide drive of the Control Staton. ( I typically use /celerra/ccc and create a subYou can process the files through "awk" to insert the timestamp at the first field of each line then interrogate the fields of interest in the output of va

Example:awk '{ if ( $1 !~ /^..:..:../ && NR 3 ) { if ( $1 ~ /^..:..:../ ) { tval=$1 ; print($0) } ; if ( $1 !~ /^..:..:../ ) { print(tval,substr($0,10)

Note: this assumes that there are 3 header lines, if there is a different number you will need to update the comparison values in the "if" statements

Section 6 : Offline data analysisThe offline data Analysis is performed by senior specialist in either support or engineering and requires a full suite of statistics in the csv format.The folowing statistics must be collected for every performance issue, concurrently with the online collection and analysis, while it is active, and pro

does not result in resolution. Every set of csv stats must have details on the times when the issue was active and when normal response times werEscalations must include the various collects and grabs and array statistics.Always pay strict attention to file system space on the control station as the collection of these csv formatted stats will result in large volumes of datThe server_stats command to be run is as follows :

Celerra 5.6 Family of Code :server_stats server_2 -summary basic,nfs,cifs,caches -table net,dvol,fsvol,nfs,cifs -interval 10 -format csv -count - > /tmp/stats.txtCelerra 6.0 & VNX 7.0 Code Families :server_stats server_X -monitorbasic-std,caches-std,nfs-std,nfs.v2,nfs.v3,nfs.v4,cifs-std,cifs.smb1,cifs.smb2,metaVolumes-std,diskVolumes-std,netDevices-std,store.ioTimeHist,st-interval 10 -format csv > some-meaningful-file-name.csv

VNX 7.1 Family of Code :server_stats server_X -monitorbasic-std,caches-std,cifs-std,cifsOps-std,nfs-std,nfsOps-std,metaVolumes-std,diskVolumes-std,netDevices-std,kernel.cpu.utilization,kernel.memory-vis eng -interval 10 -format csv > some-meaningful-file-name.csv

VNX 8.1 Family of Code :

server_stats server_X -monitorbasic-std,caches-std,cifs-std,cifs.smb1,cifs.smb2,nfs-std,nfs.v3,nfs.v4,metaVolumes-std,diskVolumes-std,netDevices-std,store.diskVolume,store.iocsv > some-meaningful-file-name.csvnote: if cifs is not started on 8.1 the command will not run if any cifs options are included, in earlier code level the server_stats command will start astarted on the mover ).Additional KB articles to analyze active performance issues.

Link to VNX/File Triage KBA (189439)

Link to KBA for online perf analysis of VNX-File/CelerraGCS Global Performance Standard Operating Procedure(SOP)What to look at when having a performance issue on the NAS

How to gather the necessary information for a CLARiiON or VNX performance analysis

Notes:

Notes(EmployeesandPartners):

Notes(Restricted):

Product: VNX/VNXe Family, VNX Application Protection Suite, VNX Event Enabler, VNX1 Series, VNX Storage Array for Mainframe, VNX VG10, VNX VG2,VNX5700, VNX5800, VNX7500, VNX7600, VNX8000, VNX for DLm2000, VNX for DLm6000, Celerra, Celerra NS, Celerra NS-120, Celerra NS-48Celerra NS40, Celerra NS40 DC NEBS Integrated, Celerra NS40 FC Integrated, Celerra NS42, Celerra NS500/NS500G, Celerra NS600/NS600G,NS701G/NS701GS, Celerra NS704/NS704G, Celerra NS80, Celerra NS80G, Celerra NS20, Celerra NS20 FC Integrated, Celerra NS20 IntegratedCelerra NX4 Integrated Storage Array, Celerra NX4 Single Data Mover Enclosure

Shared: Yes

OriginallyCreated By:

Wesley Bush

https://support.emc.com/kb/189439https://support.emc.com/kb/170271https://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004JaXhttps://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004Ilu&popup=truehttps://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004Huv&popup=truehttps://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004Huv&popup=truehttps://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004Ilu&popup=truehttps://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004JaXhttps://support.emc.com/kb/170271https://support.emc.com/kb/189439