Upload
others
View
32
Download
0
Embed Size (px)
Citation preview
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Abstract This guide helps you troubleshoot issues using the isi_gather_info command to gather and upload cluster logs.
August 13, 2019
EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE
TROUBLESHOOT ISSUES GATHERING CLUSTER LOGS
OneFS 7.2 - 8.2
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
2 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Contents and overview
Before you beginPage 3
Appendix A If you need further assistance
Start troubleshootingPage 4
RoutingPage 5
Issues gathering log filesPage 6
Upload logs using HTTPPage 10
Note Follow all of these steps, in order, until you reach a resolution.
1. Follow these steps.
2. Perform troubleshooting steps in order.
Appendix BHow to use this flowchart
Upload logs using FTPPage 13
Debug log gatherPage 15
Appendix CExample output
3. Appendixes
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Before you begin
CAUTION!If the node, subnet, or pool that you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.
Therefore, make sure that you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial console connection available. This way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.
For specific requirements and instructions for making a physical connection to the cluster, see article 304071 on the Online Support site.
Before you begin troubleshooting, confirm that you can connect through either another subnet or pool, or that you have physical access to the cluster.
Configure screen logging through SSHWe recommend that you configure screen logging to log all session input and output during your troubleshooting session. This log file can be shared with EMC Isilon Technical Support, if you require assistance at any point during troubleshooting.
1. Open an SSH connection to the clsuter and log in by using the root account.
Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.
2. Change the directory to /ifs/data/Isilon_Support by running the following command:
cd /ifs/data/Isilon_Support
3. Run the following command to capture all input and output from the session:
screen -L
This will create a file named screenlog.0 that will be appended to during your session.
4. Perform troubleshooting.
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
4 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Start troubleshooting
Did you receive a Per mi ssi on deni ed
error?
Start
IntroductionStart troubleshooting here. For an overview of the conventions used in this flowchart, see Appendix B: How to use this flowchart.
If you have not done so already, log in to the cluster and configure screen logging through SSH, as described on page 3.
No
Are you having an issue gathering
logs?
Yes
Refer toOneFS: when running isi_gather_info you get Permission denied, article 499425
Go to Page 5
Go to Page 6Yes
No
Is the output from i si _gat her _i nf o
showing that all steps have been SKI PPED?
No
Refer toIsilon OneFS 8.2 isi_gather_info does not run and you see SKIPPED across each line item, article 535679
YesExample outputConf i gur i ng SKI PPEDr unni ng l ocal commands SKI PPEDr unni ng node commands SKI PPEDcol l ect i ng f i l es SKI PPEDgener at i ng package_i nf o. xml SKI PPEDt ar r i ng gat her SKI PPEDupl oadi ng gat her SKI PPEDcl eani ng up I N PROGRESS \
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
5 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Routing
Page 5
You could have arrived here from:
- Page 4 - Start troubleshooting
Are you having an issue uploading log
files?
No
Note the page number that you are currently on.
Upload log files and contact Isilon Technical Support, as instructed in
Appendix A.
Yes
Which method is being used to transfer logs?
Go to Page 10 Go to Page 13
HTTP
FTP
ESRS
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
6 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Issues gathering log files
Page 6
You could have arrived here from:
- Page 4 - Start troubleshooting
Are you using the OneFS web administration
interface?
YesWe suggest using the
command-line interface (CLI) for all log gathers.
Are you using options such as
- - i ncr ement al , - a, - A, etc.?
No
No
Did the log gather complete
successfully?
YesRun the following command
without any options:
isi_gather_info
YesEnd troubleshooting
Go to Page 7 Go to Page 15
NoYes, but withperformance
issues
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
7 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Issues gathering log files (2)
Page 7
You could have arrived here from:
- Page 6 - Issues gathering log files
Verify the health of the nodes in your cluster by running:
isi status
The Heal t h DASR column will indicate node health. See Appendix C for example output.
Are all nodes healthy?
Go to Page 9Yes
No
Run the following command and exclude the unhealthy nodes, where < healthy> is the list of nodes that have a status of OK:
isi_gather_info -n '< healthy> '
Using the output in Appendix C as an example, the command would look like this:i si _gat her _i nf o - n ' 2'
The - n flag indicates that the command will gather information only from the listed nodes. The list must be a comma-separated list or range of logical node numbers (LNNs), which is indicated in the ID column of the i si st at us output. If no nodes are explicitly listed,
the command gathers information from all cluster nodes. Note that nodes are automatically excluded if they are marked as Down.
Did the command complete
successfully?Go to Page 9 Go to Page 8YesNo
__________
_________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
8 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Issues gathering log files (3)
Page 8
You could have arrived here from:
- Page 7 - Issues gathering log files
This indicates that the problem is the unhealthy nodes. On the nodes that have a status of At t ent i on, run the
following command and try to resolve any issues that are presented:
isi events
Once you have cleared up any issues, run:
isi_gather_info
Did the log gather complete
successfully?YesEnd troubleshooting
Go to Page 9 Go to Page 15
NoYes, but withperformance
issues
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
9 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Issues gathering log files (4)
Page 9
You could have arrived here from:
- Page 7 - Issues gathering log files (2)- Page 8 - Issues gathering log files (3)
Does the log gather seem to be
hanging?No
Review the /var/log/messages log for error messages. Search the knowledge base for
known issues relating to any error messages that you find in the log. Try to resolve issues
with any articles that you find.
Yes
Run this command to see which nodes are still processing the log gather:
isi_for_array "ps auxww | grep _gather | grep -v grep"
For every node that is still processing the log gather, open an SSH connection to that node and run the following command, where
< directory> is the name of the directory that resulted from running the previous command. The directory name changes every time you
run the command:
bash -x /ifs/data/Isilon_Support /tm p/< directory> /_gather
Note the page number that you are currently on.
Upload log files and contact Isilon Technical Support, as instructed in
Appendix A.
Were you able to find a
solution?
End troubleshooting
No
Yes
____________________________________________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
10 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Upload logs using HTTP
Page 10
You could have arrived here from:
- Page 5 - Routing
Are you using the OneFS web administration
interface?
YesWe suggest using the
command-line interface (CLI) for all log gathers.
No
Find the ht t p_pr oxy_host line and note whether an IP address or host name is
listed there. If there is a value listed, that indicates that a proxy server is being used.
Go to Page 11
Run the following command to verify whether an HTTP proxy server is being used:
grep ht tp_ /etc/m cp/override/gather.xm l
If there is a proxy server, the output will appear similar to the following:
<set t i ng name=" ht t p_enabl e" >Tr ue<i si - met a- t ag i d=" ht t p_enabl e" can- mod- t ext =" yes" / >Tr ue</ set t i ng><set t i ng name=" ht t p_host " >178. 178. 178. 221<i si - met a- t ag i d=" ht t p_host " can- mod- t ext =" yes" / ></ set t i ng><set t i ng name=" ht t p_pat h" >/ dat a<i si - met a- t ag i d=" ht t p_pat h" can- mod- t ext =" yes" / ></ set t i ng><set t i ng name=" ht t p_pr oxy_host " >178. 178. 178. 222<i si - met a- t ag i d=" ht t p_pr oxy_host " can- mod- t ext =" yes" / ></ set t i ng><set t i ng name=" ht t p_pr oxy_por t " >1313<i si - met a- t ag i d=" ht t p_pr oxy_por t " can- mod- t ext =" yes" / ></ set t i ng>
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
11 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Upload logs using HTTP (2)
Page 11
You could have arrived here from:
- Page 10 - Upload logs using HTTP
Is a proxy server being
used?Go to Page 12
Run the following command, where < ht tp_proxy_host> and < ht tp_proxy_port> are the values from the output in the previous step:
export ht tp_proxy= ht tp://< ht tp_proxy_host> :< ht tp_proxy_port>
For example:expor t ht t p_pr oxy=ht t p: / / 178. 178. 178. 222: 1313
This command sets a variable in the current shell session so when it tries to upload it will use the specified proxy.
No
Yes
Go to Page 12
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
12 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Upload logs using HTTP (3)
Page 12
You could have arrived here from:
- Page 11 - Upload logs using HTTP (2)
Run
isi_gather_info
The output of the command gives you the path to the gathered log set on your cluster and will look similar to the following:
Packagi ng compl et e. . . Package: / i f s/ dat a/ I si l on_Suppor t / pkg/ I si l onLogs- Cl ust er 1- 20190611- 115485. t gz
Perform a test upload using the curl command to reproduce the upload that previously failed when you ran i si _gat her _i nf o. Run the command, where:
- < path to log file> is the same path to the gathered log set obtained from the output of i si _gat her _i nf o above.
- < ht tp_host> and < ht tp_path> are the values specified in the output of the grep ht tp_ /etc/m cp/override/gather.xm l command you ran on page 11.
- < log file nam e> is the name of the file from < path to log file> .
curl -T < path to log file> "ht tp://< ht tp_host> /< ht tp_path> /< log file nam e> "
Using the output from previous pages, here is an example:
cur l - T / i f s/ dat a/ I si l on_Suppor t / pkg/ I si l onLogs- Cl ust er 1- 20190611- 115485. t gz ht t p: / / 178. 178. 178. 221/ dat a/ I si l onLogs- Cl ust er 1- 20190611- 115485. t gz
Note the page number that you are currently on.
Upload log files and contact Isilon Technical Support, as instructed in
Appendix A.
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
13 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Upload logs using FTP
Page 13
You could have arrived here from:
- Page 5- Routing- Page 14 - Upload logs using FTP (2)
Run the following command to upload logs using FTP:
isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support
Was the upload
successful?Yes End troubleshooting
No
Refer toHow to upload files to Isilon Technical Support, article 304567
Go to Page 14
__________________________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
14 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Upload logs using FTP (2)
Page 14
You could have arrived here from:
- Page 13 - Upload logs using FTP
Is this the second time you have been at this point in
this guide?
YesThis indicates a possible issue with your environment. Engage
your FTP administrators to troubleshoot the issue further.
Change FTP to passive mode by running the following command:
export ftp_passive_m ode= yes
Note Changing FTP to passive mode only persists through the current SSH session. The default mode is active mode. If you wish to change it back to active mode before the SSH session ends, run:
export ftp_passive_m ode= no
No
Return to Page 13
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
15 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Debug log gather
Page 15
You could have arrived here from:
- Page 6 - Issues gathering log files- Page 8 - Issues gathering log files (3)
Debug while gathering logs by running the following command:
isi_gather_info --debug --verbose
This log gather will help Isilon Technical Support determine the source of your issue.
Note the page number that you are currently on.
Upload log files and contact Isilon Technical Support, as instructed in
Appendix A.
_________________________________________________________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
16 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Appendix A: If you need further assistance
Contact Isilon Technical SupportIf you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with. This information and the log file will help Isilon Technical Support staff resolve your case more quickly.
Upload node log files and the screen log file to EMC Isilon Technical Support1. When troubleshooting is complete, type exit to end your screen session.
2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your method of uploading files. If you are not sure which method to use, use FTP.
ESRS: isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support /screenlog.0
FTP: isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support /screenlog.0
HTTP: isi_gather_info --ht tp --local-only -f /ifs/data/Isilon_Support /screenlog.0
SupportIQ:Copy and paste the following command.Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly as it appears on the page), but when you press Enter, the command will run as it should.
isi_gather_info --local-only -f /ifs/data/Isilon_Support /screenlog.0 --noupload \--sym link /var/crash/Support IQ/upload/ftp
3. If you receive a message that the upload was unsuccessful, refer to article 304567 for directions on how to upload files over FTP.
____________
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
17 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Decision diamondYes No
Process stepProcess step with command:
com m and xyz
Go to Page #
Page # Note
Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.
CAUTION!Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.
End pointDocument ShapeCalls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.Sometimes linked to a process step with a colored dot.
Optional process step
IntroductionDescribes what the section helps you to accomplish.
Appendix B: How to use this flowchart
You could have arrived here from:
- Page 4 - Start Troubleshooting
Directional arrows indicate the path through
the process flow.
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
18 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Appendix C: Example output
You could have arrived here from:
- Page 7 - Issues gathering log files (2)
Example outputStatus of - A- - indicates At t ent i on.
cl ust er - 1# i si st at usCl ust er Name: cl ust erCl ust er Heal t h: [ ATTN]Cl ust er St or age: HDD SSD St or ageSi ze: 17. 4T ( 18. 1T Raw) 0 ( 0 Raw)VHS Si ze: 644. 6GUsed: 4. 4G ( < 1%) 0 ( n/ a)Avai l : 17. 4T ( > 99%) 0 ( n/ a) Heal t h Thr oughput ( bps) HDD St or age SSD St or ageI D | I P Addr ess | DASR | I n Out Tot al | Used / Si ze | Used / Si ze- - - +- - - - - - - - - - - - - - - +- - - - - +- - - - - +- - - - - +- - - - - +- - - - - - - - - - - - - - - - - +- - - - - - - - - - - - - - - - - 1| 11. 22. 33. 44 | - A- - | 0| 22. 7k| 22. 7k| 1. 4G/ 5. 7T( < 1%) | L3: 186G 2| 11. 22. 33. 44 | OK | 0| 0| 0| 1. 5G/ 6. 0T( < 1%) | L3: 186G 3| 11. 22. 33. 44 | - A- - | 0| 77. 7k| 77. 7k| 1. 4G/ 5. 7T( < 1%) | L3: 186G- - - +- - - - - - - - - - - - - - - +- - - - - +- - - - - +- - - - - +- - - - - +- - - - - - - - - - - - - - - - - +- - - - - - - - - - - - - - - - -Cl ust er Tot al s: | 0| 100k| 100k| 4. 4G/ 17. 4T( < 1%) | L3: 559G Heal t h Fi el ds: D = Down, A = At t ent i on, S = Smar t f ai l ed, R = Read- Onl y
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
___________________________
19 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot Issues Gathering Cluster Logs
___________________
Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.
Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com