Citrix TechEdge 2014 - Advanced Tools and Techniques for Troubleshooting NetScaler Appliances

SYN402: Advanced Tools and Techniques for Troubleshooting NetScaler AppliancesAndrew Redman | Lead Escalation Engineer

May 8, 2014

© 2014 Citrix. Confidential.2

Tweet about this session with hashtag #SYN402 and #citrixsynergy

© 2014 Citrix. Confidential.3

NetScaler System Overview

Troubleshooting Tools & Techniques

Case Studies

Resources

Conclusion

Q&A

Agenda

NetScaler System Overview

© 2014 Citrix. Confidential.5

Key NetScaler Processes

ns_master/NSPPEnsvpndnsaaadnsconf

nsauthdnslog.sh

nssyncnsreadfile

nslcdnsfsyncdnsnetsvc

nsconmsgnscollect

Runs Citrix NetScaler OSSSL VPN File TransferRBA and SSL VPN external authorizationWrites the ns.conf fileCLI authenticationControls logging for the newnslogHA synchronizationUsed to read SSL certificate filesRuns the front panel LCDSynchronizes bookmarks and SSL certificatesUsed by the GUI for configuration changesControls writing of the newnslogStatistics gathering for historical purposes

Process Description

© 2014 Citrix. Confidential.6

NetScaler File System

/var (hard drive) Logs - /var/log & /var/nslog

Install - /var/nsinstall

Trace - /var/nstrace

Core Dumps - /var/crash & /var/core

/flash (flash drive) Config - /flash/nsconfig

SSL Certificates - /flash/nsconfig/ssl

© 2014 Citrix. Confidential.7

NetScaler File System (cont.)

/flash (cont.) User Monitors - /flash/nsconfig/monitors

Custom Options - /flash/nsconfig

/ (ram drive) OS - (operating system)

Troubleshooting Tools & Techniques

NetScaler

> show techsupport

Critical System Data

In-Depth Performance Monitoring

Stats

Detailed Log Files

USER Command Logging

/var/tmp/support/collector_P_10.10.10.10_21Apr2014_21_42_tar.gz

NetScaler Tech Support Bundle

© 2014 Citrix. Confidential.10

The NetScaler Tech Support Bundle

© 2014 Citrix. Confidential.11

Citrix Predictive Support

© 2014 Citrix. Confidential.12

‘Single Mission … Data Collection’ FAQ: http://support.citrix.com/article/CTX131233

Data Collection Analyze DataTailored

Recommendations

© 2014 Citrix. Confidential.13

Use this box as a guideline for proper image placement. Place your image on top and align to box.

https://taas.citrix.com

© 2014 Citrix. Confidential.14

Technical Issues Flagged

© 2014 Citrix. Confidential.15

Investigate Issues Further

The ‘BELL’

© 2014 Citrix. Confidential.16

Crash File(s) Location

nscollect-542.gz

© 2014 Citrix. Confidential.17

Intuitive Navigation

Select a different newnslog file to view

© 2014 Citrix. Confidential.18

Detailed Graphs

Mouse over graph to see more detail

Informative reference legend

Download the data as an excel sheet

© 2014 Citrix. Confidential.19

IPMI – Intelligent Platform Management Interface

Default LOM IP Address: http://192.168.1.3

Change NetScaler IP Address

Obtain Health Monitoring Detail

Harvest Serial Number

Determine MAC Address

© 2014 Citrix. Confidential.20

Common CLI Show Commands

Common show commands for system information:

show node, show info, show license

Common show commands for vserver and service:

show lb vserver, show cs vserver, show service, show persistencesession

show connectiontable

Other common show commands:

show route, show ip

© 2014 Citrix. Confidential.21

Common CLI Stat Commands

Common stat commands for system information:

stat ns, stat cpu, stat interface

Common stat commands for vserver and service:

stat lb vserver, stat cs vserver, stat service

Other common stat commands:

stat dns, stat ssl, stat http

© 2014 Citrix. Confidential.22

Leveraging ‘nsconmsg’

Nsconmsg common use cases:

View events

View console messages

View statistics

Debug system counters

Debug load balancing issues

Debug CPU/Memory utilization

Make absolutely sure that you

use a capital -K and NOT a

lower-case -k

© 2014 Citrix. Confidential.23

Example ‘nsconmsg’ Usage

# cd /var/nslog

# nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg

current log file

newnslog

name of vserver

fqdn-ssl-vip

LB stats

ConLb=1

© 2014 Citrix. Confidential.24

Displaying debug performance informationNetScaler V20 Performance DataNetScaler NS10.1: Build 123.11.nc, Date: Feb 24 2014, 17:30:43

current time is Sun Mar 23 18:33:43 2014-------------------------------------------------------NATSession : Free(6553)A(6553)InUse(0)NATSession: Cur(Tcp[0] Udp[0] Icmp[0] Other[0])NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0])Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0SSF: Conn (Srvr 0 Clnt 0) U:0CM: Conn (Srvr 0 Clnt 0) Sessions PCB 0 NATPCB 0Z(SIP[0], C[0], SSL[0] Server[0] SIPDIP[0] DIP[0] SO[0])Mon: Probes: 434562009, Failed: 15VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 0S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms) Load(0) LConn_Idx: (C:0; V:0,I:1)-------------------------------------------------------CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014

current time is Sun Mar 23 18:33:43 2014

Mon: Probes: 434562009, Failed: 15VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF)S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms)

CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014

© 2014 Citrix. Confidential.25

# nsconmsg -K newnslog -j <name of VIP> -s ConLb=1(2 or 3) -d oldconmsg | more

# nsconmsg -K newnslog -s ConMon=1 -d oldconmsg

# nsconmsg -K newnslog -s ConMEM=1 -d oldconmsg

# nsconmsg -K newnslog -s ConSSL=1 -d oldconmsg

ConDebug - DebuggingConLb - Load BalancingConMon - Monitoring ProbesConMEM - Memory ManagementConCSW - Content SwitchingConSSL - SSL OffloadConCMP - CompressionConIC - Integrated Caching

© 2014 Citrix. Confidential.26

Log File Analysis

# cd /var/log

ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2947 0 : User nsroot - Remote_ip 10.13.73.65 - Command "login nsroot "********"" - Status "Success"

ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2948 0 : User nsroot - Remote_ip 10.13.73.65 - Command "show ns license" - Status "Success"

# zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE)

interface down vServer down panic signaled

© 2014 Citrix. Confidential.27

NetScaler + Wireshark = ‘thumbs up’http://www.wireshark.org

© 2014 Citrix. Confidential.28

RED HOT Wireshark Tip

Custom Columns

Custom Menu Options

Pre-build Custom Filters

Much Faster Analysis

Get the red hot details on how to empower your default Wireshark configuration in the Reference Section at the end of this presentation.

Troubleshooting Techniques & Case Studies

© 2014 Citrix. Confidential.30

Top Tips

Use Citrix Predictive Support

Pay attention to the issues!

Note the highlighted counter(s)

Use nsconmsg to see even more detail

Correlate time-frames in other log files

Achieve root cause analysis faster!

RCA

Achieved!

Case #1 - High Availability Synchronization

© 2014 Citrix. Confidential.32

Primary NetScaler Secondary NetScaler

Same type of appliance

Same firmware version

Same nsroot password

Same RPC Node password

Open requisite TCP ports

HA Pre-requisites

© 2014 Citrix. Confidential.33

Predictive Support Flags The Issues

© 2014 Citrix. Confidential.34

The HA Pair Struggled To Synchronize

# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted)

57520 0 71837018 nic_tot_bdg_mac_moved interface(0/1)

57521 0 71837018 nic_tot_bdg_mac_moved interface(0/2)

9861 0 65 nic_err_bdg_muted interface(0/1)

9862 0 65 nic_err_bdg_muted interface(0/2)

71,837,018 MAC Moves

65 Interface Mutes

© 2014 Citrix. Confidential.35

The ‘newnslog’ Time-Frame

# nsconmsg -K newnslog -d setime

Displaying start and end time information

NetScaler V20 Performance Data

NetScaler NS9.3: Build 54.4.nc, Date: Dec 20 2011, 22:44:41

start time Fri Feb 28 21:49:58 2014

end time Fri Feb 28 21:53:28 2014

total duration 00.00:03:30

data size 1,718,949 bytes

total duration 00.00:03:30

Case #2 - XA/XD Slow Performance

© 2014 Citrix. Confidential.37

XenAppTablets

Smartphones XenDesktop

Predictive Support

Critical Insight Gleaned

Preventative Approach

Don’t Underestimate

XA/XD Slow Performance

© 2014 Citrix. Confidential.38

Predictive Support Flags The Issues

http://support.citrix.com/article/CTX136926

© 2014 Citrix. Confidential.39

Performance Was Extremely Latent

# nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved

4263 0 23 nic_tot_bdg_mac_moved interface(0/1)

4264 0 51 nic_tot_bdg_mac_moved interface(1/1)

4265 0 28 nic_tot_bdg_mac_moved interface(1/2)

23, 51 & 28 MAC Moves

© 2014 Citrix. Confidential.40

Networking Issues Again?

# nsconmsg -K newnslog -d statswt0 | grep nic_err

4274 0 1995 nic_err_rl_pkt_drops interface(1/1)

4275 0 40736 nic_err_rl_pkt_drops interface(1/2)

4276 0 1995 nic_err_rl_rate_pkt_drops interface(1/1)

4277 0 40736 nic_err_rl_rate_pkt_drops interface(1/2)

4678 0 42731 allnic_err_rl_rate_pkt_drops

System Limits Exceeded

Rate-limited Packets!

© 2014 Citrix. Confidential.41

The Moral of the Story Leverage Citrix Predictive Support

LeveragePred. Support

Pay AttentionGain Quick

InsightDig Into

‘nsconmsg’On Target for

Success!

Resources

© 2014 Citrix. Confidential.43

Helpful Resources

Comprehensive NetScaler Counters

Wireshark Developer Editions

Customizing Wireshark Tutorial

Citrix Predictive Support Forum

NSTRACE Options

How To Manage VLAN’s, Interfaces and Subnets

Conclusion

© 2014 Citrix. Confidential.45

What We’ve Actually Covered

An Overview of the NetScaler System to give you a high-level understanding of the core system.

I shared with you some excellent Troubleshooting Tools that are available at your disposal.

I also discussed a few key Troubleshooting Techniques that you can use to diagnose issues.

I then highlighted two different Case Studies leveraging the tools & techniques that I shared with you in the presentation.

In addition I provided you with a few Resources for your future reference and edification.

Q & A

© 2014 Citrix. Confidential.47

Before you leave…

Conference surveys are available online at www.citrixsynergy.com starting Thursday, May 8 at 9:00 a.m.

Provide your valued feedback by 6:00 p.m. today to be entered to win one of many prizes!

Download presentations starting Monday, May 19 from the My Event Planning tool

© 2014 Citrix. Confidential.48

WORK BETTER. LIVE BETTER.