JUNIPER NETSCREEN TROUBLESHOOTING

educat io n serv ices coursew a re

NetScreen 5000 Series

Security Systems and ISG

Series Troubleshooting

Student Guide

NetScreen 5000 Series Security Systems and ISG Series Troubleshooting

Course SERT-NS5000 © Juniper Networks, Inc. 2

NOTE: Please note this Student Guide has been developed from an audio narration. Therefore it will have

conversational English. The purpose of this transcript is to help you follow the online presentation and may require

reference to it.

Slide 1

© 2010 Juniper Networks, Inc. All rights reserved. | www.juniper.net | Proprietary and Confidential

Netscreen 5000 Series



Welcome to Juniper Networks “NetScreen 5000 Series Security Systems and ISG Series Troubleshooting” eLearning

module.



Slide 2

© 2010 Juniper Networks, Inc. All rights reserved. www.juniper.net | 2CONFIDENTIAL SERT-NS5000

Navigation

Throughout this module, you will find slides with valuable detailed information. You can stop any slide with the Pause

button to study the details. You can also read the notes by using the Notes tab. You can click the Feedback link at

anytime to submit suggestions or corrections directly to the Juniper Networks eLearning team.



Slide 3


Course Objectives

� After successfully completing this course, you will be

able to:

•Distinguish between ISG Series and NS5000 Series

hardware configuration and packet flow

•Explain the importance of the ASIC functions

•Describe First Path and Fast Path in packet flow

•Differentiate between functions processed in the CPU

versus PPU

•Use and interpret debug commands unique to high end

systems

•Explain the workarounds for 3 typical troubleshooting

examples

After successfully completing this course, you will be able to:

• Distinguish between ISG Series and NS5000 Series hardware configuration and packet flow

• Explain the importance of the ASIC functions

• Describe First Path and Fast Path in packet flow

• Differentiate between functions processed in the CPU versus PPU

• Use and interpret debug commands unique to high end systems, and

• Explain the workarounds for 3 typical troubleshooting examples



Slide 4


Agenda: Netscreen 5000 Series Security Systems and ISG Series

� The High End Systems?

� Architecture

� Packet Flow

� ASIC Functions

� Debug

� Troubleshooting Examples

This course consists of 6 sections. The 6 main sections are as follows:

• The High End Systems

• Architecture

• Packet Flow

• ASIC Functions

• Debug, and

• Troubleshooting Examples



Slide 5


The High End Systems




The High End Systems

In this section we take a look at the high end systems: the ISG Series and the NetScreen 5000 Series.



Slide 6


Section Objectives

� After successfully completing this section, you will be

able to:

•Identify the two high end system series

•List the built-in modules and the interface cards in the

platform

•Identify the types of SPMs available with each of the three

Management modules

After successfully completing this section, you will be able to:

• Identify the two high end system series

• List the built-in modules and the interface cards in the platform, and

• Identify the types of SPMs available with each of the three Management modules



Slide 7


�ISG Series

•ISG1000, ISG1000-IDP

•ISG2000, ISG2000-IDP

What are the High End Systems? (1 of 4)

What are the High End Systems?

First we have the ISG Series, which is the lower range of the high end systems, with the ISG1000, and the ISG2000.

They can also have IDP for the security module, which we are going to see is provided as a built-in card.



Slide 8


�ISG Series Modules

•Management Module (built-in)

•Security Module (built-in)

• Provides IDP functionality

•ASIC module (built-in)

•Interface Cards:

• 4-portFE

• 8-portFE

• 2-portGE

• 4-portGE (starting from ScreenOS 5.4)

• 1-portXGE (starting from ScreenOS 6.1)


We have built-in modules and also the interface card. The built-in modules are the Management module, the Security

module for the IDP, and the ASIC module. Then we have the interface cards. There are four ports and eight ports fast

Ethernet (FE), two ports gigabit Ethernet (GE), and four ports GE as well. The four port is available starting from

ScreenOS 5.4 and the one port ten gigabit is available starting with ScreenOS 6.1.


Course SERT-NS5000 © Juniper Networks, Inc.

10

Slide 9


�NS5000 Series

•NS5200

•NS5400


We also have the NS5000 Series. These are in the higher range of the high end systems, and there are two chassis

— one is the NS5200 and the other is the NS5400. The NS5400 has two more slots for the line cards.



11

Slide 10


MGTMGTMGTMGT MGT2MGT2MGT2MGT2 MGT3MGT3MGT3MGT3

2G24FE2G24FE2G24FE2G24FE YESYESYESYES YESYESYESYES NONONONO

8G8G8G8G YESYESYESYES YESYESYESYES NONONONO

8G28G28G28G2 NONONONO YESYESYESYES NONONONO

2XGE2XGE2XGE2XGE NONONONO YESYESYESYES NONONONO

8G28G28G28G2----G4G4G4G4 NONONONO NONONONO YESYESYESYES

2XGE2XGE2XGE2XGE----G4G4G4G4 NONONONO NONONONO YESYESYESYES

�NS5000 Series Modules

•Management Modules

• MGT

• MGT2

• MGT3

•Secure Port Modules (SPM)

• 2G24FE

• 8G

• 8G2

• 2XGE-2

• 8G2-G4 (ScreenOS 6.1)

• 2XGE-G4 (ScreenOS 6.1)


What sorts of modules do we have for this platform? We have the Management modules and the Secure Port

Modules (SPMs). There are three types of Management modules, referred to as Management 1, 2 and 3. For SPM,

there is the two gigabit, 24-port fast Ethernet (2G24FE). Then there is the eight port gigabit and a two port ten gigabit.

With ScreenOS 6.1 we have the latest version of the eight gigabit and ten gigabit cards. We will see that in a

subsequent slide.

In the table here, you see how they can be used. For Management 1 we can use the 24 port FE and the eight Gig 1

card. With Management 2, we can also use the eight Gig 2 card and the 10 gigabit card, and with Management 3, we

can use only the newer generation of the eight Gig and the two port 10 Gig card.



12

Slide 11


Section Summary

� In this section, we:

•Identified the two high end system series

•Listed the built-in modules and the interface cards in the

platform

•Identified the types of SPMs available with each of the three

Management modules

In this section, we:

• Identified the two high end system series

• Listed the built-in modules and the interface cards in the platform, and

• Identified the types of SPMs available with each of the three Management modules



13

Slide 12


Learning Activity 1: Question 1

The built-in modules include which of the following?

A) Interface card

B) 8 port FE

C) High end system

D) ASIC module



14

Slide 13



With the Management-3 module we can only use which

one of the following?

A) Screen OS6.1

B) Newer generation cards

C) 24 port FE

D) SPM Built-in



15

Slide 14


Architecture




Architecture



16

Slide 15


Section Objectives


able to:

•Differentiate between the ISG and NetScreen 5000 chassis

•Use the commands “get system path” and “get chassis”


• Differentiate between the ISG and NetScreen 5000 chassis, and

• Use the commands “get system path” and “get chassis”



17

Slide 16


�Why is the architecture important? •To understand the packet flow

•Troubleshooting depends on it• These components are directly involved in the process

•Debugging in the CPU level is not always enough

•System behavior depends on the architecture• E.g., in ScreenOS 5.4, TCP SYN check is done in CPU on NS5000, but it’s done in PPU on ISG

•Features depend on the architecture• AES encryption done in ASIC for GigaScreen3 and 4

Architecture (1 of 11)

Why talk about the architecture? It’s very important to understand packet flow in the system, and to be able to

troubleshoot it because these components are directly involved in the process. When we do debugging in the CPU, it

may not always be enough to find the reason a packet dropped or why the traffic is not processed as expected, etc.

Also because the system behavior depends on the architecture — depending on the card or version that’s being used,

the behavior might be different. The example here is “TCP SYN check”, which is done in the CPU for the NetScreen

5000 Series, but for the ISG it’s done in the PPU. We are going to see what the PPU is later in the course. But the

PPU is inside the ASIC chip, so it’s very important for us to understand.

Another example that shows that features depend on the architecture is the fact that AES encryption is done in the

ASIC for GigaScreen3 and 4, which we will see when we look at the schematic.



18

Slide 17


�Highlights

•Use of ASIC chips to increase performance and throughput

•ISG Series have GigaScreen3 ASIC

•NS5000 Series have 3 different ASIC’s:

• GigaScreen2 – 2G24FE/8G SPM

• GigaScreen3 – 8G2/2XGE SPM

• GigaScreen4 – 8G2-G4/2XGE-G4 SPM

•Management and Security Modules with dual CPU


Let’s cover some highlights concerning the architecture. The use of ASIC chips increases the performance and

throughput, which is one great advantage of this platform. The ISG Series uses the GigaScreen3 ASIC.

The NetScreen 5000 Series has three different types that will depend on the secure port module used. They are listed

on the slide — the GigaScreen4 is the latest one, that’s in combination with the Management3 card that we saw in the

table in a previous slide.

Another important thing is that the Management and the Security modules have dual CPUs. One CPU is used to

process the flow of traffic and the other CPU is used to perform the task — for example, OSPF routing or some other

management task in the system.



19

Slide 18


�ISG Chassis

•1 x GigaScreen3 ASIC in the

ASIC module

•ASIC module has direct

connection with Management

and Security Modules via PCI

bus

•Management and Security

Modules have dual CPU

•Security Module has

additional FPGA (Field-

Programmable Gate Array)

I/OI/OI/OI/O I/OI/OI/OI/O I/OI/OI/OI/O I/OI/OI/OI/O

ASIC ModuleASIC ModuleASIC ModuleASIC Module

Security modules Security modules Security modules Security modules

Dual 1GHz PowerPC CPUDual 1GHz PowerPC CPUDual 1GHz PowerPC CPUDual 1GHz PowerPC CPU

2 GB RAM, FPGA2 GB RAM, FPGA2 GB RAM, FPGA2 GB RAM, FPGA

Management ModuleManagement ModuleManagement ModuleManagement Module

ISG SeriesISG SeriesISG SeriesISG Series

Network TrafficNetwork TrafficNetwork TrafficNetwork Traffic


Let’s look at the ISG Series. The basic structure consists of one ASIC module. At the bottom are the interface cards

that connect to the ASIC module, and the ASIC connects to the security module. In the ISG2000 you can have three,

and the ISG1000 can have two, for the IDP functionality. Then there’s the Management module. The security module

also has an FPGA to help provide high throughput to the system.



20

Slide 19


�ISG ASIC Module• Built-in

• 1 x GigaScreen3 ASIC

• All I/O cards connect to backplane with dedicated paths to ASIC chip

• Front End Processor – FPGA chips interface between I/O and ASIC (2 in

ISG2000 and 1 in ISG-1000)

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Control Bus

Slot 3

Slot 4

Slot 1

Slot 2

FPGA

FPGA

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Control Bus

Slot 3

Slot 4

Slot 1

Slot 2

FPGA

FPGA


Let’s look specifically now into the ISG ASIC module. That’s the focus of our attention because that’s where we need

to look when we are troubleshooting the platform. We have the GigaScreen3 ASIC, we have I/O cards, and we have

connection to the I/O cards, so there is a data bus from the I/O card to the FPGA, which is a front-end processor. You

can think of a switch that’s transferring the packets from the I/O cards to the ASIC chip for processing.



21

Slide 20


�ISG2000 Architecture•ISG-1000/2000 share the similar HW architecture•Single ASIC chip, FPGA chip, IO modules are separated with chip

012

Slot 3

MGT Module

Slot 2-0

Security Modules

ASIC Module

I/O Modules

FAN Module

3 012

Slot 3

MGT Module

Slot 2-0

Security Modules

ASIC Module

I/O Modules

FAN Module

3

Slot 3

MGT Module

Slot 2-0

Security Modules

ASIC Module

I/O Modules

FAN Module

3


Here we see a feature of the chassis looking at it from the top. On the left hand side is the rear of the chassis and on

the right hand side is the front. In the front are the I/O modules. Then we see the ASIC module; then 3 empty slots for

the security modules; in the back we see in slot three the Management module



22

Slide 21


�ISG-1000 Architecture•ISG-1000/2000 share the similar HW architecture•Single ASIC, Switch Fabric FPGA, IO modules are separated with chip

2 Slot for

Security Module

ASIC Module

Slot 3

Mgt Module

FAN

Module

Power Supply

Module

2 Slot for

Security Module

ASIC Module

Slot 3

Mgt Module

FAN

Module

Power Supply

Module


The ISG1000 is very similar. Here we see the front is on the left side of the picture. We see the ASIC module — it’s

always the one that’s closest to the I/O card. Then there are two slots in the middle for the security module. Here we

see again slot 3 for the Management module. Finally, there’s the power supply in the back of the chassis.



23

Slide 22


�NS5000 Chassis

• GigaScreen ASIC in the SPM

• 15Gbps switch fabric

interconnecting SPM’s

• Dedicated bus for control

• Dedicated bus for traffic to

MGT module

• MGT1 has one CPU

• MGT2/MGT3 have 2 CPU’s

MGTMGTMGTMGT

SPMSPMSPMSPM

SPMSPMSPMSPM

SPMSPMSPMSPM

15 Gbps Switch Fabric

NetScreen 5400NetScreen 5400NetScreen 5400NetScreen 5400


Next, let’s look at the NetScreen 5000, in general. We have more capacity here. There are 3 SPMs that share the 15

gigabit switch fabric. It has a dedicated bus for traffic control in the chassis and another bus for traffic to the

Management module. Later we will show when the SPM needs to send traffic to the Management module, that

dedicated bus is used to avoid any congestion.

Management2 and Management3 cards have two CPUs for flow and tasks. For Management1 they are in same

physical CPU, separated in the architecture of the software.



24

Slide 23


�NS5000 SPM (1)

•ASIC chips reside in the SPM’s — Number and type of ASIC

depend on the SPM’s:

• 2G24FE – 1 x GigaScreen2

• 8G – 2 x GigaScreen2

• 8G2/2XGE – 2 x GigaScreen3

• 8G2-G4/2XGE-G4 – 2 x GigaScreen4

•Front End Processor – FPGA chips interface between ASICs

and backplane to MGT board/ASIC’s in other SPM’s


Here you see the Secure Port Module of the NS5000. This would be the equivalent of the ASIC module that we saw

for the ISG Series.



25

Slide 24


�NS5000 SPM (2)

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA

BackplaneGigaScreen4

I/O

FPGA

GigaScreen4

I/O

FPGA

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen4

I/O

FPGA


Here you see there are two GigaScreen ASICs in each module. There are front-end processors that do the

interconnection within the cards, between the different ASICs, and also to the backplane if the traffic needs to go to

another SPM.

At the bottom you see the I/O interface. This can be one ten gig port or four one Gig ports.



26

Slide 25


ns5400-> get system | in productProduct Name: NetScreen-5400-II

isg2000-> get system | in productProduct Name: NetScreen-2000

ns5200-> get system | in productProduct Name: NetScreen-5200-II

nsisg1000-> get system | in productProduct Name: NetScreen-ISG1000

�How to check the hardware configuration?


How do we check the hardware configuration? This simple command shows what product we are talking about: “get

system | in product”.



27

Slide 26


ns5400-> get chassis Chassis Environment:Power Supply: GoodFan Status: GoodBattery Status: GoodCPU Temperature: 141'F (61'C)

Slot Information:Slot Type S/N Assembly-No Temperature DRAM Size1 Management-III 0225032008000036 0072-001 111'F (44'C) 2048MB2 Processing-2XGE-G4 0227032008000003 0085-001 116'F (47'C) 1024MB3 Processing-8G2-G4 0226032008000055 0084-001 109'F (43'C) 1024MB

isg2000(M)-> get chassis Chassis Environment:Power Supply: GoodFan Status: Good

CPU Temperature: 113'F ( 45'C)Slot Information:Slot Type S/N Assembly-No Version Temperature 0 System Board 0079022005000207 0051-005 E01 78'F (26'C), 86'F (30'C)4 Management 0081022005000392 0049-004 D06 113'F (45'C)3 Security 0137062005000114 0049-001 A02 cpu1:Ready, cpu2:Ready5 ASIC Board 000140527B050065 0050-003 C00

Marin FPGA version 9, Jupiter ASIC version 1, Fresno FPGA version 110I/O BoardSlot Type S/N Version FPGA version

1 1 port XFP 0229062008000062 A00 32 4 port 10/100 0084042004000002 D01 63 1 port XFP 0229062008000070 A00 3

�How to check the hardware configuration?


If you want to see details, you will use the command “get chassis”. Here you see an example — first for a NetScreen

5400. Management3 is the card being used and there’s one ten Gig module, and one eight Gig module, and they are

in slots two and three in this notation. You can see the serial number for each card, the assembly number,

temperature and the DRAM size.

At the bottom, the other output is for the ISG2000. Here also is a management board, but additionally there is the

security module, and then the ASIC module — as was shown in the schematic — and also the I/O cards. Also in the

middle you can see the FPGA version information. “Jupiter” is the internal name of the ASIC and “Fresno” is the

internal name of the FPGA. Those were the names used when the command was run.



28

Slide 27


Section Summary


•Differentiated between the ISG and NetScreen 5000

chassis

•Showed how to use the commands “get system path” and

“get chassis”


• Differentiated between the ISG and NetScreen 5000 chassis, and

• Showed how to use the commands “get system path” and “get chassis”



29

Slide 28



Most troubleshooting of the ISG platform focuses on

which of the following?

A) ASIC module

B) Management module

C) IDP functionality

D) I/O cards



30

Slide 29



SPM in NS5000 is equivalent to what in the ISG

Series?

A) I/O interface

B) ASIC module

C) FPGA

D) DRAM



31

Slide 30


Packet Flow




Packet Flow



32

Slide 31


Section Objectives


able to:

•Explain the difference between packet flow in First Path and

Fast Path

•Describe packet flow in the NS5000 and ISG Series

platforms

•Identify packet types that need to be processed at the CPU

level


• Explain the difference between packet flow in First Path and Fast Path

• Describe packet flow in the NS5000 and ISG Series platforms, and

• Identify packet types that need to be processed at the CPU level



33

Slide 32


�NS5000–First Path: CPU is involved in processing

1) Packet arrives interface chip

2) Packet is forwarded to FPGA

3) FPGA forwards it to ASIC

4) ASIC checks the packet and forwards it to CPU

5) CPU processes the packet and sends it back to ASIC

6) ASIC forwards the packet to FPGA

7) FPGA forwards packet to interface chip

8) Interface chip sends the packet out

MGT3CPU CPU

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

1

2

3

4

6

7

5

6 7

Packet

MGT3CPU CPU

MGT3CPU CPUCPU CPU

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen4

I/O

FPGA

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen4

I/O

FPGA

11

22

33

44

66

77

55

66 77

Packet

Packet Flow (1 of 6)

Let’s now look at Packet Flow. We want to show how packets go through different components so you know what to

look for when you are troubleshooting. We will first start with the NetScreen 5000. The example here is for the “First

Path”. The First Path is when the CPU is involved in processing the packet. We call it First Path because this process

is most commonly used when there is a packet for a new session. A new session is always created in the CPU so the

ASIC needs to forward traffic to the CPU for processing.

You see the packet at the bottom — step number 1. The packet arrives at the interface chip, then it will go to the

FPGA, and the FPGA then forwards it to the ASIC that’s directly connected to the FPGA. The ASIC looks at the

packet and determine that this one needs to be sent to the CPU. It will send it to the CPU via the backplane and then

the CPU will do the processing. Let’s say it creates the session and then sends it back to the same ASIC chip, and

then the ASIC chip will match the packet to an existing session. When the CPU processed the packet, the session

was created and installed in the ASIC chip. The packet received matches the session and is then sent out. At that

point the FPGA gets the packet and will forward it to the correct outgoing interface. The packet goes to the interface

and then it will leave the system.



34

Slide 33


�NS5000–First Path: CPU is involved in processing

•First packet for session creation• Packets that need ALG/DI/Web Filtering

• Packets for the following protocols need to be processed by CPU:• 0 : IPv6 Hop-by-Hop Option• 1: ICMP• 2: IGMP• 4: IP-in-IP• 58: ICMPv6• 89: OSPF• 103: PIM• 112: VRRP• 132: SCTP


Here are some more details about the First Path. To repeat, first it’s for session creation. When there is a packet that

doesn’t match any existing flow, it has to be sent to the CPU for session creation. Also, when we have Application

Layer Gateway (ALG) inspection or Deep Inspection (DI) or Web Filtering, the content of the packet needs to be

inspected so that, for example, in the ALG FTP the control connection needs to be inspected so that the dynamic

ports can be opened properly by the firewall. And there are other packets that also need to be processed on the CPU

level and these are mainly: ICMP, IGMP, OSPF, PIM, VRRR, SCTP and so on.



35

Slide 34


�NS5000–Fast Path: CPU is not involved: packet matches session

1) Packet arrives interface chip



4) ASIC checks the packet, matches session and forwards it back to FPGA


6) Interface chip sends the packet out

MGT3CPU CPU

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

1

2

3

5

54

4

Packet

MGT3CPU CPU

MGT3CPU CPUCPU CPU

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen4

I/O

FPGA

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen4

I/O

FPGA

11

22

33

55

5544

44

Packet 6


Now that we have considered the First Path, that requires CPU help to process the traffic, let’s now check the “Fast

Path”. It is called Fast Path because the CPU doesn’t get involved. The GigaScreen ASIC is capable of processing the

flow and avoids burdening the CPU. The packets are processed on the ASIC level and that’s how we get very high

throughput with this system.

Let’s look at how the packet flows. It first arrives at the interface chip, it goes to the FPGA, and then the GigaScreen

ASIC checks the packet and it will check it against the session table. It will go to session lookup engine to match the

session, and then it will match the session, identify the outgoing interface, and then send it back to the FPGA. Then

the FPGA can forward it to the interface port and it will then be sent out.



36

Slide 35


�ISG2000-IDP – First Path: Traffic is sent from CPU to

Security Module 1) Packet arrives interface card



4) ASIC checks the packet and forwards it

to CPU (pass 96 bytes to MM via PCI control bus)

5) CPU processes the packet and sends it to ASIC

6) ASIC receives the packet and forwards it to IDP (A complete packet is transferred to SM through Data Bus)

7) IDP processes the packet and sends it

to ASIC

8) ASIC sends packet to FPGA


10) Interface card sends the packet out

SMCPU CPU

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Slot 3

Slot 4

Slot 1

Slot 2

FPGA

FPGA

MMCPU CPU

32

4

89

56 7

Packet

SMCPU CPU

SMCPU CPUCPU CPU

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Slot 3

Slot 4

Slot 1

Slot 2

FPGA

FPGA

MMCPU CPU

MMCPU CPUCPU CPU

3322

44

8899

5566 77

Packet

1

10


Next, we check the First Path for the ISG2000 with the IDP security module. Let’s see how the packet flows in this

case. We start at the same point — the packet arrives at the interface card and then via the data bus goes to the

FPGA. The FPGA will send it to the ASIC chip; the ASIC chip checks the session table and will not find it. It will send it

to the Management module for the session creation in this example. If it’s ALG, the session actually is matched, but it

will have a flag to say this packet needs to go to the CPU for inspection — for further processing. Then the packet is

processed and it is sent back to the GigaScreen ASIC. If this is the case — for the security module to also inspect the

traffic — then the ASIC gets the packet and sends it to the security module. At this point, the whole packet is sent to

the security module — all the packet’s content — because the security module needs to receive all the data to be able

to inspect it. Then it is inspected and then it goes back to the GigaScreen ASIC, and then finally it will go out to the

interface.



37

Slide 36


SMCPU CPU

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Slot 3

Slot 4

Slot 1

Slot 2

FPGA

FPGA

MMCPU CPU

5

321

4

67

Packet

SMCPU CPU

SMCPU CPUCPU CPU

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Slot 3

Slot 4

Slot 1

Slot 2

FPGA

FPGA

MMCPU CPU

MMCPU CPUCPU CPU

55

332211

44

6677

Packet

�ISG2000-IDP – Fast Path: Traffic is directly to Security

Module1) Packet arrives interface

card

2) Packet forwarded to FPGA


4) ASIC checks the packet, matches session and forwards it to IDP

5) IDP processes the packet and sends it to ASIC

6) ASIC sends packet to FPGA


8) Interface card sends the packet out

8


How does it work in the case of Fast Path? The CPU is not involved, but the security module still has to inspect the

traffic. Again, the packet will go to the FPGA and then the GigaScreen ASIC. It will go straight to the security module

this time — no CPU involvement. Then the packet is processed and sent back. The GigaScreen ASIC will identify the

outgoing interface and send the packet out through the FPGA and then to the interface card and then out of the

system.



38

Slide 37


�What are the possible paths?

•NS5000

• Single-ASIC

• Cross-ASIC

•ISG2000

• Always single-ASIC

• Single FPGA

• Dual FPGA

•ISG-1000

• Always single-ASIC/single-FPGA

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Control Bus

Slot3

Slot 4

Slot 1

Slot 2

FPGA

FPGA

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen4

I/O

FPGA

GigaScreen4

I/O

FPGA

8G2-G4 SPM

FPGA


I/O

FPGA

GigaScreen4

I/O

FPGA

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Control Bus

Slot3

Slot 4

Slot 1

Slot 2

FPGA

FPGA

GigaScreen3

Data Bus

SDRAM

ASIC Module

Data Bus

Control Bus

Slot3

Slot 4

Slot 1

Slot 2

FPGA

FPGA


Let’s summarize the packet flow now; let’s think of possible paths. First, let’s consider the NetScreen 5000, which can

use what we refer to as Single-ASIC or Cross-ASIC. Single-ASIC is when the incoming traffic goes this way and then

return traffic goes out this way — out of the same ASIC chip.

Then we have cross-ASIC; it’s going to be this way. For example, incoming traffic goes here, then the return traffic

goes this way. When the traffic comes from the other side, it will come here, on the other interface set. It will go to this

ASIC for processing, and then this ASIC will process the packet, and then send it this way. Thus we have Cross-ASIC.

For the ISG, it’s always Single-ASIC because in the ASIC module it’s just one chip, but we think of the FPGA in this

case. We can have traffic coming here and going out the same FPGA or we can have traffic coming into the top FPGA

and going out of the bottom FPGA. This is important when we look at the output, so that we know which FPGA to

check and we know what to expect when we look at the counters.



39

Slide 38


Section Summary


•Explained the difference between packet flow in First Path

and Fast Path

•Described packet flow in NS5000 and ISG Series platforms

•Identified packet types that need to be processed at the

CPU level


• Explained the difference between packet flow in First Path and Fast Path

• Described packet flow in the NS5000 and ISG Series platforms, and

• Identified packet types that need to be processed at the CPU level



40

Slide 39



A new session is always created in the what?

A) ASIC

B) CPU

C) PPU

D) FPGA



41

Slide 40



Cross-ASIC processing is available in which Juniper

platform?

A) ISG1000

B) ISG2000

C) NS5000

D) GigaScreen 4



42

Slide 41


ASIC Functions




ASIC Functions



43

Slide 42


Section Objectives


able to:

•Differentiate between functions performed in the CPU

versus those done in the ASIC chip and PPU

•Use the “get ASIC PPU” command to see which functions are

processed by each PPU


• Differentiate between functions performed in the CPU versus those done in the ASIC chip and the PPU, and

• Use the “get ASIC PPU” command to see which functions are processed by each PPU



44

Slide 43


�ASIC benefits: Increase Performance and Throughput•FAST PATH: Traffic forwarding without using CPU

•VPN Encryption and Decryption (AES, 3DES, DES,SHA-1, MD5)

•TCP 4-Way close

•IP fragmentation re-assembly

•Screening

•IPSec fragmentation and re-assembly with IKE acceleration

•Byte counters / data collection from local session memory

•IPv6 acceleration

ASIC Functions (1 of 3)

Let’s now look at the ASIC functions to see what the ASIC is doing. The most important objective is to increase the

performance and throughput in the system. One of the benefits that the system has is Fast Path. This enables the

system to handle traffic forwarding without using the CPU, as we saw in the packet flow.

VPN encryption and decryption is also done in the ASIC chip, so it doesn’t increase CPU utilization to do that. It also

can be responsible for processing TCP 4-Way close; also to do fragmentation re-assembly, and additionally for some

screen functions, such as IDP flood, SYN flood, ISMP flood.

It can also perform IPsec fragmentation and re-assembly with the IKE acceleration. Additionally, it can provide byte

counters for the policy and IKE acceleration for IPv6 traffic. So, the IPv6 traffic is also processed on the ASIC level

without going to the CPU.



45

Slide 44


�Packet Processing Units (PPU)•Packet Processing Units (PPU)provide additional processing capacity in ASIC level

•Provide additional processing power for ASIC chip•PPU features:

• Defragmentation (cleartext and encrypted)• TCP SYN check• SYN proxy• SYN cookie• TCP 4-way close• IPv6 acceleration• HA packet forwarding (ISG)• Interface with IDP Security Module (ISG)• DSCP copy• Policy counters


One important part of this architecture in the ASIC chip is the PPU; the packet processing unit. It gives additional

processing capacity at the ASIC level. It is an entity that can be programmed to do different things. The features that

are supported in this PPU are listed in this slide.

It can perform defragmentation for both clear text and encrypted traffic. It can perform TCP SYN check, SYN proxy

and SYN cookie, “get TCPU 4-way close” and increase the acceleration like shown previously. It also does the HA

packet forwarding in the case of ISG, and also interfaces the IDP security module in the ISGs. It can also perform the

DSCP copy for QoS and policy counters to count the number of bytes.



46

Slide 45


�How to check PPU functions

•Total of 6 PPU’s in GigaScreen3 and 4

• Example for ScreenOS 6.3

•Use “get asic # eng ppu functions” for ScreenOS 5.4 and earlier

ns5400(M)-> get asic ppu functions PPU and XTCPU functions: Defragmentation of encrypted packets: PPU-A Defragmentation of clear-text packets: PPU-C Syn-proxy function: PPU-B Tcp-3way-check function: PPU-B sdram HA and IDP packet forwarding: PPU-D IDP processing: PPU-E Syn-cookie function: PPU-F IPV6 flow processing: PPU-A IPV6 tunnel processing: PPU-C and PPU-D IPV6 parser: PPU-E


How do you check these functions in the system? It’s simple with this command “get ASIC PPU functions”. If you run

this command, you can see the PPU. We have six PPUs in GigaScreen3 and 4 — the latest models. In this example

for ScreenOS 6.3, you can see the PPUs. For example, the SYN cookie function is processed by PPU-F. We have

PPUs from PPU-A to PPU-F. Another example highlighted here: defragmentation of clear-text is done by PPU-C.

These functions might change depending on the version, because of different features that were included. You can

check using this command.



47

Slide 46


Section Summary


•Differentiated between functions performed in the CPU

versus those done in the ASIC chip and PPU

•Used the “get ASIC PPU” function to see which functions are

processed by each PPU


• Differentiated between functions performed in the CPU versus those done in the ASIC chip and PPU, and • Used the “get ASIC PPU” function to see which functions are processed by each PPU



48

Slide 47



The ASIC chip increases the performance and

throughput in the system since it does what?

A) Enables traffic forwarding without using the CPU

B) Uses First Path

C) Gets packets through the firewall

D) Eliminates the need for FPGA



49

Slide 48



The PPU gives additional processing capacity to the

ASIC by performing which of the following?

A) Re-assembly

B) Isolation

C) Management

D) Defragmentation



50

Slide 49


Debug




Debug



51

Slide 50


Section Objectives


able to:

•Review general commands used in ScreenOS

•List the most important commands specific to high end

systems

•Explain how to collect the data and interpret the output

•Run “debug tag info” when looking for problems related to

CPU


• Review general commands used in ScreenOS

• List the most important commands specific to high end systems

• Explain how to collect the data and interpret the output, and

• Run “debug tag info” when looking for problems related to the CPU



52

Slide 51


�What are the troubleshooting commands?

• Same get/debug commands from ScreenOS

•Additional commands to troubleshoot different components

in the system

• Different commands depending on platform/card type

• Different outputs depending on card type/ScreenOS version

• In ScreenOS 6.2 and 6.3 the commands are visible and

documented

Debug (1 of 49)

Let’s now discus debugging and the commands that are used to troubleshoot the platform.

The first thing to note is that we have the same “get” and “debug” commands as ScreenOS. That’s going to help us

here. But we are also going to see additional commands — specifically for this platform. In the ScreenOS 6.2 and 6.3,

the latest version, we have these commands visible in the command line interface. If it’s an earlier version then they

are hidden, but you can execute them as normal.



53

Slide 52


�Common commands in ScreenOS

•General information:• get tech• get log system• get log system saved• get event

•Performance:• get performance cpu all detail• get performance session detail

•Session Information:• get session info• get session frag• get session

Debug (2 of 49)

The first set of commands consists of general commands that we use in ScreenOS. We want to check general

information, so we use “get tech”, “get log system”, “get log system saved” and “get event”. Then, for performance, we

use “get performance CPU all detail” and “get performance session detail”. For session information, we use “get

session info”, and for information about fragmentation counters and processing we use “get session frag”. The “get

session” command can be used for the complete session table. You can use that tool to investigate the data. You can

also run the session analyzer using “get session” output.



54

Slide 53


�Common commands in ScreenOS

•Interface and Screening statistics:• get counter stat

• get pps * (if ScreenOS 6.1 and later)

• get zone <zone> screen counter

•Memory and internal resources:• get net-pak s

• get gate

• get pport

• get tcp

• get flow

* Packet per second counts have to be enabled with “set pps” command* Packet per second counts have to be enabled with “set pps” command* Packet per second counts have to be enabled with “set pps” command* Packet per second counts have to be enabled with “set pps” command

Debug (3 of 49)

There are also some other things to check: interface and screening counters. First you check with “get counter stat”.

You can use packets per second (PPS) counters as well if you enable them with check PPS. You can check screen

counters with “get zone screen counter”. If you are looking for possible attacks, such as floods, you can check this

command.

For the memory and internal resources, use the command “get net-pak s”. For statistics, use “get gate”, “get pport”,

“get tcp” and “get flow”. This provides general information about how the system is allocating resources.



55

Slide 54


�Additional Commands for High End Systems

• get session hardware• Displays the hardware sessions installed in the ASIC chip

•get sat <asicnumber> counters• Displays information about the read-write pointers and the full counters of each queue in an ASIC.

•get sat <asicnumber> demux-counter• Shows the packets sent by ASIC to the CPU and packets dropped by Screening

•get sat <asicnumber> frq1• Displays the status of free buffer queue. Use the command to check for presence of leak in the buffer queue.

•get sat <asicnumber> x-context• Displays records of various memory tables, table addresses, and reset counters in an ASIC.

Debug (4 of 49)

Now we come to what’s really special about this platform. These are the most important commands we are going to

cover here and they are most commonly used in troubleshooting.

The command “get session hardware” is going to show the session tables on the ASIC chip itself. Sometimes there

may be a problem. For example, if the session table in the CPU is not the same as in the ASIC chip. We can get the

output to compare. With the command “get sat counters” you see the read-write pointers that are used for the queues.

There are different queues in the ASIC and it’s very important to see how the queues are — if they are full or if they

are free, if there are packets dropped, you can look for “queue full”.

Then there’s “get sat demux”. This is important as it enables you to see packets going to the CPU, and packets

dropped by the screening function. Then there’s “get sat frq1”, which is a command to see the free buffer queue. This

is basically to see how the packet’s buffers are being used.

With “get sat x-context” you see the output of some memory tables, and also some reset counters that are important.

We’ll show you see an example of everything later on.



56

Slide 55



•get arp asic <asicnumber>• Displays the ARP entries in an ASIC

•If 6.0r2 or later: get asic demux-counters• Equivalent to “get sat <asicnumber> demux-counters” but for the whole system instead of one ASIC chip

•get asic ppu defrag• Displays defragmentation statistics for cleartext and encrypted traffic for all ASIC chips

•get asic ppu syn-cookie• Displays statistics for syn-cookie Screening feature (SYN flood)

Debug (5 of 49)

This second set of commands is also specific for high end systems. With “get sat session” we see how sessions are

allocated in the hardware — in the chip. With “get ARP ASIC”, we see the ARP entries in the ASIC chip. You can also

use “get ASIC demux”. It’s the same as “get sat demux” but it will be information for the whole system.

If you have NetScreen 5000, with three cards, you have six ASIC chips. When you use “get ASIC demux”, you see the

counters for all of them in aggregate.

Then we have the command “get ASIC PPU” to check how the PPU is performing. Use “get ASIC PPU defrag” for the

defragmentation and “get ASIC PPU SYN-cookie” for the SYN cookie feature.



57

Slide 56



•get asic ppu syn-proxy• Displays statistics for syn-proxy Screening feature (SYN flood)

•get asic ppu tcp-3way-check• Displays statistics for TCP SYN check feature

•get asic ppu ipv6 • Displays statistics for IPv6 traffic acceleration in PPU

•get asic ppu ha-idp-fwd (ISG only)• Displays statistics for HA and IDP packet forwarding

•get asic ppu idp (ISG only)• Displays statistics for packets forwarded/received by IDP

•debug tag info• Displays additional information about packets going to CPU

* For ScreenOS 5.4 use “get asic eng ppu <option>”For ScreenOS 5.4 use “get asic eng ppu <option>”For ScreenOS 5.4 use “get asic eng ppu <option>”For ScreenOS 5.4 use “get asic eng ppu <option>”

Debug (6 of 49)

The “get ASIC PPU SYN-proxy” command displays statistics for the SYN-proxy screening feature (SYN flood); “get

ASIC PPU TCP 3-way check” displays statistics for the TCP SYN check feature.

Use “get ASIC PPU ipv6” for IPv6 traffic acceleration in the PPU. The command “get ASIC PPU HA-IDP fwd” is used

to display HA or IDP forwarding in the ISG. In the ISG the PPU can do the HA forwarding and also send packets to

security module.

If you run the “get ASIC PPU IDP”, you also get counters for the packets sent or received by the IDP security module.

Then there’s a debug command, which is “debug tag info”. This is very useful when you need to see what’s going to

the CPU. You can run this command to see the packet tags that go to the CPU for processing.



58

Slide 57


�Specific Commands per Platform

•NS5000-2G24FE

• get michigan

• Displays specific information for front end processor in 2G24FE card

•NS5000-8G2/2XGE/8G2-G4/2XGE-G4

• get arch

• Displays counters for front end processor in the SPMs using

GigaScreen3 and 4

•ISG

• get fresno

• Displays counters for front end processor in the ASIC module

Debug (7 of 49)

Let’s go ahead and look at the specific commands for each platform as well. If you have the 24 FE card you use “get

michigan”. If you have an 8 gig card or 10 Gig card you use “get arch”, and if you have an ISG, you use “get fresno”

because these commands are for the different FPGA chips that exist in each platform. You use different commands

for each of the different FPGAs.



59

Slide 58


�Commands to Collect

•NS5000 with 2G24FE SPM

• get sat <asicnumber> d

• get sat <asicnumber> x-c

• get sat <asicnumber> fr

• get sat <asicnumber> c

• get sat <asicnumber> s

• get arp asic <asicnumber>

• get michigan <slotnumber> count

• get michigan <slotnumber> igmac

Debug (8 of 49)

This is a simple example of the commands. For example, here are commands that you’d use for the NetScreen 5000

with the 24 FE card.



60

Slide 59



•NS5000 with 8G2/2XGE/8G2-G4/2XGE-G4 SPM

• get asic demux (if 6.0r2 or later)







• get arch <slotnumber>

Debug (9 of 49)

Here you see example commands in the case of the eight Gig or 10 Gig card. The “get sat” command and the “get

ASIC” command are always common. But now we use “get arch” instead of “get michigan”.



61

Slide 60



•ISG2000

• get asic demux (if 6.0r2 or later)







• get fresno 0

• get fresno 1*

* Only for ISG2000 (two FPGA’s)Only for ISG2000 (two FPGA’s)Only for ISG2000 (two FPGA’s)Only for ISG2000 (two FPGA’s)

Debug (10 of 49)

In the ISG we use “get fresno”. In the ISG1000 there is only

“get Fresno 0” since there is only one FPGA.



62

Slide 61


�How to Collect

•Most counters are absolute -> multiple outputs needed

•Recommendation:

• Run block of commands 5 times with 30 second interval

•How:

• Copy/paste commands in console session

• Script in ScreenOS (if 6.0 or later)

• Script in external tool

Debug (11 of 49)

Now the question that we have is how do we collect this output? You know the commands but you need to know how

do you actually collect them. The tip here is that most counters are absolute, so they will always increment — every

time you run a command, they increment. The idea is to run the commands five times during a 30 second interval, so

later you can check the delta between each output, and then compare if their counter is incrementing or not.

You may see some counter with a very high number but it could be it’s not incrementing anymore. That’s why we run it

a few times — usually it is five times. How do you do that? You can do copy/paste in the session — so console or

Telnet or SSH, or you can do a script in the ScreenOS itself if you create a script for that. Alternatively, you can use an

external tool to connect to the firewall and execute the command.



63

Slide 62


�How to collect? (NS5000 only)

•How to obtain <slot number>?

• get chassis shows the physical slot numbers <Slot>

• <slotnumber> = <Slot> - 2

• E.g. “get arch 2” is for SPM installed in physical Slot 4

•How to obtain ASIC number?

• Always “0” for ISG

• For NS5000 use “get asic mapping”

• E.g. NS5400 with 8G2 in Slot 2 and 2XGE in Slot 4

ns5400-> get asic mapping0 (ethernet2/1 to ethernet2/4)1 (ethernet2/5 to ethernet2/8)2 n/a 3 n/a 4 (ethernet4/1)5 (ethernet4/2)

Debug (12 of 49)

There is one thing about NetScreen 5000. How do you know the exact numbers that need to be put in the command?

In this case, when we do “get chassis” we see the slot number is 4, so the command is going to be “get arch two”

because we need to subtract two from the slot number to get the number. For the ASIC number, we always use zero

for the ISG because there is only one, but for the 5000 Series we have to use “get ASIC mapping”. You can easily see

which ASIC you need to check. Let’s say you have a problem with Ethernet 4/1, then you go check ASIC 4.



64

Slide 63


�Example: NS5400-8G2-G4/2XGE-G4 with

ScreenOS 6.2 (1)

•ASIC numbers: 0, 1, 4 and 5•Slot numbers: 0 and 2•List of commands:

• get asic demux

• get asic ppu defrag

• get asic ppu tcp-3way-check

• get asic ppu syn-cookie

• get asic ppu syn-proxy

• get sat 0 d

• get sat 0 x-c

• get sat 0 fr

•get sat 0 c

•get sat 0 s

•get arp asic 0

•get sat 1 d

•get sat 1 x-c

•get sat 1 fr

•get sat 1 c

Debug (13 of 49)

To summarize here, we will show an example. Here’s a NetScreen 5400 with an 8 gig card and a 10 gig card and the

ASIC numbers are 0, 1, 4 and 5. This means there is one card in slot zero and one card in slot two. Here are the

commands to run to get the data for all the system. We see the “get ASIC PPU” and the “get ASIC demux” are

common — you run it only once. With the “get sat” command and the “get arp” command you have to run it for each

ASIC.



65

Slide 64


�Example: NS5400-8G2-G4/2XGE-G4 with

ScreenOS 6.2 (2)

•ASIC numbers: 0, 1, 4 and 5•Slot numbers: 0 and 2•List of commands, cont’d:

• get sat 1 s

• get arp asic 1

• get arch 0

• get sat 4 d

• get sat 4 x-c

• get sat 4 fr

KB13216 - How to troubleshoot ASIC issues on Juniper Firewalls: NS5000 and ISG Series

• get arp asic 4• get sat 5 d• get sat 5 x-c• get sat 5 fr• get sat 5 c• get sat 5 s• get arp asic 5

get arch 2

get sat 5 c

get sat 5 s

Debug (14 of 49)

The “get arch” command is for each card, so get arch zero and get arch two. Refer to the Knowledge Base reference

document KB13216 for a more detailed explanation, as well as other examples.



66

Slide 65


�How to interpret the outputs?

•get asic demux (or get sat <asic> demux)nsisg2000(M)-> get asic demux-counters

Current(3d;13:27:15) Last(3d;13:27:15) PPS( 21s)to_host_packet: 928686 928632 2

SYN/ACK: 10577 10574 0FIN: 53221 53221 0RST: 26713 26708 0OTHERS: 838175 838129 2

first_packet: 1366460414 1366346964 5268brcst: 53933 53930 0no_ip_ether_net: 310335 310312 1ttl_zero: 978 978 0invalid_src_adr: 1300 1300 0udp_hdr_len_err: 159 159 0tcp_data_off_err: 1562 1561 0tiny_tcp_err: 29 29 0lan_attk: 211 211 0ping_of_death: 15 15 0tcp_chksum_err: 203246 203228 0udp_chksum_err: 56053 56039 0defragged_proc: 12578 12574 0total packet: 1368029499 1367915932 5274

clsf counters:fragment pak 76212 76206 0unknown protocol 225 225 0icmp 43214361 43210876 161

Debug (15 of 49)

Now that you have seen how to collect the data, even more importantly, you need to see how to interpret this output.

It’s very important that you know what you are looking at. The “get ASIC demux” output or “get sat demux” will provide

a similar output. Here you see the packets going to the CPU. You can see on the right-most column the PPS count —

the packets per second. This is the most important thing you need to check in this output. The slide is highlighted to

show there are 5000+ packets going to the CPU per second. This is something we consider very important when we

are looking at problems of performance. For example, in case we are having high CPU processing in the system, we

want to know why. We can run this command to see how many packets per second are going to the CPU. Then you

can understand whether that is expected or if that is overloading the system and you can make a decision about what

to do next. For example, we also see here a breakdown of the packets that go to the CPU. It can be packets to the

host or packets for the First Session. In this case, most of the packets that are going to the CPU are for First Session,

so they are packets that don’t match any session of the ASIC chip and were sent to the CPU for further processing.

Here we also see the counters of the packets that somehow were dropped. So, “ttl_zero” or “invalid source address”

or “TCP checksum error”, “UDP checksum error”. These were all packets that were dropped.



67

Slide 66


�get asic demux (or get sat <asic> demux)

•Shows packets going to CPU and dropped by Screening

•Check PPS counters on rightmost column

•What is important? • Find out how many packets per second are going to CPU

•Why is it important?• Troubleshooting of high CPU issues

•What to do next? • Determine if the pps observed is expected or solve problem in the network to reduce the load

• Investigate the type of packet that is going to CPU with high pps

Debug (16 of 49)

We look at the PPS counters and thereby understand what’s going to the CPU, and this is important for us to see if

there’s an attack or why the traffic is going to the CPU.



68

Slide 67



•get asic ppu defrag

nsisg2000-> get asic ppu defragShow ASIC 1 PPU information: —— Defragmentation of Encrypted Packets ——Total input packets: 0, Total Fragments: 0 First frag: 0, None-first Frag: 0 Defrag pass: 0, ESP frag: 0 Unexpedted packet: 0, To RSMQ: 0 AH frag: 0 —— Defragmentation of Clear-Text Packets ——Total input packets: 934463, First frag: 455095 Defrag pass: 905668, Defrag fail: 1301 Null Session Error: 643, Out-of node buffer: 0 PPU merge: 0

Debug (17 of 49)

Then there is “get ASIC PPU defrag”. You use that to check statistics about fragmentation. What is important here is

to check the new session error and the defrag fail. Usually, when there is a problem with defragmentation, that’s

where the counters increment.



69

Slide 68


�get asic ppu defrag

•Shows defragmentation in the PPU

•Check “Defrag Fail” and “Null Session Error”

•What is important?

• Find out if there are dropped or failed fragments increasing

•Why is it important?

• Fragmented traffic may be getting dropped

• Detect fragmentation in the network

Debug (18 of 49)

What else can you do for this case? You can check whether you really expect this defragmentation? Do you want this

fragmented traffic in the network?



70

Slide 69


�get asic ppu defrag

•What to do next?

• Determine if fragmentation is expected

• Use also “get session frag” to check fragment counts

• Check the other ASIC commands

• Capture packets to see which device in the network is dropping the

fragments

• Enable “no-hw-session” in the policy and check if the problem

stops

Debug (19 of 49)

Next, you can check “get session frag” output to look for the fragmentation count to see how many packets arrived as

first fragment, or no first fragment; fragments that couldn’t be re-assembled can also be checked with this command.

You can also correlate the data with the other ASIC commands to help you pinpoint the issue and you can also do

some packet captures. You want to see, did you really receive all the fragments that were sent to the firewall? Maybe

the firewall is not receiving all the fragments.

Then you can also tweak the policy configuration. Set “no hardware session” to see if that solves the problem. When

you do that you bypass the PPU defragmentation processing, and you can possibly isolate the issue.



71

Slide 70


�How to interpret the outputs?•get asic ppu tcp-3way-check

Ns54000-> get asic ppu tcpShow ASIC 1 PPU information: total input: 355742, total fwd: 355740 total drop: 3, redirect to client: 0 packet from server: 118611, msg send to server: 118555 msg rcv stage 4: 0, msg rcv stage 5: 0 Invalid session count: 0

Show ASIC 2 PPU information: total input: 118611, total fwd: 0 total drop: 0, redirect to client: 118611 packet from server: 0, msg send to server: 0 msg rcv stage 4: 118548, msg rcv stage 5: 3 Invalid session count: 0

Debug (20 of 49)

Similarly, you can use “get ASIC PPU TCP-3-way check”. Most important here are “total drop” and “invalid session

count”. This is to help you understand how the ASIC is processing the “3-way handshake”. You can see here there is

a total drop of three in ASIC one, and you have ASIC two receive stage five and also three.



72

Slide 71


�get asic ppu tcp-3way-check

• Shows TCP SYN check counters (set flow tcp-syn-check)•Check “total drop” and “invalid session”

•What is important?•Find out if there are dropped packets


•TCP sessions are not being established due to TCP SYN check

•TCP SYN check feature is faulty

Debug (21 of 49)

This is an example of a problem that “TCP 3-way check” was not working properly when the session involved two

ASIC chips. It was being dropped by one chip and the other was waiting stage 5.



73

Slide 72


�get asic ppu tcp-3way-check

•What to do next?

•Determine the conditions in which the problem occurs:

•Is it any TCP traffic or specific src/dst/service?

•Is there asymmetric traffic in the network?

•Check the other ASIC commands

•Disable TCP SYN check feature to see if the problem stops

•Get the session information of a connection test

• get session id <index>

Debug (22 of 49)

What else can you check with this output? You can try to understand the condition — is it all TCP traffic or is it a

specific source, destination, or service? In the problem we looked at there was traffic going through both ASIC chips,

so it was a special case.

Also check if there is asymmetric traffic — whether only one direction of the flow is going through the firewall. This

could be something that’s having an influence.

Also check the other ASIC commands. Look at the data of not only one output but also as a whole. One thing that can

be done as an action is “disable TCP SYN check” to see if that can help.

You can use “get session ID” because to see the status of the session — if it’s going normally or if it is not completing

properly.



74

Slide 73


�How to interpret the outputs?•get asic ppu syn-cookie

nsISG2000-> get asic ppu syn-cookieShow ASIC 1 PPU information: Syn-Cookie process statistics: Total input packets: 261628, Non-TCP first packets: 0 VLAN check fail: 0, TCP ACK: 0 TCP SYN: 26471, ACK decryption: 0 SYN encryption: 0, BGP bypass: 26471 From VPN engine: 0, Invalid ACK: 0

Debug (23 of 49)

The other command is “get ASIC PPU SYN-cookie”. It’s the same idea, so the most important things to check are

“VLAN check fail” and “invalid ack”.



75

Slide 74


�get asic ppu syn-cookie

•Shows counters for SYN cookie feature

•Check “Invalid ACK” • It doesn’t mean packet drop.

• ACK packet is not a cookie ACK but a first packet of the TCP connection

•What is important? • Find out if there are packets dropped by SYN cookie feature

•Why is it important?• Unable to pass TCP traffic

• Network under attack

Debug (24 of 49)

Here we can look at some attacks.



76

Slide 75


�get asic ppu syn-cookie

•What to do next? • Determine if there is an attack

• Determine if SYN flood thresholds are set correctly

• Check other ASIC commands

• Disable SYN cookie to see if the problem is solved

Debug (25 of 49)

Do we have a SYN flood attack or do we have the proper settings for SYN flood protection. We can also take action to

disable it for troubleshooting — to see if that will avoid the problem. Usually you may have a packet drop, and then

you can disable it and check.



77

Slide 76


�How to interpret the outputs?•get asic ppu syn-proxy

nsisg2000-> get asic ppu syn-proxy Show ASIC 1 PPU information: Syn-proxy process statistics: Total input packts: 615701, Xport-ESP input: 0 Xmit to client: 0 Xmit to server: 0 Xmit SYN/ACK: 0, Xmit RST: 0 Rcv SYN: 0, Rcv RST: 0 Rcv FIN: 0, From VPN engine: 0 VPN process drop: 0, Unexpected pack drop: 0

Debug (26 of 49)

For SYN-proxy, the counter to usually check is the unexpected packet drop, which will tell you if there is a problem.



78

Slide 77


�get asic ppu syn-proxy

•Shows SYN Proxy counters•Check “VPN process drop” and “Unexpected pack drop” •What is important?

• Find out if are dropped packets due to SYN Proxy

•Why is it important?• Packets are being dropped due to SYN Proxy

• SYN Proxy feature is being triggered

•What to do next? • Determine if SYN flood thresholds are expected

• Check syn cookie counters if enabled

• Determine if there isn’t any SYN flood attack

• Disable SYN Proxy to see if the problem is solved


Debug (27 of 49)

We can look further at the SYN flood attacks. Look at the configuration, see if the threshold is as expected; have a

look at the traffic to see if the load is expected or if it may be some kind of attack.



79

Slide 78


�How to interpret the outputs?•get sat <asic> countersnsisg2000(M)-> get sat 0 cQ name wrptr rdptr full emp size q_full_cnt0 frq1 001d 0039 0 0 0064 01 psra1 000b 000b 0 1 0000 02 psra2 006b 006b 0 1 0000 03 psra3 0000 0000 0 1 0000 04 psra4 0000 0000 0 1 0000 05 psrb 0000 0000 0 1 0000 06 cpu fifo 0002 0002 0 1 0000 06 cpu1 06f9 06f9 0 1 0000 77 slu 0007 0000 1 0 0007 959 8 spi 0001 0001 0 1 0000 09 rsm fifo 0019 0019 0 1 0000 09 rsm2 0000 0000 0 1 0000 010 xmt1 000f 000f 0 1 0000 011 xmt2 0004 0004 0 1 0000 3312 xmt3 0000 0000 0 1 0000 013 xmt4 0000 0000 0 1 0000 014 cpu3 0000 0000 0 1 0000 015 cpu4 0000 0000 0 1 0000 0

Debug (28 of 49)

Now let’s go to “get sat counters”. This is also a very important command, because here you look at the status of the

queue. Each line is one queue in the ASIC chip, and they send packets to each other. You see in the example the

session lookup queue is the one that is highlighted with a high “queue full count” number. You need to look at the

“queue full count” to see if it is incrementing. Queue full means the queue has reached capacity and cannot process

any more packets. There can be packets dropped because the queue was full and couldn’t receive more packets.

Also, it’s important to check the full column because, if this is “1”, it means the queue is full and then it may block all

the traffic. If the queue is full all the time, it will block the traffic all the time. We’ll see that in an example further on.



80

Slide 79


�get sat <asic> counters

•Shows status of each queue in ASIC chip•Each queue has different function:

• psr: parser

• xmt: transmit

• cpu: queue from CPU

• host: queue to CPU

• slu: session lookup engine

• ppb: PPU-B queue

• frq2: free buffer queue

•Check “full” and “q_full_cnt” columns•What is important?

• If full = 1 queue is full and can’t forward packets

• If q_full_cnt increments queue was full and reset

Debug (29 of 49)

As was mentioned, each line is for a different queue. They exist inside the chip, so we have “parser queue”, “transmit

queue”, “CPU queue”, “host queue”, “session lookup engine queue”, “PPU queue”, and “free buffer queue”.



81

Slide 80


�get sat <asic> counters

•Why is it important?• System is not forwarding traffic

• NSRP cluster went into split-brain scenario

• Traffic load is reaching system maximum capacity

• Traffic to IDP is being dropped

•What to do next? • Determine which traffic/services are being affected

• Disable the feature corresponding to the queue to see if the problem stops


• Check PPS to determine if traffic load is too high

• Check if “full” goes back to “0” – if not system reset is required

• Check “get log sys” for ASIC reinit messages

Debug (30 of 49)

If “queue full” is always “1” and it doesn’t go back to zero, it may require a reset to recover the system from the failure.



82

Slide 81



• get sat <asic> x-contextnsisg2000-> get sat 0 x-csaturn context: 0x03b3fdd8(80000000)sess pool, hdr:0x8c5c8600, tail:0x925ee700session: in use:214882, alloc:830920960, free:830706078, total:1048575sess shadow base: 0x63543980, size: 56soft session base: 0x07fd6650, size: 288ageout_fifo: 0x25974560ager: rd:0x2940ef, wr:0x2940efager wrap count: rd:198, wr:198, catchup:0ageout counters: rd:833175791, wr:145036538, not valid:1skip:0, never:0, twin active:0, dma miss:0 unlink err:0dma miss retry fail:0, dma miss retry succ:0cleanup:0, proc:830711627, by twin:0, batch:131072rsm rcv: 0, 2vpn: 0rsm onhold: 0, freed: 0rsm hash: 0x6326d3e0, pool: 0x0378b994/0x0378b9a4ras hold: 0, total packet after ras is 0hostq base: 0x04c00000, 0x6e000000hq2 rcv: 0x04d80000, xmt: 0x04d82000

saturn free buffer reinit count: 1saturn engine reset count: 1

st_dbg_asic_reinit: 0x04a2f8a8, val 0packet up/down between CPU and ASIC: 1tcp-syn-bit-check drop count: 1128272, tcp-syn-bit-check fragment drop count: 0

Debug (31 of 49)

This is a very important command as well: “get sat <asic> x-context”. Here you look for “free buffer reinit” and “engine

reset” counts. These two counters help us understand if there was any reset in the ASIC chip for any reason. If the

ASIC had to reset, you will see it here with these counters. If you are seeing packet drops in the network, you can look

at these and see if it was “reinit”, which means they were dropped.

Also, you check “packet up/down between CPU and ASIC” to see if, for any reason, there was some loop between the

CPU and the ASIC. One example could be the session exists in the CPU but doesn’t exist in the ASIC. So, the ASIC

receives a packet from the CPU and doesn’t know where to send it, it will send it back to the CPU. Then it stays in a

loop, and these are the counters you can check. This is good to check in the case of high CPU — you might have a

packet looping inside the system.



83

Slide 82


�get sat <asic> x-context

•Shows memory tables, addresses and asic status

•Look for “reinit” or “reset”


• Find out if there are ASIC reinits


• ASIC reinits drop traffic

• System may be overloaded

• To understand if there is ASIC failure

Debug (32 of 49)

Basically that’s what we check in this output.



84

Slide 83


�get sat <asic> x-context

•What to do next?

• Check “get sat <asic> c” for full queues or queue full increments

• Disable the feature using the PPU affected


• Check other ASIC outputs

•Output changes in 6.1 and later

• Defrag info

• Buffers

• Port information (Jupiter chip has 32 ports)

• Interface mac table

Debug (33 of 49)

In the 6.1 release and later, you can also see with this output defragmentation information — some additional buffers

that you usually don’t need to check — only when you get a special request via our engineering team.



85

Slide 84



•get sat <asic> frqns5400-> get sat 4 frqJPT 4 FRQ buffers...FRQ1 (4/97) buffer 29 duplicated 1 times!buffer 62 missing(0x000a3000)!buffer 104 duplicated 1 times!buffer 117 missing(0x000be800)!Buf allocated:cpu : 00000000 cpu1: 00000000 cpu2: 00000000 rsm : 800a3902 ppa : 80088902 ppb : 800bb902 ppc : 80088102 ppd : 00000000 ppe : 00000000 ppf : 00000000 pdma: 8008f102 fb0 : 80092902 fb1 : 80095102 CH00: 0009e902 CH01: 000ad102 CH02: 00000000 CH03: 00000000 CH10: 00095102 CH11: 00096902 CH12: 00000000 CH13: 00000000

FRQ2 buf allocated:cpu : 802d4100 cpu1: 80202100 cpu2: 80202900 rsm : 807fb100 ppa : 80200900 ppb : 80205100 ppc : 80203100 ppd : 80203900 ppe : 80204100 ppf : 80204900

wr=0x0000f29a, rd=0x0000e6a5, 0xbf5 bufs in frq2.FRQ2 buf HEALTHY, 11 bufs held expected:

No.1 buf 0x00200902No.2 buf 0x00201102No.4 buf 0x00202102No.5 buf 0x00202902No.6 buf 0x00203102No.7 buf 0x00203902

Debug (34 of 49)

Let’s now check another very important command, “get SAT FRQ”. This shows the state of the free buffers that are

used to store the packets. When you look here you see “buffer missing” messages, but please note that they might not

always indicate an issue. They are here but the ASIC itself can deal with that and avoid any problem. Also, you can

see here that the state is “HEALTHY”, so you don’t need to really worry about it.



86

Slide 85


�get sat <asic> frq

•Shows status of free buffer queue

•Look for “missing” buffers; “leak” and “Err:” -> do not

necessary indicate problem as ASIC can recover from it

•Do “get sat 0 frq | in bufs” few times and check if read/write

pointers are always the same -> LEAKns5400(M)-> get sat 0 frq | in bufs

wr=0x0000c276, rd=0x0000b681, 0xbf5 bufs in frq2.

FRQ2 buf HEALTHY, 11 bufs held expected:

Debug (35 of 49)

The condition you do need to worry about is when you do a “get SAT 0 FRQ | include bufs” and you see the read and

write pointers are always the same.



87

Slide 86




• Find out if there is buffer leak

•Missing buffers keep incrementing

• Status shows “LEAK”


• Buffer leak eventually can cause ASIC reinit

• Performance is affected

• System may be overloaded

• To understand if there is ASIC failure

Debug (36 of 49)

When the read and write pointers are always the same it means you might have a leak. It means all the buffers are

used and no more buffers are available, so no more packets can be processed. The consequence for the network is

that the system just stops forwarding the traffic.



88

Slide 87



•What to do next?

• If showing “LEAK” check multiple times to see if buffer list is always

increasing – it’s only a real leak if the buffer list is extremely long and

no buffers are being freed




Debug (37 of 49)

You can always correlate that with the “get sat counter” command, because it will tell you if there is any queue full. If

you have “frq full” in the “get sat counter”, you are going to see “frq is full”.



89

Slide 88



•get sat <asic> session• Shows session allocation information• Look for “leaked” counts

•What is important? • Find out if there are sessions leaking in the ASIC session table


• Session leak can cause packet loop between CPU and ASIC -> high

CPU problem

ns5400-> get sat 4 session Saturn chip 4 free session link list sanity check:session: total 524287, alloc 3013104, released 3001124, free 512307, checked_free 512307, leaked 0

Debug (38 of 49)

Then you have “get sat session”. This one usually is not a problem, but sometimes you may have a leak, so you have

sessions in the ASIC that are mismatching from the CPU session table.



90

Slide 89



•What to do next?• Check “get sat <asic> c” for full queues or queue full increments• Disable the feature using the PPU affected • Check “get log sys” for ASIC reinit messages• Check other ASIC outputs• Run “debug tag info” and “debug flow basic”

Debug (39 of 49)

This is nothing to worry about, because the ASIC can also deal with that, and the CPU as well can correct. It’s only a

problem if this output, this number of leak sessions, really starts increasing very high.



91

Slide 90



•get michigan

ns5400-> get michigan 1 countP3 rx count 53843, tx count 20896P4 rx count 0, tx count 0P5 rx count 59496, tx count 47859P6 rx count 0, tx count 47859P5 drop count 0, P6 drop count 0iTxrdy 3c, iRxrdy 0x0, Txrdy 0xf, Rxrdy 0x0

Debug (40 of 49)

Now let’s check some specific commands introduced earlier. The command, “get michigan”, for the FPGA for the 24-

FE card, looks for the drop counters.



92

Slide 91


�get michigan

•Shows counters for 2G24FE SPM front end processor

•Look for drops

•What is important? • Find out if there are drops

•Why is it important?• System capacity is being reached

• Hardware fault

Debug (41 of 49)

This usually is not a problem. When you have drops at this level of the FPGA chip most of the time there are hardware

issues.



93

Slide 92


�get michigan

•What to do next? • Determine the traffic load that is arriving the system

• Check “get count stat” and correlate the information




• Possible RMA

Debug (42 of 49)

In such cases, you can do a replacement or, if system capacity is being reached, then there is nothing else to do but

to increase the number of cards or change the design.



94

Slide 93



•get archns5400-> get arch 2

—————-- I/O Card 2, BigSur (0xf6c00000) —————-0 1

Alpine0 RxPktCnt 5a344e01 1fd0f200 Alpine0 RxErrCnt 00000000 00000000 Alpine0 TxPktCnt 37869601 62647801 Alpine0 TxErrCnt 00000000 00000000 Alpine1 RxPktCnt 39869601 62647801 Alpine1 RxErrCnt 00000000 00000000 Alpine1 TxPktCnt 74344e01 20d0f200 Alpine1 TxErrCnt 00000000 00000000

—————-- I/O Card 2, Alpine 0 (0xf6c0c000) —————-0 1 2 3

MacRxPktCnt bb9a bd6e bfd3 c308 MacRxErrPktCnt 0000 0000 0000 0000 MacTxPktCnt 5871 329e 2e28 31c8 MacTxErrPktCnt 0000 0000 0000 0000 JRxPktCnt 00a6e568 0088bd6e 0088bfd4 0088c309 JRxErrPktCnt 00000000 00000000 00000000 00000000 JTxPktCnt 00d4cf47 00978ca2 007967e0 00796ae3 JTxErrPktCnt 00000000 00000000 00000000 00000000 SRxPktCnt 0196c7b4 0178a0fd SRxPktErrCnt 00000000 00000000 STxPktCnt 014e9dfb 00f30c6a STxPktErrCnt 00000000 00000000

Debug (43 of 49)

The other specific command is “get arch” for the 8 gig or 10 gig card. Here in this command you see the names

“BigSur” and “Alpine”, which are the FPGA chips. Here you see the counters rx, tx, packet and error. What you look

for here are errors; you need to pay attention to that.

Another thing that might help here is to check if all the expected counters are incrementing. For example, you have

here four channels. If you have the eight Gig card you expect each channel to be related to one port, so you can see

here, you can run this command and see how they are incrementing, when you send traffic through the system.



95

Slide 94


�get arch

•Shows status of 8G2/2XGE/8G2-G4/2XGE-G4 SPM front end processor

•Look for “err”•What is important?

• Find out if there are error or drops in front end processor•Why is it important?

• Throughput is not as high as expected• Hardware failure• System capacity is being reached

•What to do next?• Determine if traffic load is not reaching system capacity• Check “get sat <asic> c” for full queues or queue full increments• Check “get log sys” for ASIC reinit messages• Check other ASIC outputs• Possible RMA

Debug (44 of 49)

Most of the time, when you look for errors, they are going to be hardware errors, in which case you do an RMA. While

it is certainly possible there may be a problem in how the packets are sent, that’s not very common.



96

Slide 95



•get fresno

nsisg2000-> get fresno 0fresno version is 0x66, Rocket IO modeiorx_pkt_cnt0/1/2/3 is 0x9a74, 0x0000, 0x7284, 0x31ffiotx_pkt_cnt0/1/2/3 is 0x2a83, 0x0000, 0x252a, 0x0000iorx_ipb_timeout_cnt0/1/2/3/4/5/6/7/8/9 is 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00jrx_pkt_cnt0/1/2/3 is 0x0000, 0x0000, 0x9a87, 0xa483jtx_pkt_cnt0/1/2/3 is 0x0000, 0x0000, 0x2a86, 0x252cjrx_pkt_sop_cnt0/1/2/3 is 0x0000, 0x0000, 0x0000, 0x0000jtx_pkt_sop_cnt0/1/2/3 is 0x0000, 0x0000, 0x0000, 0x0000rio_ipb_status0/1/2/3 is 0x00, 0x00, 0x00, 0x00rio_opb_status0/1/2/3 is 0x00, 0x00, 0x00, 0x00cross_fresno_rx0/1 is 0x0000, 0x0000cross_fresno_tx0/1 is 0x0000, 0x0000

SYNC , NO LINK , NO LINK , SYNC tx_total_frame_cnt = 00000000, 00000000, 00000000, 00000000tx_err_frame_cnt = 00000000, 00000000, 00000000, 00000000Rx_crc_frame_cnt = 00000000, 00000000, 00000000, 00000000Rx_err_frame_cnt = 00000000, 00000000, 00000000, 00000000Tx_real_error_pktcnt = 00000000, 00000000, 00000000, 00000000Tx_real_total_pktcnt = 00000000, 00000000, 00000000, 00000000Rx_real_error_pktcnt = 00000000, 00000000, 00000000, 00000000Rx_real_total_pktcnt = 00000000, 00000000, 00000000, 00000000Rx_real_illgl_pktcnt = 00000000, 00000000, 00000000, 00000000slot0 slot1 slot2 slot3XMTQ7 XMTQ6 XMTQ3 XMTQ4 XMTQ2

Debug (45 of 49)

The “get fresno” output is similar. For ISG you check the FPGA counters on the ISG platform. You also look for errors

to see if they are incrementing and here there is one extra detail so that you see the transmit queues. If you remember

from “get sat counters”, that output shows the queues. Here you see how the queues are used, so slot 2 is using the

“transmit queue three” (XMTQ3), for example.



97

Slide 96


�get fresno

•Shows status of ISG front end processor•Look for “err”•What is important?

• Find out if there are error or drops in front end processor

•Why is it important?• Throughput is not as high as expected• Hardware failure• System capacity is being reached

•What to do next? • Determine if traffic load is not reaching system capacity• Check “get sat <asic> c” for full queues or queue full increments• Check “get log sys” for ASIC reinit messages• Check other ASIC outputs• Possible RMA

Debug (46 of 49)

That’s what you look for with “get fresno”. The errors are basically the same idea as with “get arch”. Most of the time,

it’s either a hardware failure or you are really reaching the system capacity.



98

Slide 97


�How to run debug tag info•Example - telnet

• demux 4: first packet for the session

• Src-ip: 10.227.5.200 -> dst-ip: 4.4.4.4

• Src port: c52e -> dst port: 17

• Incoming interface eth2/1.400

• IPID = 0xe4ca

****************** 03167.0: tag (03a06f80) ******************pak length: 48 vlan qidx:6 slot:0 port:0 buffer:0x8028d91cprotcol:6 demux:4 l2idx:5190 ipid:e4ca flags:0x40008007 session pointer:0x00029247src:10.227.5.200 dst:4.4.4.4 sport:c52e dport:17********************** end tag info *************************st_tag_2_ifp: 10.227.5.200 -> 4.4.4.4, incoming ifp=ethernet2/1.400start demux process 4

Debug (47 of 49)

Now we come to the debug command, “debug tag info”. This is very important. You run it when you are looking for

problems related to the CPU. This command will show us only packets going to the CPU. If packets are being

processed only by the ASIC chip, we don’t see them in the debug.

The “debug flow basic” is the same; it only shows packets going to the CPU.

Why do we do “debug tag info”? Here you see the information from the packet going to the CPU and a lot of detail.

You see packet length and also the queue index that shows which queue sent the packet to the CPU. If you go to the

“get sat counters” you can see which queue has queue index 6. You see the address of the buffer, so if you want to

see the whole packet’s content, you can look at this buffer.

The protocol is six and then the demux tag, which is very important since it indicates why the packets went to the

CPU. “Demux 4” means, it’s the first packet for the session. If there was no session in the table in the ASIC chip, it has

to send it to the CPU for session creation.

You also see source address, destination address, source port and destination port here, in abbreviated notation.

Another important thing is the IPID of the packet. If you are looking for a packet loop, you can do this debug and then

you see it all — you see the same packet ID — five, ten, or 100 times; the same packet — so, there is a loop.



99

Please remember that the debug command can be service affecting — depending on the load in the system —

because it takes a lot of CPU time to do this debug. If the load is very high, you might create some interference.



100

Slide 98


�How to run debug tag info

•Run it for ~10seconds when the problem is happening• Set debug buffer to maximum size: set dbuf size 4096

• Clear debug buffer: clear db

• Run debug command: debug tag info

• Wait 10seconds

• Type “Esc” to abort

• Collect output: get db stream

•CPU intensive, affects system performance

•Look for “demux” number• 1: packet has to be sent to CPU for processing (e.g. ALG)

• 4: first packet

• 25: ICMP

Debug (48 of 49)

What we usually do is run debug for 10 seconds, and then type ESC to abort immediately, and then inspect the

output.

Another example is tag. We have “1”, which is a packet that had to be sent to the CPU for processing. Even if there is

a session, the packet needs to go to the CPU — for example, in the case of ALG — also, 25 is for ICMP, and ICMP

always goes to the CPU.



101

Slide 99


�How to run debug tag info

•What’s important?• Find out if there are too many packets going to CPU

•Why is it important?• Investigation of high CPU

• Packets that should processed only by ASIC are going to CPU incorrectly

• Packet loop between ASIC and CPU

•What to do next?• Determine if the packets going to CPU are expected

• If not, investigate the traffic pattern and policy configuration

• Check ASIC commands for queue full increments or reinits

Debug (49 of 49)

That’s it for this debug command. We always do correlation, so we check also the “get sat command”, especially “get

sat demux”, because then we know how many packets are going to the CPU per second.



102

Slide 100


Section Summary


•Reviewed general commands used in ScreenOS

•Listed the most important commands specific to high end

systems

•Explained how to collect the data and interpret the output

•Showed how to run “debug tag info” when looking for

problems related to CPU


• Reviewed general commands used in ScreenOS

• Listed the most important commands specific to high end systems

• Explained how to collect the data and interpret the output, and

• Showed how to run “debug tag info” when looking for problems related to CPU



103

Slide 101



We run the command “get sat counters” to do what?

A) Look at the status of the queue

B) See if there was any reset in the ASIC chip

C) Check for high CPU

D) Find SYN flood attacks



104

Slide 102



When the command “get michigan” shows drops on the

FPGA chip, most of the time it indicates what?

A) Mismatched sessions in ASIC and CPU

B) Leaked sessions

C) Hardware issues

D) Output defragmentation errors



105

Slide 103


Troubleshooting Examples




Troubleshooting Examples



106

Slide 104


Section Objectives


able to:

•Describe workarounds provided in the three most critical

troubleshooting examples occurring in the field

•Apply the commands described in each troubleshooting

example


• Describe workarounds provided in the three most critical troubleshooting examples occurring in the field, and

• Apply the commands described in each troubleshooting example



107

Slide 105


�Example 1 – System stops forwarding traffic•Scenario

• NS5400-MGT2-2XGE

• NSRP Active/Passive cluster

• ScreenOS 6.2r1

•Problem

• Master unit stops forwarding traffic

• Failover to backup unit doesn’t occur

• Manual failover needed to recover the services

• Reset needed to recover the system

ns5400-> get chass | in mbSlot Type S/N Assembly-No Temperature DRAM Size1 Management 0102032007000009 0058-005 109'F (43'C) 2048MB2 Processing-2XGE 0143072006000013 0063-003 114'F (46'C) 1024MB

Troubleshooting Examples (1 of 12)

The first real world troubleshooting example here, and one that is most service affecting, is when the system stops

forwarding the traffic. This example was with NetScreen 5400 Management 2, with the two port, 10 gigabit card, and it

was an active/passive cluster running the 6.2r1 release. What was the problem? The master unit just stopped

forwarding traffic; no traffic was being processed. It was service affecting because no failover to the backup unit was

triggered, so the traffic was not being processed. But they were still exchanging heartbeats, so there was no failover

that was triggered.

How was the situation resolved? A manual failover was done to the backup unit, so the backup unit was running well

— it recovered the services. Then the old master had to be reset to recover from that situation. Here we show the “get

chassis” output so you can see the information about the card.



108

Slide 106


�Example 1 – System stops forwarding traffic•Commands collected

• get sat 0 d

• get sat 0 x-c

• get sat 0 fr

• get sat 0 c

• get sat 0 s

• get arp asic 0

• get sat 1 d

• get sat 1 x-c

• get sat 1 fr

• get sat 1 c

• get sat 1 s

• get arp asic 1

• get arch 0

ns5400-> get asic mapping0 (ethernet4/1)1 (ethernet4/2) 2 n/a 3 n/a 4 n/a5 n/a


How did we investigate this problem? We collected the “get sat” commands. To look at the ARP table, these are the

most important commands: “get sat demux”, “get sat x-compact”, “get sat frq”, “get sat counter”, “get sat session” and

“get arp asic”. Also, use “get arch zero” to see the counters in the front-end processor.

Use the command “get asic mapping” to know which ASIC you need to check. You have to check zero and one, so

that’s why you see here both “get sat 0” and “get sat 1.”



109

Slide 107


�Example 1 – System stops forwarding traffic•Analysis

• “slu” ASIC queue full and not getting freed

• Packet loop between ASIC/CPU

• ASIC reinits

• Problem didn’t happen after disabling TCP SYN check

LISNS5400:FW1(M)-> get sat 0 x-c | in betweenpacket up/down between CPU and ASIC: 72

ns5400(M)-> get sat 0 cQ name wrptr rdptr full emp size q_full_cnt(…)7 slu 0007 0003 1 0 0007 349 (…)

LISNS5400:FW1(M)-> get log sys | in reinit## 2008-12-08 13:40:42 : reinit chip 0, invalid buf (380a7100).## 2008-12-08 13:41:42 : reinit chip 0, invalid buf (380bf900).


What did we see with this output? We were looking for the counters, so the first thing we note is the “slu” queue in the

“get sat counter” command was showing a lot of queue full. This was incrementing constantly. Every time we ran the

command the number was higher. Then we also noted that the queue full was always “1”, so that meant no packets

were being processed, the queue was full and stuck. It was dropping all the traffic. That’s why no packets were being

processed; no traffic was running.

Then we kept on checking the data and we also saw a lot of packets up and down between the CPU and ASIC. Also

we see that re-initialization in the ASIC chip. With the “get log sys” command, we saw “reinit chip zero” so there was

an invalid buffer.

So, we obtained these three pieces of evidence that there were problems on the ASIC chip. Then we did one try of

disabling the TCP SYN check, and we noted the problem was not happening anymore.



110

Slide 108


�Example 1 – System stops forwarding traffic

• Workaround

• Disable TCP SYN check

• unset flow tcp-syn-check

• unset flow tcp-syn-bit-check

•Root Cause

• Software defect: TCP SYN check was corrupting packets for cross-ASIC

sessions, causing packet loop between ASIC/CPU and slu queue

stuck.

•Solution

• Code was modified to implement the necesssary corrections


Then we know the workaround is to disable the TCP SYN check, but what’s important here is the investigation that we

did with engineering determined that TCP SYN check was corrupting the packets in the case of cross ASIC sessions.

Then we saw that because of the packet loop — between the ASIC and the CPU — the session lookup queue got

stuck and couldn’t recover — and then it couldn’t process any more packets. That’s why the system stopped

forwarding the traffic.

The solution in this case was to modify the code to avoid this problem of corrupting the packets, and then the problem

was solved. Now we don’t have this issue anymore.



111

Slide 109


�Example 2 – TFTP transfers not working •Scenario

• NS5400-MGT3-2XGE-G4/8G2-G4

• NSRP Active/Active cluster

• ScreenOS 6.1r4

• Sessions are cross-ASIC (2XGE-G4 to 8G2-G4)

•Problem• Specific users cannot do TFTP transfers through the cluster

• Transfer starts but after a few seconds it hangs

• If “no-hw-session” is enabled in policy transfer is successful

SDU:Jabbar-NS5400(M)-> get chas | in mbSlot Type S/N Assembly-No Temperature DRAM Size1 Management-III 0225082008000060 0072-001 109'F (43'C) 2048MB2 Processing-2XGE-G4 0227062008000032 0085-001 123'F (51'C) 1024MB3 Processing-8G2-G4 0226092008000027 0084-001 116'F (47'C) 1024MB


The second example is also with the NetScreen 5400, but now with the Management-3 card, and with the new

interface cards — ten Gig and also eight Gig. Also, we have in this case active/active cluster, ScreenOS 6.1r4, and we

saw that all the sessions were cross-ASIC — going from a 10 Gig port to an eight Gig port. The problem is we had

some specific users that couldn’t do TFTP transfers through the cluster. From the client side, we could see the

transfers were starting but after a few seconds they would just hang. We suspected some of those problems were in

the ASIC level, so we enabled “no hardware session” in the policy, especially for that client, and then we saw that port.

We could then see that we had something in the ASIC that’s causing the problem, because the “no hardware session”

will bypass the processing in the PPU.



112

Slide 110


�Example 2 – TFTP transfers not working •Analysis (1)• Packet captures showed that only tranfers with fragmented packets were unsuccessful

• No fragment drops were detected in the system

• ASIC commands didn’t show any anomaly

• No ASIC queue full, no reinits

ns5400(M)-> get asic ppu defragShow ASIC 1 PPU information: —— Defragmentation of Encrypted Packets ——Total input packets: 0, Total Fragments: 0 First frag: 0, None-first Frag: 0 Defrag pass: 0, ESP frag: 0 Unexpedted packet: 0, To RSMQ: 0 AH frag: 0 —— Defragmentation of Clear-Text Packets ——Total input packets: 353294, First frag: 82415 Defrag pass: 352369, Defrag fail: 0 Null Session Error: 0, Out-of node buffer: 0 PPU merge: 0


What’s the analysis we did here? We did some packet captures to see why only that specific client was having a

problem. We saw that those clients were doing transfers with fragmented packets. The block size of the TFTP was

8000 bytes or so, so it was causing fragmentation. Then what do we do? Let’s check “get ASIC PPU defrag”, because

that’s where the defragmentation is done. But here we see zero — no defragmentation error; no null session error. So

the PPU processing seemed to be fine. We continued to look at the other ASIC commands. They also didn’t show

anything that could really pinpoint the problem. What do we do next? We did a “debug tag info”.



113

Slide 111


�Example 2 – TFTP transfers not working

•Analysis (2)

• debug tag info showed the packets going to CPU incorrectly

• Session already created, fragmented traffic is processed only by ASIC

• Demux = 4 -> packets were considered first packets incorrectly

****************** 03167.0: tag (03a06f80) ******************pak length: 48 vlan qidx:6 slot:0 port:0 buffer:0x8028d91cprotcol:6 demux:4 l2idx:5190 ipid:e4ca flags:0x40008007 session pointer:0x00029247src:10.227.5.200 dst:4.4.4.4 sport:c52e dport:45********************** end tag info *************************st_tag_2_ifp: 192.168.25.30 -> 192.168.33.43, incoming ifp=ethernet2/1.43start demux process 4


We decided to see whether there was something wrong going to the CPU. We did “debug tag info” and then we saw

what the problem was. We saw these fragments were going up to the CPU. They belonged to a flow that didn’t exist,

but they were being sent to the CPU with demux tag four; they were considered first packets for a new session. It was

confusing the CPU because the CPU already had a session for that traffic. The packet was not sent out. It was being

dropped when the ASIC received it. That was the issue.



114

Slide 112


�Example 2 – TFTP transfers not working

•Workaround

• Enable “no-hw-session” in the policy

• All packets are processed by CPU

• Debug flow basic confirmed correct processing

•Root Cause

• Software defect: PPUC fragment handling was incorrect, causing ASIC

session matching to fail and send packet to CPU

•Solution

• Code was modified to implement the necesssary corrections


What we did as a workaround is we used “no hardware session” in the policy. In that case, the packets are processed

in the CPU, and we saw from the root cause that the PPUC, which is the one that handles defragmentation, was

incorrect. We saw zero errors, but that was incorrect, so it was using a bad hashing mechanism to match the session

table in the ASIC. This was causing session matching fail in the ASIC. Then, because no session was found in the

ASIC, it was sent to the CPU. The CPU was confused and the packet was not sent out. The solution here was also to

modify the code and now we don’t have this problem anymore in the latest version.



115

Slide 113


� Example 3 – System showing abnormal high CPU

•Scenario

• NS5400-MGT2-2XGE

• ScreenOS 6.2r1

•Problem

• System showing high CPU

• Determine the reason for this behavior

ns5400-> get perf cpu all detailAverage System Utilization: 5% (flow 6 task 3)Last 60 seconds:59: 20(30 2) 58: 20(30 1) 57: 29(39 3) 56: 79(89 7)** 55: 78(88 8)** 54: 78(88 7)** 53: 77(87 6)** 52: 77(87 6)** 51: 77(87 6)** 50: 77(87 6)** 49: 77(87 6)** 48: 77(87 6)** 47: 77(87 6)** 46: 77(87 6)** 45: 76(86 5)** 44: 77(87 7)** 43: 76(86 6)** 42: 77(87 6)** 41: 76(86 6)** 40: 76(86 5)**


Here’s another example, which is regarding abnormally high CPU. This is something that is also important for the

system.

What causes high CPU? In this example we have NetScreen 5400 with the 10 gigabit card running ScreenOS 6.2r1.

We have a system showing high CPU. The first command to use when high CPU exists is “get perf CPU all detail”.

The word “all” is critical since, when using it, it will break down the CPU utilization.

The output shows both flow and task CPU utilization. This reveals, in this case, that we had flow CPU high, but not

task. What does this tell you? It tells you that the flow processing is the one that’s causing the high CPU utilization and

that means it’s traffic — we are processing a lot of traffic. Let’s focus on the traffic that’s being processed.



116

Slide 114


�Example 3 – System showing abnormal high CPU

• Analysis (1)

• “flow” is the CPU running high

• Related to traffic processing/forwarding

• ~8000 packets per second were sent to CPU because of ALG

processingns5400-> get asic demux

Current(02:57:15) Last(02:57:15) PPS( 17s)to_host_packet: 612430 612430 0first_packet: 13400782 13258685 8147brcst: 243 243 0no_ip_ether_net: 708 708 0total packet: 14014163 13872066 8147

clsf counters:icmp 40 40 0

To CPU traffic analysis:ALG: 4152761 4010664 8147DMA required: 59 59 0


The next thing we did was to look at “get ASIC demux”. We checked the PPS and saw we have 8,000 packets per

second going to the CPU for ALG processing. We had all these packets going to the CPU for ALG. The next question

that we asked was, which ALG is being triggered? Which traffic is this? We didn’t expect to have this amount of traffic

for the ALG.



117

Slide 115


� Example 3 – System showing abnormal high CPU

• Analysis (2)

• debug tag info showed packets going to CPU

• Demux = 4 -> first packets

• Destination ports were identified

• There were services using well-known ports and matching ALGs

• Packets go to CPU if needed to be processed by ALG

****************** 11236.0: tag (03a15f00) ******************pak length: 46 vlan qidx:6 slot:0 port:0 buffer:0x806e191cprotcol:17 demux:4 l2idx:5190 ipid:0 flags:0x00000007 session pointer:0x000e1d91src:192.134.71.124 dst:212.60.215.99 sport:13c4 dport:13c4********************** end tag info *************************st_tag_2_ifp: 192.134.71.124 -> 212.60.215.99, incoming ifp=ethernet2/1.400start demux process 4


Next we ran “debug tag info”, which shows the packet tags going to the CPU. In the tag, we can see the destination

port. We can match to a service and then understand which ALG is being triggered.

In this case, 13c4 is 50/60, which is the port for the SIP service for Voice over IP. We then knew why the CPU was

high. There was a lot of traffic going through the firewall for the SIP service.

We asked ourselves, “Do we expect this high amount of traffic for SIP service?” We can try a packet capture in the

network or check, for example, the source, to see why it’s sending all the traffic, and hopefully understand what’s

going wrong.

In this case there was no problem in the system. The traffic load was high because the packets sent to this condition

represented a relatively high load and what happened was that port was being used by a different service using that

port and that service didn’t need any ALG processing. But, because it was using the port that was for SIP, it was going

to the CPU for the ALG processing.



118

Slide 116


�Example 3 – System showing abnormal high CPU

• Workaround

• Disable the ALGs being triggered

• unset alg <algname> enable

•Root Cause

• System working as expected, traffic load for CPU processed packets

was too high.

•Solution

• Change services to use non well-known ports

• Or disable the ALGs if not needed

KB9453 - Troubleshooting High CPU on a firewall device


The idea here was to either change the port that serves that application from that specific network, or disable the ALG

if you don’t need to use it; if you don’t have any SIP service in the network.

With these three examples, we saw the most important problems that we had in the field. First, system stopped

forwarding the traffic, then second, certain applications or certain services are dropped and we needed to check

exactly which service it is and check the details. Then the third one was the high CPU. Again, these three are the most

important types of problems we have had.

We also have this Knowledge Base reference document KB 9453, which provides a good starting point, and also

covers the analysis that we covered.



119

Slide 117


More Information

� Juniper Knowledge Base: http://kb.juniper.net

•Ask a question and get answers

� Technical Documentation: http://www.juniper.net/techpubs/software/screenos/index.html

•ScreenOS Concepts and Examples Guide

•ScreenOS CLI Guide

� J-Net Forum: http://forums.juniper.net/jnet

•Sign up and participate

You have these additional sources of information.

The Knowledge Base has several articles that can help you.

Via the Technical Documentation link you can get to the ScreenOS Concepts and Examples Guide, which can help

you understand the expected behavior, and also the ScreenOS CLI Guide can help you review the syntax of the

commands.

You can also use J-NET to discuss problems you may encounter.



120

Slide 118



•Described workarounds provided in the three most critical

troubleshooting examples occurring in the field

•Showed how to apply the commands described in each

troubleshooting example

Section Summary


• Described workarounds provided in the three most critical troubleshooting examples occurring in the field, and

• Showed how to apply the commands described in each troubleshooting example



121

Slide 119



Which of the following is an indication that the system

has stopped forwarding traffic?

A) Fragmented packets

B) Queue full & full always “1”

C) Session matching fail

D) High CPU



122

Slide 120



The first command to use when high CPU exists is:

A) “get ASIC demux”

B) “get sat counter”

C) “get sat session”

D) “get perf CPU all detail”



123

Slide 121


Course Summary

� In this Course, we:

•Distinguished between ISG Series and NS5000 Series

hardware configuration and packet flow

•Explained the importance of the ASIC functions

•Described First Path and Fast Path in packet flow

•Differentiated between functions processed in the CPU

versus PPU

•Used and interpreted debug commands unique to high end

systems

•Explained the workarounds for three typical troubleshooting

examples

In this Course, we:

• Distinguished between ISG Series and NS5000 Series hardware configuration and packet flow

• Explained the importance of the ASIC functions

• Described First Path and Fast Path in packet flow

• Differentiated between functions processed in the CPU versus PPU

• Used and interpreted debug commands unique to high end systems, and

• Explained the workarounds for 3 typical troubleshooting examples



124

Slide 122


Additional Resources

� Education Services training classes

•http://www.juniper.net/training/technical_education/

� Juniper Networks Certification Program Web site

•www.juniper.net/certification

� Juniper Networks documentation and white papers

•www.juniper.net/techpubs

� To submit errata or for general questions

•[email protected]

For additional resources or to contact the Juniper Networks eLearning team, click the links on the screen.



125

Slide 123


Evaluation and Survey

� You have reached the end of this Juniper Networks

eLearning module

� You should now return to your Juniper Learning

Center to take the Practice Test and the Student

Survey

•The test will allow you to gauge your knowledge

of the material covered in this course

•The survey will allow you to give feedback on

the quality and usefulness of the course

You have reached the end of this Juniper eLearning module. You should now return to your Juniper Learning Center

to take the Practice Test and the Student Survey. The test will allow you to gauge your knowledge of the material

covered in this course. The survey will allow you to give feedback on the quality and usefulness of the course.



126

Slide 124


© 2010 Juniper Networks, Inc.

Juniper Networks, Junos, Steel-Belted Radius, NetScreen, and

ScreenOS are registered trademarks of Juniper Networks, Inc. in the

United States and other countries. The Juniper Networks Logo, the

Junos logo, and JunosE are trademarks of Juniper Networks, Inc. All

other trademarks, service marks, registered trademarks, or registered

service marks are the property of their respective owners. Juniper

Networks reserves the right to change, modify, transfer, or otherwise

revise this publication without notice.

Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos, NetScreen and

ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and other countries. JunosE is a

trademark of Juniper Networks, Inc. All other trademarks, service marks, registered trademarks or registered service

marks are the property of their respective owners. Juniper Networks reserves the right to change, modify, transfer or

otherwise revise this publication without notice.



127

Slide 125

CONFIDENTIAL

Corporat e and Sales Headquart ers

Juniper Networks, Inc.

1194 Nort h Mat hilda Avenue

Sunnyvale, CA 940 89 USA

Phone: 888 .JUNIPER

( 888 .586.4737)

or 40 8 .745.20 00

Fax: 408 .745.210 0

www.juniper.net

APAC Headquart ers

Juniper Networks (Hong Kong)

26/ F, Cit yplaza One

1111King’s Road

Taikoo Shing, Hong Kong

Phone: 852.2332.3636

Fax: 8 52.2574.780 3

EMEA Headquart ers

Juniper Networks Ireland

Airside Business Park

Swords, Count y Dubl in, Ireland

Phone: 35.31.890 3.600

EMEA Sales: 0 0 80 0 .4586.4737

Fax: 35.31.8 90 3.60 1

Copyright 20 10 Juniper Networks, Inc.

Al l right s reserved. Juniper Networks,

t he Juniper Networks logo, Junos,

NetScreen, and ScreenOS are regist ered

t rademarks of Juniper Networks, Inc. in

t he Unit ed States and ot her count ries.

Al l ot her t rademarks, service marks,

regist ered marks, or regist ered service

marks are t he propert y of t heir

respect ive owners. Juniper Networks

assumes no responsib il it y for any

inaccuracies in t his document . Juniper

Networks reserves t he right t o change,

modif y, t ransfer, or ot herw ise revise t his

publ icat ion w it hout not ice.

educat ion serv ices coursew are

Documents

JUNIPER NETSCREEN TROUBLESHOOTING