Embed Size (px)
DESCRIPTION
Piotr Kupisiewicz – Technical Expert in Krakow’s TAC VPN team. In IT for more than 10 years, out of which 5 years is mostly software engineering experience. Last 5 years spent mostly in networking area interested mostly in Network Security. His hobby are drums and very heavy music. CCIE Security 39762. Olivier Pelerin – as a key member of the escalation team at Cisco’s Technical Assistance Center, he handles world-wide escalations on VPN technologies pertaining to IPSEC, DMVPN, EzVPN, GetVPN, FlexVPN, PKI. Olivier has spent years troubleshooting and diagnosing issues on some of largest, and most complex VPN deployments Olivier have a CCIE in security #20306 Topic of Presentation: Make IOS-XE Troubleshooting Easy – Packet-Tracer Language: English Abstract: “IOS-XE is operating system running on Service Provider devices like ASR series and ISR-4451. Aim of this session is to show how very complicated Service Provider’s configurations can be easily troubleshoted using packet-tracer tool.”
Citation preview
Make IOS XE Troubleshooting Easy:Packet tracerPiotr Kupisiewicz -- [email protected]
VPN Lead, Cisco TAC Krakow CCIE Security #39762
September 30th, 2014
PLNOG 2014 Breakout
Olivier Pelerin – [email protected]
VPN Escalation, Cisco TAC Brussels CCIE Security #20306
Cisco Public 2© 2013-2014 Cisco and/or its affiliates. All rights reserved.
System architecture
Day in life of normal packet
Debugging strategies
Packet tracer and conditional filters
Live Demo
Wrap up
Session Agenda
Cisco Public 3© 2013-2014 Cisco and/or its affiliates. All rights reserved.
System Architecture
Cisco Public 4© 2013-2014 Cisco and/or its affiliates. All rights reserved.
4
ESP
FECP
QFPCryptoAssist.
interconn.
PPE BQS
ESP
FECP
QFPCryptoAssist.
interconn.
PPE BQS
System Architecture Forwarding Plane
RP
CPU
interconn.
GE switch
SIP
SPA SPA
IOCPSPA
Aggreg.
interconn.
RP
CPU
interconn.
GE switch
Midplane
SIP
SPA SPA
IOCPSPA
Aggreg.
interconn.
SIP
SPA SPA
IOCPSPA
Aggreg.
interconn.
Activ
e
Activ
e
Stb
y
Stb
y
Hypertransport10 Gbps Ethernet
Embedded Service Interconnectaka ESI Bus11.2 – 40 Gbps Forwarding Bus
Centralized ArchitectureAll traffic flows through ESP
Cisco Public 5© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ESP
FECP
QFPCryptoAssist.
interconn.
RP
CPU
interconn.
GE switch
SIP
SPA SPA
IOCPSPA
Aggreg.
interconn.
ASR1K Software Architecture
RPCPU
IOSChassis Manager
Forwarding Manager
Linux Kernel
EO
BC
(1 G
bp
s)
ESI
(10
-40
Gb
ps)
ESP FECP
Linux Kernel
Chassis Manager
Forwarding Manager
QFPCryptoAssist.
µµ
µBQS
µµ
µ
DriversDriversDrivers
SIPIOCP
Linux Kernel
Chassis Manager
SPA
SPA DriverSPA Driver
SPA Driver
SPA SPA
ESI
(10
-40
Gb
ps)
I2C
Cisco Public 8© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Day in life of normal packet
Cisco Public 9© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ingress Packet Through SIPSIP
SPA SPA
IOCPSPA
Aggreg.
intercon.
…
ESPs
C2W
EV-FC
EV-RP
In ref clocks
Network clocks
SPA Agg.
SPA Aggregation ASIC (Marmot)
Ingress Scheduler
Egress Buffer Status
Ingress Classifier
Egress buffers(per port)
Network clock
distribution
IOCP(SC854x SOC)
…
Ingress buffers(per port)
…
Interconnect
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Reset / Pwr Ctrl
SPA
Cisco Public 10© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ingress Packet Through ESPESP
FECP
QFPCryptoAssist.
intercon.
PPE BQS
Crypto
FECP
RPs RPs RPsESP SIPs
QFP Complex
TCAMResource
DRAMPacket Buffer
DRAMPart Len / BW
SRAM
SA tableDRAM
DispatcherPacket Buffer
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Packet Processor Engine
…
PPE1 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPEN
BQS
Reset / Pwr Ctrl
Interconnect
SPI Mux
PPE2PPE2
Cisco Public 11© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Crypto
FECP
Packet Dispatched to PPE Core
RPs RPs RPsESP SIPs
QFP Complex
TCAMResource
DRAMPacket Buffer
DRAMPart Len / BW
SRAM
SA tableDRAM
DispatcherPacket Buffer
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Packet Processor Engine
…
PPE1 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPEN
BQS
Reset / Pwr Ctrl
SPI Mux
ESPFECP
QFPCryptoAssist.
intercon.
PPE BQS
PPE2
Interconnect
PPE2
Th
read
1
Th
read
2
Th
read
3
Th
read
4
Cisco Public 12© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Crypto
FECP
Packet Dispatched to PPE Thread
RPs RPs RPsESP SIPs
QFP Complex
TCAMResource
DRAMPacket Buffer
DRAMPart Len / BW
SRAM
SA tableDRAM
DispatcherPacket Buffer
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Packet Processor Engine
…
PPE1 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPEN
BQS
Reset / Pwr Ctrl
SPI Mux
ESPFECP
QFPCryptoAssist.
intercon.
PPE BQS
PPE2
Interconnect
PPE2
Th
read
1
Th
read
2
Th
read
4
Th
read
3
Cisco Public 13© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Crypto
FECP
FIA’s Applied on Packet by PPE Thread
RPs RPs RPsESP SIPs
QFP Complex
TCAMResource
DRAMPacket Buffer
DRAMPart Len / BW
SRAM
SA tableDRAM
DispatcherPacket Buffer
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Packet Processor Engine
…
PPE1 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPEN
BQS
Reset / Pwr Ctrl
SPI Mux
ESPFECP
QFPCryptoAssist.
intercon.
PPE BQS
PPE2
Interconnect
PPE2
Th
read
2
Th
read
1
Th
read
4
Th
read
3
X-Connect L2 Switch IPv4 IPv6 MPLS
Netflow
Input ACL
NBAR Classify
MQC Classify
…
NAT
PBR
Dialer IDLE Rst
URD
IP Unicast
IP Multicast
Packet For Us
Netflow
NAT
NBAR Classify
…
MQC Policing
MAC Accounting
Output ACL
Input FIA Output FIA
Cisco Public 14© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Crypto
FECP
Leaving the PPE Thread
RPs RPs RPsESP SIPs
QFP Complex
TCAMResource
DRAMPacket Buffer
DRAMPart Len / BW
SRAM
SA tableDRAM
DispatcherPacket Buffer
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Packet Processor Engine
…
PPE1 PPE2 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPEN
BQS
Reset / Pwr Ctrl
SPI Mux
ESPFECP
QFPCryptoAssist.
intercon.
PPE BQS
PPE2
Th
read
1
Th
read
2
Th
read
4
Th
read
3
Interconnect
X-Connect L2 Switch IPv4 IPv6 MPLS
NetflowBGP AccountingNBAR ClassifyMQC Classify
…
NAT
PBRDialer IDLE RstURD
IP UnicastIP MulticastPacket For Us
Netflow
NATNBAR Classify…MQC PolicingWREDOutput ACL
Input FIA Output FIA
PPE2
Thread 3
Cisco Public 15© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Crypto
FECP
Packet proceeding to BQS then SIP
RPs RPs RPsESP SIPs
QFP Complex
TCAMResource
DRAMPacket Buffer
DRAMPart Len / BW
SRAM
SA tableDRAM
DispatcherPacket Buffer
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Packet Processor Engine
…
PPE1 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPEN
BQS
Reset / Pwr Ctrl
Interconnect
SPI Mux
ESPFECP
QFPCryptoAssist.
intercon.
PPE BQS
PPE2PPE2
Cisco Public 16© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Egress Packet Through SIPESPs
C2W
EV-FC
EV-RP
In ref clocks
Network clocks
SPA Agg.
SPA Aggregation ASIC (Marmot)
Ingress Scheduler
Egress Buffer Status
Ingress Classifier
Egress buffers(per port)
Network clock
distribution
IOCP(SC854x SOC)
…
Ingress buffers(per port)
…
Interconnect
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Reset / Pwr Ctrl
SIP
SPA SPA
IOCPSPA
Aggreg.
intercon.
SPA
Cisco Public 17© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Debugging Strategies
Cisco Public 18© 2013-2014 Cisco and/or its affiliates. All rights reserved.
18
Everyday situations
Which feature went wrong ?
NATZBFIPsec
Routing
WAASOTV
SNMP
Cisco Public 19© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Using statistics for troubleshooting packet drops
• SPA
• show interfaces <interface-name>
• show interfaces <interface-name> accounting
• show interfaces <interface-name> stats
• SIP
• show platform hardware port <slot/card/port> plim statistics
• show platform hardware subslot {slot/card} plim statistics
• show platform hardware slot {slot} plim statistics
• show platform hardware slot {0|1|2} plim status internal
• show platform hardware slot {0|1|2} serdes statistics
• RP
• show platform hardware slot {r0|r1} serdes statistics
• show platform software infrastructure lsmpi
• ESP
• show platform hardware slot {f0|f1} serdes statistics
• show platform hardware slot {f0|f1} serdes statistics internal
• show platform hardware qfp active bqs 0 ipm mapping
• show platform hardware qfp active bqs 0 ipm statistics channel all
• show platform hardware qfp active bqs 0 opm mapping
• show platform hardware qfp active bqs 0 opm statistics channel all
• show platform hardware qfp active statistics drop [detail]
• show platform hardware qfp active interface if-name <Interface-name> statistics
• show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_
• show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude _0_
• show platform hardware qfp active infrastructure punt statistics type inject-drop | exclude _0_
• show platform hardware qfp active infrastructure punt statistics type global-drop | exclude _0_
• show platform hardware qfp active infrastructure bqs queue output default all
• show platform hardware qfp active infrastructure bqs queue output recycle all
Not easy… not very practical either.Let’s dig deeper before making it simpler
Cisco Public 20© 2013-2014 Cisco and/or its affiliates. All rights reserved.
20
Debugging Strategies to Date
IOS Control Plane• ACL + show access-list,…• show interface / ip route / bgp …
Platform Control Plane• ESP “stuff”• e.g. show platform … hard to
remember
Data Plane• ESP “stuff”• More arcane show platform …
Top
Dow
n
Very Difficult
Well Known
Let’s change that!!
Rock bottom
Cisco Public 21© 2013-2014 Cisco and/or its affiliates. All rights reserved.
The road to simplification:The Packet Tracer
Cisco Public 22© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Crypto
FECP
The Packet Tracer and FIA Debugger
RPs RPs RPsESP SIPs
QFP Complex
TCAMResource
DRAMPacket Buffer
DRAMPart Len / BW
SRAM
SA tableDRAM
DispatcherPacket Buffer
DDRAM
Boot Flash(OBFL,…)
JTAG Ctrl
EEPROM
Temp Sensor
Reset / Pwr Ctrl
Packet Processor Engine
…
PPE1 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPEN
BQS
Reset / Pwr Ctrl
SPI Mux
PPE2
Interconnect
PPE2
Th
read
2
Th
read
1
Th
read
4
Th
read
3
X-Connect L2 Switch IPv4 IPv6 MPLS
Input ACL
MQC Classify
NAT
PBR
IP Unicast
Output ACL
NAT
Encaps
Crypto
Input FIA Output FIA
Pak Match ?
Packet # 16
Input ACL
MQC Classify
NAT
PBR
Output ACL
NAT
Encaps
Crypto
Optionally, FIA actions can logged per packet
System can capture several packets flowsPacket flows can be reviewed in show
commands
Condition determines packets to be traced
IOS 3.10+
Statistics and final action will be collected (matched packets
dropped, punted to RP, forwarded to output interface …)
Optionally match on the egress FIA
Cisco Public 23© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Conditionally Matching PacketsIdentifying Interesting Packets
asr-1k# debug platform condition ?
debug platform condition ?
both Simultaneous ingress and egress debug
egress Egress only debug
…
ingress Ingress only debug
interface Set interface for conditional debug
ipv4 Debug IPv4 conditions
ipv6 Debug IPv6 conditions
mpls Debug MPLS conditions
…
asr-1k#debug platform condition ingress
asr-1k#debug platform condition interface gig0/0/3 ingress
asr-1k#debug platform condition ipv4 10.0.0.1/32 both
asr-1k#debug platform condition ipv4 access-list 100 egress
asr-1k#debug platform condition mpls 10 1 ingress
Match all ingress packets
Match MPLS packets with top ingress label
10
Match all ingress packets on interface gig0/0/3
Match in & out packets with source or destination 10.0.0.1
Match egress packets passing access-list 100
Cisco Public 24© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Activating the Packet TracerFollowing packets through IOS-XE – Basic Statistics
asr-1k# debug platform condition interface gig0/0/0 ingressasr-1k# debug platform condition startasr-1k# debug platform packet-trace enableasr-1k# … !send trafficasr-1k# show platform packet-trace statisticsPackets Summary Matched 102 Traced 0Packets Received Ingress 12 Inject 90 Count Code Cause 90 9 QFP ICMP generated packetPackets Processed Forward 12 Punt 0 Drop 90 Count Code Cause 13 92 Ipv4Null0 17 47 FirewallInvalidZone 60 184 FirewallL4 Consume 0
102 packets were matched by the condition
12 packets were forwarded90 packets were
dropped 13 packets were dropped due to no route
17 packets were dropped due to absence of zone pair60 packets dropped by L4
inspection (e.g. receiving window)
asr-1k# debug platform packet-trace ? copy Copy packet data drop Trace drops only enable Enable packet trace packet Packet count
The packet tracer follows a set of packets in details through the
FIA
Cisco Public 25© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Packet Tracer – Tracing Packets…The fate of 16 packets
asr-1k# debug platform condition interface gig0/0/0 ingress
asr-1k# debug platform condition start
asr-1k# debug platform packet-trace packet 16
asr-1k# debug platform packet-trace enable
asr-1k# … !send traffic
asr-1k# show platform packet-trace summary
Pkt Input Output State Reason
0 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
1 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
2 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
3 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
4 INJ.7 Gi0/0/2 FWD
5 INJ.7 Gi0/0/2 FWD
6 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
7 INJ.7 Gi0/0/2 FWD
8 …
Automatically stops tracing after 16 packets
16 packets were traced; we can zoom in
INJ.7: Packet injected by the RPinternal0/0/rp:0: Packet punted to the RP
Cisco Public 26© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Packet Tracer – Tracing Packets…The fate of an individual packet
asr-1k# show platform packet-trace packet 1
Packet: 1 CBUG ID: 109056985
Summary
Input : GigabitEthernet0/0/2
Output : internal0/0/rp:0
State : PUNT 55 (For-us control)
Timestamp
Start : 334771580191282 ns (04/29/2014 08:01:38.017738 UTC)
Stop : 334771580487612 ns (04/29/2014 08:01:38.018035 UTC)
Path Trace
Feature: IPV4
Source : 17.0.0.196
Destination : 172.18.0.1
Protocol : 50 (ESP)
Feature: IPSec
Action : DECRYPT
SA Handle : 753
SPI : 0x30ba5940
Peer Addr : 17.0.0.196
Local Addr: 172.18.0.1
Zooming on packet 1
Only major featuresare shown
Feature specific details are displayed
Cisco Public 27© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Packet Tracer – Focus on DropsDropped packets – nothing else
asr-1k# debug platform condition interface gig0/0/0 ingress
asr-1k# debug platform condition start
asr-1k# debug platform packet-trace packet 16
asr-1k# debug platform packet-trace drop [code <dropcode>]
asr-1k# debug platform packet-trace enable
asr-1k# … !send traffic
asr-1k# debug platform condition stop
asr-1k# show platform packet-trace summary
Pkt Input Output State Reason
0 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
1 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
2 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
3 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
4 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
5 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
6 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
7 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
8 …
Only save dropped packets
Focus on specific drop codes(find codes in packet-trace
statistics)Stop tracing before dumping the summary (code
limitation)Admire dropped packets… real
close
asr-1k#show platform packet-trace packet 1 Packet: 1 CBUG ID: 148787639Summary Input : GigabitEthernet0/0/2 Output : GigabitEthernet0/0/2 State : DROP 53 (IpsecInput) Timestamp Start : 361426338620013 ns (04/29/2014 15:25:52.785406 UTC) Stop : 361426338684993 ns (04/29/2014 15:25:52.785471 UTC)Path Trace Feature: IPV4 Source : 17.0.1.34 Destination : 172.18.0.1 Protocol : 50 (ESP)Packet Copy Out 002304bb 72020007 7dfbe301 080045c0 0088d135 0000fe32 2c191100 0122ac12 0001085e 1d620000 00c8172c e8010c3e 44726e6f 3eb231d5 166298c1 f519313c
For drops, condition is optional…
IOS 3.11+
Cisco Public 28© 2013-2014 Cisco and/or its affiliates. All rights reserved.
The packet tracer demonstration
Cisco Public 29© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Demo Network Diagram
29
ASR1000
leased
MPLS Internet
DMZ
Spoke 1
Spoke 11
Spoke 2Spoke 3
Spoke …
GE 2 GE 1
GE 3
This Internet based client PC can not connect to the server in the DMZ.
192.168.1.0/24
192.168.11.0/24 172.16.0.
11
10.1.1.71
172.16.0.1
192.168.0.254
Cisco Public 30© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Wrapping up…
30
Cisco Public 31© 2013-2014 Cisco and/or its affiliates. All rights reserved.
31
New Debugging Strategy
IOS Control Plane• show interface, show ip route, show bgp
…• Feature debuggingPlatform Control Plane• Unified show commands• Platform show commands• Future: control plane conditional
debuggingData Plane• Packet Tracer• Forwarding plane conditional debugging• Embedded Packet Capture
Still Difficult(not overly)
Well Known
Cisco Public 32© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Title Goes Here
Questions?
Cisco Public 33© 2013-2014 Cisco and/or its affiliates. All rights reserved.
N7K-M148GS-11
Cisco Public 34© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Thank you.
