4-CEN Fault Isolation

CEN Fault Isolation

1. Checking Services using OTNM2000 GUI

2. Checking Services using Command Line

Interface (CLI)

3. Fault Isolation

Contents

Interface (CLI)

3. Fault Isolation

Contents

Checking Services using OTNM2000 GUI

1. The Task Panel

ALARM: To view the current alarms

ALARM_HIS: To view the historical alarms

PERFORMANCE: To view NE current performance (e.g.

IOP(Input Optical Power/Received) and OOP(Output Optical

Power/Transmit), Rx/Tx , NE/FAN Temperature, Laser Bias,

Voltage, Memory Utilization

PERFORMANCE_HIS: Querying Performance Code

STATUS: To view the current port Link State, Optical Module

Information, Received and Transmitted Packets.

Card Configuration: FAN Speed Configuration, Clock Config.

MASK SETTING: Alarm and Performance masking/hiding

Control Command: Loopback Testing, Card Soft Reset, Card

Switching to Main/Protection, Port Laser ON/Off,

2. The Help Toolbar

Alarm and Performance Help: Views the relevant information of the alarm

and performance.

To view the current and

timestamp of alarm: Right

click at the NE and select

“Open”. Click the alarmed

Click the ALARM icon found

on the Task Panel.

ALARM_HIS

Users can know about the faults of

network via viewing alarm history, so

as to facilitate the fault analysis.

PERFORMANCE Users can know whether the equipment or service in the network is abnormal or not

by viewing performances. Viewing performance also help the maintainers to handle

the faults in a timely manner and prevent faults.

PERFORMANCE_HIS View the performance history to obtain the abnormal performance data, in order to

instruct the current routine operations.

STATUS

Via viewing card status, users can

check whether the cards are

present and the software /

hardware version complies with

the planning.

Card Configuration

Via card configuration, users can enable E1 ports, configure clock and time

synchronization.

Control Command

Via control command, users can perform software reset, card forced switch, E1

loop testing and laser on/off of optical ports.

Interface (CLI)

3. Fault Isolation

Contents

Checking Services of Dynamic Networking

Check IS-IS, OSPF, and BGP neighbor establishment.

Check RSVP and LDP session setup.

Check whether BGP distributes direct routes.

Check VC establishment.

Check the BFD status.

Check the MPLS and VRF ILM forwarding tables.

Check services through LSP ping, VRF ping, and Internet Control

Message Protocol (ICMP) ping.

OSPF check command:

router#show ip os ne

Output:

OSPF process 100:

Neighbor ID Pri State Dead Time Address Interface Instance

ID Up Time

6.6.6.6 1 Full/ - 00:00:30 56.1.1.2 XGE0/6/1.31 0

01d00h00m

22.22.22.22 1 Full/DR 00:00:34 115.1.1.2 GE0/11/3.31 0

01d00h31m

Service Self-test

Perform service self-test after service configuration.

IS-IS check command:

router#show clns is-neighbors

Output:

Area 300:

System Id Interface State Type Priority Circuit Id Up

0101.1213.6185 XGE0/3/1 Up L1 64 0101.1213.6185.03

83d05h51

0101.1213.6008 XGE0/4/1 Up L1 64 0101.1213.6186.03

12d19h58

Service Self-test

Check RSVP and LDP session setup.

Command:

router#show rsvp session

Output:

Ingress RSVP:

To From State Pri Rt Style Labelin Labelout LSPName DSType UpTime

1.1.1.1 5.5.5.5 Up Yes 1 1 SE - 131587 1-5.1-1.1.1e00000f ELSP_SIG 00:01:32

1.1.1.1 5.5.5.5 Up No 1 1 SE - 131584 1-5.1-1.1.1e00000f ELSP_SIG 00:02:26

Total 2 displayed, Up 2, Down 0.

Egress RSVP:

To From State Pri Rt Style Labelin Labelout LSPName DSType UpTime

5.5.5.5 1.1.1.1 Up Yes 1 1 SE 131589 - 1-5.1-1.1.1e00000f ELSP_SIG 00:01:30

5.5.5.5 1.1.1.1 Up Yes 1 1 SE 131585 - 1-5.1-1.1.1e00000f ELSP_SIG 00:02:17

Command:

router#show ldp session

Output :

Peer IP Address IF Name My Role State KeepAlive Up Time

8.8.8.8 GE0/11/3.31Passive OPERATIONAL 60 01d00h44m

6.6.6.6 XGE0/6/1.31Passive OPERATIONAL 60 01d00h12m

22.22.22.22 GE0/11/3.31Passive OPERATIONAL 60 01d00h44m

Service Self-test

Check VC establishment.

Command:

router#show mpls vc-table

Output:

VC-Name VC-ID Access-Intf Network-Intf In Label Out Label Tunnel-Label Nexthop

Status Oam-Status Up-Time Y1731

1-8.1-5.1 1 GE0/7/3.32 GE0/7/1.31 8713 132364 9219 5.5.5.5

Inactive NULL 00:49:30 0

1-8.1-6.2 2 GE0/7/3.32 GE0/7/2.31 8715 132096 9223 6.6.6.6

Active Bfd-UP 00:49:30 0

1-8.1-5.3 3 GE0/7/3.33 GE0/7/1.31 8714 132365 9216 5.5.5.5

Active Bfd-UP 00:49:30 0

1-8.1-6.4 4 GE0/7/3.33 GE0/7/2.31 8718 132097 9224 6.6.6.6

Standby Bfd-UP 01d20h52m 0

Service Self-test

Check the BFD status.

Command:

router#show bfd session

Output:

Sess-Idx Remote-Disc Lower-Layer Sess-Type Sess-State UP-Time Remote-Addr

1801 1806 IPv4 Single-Hop Up 1d01h00m 45.1.1.1/32

1803 802 IPv4 Single-Hop Up 1d01h00m 115.1.1.2/32

1 1 MPLS LSP Single-Hop Up 1d00h30m 127.0.0.12/32

1804 0 IPv4 Single-Hop Down 00:00:00 56.1.1.2/32

7 6 MPLS VCCV Single-Hop Up 00:53:30 127.0.0.12/32

Service Self-test

Check the MPLS and forwarding tables.

Command:

router#show mpls forwarding-table

Output:

Codes: > - selected FTN, p - stale FTN, B - BGP FTN, K - CLI FTN,

L - LDP FTN, R - RSVP-TE FTN, S - SNMP FTN, I - IGP-Shortcut,

U - unknown FTN

Code FEC Tunnel-id FTN-ID Pri Nexthop Out-Label Out-Intf LSP-Type LSP-

ID Ingress-IDUp-Time

R> 1.1.1.1/32 1 1 Yes 45.1.1.1 131587 XGE0/6/4.31 1 5.5.5.5

ELSP_SIGNAL 00:25:54

R 1.1.1.1/32 1 20 No 56.1.1.2 131592 XGE0/6/1.31 2 5.5.5.5

R> 1.1.1.1/32 5 17 Yes 45.1.1.1 131588 XGE0/6/4.31 1 5.5.5.5

R> 2.2.2.2/32 2 19 Yes 56.1.1.2 131593 XGE0/6/1.31 1 5.5.5.5

ELSP_SIGNAL 01d00h31m

L> 6.6.6.6/32 0 2 Yes 56.1.1.2 132102 XGE0/6/1.31 LSP_DEFAULT

01d00h20m

Service Self-test

Command: FEC to Next Hop Label Forwarding Entry Table

router#show mpls ftn-table

Output:

Primary FTN entry with FEC: 1.1.1.1/32, id: 1, row status: Active

Owner: RSVP, Action-type: Redirect to Tunnel, Exp-bits: 0x0, Incoming DSCP: none

Tunnel id: 1, Protected LSP id: 1, QoS Resource id: 71, Description: 1-5.1-1.1.1e00000f

Select: 0

, Bfd id:2, MPLS-TP OAM Y1731 Initiative id:0, Lock:0 Cross connect ix: 290, in intf: - in

label: 0 out-segment ix: 269

Owner: RSVP, Persistent: No, Admin Status: Up, Oper Status: Up

Out-segment with ix: 269, owner: RSVP, out intf: XGE0/6/4.31, out label: 131587

Nexthop addr: 45.1.1.1 cross connect ix: 290, op code: Push

LSP Type: ELSP_SIGNAL

Class_Exp mapping:

CLASS DSCP_value EXP_value

be 000000 0

Bidirectional Forwarding Detection is configured

Bidirectional Forwarding Detection is enabled

Up Time: 00:26:05

Service Self-test

Command: MPLS Incoming Label Map

router#show mpls ilm-table

Output:

Codes: > - selected ILM, p - stale ILM, K - CLI ILM

Code In-Label Out-Label In-Intf Out-Intf Nexthop FEC ILM-ID

LSP-Type Up-Time

> 132370 8197 XGE0/6/1.31 GE0/11/3.31 115.1.1.2 22.22.22.22/32

579 LSP_DEFAULT 01d00h36m

> 132262 N/A XGE0/6/1.31 inloopback 0.0.0.0 192.168.184.0/24

747 LSP_DEFAULT 01d00h36m

> 132166 N/A XGE0/6/1.31 inloopback 0.0.0.0 192.168.88.0/24

651 LSP_DEFAULT

Service Configuration

Before using the show running command to query single-station configurations,

log in the device through the SecureCRT software.

SecureCRT-based login:

Both the username and password are root.

IP address of the

loopback interface

of the device

The following page is displayed after successful login to the core switch card.

Enter the following command and press Enter to log in to the control program.

Log in to the control

program.

Enter the password (a) and run en, as shown in the following figure.

The following describes the outputs of the show running-config command at

stations A1 and B1.

no service password-encryption

hostname A1_YZXTD

password a

log file rcu.log

! mpls l2-circuit YZXTD_A11-LXRJ_B1-pw-main-1X 101 222.173.0.168 control-word

mpls l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-1X 102 222.173.0.167 control-

mpls l2-circuit YZXTD_A11-LXRJ_B1-pw-main-DO 201 222.173.0.168 control-word

mpls l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-DO 202 222.173.0.167 control-

Username

Password

Active VC for 1X services

Standby VC for 1X services

mpls class-type ct0 default

mpls class-type ct1 A11

mpls class-type ct3 a3

mpls te-class te0 default 0

mpls te-class te1 A11 1

mpls te-class te2 A11 2

mpls te-class te3 a3 3

Configure BFD FOR PW

mpls te-class te7 default 7

mpls bfd l2-circuit 101 local-discriminator 101 remote-discriminator 101 min-tx 50 min-

mpls propagate-ttl

mpls vpws YZXTD_A11-LXRJ_B1-vpws-1X 101

vpws-vc YZXTD_A11-LXRJ_B1-pw-main-1X

vpws-vc YZXTD_A11-QLGJ_B2-pw-backup-1X

mpls vpws YZXTD_A11-LXRJ_B1-vpws-DO 201

vpws-vc YZXTD_A11-LXRJ_B1-pw-main-DO

vpws-vc YZXTD_A11-QLGJ_B2-pw-backup-DO

Configured BFD for PW.

Configured the VPWS.

Bound the VC to the VPWS.

Service Configuration router ldp

router-id 3.76.1.201

explicit-null

no-php

global-merge-capability merge-capable

control-mode ordered

label-retention-mode liberal

pw-status-tlv

targeted-peer ipv4 222.173.0.168

exit-targeted-peer-mode

targeted-peer ipv4 222.173.0.167

exit-targeted-peer-mode

transport-address ipv4 3.76.1.201

hold-time 15

hello-interval 5

targeted-peer-hold-time 60

targeted-peer-hello-interval 20

keepalive-timeout 60 keepalive-interval 20

targeted-peer-hello-receipt

advertisement-mode downstream-unsolicited

mpls ldp lsp trigger host-only

Configured LDP. The IP address of the loopback

interface is configured here.

Configured the IP address for

loopback in the local station.

Quit the TAR-PEER configuration mode.

Configured the IP address of the loopback

interface for oppostation end B1.

all: LDP advertises labels for all.

host-only: LDP advertises label for hosts only.

for vc: LDP advertises labels for VCs only.

interface FE0/5/5

mtu-board 9600

no shutdown

switchport

physical basic oe-mode electric

commit

! interface FE0/5/5.33

description for QoS-NMS

no shutdown

switchport

uni stream-type 1

uni stream-id 1 ploy 0 assign 1

uni stream-id 1 ovlan 33

mpls-l2-circuit YZXTD_A11-LXRJ_B1-pw-main-1X ethernet

mpls-l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-1X ethernet secondary

vc-mode revertive

commit

Configured the physical

interface.

Configured the subinterface of VLAN

33 in 1X services.

Enabled the interface.

Configured the interface

on the UNI side.

Bound the VC to this interface.

Service Configuration Station A1:

interface FE0/5/6

mtu-board 9600

no shutdown

switchport

physical basic oe-mode electric

commit

interface FE0/5/6.34

description for QoS-other

no shutdown

switchport

uni stream-type 1

uni stream-id 1 ovlan 34

mpls-l2-circuit YZXTD_A11-LXRJ_B1-pw-main-DO ethernet

mpls-l2-circuit YZXTD_A11-QLGJ_B2-pw-backup-DO ethernet secondary

vc-mode revertive

commit

Service Configuration !

interface GE0/5/1

mtu-board 9600

no shutdown

physical advance out-limit no-shutdown

physical queue-dispatch queue-no 0 mode 1 weight 1

physical queue-dispatch queue-no 0 cir 0 pir 1000 cbs 256 pbs 256

label-switching

commit

Enable label switching.

interface GE0/5/1.31

description SD-JN-SJ-A-1.163 GE0/0/0 2.5G(S-16N0001IP)

ip address 10.1.1.5/30

no shutdown

label-switching

commit

ip ospf network point-to-point

ip ospf cost 100

enable-ldp ipv4 no-igp-sync

Configure dthe IP address of

the subinterface.

Enable OSPF on this

interface.

Enable LDP on the NNI. A1

Service Configuration !

interface GE0/2/1

mtu-board 9600

no shutdown

physical advance out-limit no-shutdown

label-switching

commit

interface GE0/2/1.31

description SD-JN-SJ-A-1.163 GE0/0/0 2.5G(S-16N0001IP)

ip address 10.1.1.1/30

no shutdown

label-switching

commit

ip ospf network point-to-point

ip ospf cost 100

enable-ldp ipv4 no-igp-sync

router ospf 2

ospf router-id 3.76.1.201

timers spf exp 50 50000

area 0.0.0.1 stub

network 3.76.1.201/32 area 0.0.0.1

network 10.1.1.0/30 area 0.0.0.1

network 10.1.1.4/30 area 0.0.0.1

cspf disable-better-protection

telnet max-clients 10

line vty

exec-timeout 0 0

dhcp enable

dhcp hops threshold 16

Configured OSPF. Set the IP address of the OSPF

interface to the IP address of the

loopback interface.

Added the loopback

interface to area 1.

Added the link interface to area 1.

CLI basic commands in total

CLI : show running

Screenshot below: (CLI in the red box)

View global configuration

CLI : show arp

Screenshot below: (CLI and Content displayed as red font)

View all ARP

CLI : show ip interface brief

View port information

CLI : show ip route

View the IGP route table

CLI : show rsvp session

View RSVP session information

CLI : show rsvp interface

View RSVP enabled interface information

CLI : show mpls forwarding-table

View all LSP initiated by this router

CLI : show mpls ilm-table

View all mpls ilm table information

CLI : show bfd session

View bfd session information

CLI : show bfd interface

View bfd enabled interface

CLI : show ldp session

View LDP session information

CLI : show ldp interface

View LDP enabled interface information

CLI : show mpls vc-table

View all VC FEC statistics

Interface (CLI)

3. Fault Isolation based on different scenario

Contents

Fault Isolation

E-CES(E1) Service

Situation : Field Operations is conducting loop testing end-to-end(RNC to end site) but they

cant see the loop provided or the data cant pass thru RNC from the Source Site.

Checking:

1. Check the slot/port of the Source Site and Destination Site if mapping is created on

Business Management View.

2. Check the NE E1K5/E101/S101/S102 Card for current alarms. If seen PPI_LOS alarm,

physical termination fault. If seen AIS alarm, possible that alarm is on far end site.

3. If no alarm seen, check the port if unshut. Go to Secure CRT, type show run int <port>

command or go to STATUS>Link State must be with “on” status.

4. If port is up, check the status of VC by running the command show mpls vc-table and

check its uptime status. If with INACTIVE status, check the status of the tunnel by running

the command show rsvp session.

5. If no result show upon running the mpls vc-table command, check the LDP session if

configured by running show ldp session.

6. If RSVP session is down, check for active fiber cut.

6. If all protocols are up, conduct loop testing with the Engineer on site.

Control Command

E1 Loop Testing: Verify the configured E1 ports on Business Management .

Accessing C31 Ilagan on FH NMS:

LLB: Local Loopback = enable loop at Site A DDF

facing the BER tester(also installed in Site A)

DLB: Device Loopback = enable loop at Site A

facing the DDF of Site B(depending on the created

tunnel/trail, where BER tester is installed in Site B).

SITE A SITE B

Fault Isolation

E-LAN(Data Services (3G, BRO, Wimax, FD/TD-LTE)

Down Service

Checking:

1. Check the slot/port of the Source Site and Destination Site if mapping is

created on Business Management View.

2. Check the port for current alarms. If seen RLOS alarm, possible physical cable

fault. If seen LINK_LOS alarm, possible port setting problem(duplex/speed).

3. If no alarm seen, check the port if unshut. For R865, select the card, view the

STATUS>>Port Status to check for link status, received/transmit packets, current

speed/duplex mode.

4. If port is up but no packets, check the status of VC by running the command

show mpls vc-table and check its uptime status. If with INACTIVE status, check

the status of the tunnel by running the command show rsvp session.

5. If no result show upon running the mpls vc-table command, check the LDP

session if configured by running show ldp session.

6. If RSVP session is down, check for active fiber cut.

7. If no fiber cut, check for broken tunnel by running show run rsvp-path.

Fault Isolation

L2 services are unavailable.

1) Run show mpls vc-table to check whether the VC is normal.

Vc-name: VC name Vc-id: VC ID Access-intf: bound interface

Network-intf: network interface In label: incoming label Out label: outgoing label

Tunnel-label: label of the bound tunnel Status: status

Nexthop: IP address of the next-hop interface Up-time: establishment time

Fault Isolation

The VC is abnormal.

Fault Isolation

2) If the VC is abnormal, run show mpls forwarding-table to check the LSP

labels generated by LDP. The outgoing label is normal.

Fault Isolation

The outgoing label is abnormal.

Fault Isolation Run show mpls ilm-table to check the incoming label. The incoming label is normal.

Fault Isolation

The incoming label is abnormal.

Fault Isolation

3) If the LDP connection is abnormal, run show ip ospf neighbor to check OSPF.

Neighbor id: IP address of the peer loopback interface Pri: priority State: status

Dead time: The connection is interrupted if no Hello packet is received from the peer end within 30

seconds.

Address: IP address of the peer interface Interface: local interface

Instance id: local OSPF process ID (0 by default) Up time: establishment time

Fault Isolation The OSPF connection is abnormal.

Fault Isolation

If the OSPF connection is abnormal, ping the directly-connected interface. If the

directly-connected interface can be successfully pinged, OSPF is faulty.

Fault Isolation The directly-connected interface cannot be pinged.

5) If the directly-connected interface cannot be pinged, check the link state to

see whether the optical fiber is correctly connected.

THANK YOU

4-CEN Fault Isolation

Documents

Efficient Software-Based Fault Isolation

nnovative Fault Detection, Isolation and Recovery ... · INNOVATIVE FAULT DETECTION, ISOLATION AND RECOVERY ... of the art in spacecraft fault detection, ... INTRODUCTION . Fault

Fault detection and isolation from uninterpreted data in

Efficient Software Fault Isolation Maitree · Why Software fault isolation: Hardware-based Fault Isolation Run modules in separate address spaces Communicate via Remote Procedure

Distributed processing based fault location, isolation

FAULT DETECTION AND ISOLATION FOR WIND TURBINE …

GAS TURBINE PERFORMANCE DIAGNOSTICS AND FAULT ISOLATION … · · 2016-03-08GAS TURBINE PERFORMANCE DIAGNOSTICS AND FAULT ISOLATION BASED ON MULTIDIMENSIONAL COMPLEX HEALTH VECTOR

FAULT DETECTION AND ISOLATION BASED ON BOND GRAPH … · FAULT DETECTION AND ISOLATION (FDI) Fault detection: to determine if the system behavior has departed from the acceptable

Fault Detection, Isolation and Accommodation Using the

Hybrid RNC-Isolation of Structures under Near-Fault

Fault Detection, Isolation, and Recovery for Autonomous

Fault Isolation and Quick Recovery in Isolation File Systems · Fault Isolation and Quick Recovery in Isolation File Systems ... mobile devices, ... even the operating system 12

VLAN Fault Isolation User's Guide (9033543)ehealth-spectrum.ca.com/support/secure/products/Spectrum_Doc/spec... · VLAN Fault Isolation Page 3 Restricted Rights Notice (Applicable

Fault Detection and Isolation: an overview€¦ · •Fault isolation: Find the root cause, by isolating the system component(s) whose operation mode is not nominal •Fault identification:

Control Flow Integrity & Software Fault Isolation

Intermittent Fault Detection & Isolation System (IFDIS

Fault Diagnosis – Isolation of Malfunctions in Power Transformers

Efficient Software Based Fault Isolation

Fault Detection, Isolation and Quantification from ... · Fault Detection, Isolation and Quantiﬁcation from Gaussian Residuals with Application to Structural Damage DiagnosisI Michael

Fault Detection, Isolation and Fault-Tolerant Control of