11
FOP-TSTM Maxis Confidential Restricted I-LINK SLOW BROWSING FAULT RESOLUTION SUMMARY FOP - TSTM December 2009

I-link Slow Browsing

Embed Size (px)

Citation preview

Page 1: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

I-LINK SLOW BROWSING

FAULT RESOLUTION SUMMARYFOP - TSTM

December 2009

Page 2: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted04/29/08 2

End to end Chorological Events

Received email from FCM on slow browsing issue

7 /12/09 10.50 AM

Analyze trace route result and fine tune IP advertisement to I-Link and various optimization efforts executed but with no improvement on the downlink

Escalated to HP Rizal for further checking. Slow due to MTU possibility ruled out after various extensive tests. IX confirmed to be normal

7 /12/09 13.00 PM

7 /12/09 19.10 PM

8 /12/09 07.00 PM

Proceed with various checks for LDP issues. Changed interfaces with LDP to multimode cable

9/12/09 22.00 PM

Pathma continued with further verification on the IPS . It was removed at 8am on 10/12/09 but no improvement found

10/12/09 10.00 AM

Complains worsen as some customers could not browse at all

Daniel TSTM was engaged to joint troubleshoot on the access portion. FCM informed that only WP customers are affected. No alarm detected at the access portion; i.e. ATM and Anymedia

10/12/09 10.30 AM

10/12/09 6.30 PM

Re-group and briefing at warrooms. Discussion point to problem that may occurred between ATM KLCC and ATM SHTI

10/12/09 20.00 PM

Switch ATM KLCC to redundant shelves. Test browse at site indicates improvement and so do customers at site but problem re-appear after half an hour

Preparations done for extensive testing of fibers to isolate fiber issue. A new patch cord (10 meters) between ODF and ATM at SHTI was laid as one of them was found to inhibit significant loses

10/12/09 21.00 PM

11/12/09 02.30 AM

New patch cord terminated by 2.15 AM and after various testings the service is confirmed to be back to normal by 2.30 AM

- Daniel coordinated with Zuhari to filter customer complaint to further map areas/services that were affected as customers in Selangor is also complaining- Technicians despatched to KLCC site for test browsing- Preliminary check with FM on fiber health- Ping test to various ip adresses show inconsistent results (intermittent)- E3 cards at ATM were reset but problem still persists

10/12/09 11.00 AM

Page 3: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

-TSTM engineer Daniel was engaged to joint troubleshoot on the access portion – i.e. ATM/MSAN.

- Daniel informed there was no alarm detected at the access portion; and he started to arrange field test try to isolate issue.

10/12/09 10.30 AM

10/12/09 18.30 PM

- Test results was discussed and compared.-Re-group and briefing at warrooms. -Discussion pointed to problem that may occurred between ATM KLCC and ATM SHTI

10/12/09 20.00 PM

-Switch ATM KLCC to redundant shelves.

-Test browse at site indicates improvement and so do customers at site but problem re-appear after half an hour

-Preparations done for extensive testing including power & OTDR measurement, and physical check on patch cord to isolate fiber issue at both ends. -We found high fiber loss between ODF & ATM at SHTI end;-A new 10m patch cord was laid to replaced the old one;

10/12/09 21.00 PM 11/12/09

02.30 AM

-Problem resolved after fiber termination completed at 2:15am;

-After various testing the service is confirmed to be back to normal by 2.30 AM

Simplified Chorological Events

10/12/09 11.00 AM

- Daniel coordinated with Zuhari to filter customer complaint to further map areas/services that were affected as customers in Selangor is also complaining- Technicians despatched to KLCC site for test browsing- Preliminary check with FM on fiber health- Ping test to various ip adresses show inconsistent results (intermittent)- E3 cards at ATM were reset but problem still persists

Page 4: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted04/29/08 4

Anymedia Network Diagram

direct fiber connection from ATM KLCC and ATM SHTI (STM- 4)

High loss fiber patch cord

Page 5: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

Challenges and Wayforward

CHALLENGES (Delay in isolation and fault restoration)

Multiple occurrence of faults to multiple services Initial focus was more on Core due to the above as the common point

(Maxisbb on Sunday, VSAT internet interrruption) Mapping the customers complains by areas/services to identify common

point

WAYFORWARD

Simulate customer experience to be performed for major cases Anymedia migration to Metro-E should have been completed Periodic case update on the complete customers mapping as cases piling

up during crisis

15/12/2009

Page 6: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted04/29/08

The ATM KLCC was not planned to be used beyond June 2009 as per commitment given by PSE to finish migrating all Fixed Broadband customers of Anymedia/MSAN to the Metro-E network.

We did not sign the EWP beyond June 2009 in view of the above

Fast completion of these migration is needed to ensure the system availability for Anymedia/MSAN if such failures on ATM re-occur

BCP needs to be planned considering that ATM KLCC is a single point of failure node for I-Link customer in Wilayah Persekutuan.

Delay of Anymedia migration to Metro-E

Page 7: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

END OF SLIDES…

Page 8: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

BACK UP SLIDES

Page 9: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

I-Link Customer Complains Statistics

Page 10: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

No. Of Customer Complains Vs Date

Page 11: I-link Slow Browsing

FOP-TSTMMaxis Confidential Restricted

Customer Complains (Selected Details)