open eir Training Day

Guide to UG, Fault Handling & FTTH

IntroductionSinead O’Gorman Head of Customer Service

The open eir customer care team continually looks for ways to improve our

performance.

The overall objective is always clear in everyone’s mind, to streamline and

optimize the delivery of faults, regardless of which operator is

serving the end customer.

Introduction - Welcome

Introduction – open eir (oeCC) Team Structure

Fault Handling

Joe Phelan & Gerry Loughney

01

FAO Repair Organisation

Field- Centre Model, with centralised

management of the repair workload from the

Works Management Centre in Sligo.

Field service repair resources = 868 with

up to double this resource in winter storms.

Structure and organisation:

1250 exchanges.

33 teams areas, geographic areas

containing 10 to 55 exchanges.

13 nationally mobile teams (hit squad)

of technicians.

5 cabling teams for infrastructure

repair.

FAO Repair Organisation

Technicians cover ~50km radius to faults, less in the large urban centres.

Default in to a subgroup (primary area) of ~10 exchanges.

Jobs loaded on to technician’s mobile device from 5am.

Technicians go directly from home to their first job.

Hit squad Technicians are mobile and can deploy to anywhere in the country at

24 hours notice.

On going support from our Contractors for civils work, poling & cabling work

Capacity planning function integrated into work management:

– Resources deployed to match workload on a pre-planned basis

– Dynamic re-allocation of resources in response to workload variations

Products repaired

PSTN - Basic telephone service

ISDN - Digital line usually used by companies as part of a integrated phone

system

ADSL - Normal broadband anything from 2 megs to 24 megs

LLU - Local loop unbundling where operators use their own core equip in

the exchange

LS - Products are split between operators (Voice with one BB with another)

FTTC - Fibre broadband served from a Cabinet near the premises

EVDSL - Fibre broadband served directly from the exch (line not fed

through a cab)

FTTH - Fibre to the Home – high speed broadband delivered over fibre to

the customer’s premises

Exchange

Exchange

This is where we connect all our

customer’s to switching

equipment.

Each individual customer’s

cable pair is connected to

exchange equipment (LI and

ADSL or eVDSL ports.)

All customer services are routed

through our exchange and core

networks

Cabinet – Distribution point

Cabinet

Roadside Cabinets are used to

distribute cable pair’s to

individual customer premises

The recent roll out of superfast

broadband (NGA) is provided

using the FTTC cab situated

alongside. Powered by electrical

feed.

FTTC cabinets are fibre powered

and contain NGA ports to

connect to our copper network

DSL Splitters

FTTC cab

Centralised work management summary

During storm recovery periods when backlogs are high, daily reports are issued showing average SLA

days per county in the storm impacted areas

During normal workload periods we clear 84% + of faults within 2 working days

The remaining 16% generally fall into the non-standard fault category (e.g. survey, cabling, poling,

DUG, etc)

Daily service plans (based on team fault volumes, age profile, resource availability, etc) are used as a

tool to optimise our SLA performance

Jeopardy management of faults approaching their SLA expiry times

Jeopardy management of customer appointments

Escalation management

Field resource utilisation:

Pre-planned deployments

Reactive re-allocation of resources when required to manage local peaks

Deployment of support / incremental resources

Overtime allocation and control

Standard faults - most fault reports are dispatched directly to our repair crews

around the country and are completed by them within SLA

Non-standard faults - some faults cannot be completed directly by our repair

crews and require additional network intervention. Examples of non-standard

Faults are:

Poles required

Cable replacement required

Civils – underground duct and boxes

DUG lead replacement

Faults that require survey, works orders, specific customer interaction or

investigation, etc, are assigned to the local field manager

Standard / non-Standard faults……

Non Standard - Overview

Non standard - Poles

Non standard - Cabling

Non standard – Civils/UG

• CUD – underground civils work, e.g. duct and boxes (will require a LA licence in

practically all cases)

• CUC / MLU – underground cabling/jointing (may require a LA licence in some cases if

blockages in the duct line are encountered)

• CPO – poling (may require a LA licence in some cases if the pole is on public property)

• CCO – overhead cabling (generally does not require a LA licence)

• Faults that are dependent on works orders will have a note pasted into them from the

works order system – the time stamp on these notes will be 2am

• This note will include an indication as to whether the fault is awaiting a LA licence at

that time

• Information on how to interpret the acronyms in the works order note is included in the

glossary of terms

• Escalations can be raised on faults associated with works orders – however the

dependency on the completion of the works order will remain

Works Order Types

Examples of W/O notes on faults…..

Logging a fault report

Testing & Screening / Troubleshooting with End User

• Electrical test completed

• CPE isolation

• Synch test completed if BB fault

• Interpret the troubleshooting results

Determine if the fault is in the open eir network

If yes, proceed to log the fault report and confirm that access will be available if

required and customer contact details are correct

Adding value to a fault report

• If indications are that access will be needed to the End User’s premises -

book a customer appointment on the fault

• If the End User lives in a gated development or an apartment complex – the

customer must ensure that access to the development / complex is available

for the repair technician when he calls

• When booking customer appointments in UG – make sure that the End User

is aware of the booked appointment date / time

• Check that the End User liaison number is up-to-date and accurate and that

the End User can answer an incoming call on that number during working

hours: 9am – 5pm

• If a fault report is cleared 003 with a note indicating that the End User’s

premises cannot be found – the End User’s eir code should be included in

the fault notes when it is sent back to open eir

Glossary of Terms

Term Explanation

Access Network The network from the local exchange to the customer’s premises

O/H Overhead cable on a pole route

U/G Underground cable or plant

Chamber / JB / manhole

/ Joint Box

All terms to describe an underground box (may be in the footway or

carriageway) that contains open eir cables & joints

Poly Polythene insulated cable - may be overhead or underground

W/O (or Works Order) A job-pack to carry out a network repair job. May involve cable and/or

poles and/or underground plant replacement

KNN / KNIS Contractor that carries out work for open eir

Civils Repair work on underground duct or chambers

Rurtel Analogue radio based system for the provision of telephone service

FWA Fixed Wireless Access: radio based system for the provision of telephone

service

FCS Fixed Cellular Service: telephony service provided at a specific location

using the Meteor mobile network

Glossary of Terms

Term Explanation

MDF Main Distribution Frame (the cross connection frame in every exchange

where the exchange equipment (LI’s) are cross-connected to the cables

serving customer’s premises.

Cab or Cabinet The cross connection point where main cables from the exchange are

jumpered to smaller distribution cables serving DP’s

DP Distribution point (the point to which individual customer drop-leads are

connected to exchange / cabinet cables

CD Wall mounted distribution point (external or internal)

NTU Network Termination Unit (eircom point/socket)

For BU outside WM For Business Unit outside of Works Management (fault not appropriate for

dispatch to field access network technicians)

S/C Short Circuit (Fault condition on line where both legs of the copper pair

are in contact)

DIS Line Disconnected (line broken at some point between exchange and

customer premises)

DW Drop Wire (wire between pole and customer premises)

LI Line Interface (the termination that a customer’s line is connected to at the

exchange MDF for dial tone)

Glossary of Terms – W/O related

Term (Works Order

Related)

Explanation

CoCo County Council

Contr contractor

COS crew on site

ETA estimated time of arrival

FC forecast or re-forecast

FFPU fit for purpose (works order completed)

LA local authority

LIC APP licence applied for (+ date)

LIC GRA licence granted

NONF A awaiting third party

NONF D dependent order

NONF L private wayleave

NONF M awaiting customer

NONF R awaiting re-plan

NONF S awaiting stores

Glossary of Terms – W/O related

Term (Works Order

Related)

Explanation

NONF T awaiting DFE approval

NONF W licence application (LA)

NONF X awaiting cancellation

NRA National Roads Authority (see TII below)

OPFF operation freeflow

REFC re-forecast

TII Transport Infrastructure Ireland (formed when the National Roads Authority

and the Railway Procurement Agency merged)

WO or w/o works order

LA Licence Local Authority licence or permit to carry out works on underground duct or

chambers

REL_ASIG SAP statuses on Works Orders. Effectively means that the works order is

assigned to a resource for attention.

REL_NONF Works order is non-fluid and cannot progress at present. Usually awaiting

LA Licence. Note in FHS should clarify

Fault Process and

Routines

02

What is a Fault Report?

A Fault Report is where an End Customer has an issue with a Product or Service

and contacts their Operator – the Operator’s Reception area must determine if

the customer’s issue is a valid open eir Network Fault.

All product fault reports can be made via our Unified Gateway. The Service

Assurance Team within our Wholesale customer contact centre provide a fault

reception and end to end fault management facility and act as a single point of

contact throughout this process.

For interconnect faults please contact our “OLO desk” at our Network

Management Centre in Citywest, at Olodesk@openeir.ie

Benefits of improved Analysis at First Point of Contact

The End Customer experience of how they are treated by their Service Provider is paramount

in

securing customer satisfaction & customer retention

Fault management is about treating every customer’s Fault report with respect and

consistency

in understanding what their issues are and dealing effectively with them to have them

resolved

Effective screening and lower non faults will improve customer satisfaction

Benefits to the Operators are:

Improved fault identification and technical support

Operators can focus efforts on allocating resources where they are genuinely

required

Improved fault clearance time

Benefits to open eir are:

Ability to focus efforts on genuine faults

Improve fault clearance time

What is a Decision Tree?

The Decision Tree is a set of Routines that the

Customer Facing Centre should follow in order

to analyse the issue that the End Customer has

–

It looks at how an agent should deal with a call

when presented with Line Test Results

including when there is no test result or when

the line is testing ok.

Open eir developed this by identifying and

documenting the “Best in Class” practice for

dealing with Narrowband and Broadband

issues.

Front End - Decision Tree

Line Test Conditions

• DC Voltage

• AC Voltage

• DIS

• Earth

• Short Circuit

• No Test

• Testing OK

Routines

• Outage

• Dangerous Plant

• ELF

• ISDN

• CPE

• Repeat

• Carrier

• FCS

Testing:

The objective of the Testing aspect of Decision Tree Implementation is to ensure that

Customer Facing Centres understand how to interpret Test Results and know what to

do when a line is testing faulty, testing ok or where no test is available

Routines:

The objective of the Routines aspect of the Decision Tree implementation is to ensure

that CFCs know how to process a report when dealing with certain specific elements

of a report

Overview Reception PSTN Process

Report Codes

Wholesale Operators use the Unified Gateway to report a fault. It is the responsibility of all Operators

to ensure that only valid Network Faults are reported to open eir Field Operations.

Report Codes – Report Codes are selected by the Agent taking the Call from the End Customer and

must reflect the issue being reported. The Report Codes are selected via UG or Web Services. The

Report Codes should be supported by clear and concise Comments based on the analysis at the first

point of contact.

High Level Flow

Outages

Scheduled Outages

• Change Management

• Planned event

• Notification email to relevant group e.g. upgrade of exchange equipment

Unscheduled Outages

• Unexpected event

• Incident Management e.g. 100 pair cable dug up

CPE Isolation Routine

• The termination point at the customer end of the open eir network is the NTU (Network

Terminating Unit)

• Faults in the network up to and including the NTU is the responsibility of open eir.

• Faults in the internal wiring after the NTU are the responsibility of the customer.

• Faults in the CPE equipment are the responsibility of the customer unless the equipment is

rented or under warranty.

Useful Tips re CPE

• Customer should be at the premises

• Check for recent CPE changes

• Check for recent construction work at the premises

• Confirm the fault status at the NTU

• Confirm that alarm is wired correctly

• Advise Customer of Possible Charge if no fault found

CPE Isolation – Step by Step Process

Step 1 – Is customer at fault location ?

If No – ask customer to ring back from fault location

If Yes - ask customer for mobile number and call back

Step 2 - Is problem with incoming calls?

If Yes – Check ringer switch on phone is correctly set

If fault is cleared - inform customer and clear fault report

Step 3 – Is CPE rented?

If Yes and CPE is faulty – inform customer to change CPE or contact CPE supplier

If No – Continue

Step 4 – Ask customer to unplug phones & equipment (e.g. Sky Box, Alarm, other CPE etc)

Retest and compare with previous test

Step 5 – Is fault cleared?

If Yes – inform customer and clear fault report

If No - Ask customer to plug phone into NTU & resume analysis

Step 6 – Is fault cleared?

If Yes – inform customer and clear fault report

Otherwise resume analysis

Repeat Fault - Fault that is reported for a second time within 28

days

Step 1 - Reception Agent checks the Fault Report History – looking at the previous

Clear and Report Codes including Comments and establish any related issues.

Does Repeat History invalidate current Report?

• If Yes – Reject Fault Report

• If No - Continue

Step 2 – Was previous Clear 003/002?

• If No - Log Fault if appropriate with supporting Notes

• If Yes – Continue, paying careful attention to the note from the

technician on the previous fault

Step 3 – Is Line Test indicating Fault in the Open Eir network at the customer’s

premises?

• If Yes – Create an Appointment

• If No – Contact or Network DIS – Log Fault using appropriate Report

Codes with Supporting Notes

Repeat Routine - <28 days

Line Testing03

Testing

Short

Circuit

Earth

DIS

No Test

Result

Testing OK

AC

Voltage

DC

Voltage

Testing

In this section we are looking at Routines that should be followed by the Agent when presented

with certain line test conditions:

DC – Refers to Direct Current – an exchange generates 50volts DC on a PSTN Circuit

AC – Refers to Alternating Current – generally this is 220volts AC generated by Power Supply

Note always a Health and Safety issue and requires a “Warm Handover” (Threshold 10 volts

AC)

Short Circuit – is when the two wires of the same circuit are touching

Earth is where one or both legs of the Copper Pair are in contact with Earth

DIS – A Dis is where one or both legs of a circuit have been disconnected.

The Copper Pair Test will display the above Test Conditions – there are Routines below in relation

to each Test Result

Note: that the Line could be Testing OK or No Test Result may be presented in both cases there

are Routines to be followed.

Testing

Most faults are caused by the deterioration of the Insulation around the copper wire – this

can lead to Earths, Short Circuits, DC Contacts and even AC contacts and eventually a

DIS condition.

We can have Outages where more than one customer is affected by a common single

fault – e.g. faulty exchange cards, cable damaged by builders or contractors - some

outages may be Scheduled and others Unscheduled such as Storm Damage or where

there has been an accident.

One other aspect of testing that must be taken in to consideration is Capacitance

Balance Readings which look in particular at the A to B element of the Test.

Note: Line Test interpretation is a key enabler in prevention of wasted “Truck

Rolls”

Line Test Characteristics

Short Circuit

Short

Circuit

Short Circuit – UG Screen

Short Circuit - Insulation value < .17M (170K)

Short Circuit

Useful Tips on Short Circuits

• Some exchanges can switch off Dial Tone when presented with a Short Circuit – you may

hear the terms Line Lockout or ABL (Auto Blocked Lines)

(Note this can be caused by a Phone Off Hook or by some customer premises equipment

as well as a Line Fault Condition)

• If the customer is not calling from their premises – then the Agent should request that they

report the Fault from their premises

• When the Agent is presented with a S/C the agent should request the customer to

disconnect all CPE and re-test

DC Voltage

DC

Voltage

DC Voltage – UG Screen

DC voltage

DC voltage - Value > 5V

DC voltage > 5 volts - Step by Step Process

Compare Line test result with line Signature

Step 1 – If it is better or same Then

• If Yes – Check for evidence of electrical deterioration

• If No –

• Follow CPE Isolation Process

• If this resolves the fault then issue Clear ticket

Step 2 – Is this a repeat Fault?

• If Yes – Follow Repeat fault process

• If No – Log fault using appropriate Report Code/s Plus supporting notes

DC Voltage

NGA – Fibre to the Cab

(FTTC)

04

FTTC Next Generation Access (NGA) Overview

FTTC NGA ( Next Generation Access) refers to fibre based networks that offer

Broadband Speeds up to 100Mbs.

These speeds will support Next Generation products

Fibre to the Cabinet with POTS (PSTN) – FTTC + POTS Based (PB FTTC)

Fibre to the Cabinet Stand Alone – FTTC Stand Alone (SA FTTC)

• Fibre to the Home with POTS – FTTH + POTS Based (PB FTTH)

• Fibre to the Home Stand Alone – FTTH Stand Alone (SA FTTH)

• Exchange Launched VDSL – eVDSL

• Standalone ADSL – SA ADSL

Product Variants POTS based & Stand

Alone Broadband

53

Demarcation Points

Internet Provider Network

NGN Network

PSTN Network

Exchange

NTU/ Splitter

FTTC Customer

PSTN

Switch

DSLAM

Phone

Modem/PC

Roadside FTTC Cab

Fibre Cable

Copper Cable

Roadside Copper Cab

Splitter

RAP (Regulated Access Product) NON RAP

DPE Only

VDSL Decision Tree.

Testing on FTTC

• MELT Test (Copper Pair)

• Sync Test – Sync History

• Performance

• SELT Test

CPE Checks

Faults on Copper Pair

DC – Refers to Direct Current – an exchange generates 50volts DC on a PSTN

Circuit

AC – Refers to Alternating Current – generally this is 220volts AC generated by

Power Supply Note always a Health and Safety issue and requires a “Warm

Handover” (Threshold 10 volts AC)

Short Circuit – is when the two wires of the same circuit are touching

Earth is where one or both legs of the Copper Pair are in contact with Earth

DIS – A Dis is where one or both legs of a circuit have been disconnected.

Metallic Line Test

SYNC Test

The following information is retrieved:

Port status

Actual bit rates upstream and downstream

Attainable bit rates upstream and downstream

Attenuation

Noise Margins

Last Synch time

Last loss of Synch

Showtime (seconds)

This information gives the user a snapshot of the state of the line at the point that the test is

done.

Sync Test – used for FTTC, FTTH and SADSL

Operator Screen on UG – synch check

Profile CodeDownstream_Upstream_Typei.e. Provisioned

bitrate_bitrate_ratetype

Noise Margin: :difference in strength between the VDSL signal and the background noise on the line. Higher better

Attenuation: :signal loss over copper pair –lower better

Attainable bitrate: :maximum current bitrate capability of the local loop –may never be reached in practice

Actual Bitrate: current bitrate at which modem is synched

Last Synch time: date and time of last synch

SYNC Test History

Melt Test

Performance Test

Performance Test

This test interrogates the DSLAM and retrieves the performance counters for the NGA

VDSL2 port.

These measure line performance characteristics over 15 minute and 24 hour intervals.

Typical line performance parameters are:

Corrected Blocks: The number of data blocks which had errors corrected in the period

(not seen by end-user)

Uncorrected Blocks: The number of data blocks which had errors not corrected in the

period

Errored seconds: Number of seconds containing at least one error

Severely errored seconds: Number of seconds with errors above a certain threshold

(typically 10-3)

CRC errors : Errors in Cyclic Redundancy Check Checksum in digital bitstream

Number of initialisations (Synchronisation): Number of re-synchs that the VDSL

modem has completed with the network DSLA

Performance Test

Performance Test

Test Performance for NGA (TPN)

This data comes from 15

minute and 24 hour error

counters in the DSLAM

Low numbers are better

Results for

Current 15 min period,

Previous 15 min period

Current 24 hour period

Previous 24 Hour Period

Test Performance for NGA (TPN) (2)

Parameter Explanation

Corrected blocks in Current 15

Min

Number Data blocks which have had errors corrected in them

in the present 15 min period

Uncorrected Blocks Current 15

Min

Number Data blocks which have had errors not corrected in

them in the present 15 min period

Errored Second Number of Seconds containing at least 1 error

Severely Errored Second Number of Seconds with errors above a certain threshold

Code Violation Errors within a Data Block

CRC Errors Error in CRC Checksum in digital bitstream

Initialisation count Number of resynchs

Failed Initialisation Count Number of resynch attempts where the modem failed to come

into showtime first time around

Nil

Low

High

End-user

Impact

Fibre to the home (FTTH)

05

Overview of Design

Completely new network.

FTTH Fault diagnosis

The service provider will carry out the following:

Establish that they are dealing with an FTTH service

Correlate the fault against network faults or incidents which open eir has raised as an

external trouble ticket on UG via the Outage Alerts Tab

Complete Level 1 Fault Management and proves the fault out of its network and CPE

Level 1 Fault Management:

Level 1 Fault Management consists of high level diagnosis that takes place between the

service provider and the customer It is the process whereby the service provider determines

that the fault lies with one of the following:

The open eir network

The service provider network

CPE

In-house installation In-house installation

Modem

NTU

ONT

Troubleshooting Fibre network faults

• One test required – synchronisation (sync) test

• If ONT is active and synched and RX power level is >= -22dB there is no

fault in the FTTH access network

• Based on our current product offering – additional information is presented in

the synch test result that can assist the Operator during in-home

troubleshooting with their customer:

Line profile

Last up time and last down time

Last down cause

MAC address

ONT serial number

LOSI

Fibre fault causes

ONT LEDs – Part 1

LAN

Power

Overview Reception Process

Overview FTTH decision Tree

ONT Routine

ONT-Modem Connections Routine

ONT-NTU Connections Routine

Power Routine

Synch test results

The result shows that the ONT is up and a signal is being received

The result indicates that there are no issues on the network side

A fault should not be logged without troubleshooting of the CPE

MAC address:From modem confirms

modem is connected :

ONT serial

number :

Last Down Cause: reason broadband dropped the last time it was offline

Receive power level: divide by 100 to get value in dBm

Port status

Current profile:Profile cannot be changed, Package can only be

upgraded or downgraded:

Power level threshold is the same for all packages

Validation steps to be taken by agent before logging a fault:

Query fault with end user and status of equipment in customer’s premise

Reset ONT

Synch Check

Synch History

Check ONT port profile

Fault Validation rules on UG

Fault rejected if operator has not attempted the TSB test in the last hour when

logging a FTTH fault.

All other report codes a synch test is performed to check the ONT status and the RX

power level – if ONT is UP and RX power is better than -25dBm UG will reject the

fault.

If the ONT is down and the last down cause is dying gasp (mains loss)

Report code 386(ONT not responding) -> fault is accepted

Any other report code used -> fault is rejected

Fault validation – Front End & UG

Fault Reporting Form - FNH

It is imperative that the correct report code is used when logging

a fault and that all questions are correctly answered to ensure

an efficient resolution of the issue

Fault Reporting Form - FNH

It is imperative that the correct report code is used when logging a fault and

that all questions are correctly answered to ensure an efficient resolution of

the issue

• Eircode – add to fault notes especially

• for rural unstructured addresses

• if last clear was 003 stating premise could not be found

• Confirm customer liaison number is valid and that calls will be answered

during normal working hours

• In the case of Short Circuit faults and on ADSL/VDSL faults access to

premises will probably be required – confirm access arrangements with

customer and consider booking an appointment

• In gated communities or apartment blocks access to the DP or cable

distribution will be required – agree access arrangements with customer

when logging fault rep

Value Add for Call Reception Centres

• Evaluation survey will be sent out shortly

• All Training Material is listed on the open eir website

Thank You!!!

Wrap Up.

www.openeir.ie

Thank You