1

Embracing next generation telecommunications technologies enabling smart grids and substation automation

20 %

Francis Baestaens

3

Agenda

• Introduction to OTN Systems

•Choosing the best telecom technology for power utilities

•SDH migration

•Network Management

•Network hardware

•Features for power utilities

•Who is using it?

4

Introduction to OTN Systems

March, 2018

5

27 years experience in creating industrial network solutions

• A worldwide market leader o Electricity, water and gas utilities

o Metro & Railways

o Most Oil & Gas Majors

o ITS, Airports, Mining, …

• 500+ customers in 75+ countries

o High loyality & recurring

• 17.000 nodes in operations

o Running flawlessly for many years

Power Utilities

Electric

Transport

Gas&

Oil

6

Delivering the critical network element for industrial networks

Power Utilities

Electric

Transport

Gas&

Oil

OTN Systems

Network Solutions

7

… as part of an end to end solution for Operational Telecom

Power Utilities

Electric

Transport

Gas&

Oil

Network Equipment

Network Management

Services& Support

8

Choosing a next generation telecommunicationstechnology for power utilities

9

Smart Grids

• Power o production

o Transport

o consumption

• Telecom required foro Managing Energy flows

o Control, metering

o Grid Protection

o Auxiliary services

10

Telecom should be “invisible” for Industrial customers

• Reliable – always on

• Secure and ruggedized

• Deterministic – no surprises

• Easy to install and manage – also for non-Telecom experts

• Safe – with an extended life time up to 25 years or more

• Enabling smart grid applications & substation automation

Power Utilities

Electric

Transport

Gas&

Oil

SDH

11

TDM technology phasing out

What ?

• SDH = used for leased lines, backbone infrastructure of telcos, but already phased out since 10 yearso In SA Telkom is closing down the last Diginet services

• PDH = access technology for appliances, telemetry, SCADA, voice, HVM, teleprotection, signalling

Why ?

• 95% has already phased out

• Closing of component factories

• Need for more bandwidth & flexibility

• Applications already using packet technologies

12

The right technology pick: MPLS-TP

IP/MPLS Ethernet MPLS-TP

Reliability ++<50ms convergence per

service

+ <50ms network protection

++<50ms convergence per

service

Security +L2 and L3 toolset

+L2 toolset

+L2 toolset

Traffic

engineering++

Flow based

engineering/QoS

-No network wide QoS

++Flow based

engineering/QoS

Deterministic -Dynamic behavior control

plane

-Only applicable in limited

topologies

++Static provisioned APS via

OAM

Monitoring -Need deep network

knowledge

-Lack of NMS

++Full NMS and OAM

Ease of Use -Complex stack of

protocols

+ Pure Ethernet based

++ NMS driven approach

• Telecom is a rapid changing environmento Driven by large Service Providers and Enterprise

customers

• MPLS-TP is the enabling technology to support a smooth operationo Confirmed by many users all over the world

o Complex Service Provider solutions (e.g. IP/MPLS)

put a lot of pressure on the operational staff

13

Main telecommunictions requirements

• Connection orientedo Paths through the network are predefined

• Bidirectional pathso Traffic flows in a symmetrical way

• Static control planeo The full control is in the Network Management System

• Logical topologieso Ring, subring, point-to-point, point-to-multipoint

• Complete monitoring of each link by default (“hello” packets every 1-5ms)

VC-12/2Mbps

STM-1/155Mbps VC-4/140Mbps

SDH

14

Technology comparison

SDH

IP/MPLSMPLS-TPCarrier Ethernet

15

Challenges for Operational Telecom (1)

Introduction Growth Maturity

MPLS for OT MPLS for IT

Decline

SpecializedSDH/SONET

SDH/SONET

Forced to a technology leap by much bigger industries

Power Utilities

Electric

Transport

Gas&

Oil

Transformation is inevitable

16

Challenges for Operational Telecom (2)

Struggle for talent

Power Utilities

Electric

Transport

Gas&

Oil

Responsible for OT and many other things30y+ experience in OT but knows little about packetWill retire in less than 10 years

IT expertExperience in routing and switchingNo clue about OT

New packet platform must have an easy step-in

17

The optimal product for Operational Telecom

20 %

ProductizationMPLS-TP

Technology

Power Utilities

Electric

Transport

Gas&

Oil

18

Migration from SDH to MPLS-TP

19

Migration from SDH to MPLS-TP: How ?

Transparent: • connect all existing applications

• Seamless integration

Simple:• No overly complex telco solutions

• Short training

• Intuitive to manage

• Single core

Future proof:• Supporting latest application

evolutions

• Scalable

• Long lasting (at least 15 years)

20

TDM

Packet

TDM

TDM

Packet

TDM

Packet

PacketNow

1. Keep SDH in service during transition 2. Build a parallel network or extend with XTran3. Migrate to packet

Service continuityFall back (risk mitigation)Optimized MPLS network• Efficient• Flexible• Future proof

Future

SDH Migration and extension to MPLS-TP – 1-step migration

Transparent forapplications

21

Recent CIGRE publication June 2016

“MPLS-TP in the context of operational telecom seems to be the appropriate technology when going to packet”

• Migration from SDH

• Maintaining full control of the Network

• Quality of Service and deterministic behavior

• Size of network and type of traffic

• Capability versus complexity

22

MPLS-TP node deployment environment – typical topology

Distribution

Substation

Transmission

Substation

Generation

Substation

Distribution network

High voltage grid

Dispatching center

SCADA

VOICE

Data

Power ranges 220-240VAC, 18-300VDC

Low consumption from 33-150W

Operational Telecom

Operational Telecom Operational Telecom

Mission critical

Operational applications

Optional DWDM network

Wholesale

Datacenter traffic

IT traffic

…

Easy

NMS

SNMP

Umbrella

Management

Platform

23

Network availability & Easy network management

24

What is network availability?

Availability is determined by:

• MTBF of the hardware (mean time between failure) – expressed in years (e.g. 35)

• Redundancy of the hardware (power supply, controller card)

• Redundancy of the network (multiple paths, backup)

Availability Unavailable time (31 days)

99,9% 2678 seconds or 44,6 minutes

99,99% 268 seconds or 4,5 minutes

99,999% 27 seconds or ½ minute

Switch-over time between controller cards shouldbe in order of miliseconds to guarantee service continuity

25

Easy network management

A management system should be

• Intuitive with short learning cycle (a few days)

• Clear and easy to use

• Application oriented

• Not requiring direct CLI access foro Troubleshooting

o Configuration

26

Total Cost of Ownership

Maintenance

Installation

Training

Updating

Complexity

Equipment Cost

Unavailability

CAPEX

OPEX

MPLS-TP / XTRAN Other Equipment

27

TXCare: Easy network management & operation

Plan & design

TX = Takes care of your network

C = Configuration Management

A = Assurance Management

R = Resilience Management

E = End-to-end Management

Provision

& Program

Troubleshoot

Assure &

measure

28

Network hardware

29

XTRAN: MPLS-TP platform made of Industrial hardware

• Fan-less / extended temperature range

o -30°C to +65°C

• EMC hardened

o IEC61850-3 / IEEE 1613 / EN 50121-4

• Ethernet and IP interfaces

• PDH interfaces: G703, Serial, E1, FXS, E&M

• Teleprotection interface: C37.94

• Flexible power options (wide input range)

• Rugged industrial design

o Stainless steel

o Compact 19” or DIN rail mountable

30

IEEE 1613 – Vibration/Chock testing

31

Hardware & firmware operation

• Deterministico SDH-like

o No surprizes

• Total controlo Complete Provisioning and monitoring

via NMS

o Full control

• High availability due to redundancyo Achieving 99.999% availability

• Cyber secure and virtually invisibleo OSI Layer 2.5

o Door-closing measures

32

Features for power utilities

33

Substation communication

Protection

relay/SCADA

C 37.94, G.703, E1…Substation Switch

RTU

Function Function Function

IEC 61850 Process Bus

Substation Switch

Network Transport

node

Ethernet (10/100/1000)

Ethernet (10/100/1000/10000)

IEC104 (Ethernet)

SCADA, HVM, telemetry,…

IEC101 (RS232/422/485, E&M, G703, E1) Ethernet, IP

PMU

Synchrophasors

34

Tuned For Time Critical Applications

Enhanced Delay

Control©

NMS Driven Emulation Settings

Strict Traffic Engineering

Bidirectional Co-routed

LSPs

Buffer Equalization

Clock Drift Prevention

Hitless Switching

35

Can delay be controlled ?

Factors that influence delay/latency

• Link distance: every km of fiber introduces 5µs of delay

• Industrial MPLS-TP nodes averagely introduce 8µs of delay

• Dejitter buffer (allow a high-speed network to connect to a low-speed network)

• Delay Compensation buffer in case 1+1 scenario is usedo Paths with different length

Delay/latency CAN and MUST be controlled

to have reliable transmission over MPLS

36

Most sensitive application: current differential protection

• Based on comparison of measured values

• Difference means that there is a fault existing

• Comparison of similar measurement is essential

• Variation of transmission time “looks” like a fault current for the relay

• Performance criteria are based on• End-to-end delay variation

• Differential delayBalance between DEPENDABILITY and RELIABILITY

37

Evaluated by the experts

• The University of Strathclyde, in Glasgow, UK, hosts the Institute for Energy and Environment,

• The Technology and Innovation Centre (TIC) has a laboratory for testing novel power systems research

• Worldwide recognition

Differential Protection

Relay

Differential Protection

Relay

C37

.94

1G 1GXTRAN (200)

C37

.94

SDH SDHSTM-11G 1G

XTRAN (100) XTRAN (300)

10G

38

Evaluation results & conclusion

• MPLS-TP using hitless switching is thebest technology for power

• All expected results were reached

• MPLS-TP remains easy to operate even in large networks

End-to-end delay < 6ms

Differential delay < 400 µs

MPLS-TP performed as predicted and industry

values were respected under all conditions

39

Tuned For Time Critical Applications

Enhanced Delay

Control©

NMS Driven Emulation Settings

Strict Traffic Engineering

Bidirectional Co-routed

LSPs

Buffer Equalization

Clock Drift Prevention

Hitless Switching

40

NMS Driven Emulation Settings

• Smart Network Management Software

• Service based settings –network wide

• Automatic Calculation:o Bandwidth & delay

• Prevent asymmetric delay

• Hitless switching settings

• Offline & online mode

41

Control & design the delay to your applications

Optimized for

delay

PDH C37.9

4

42

Tuned For Time Critical Applications

Enhanced Delay

Control©

NMS Driven Emulation Settings

Strict Traffic Engineering

Bidirectional Co-routed

LSPs

Buffer Equalization

Clock Drift Prevention

Hitless Switching

43

Strict Traffic Engineering

• XTran is designed to provide full control of the traffic behavior

• Prevent network wide asymmetric delay

• External data will not influence the behavior of the system

• Traffic Engineering is managed network wide by TXCare NMS

• Traffic policing done on the ingressof the XTRAN network

44

Tuned For Time Critical Applications

Enhanced Delay

Control©

NMS Driven Emulation Settings

Strict Traffic Engineering

Bidirectional Co-routed

LSPs

Buffer Equalization

Clock Drift Prevention

Hitless Switching

45

Bidirectional Co-routed Paths

3ms

5ms

3ms

• MPLS-TP Characteristic (standard RFC 5654)

o Associated Bidirectional path

o Co-routed

• Both directions are setup, monitored and protected as a single entity through TXCare NMS

• Prevent asymmetric delay

• Always & native behavior

3ms

46

Tuned For Time Critical Applications

Enhanced Delay

Control©

NMS Driven Emulation Settings

Strict Traffic Engineering

Bidirectional Co-routed

LSPs

Buffer Equalization

Clock Drift Prevention

Hitless Switching

47

Buffer Equalization

2 ms

3 ms

2 ms

Equalization

2 ms

• Buffers are equalized to guarantee stable & lossless operation

• Prevent asymmetric delay

48

Tuned For Time Critical Applications

Enhanced Delay

Control©

NMS Driven Emulation Settings

Strict Traffic Engineering

Bidirectional Co-routed

LSPs

Buffer Equalization

Clock Drift Prevention

Hitless Switching

49

Clock Drift prevention

• In some use cases there may occur a clock difference between two directions, eg. below where RX is disconnected (A-side).

• To avoid this, XTran maintains the timing over the circuit emulation stream but using a slip buffer on the packetization side A.

• Result is that the clock towards B remains constant and no differential delay is build up.

• Prevent asymmetric delay

X

Internal Clock

SLIP BUFFER

Adaptive

A B

A-Clock

50

Tuned For Time Critical Applications

Enhanced Delay

Control©

NMS Driven Emulation Settings

Strict Traffic Engineering

Bidirectional Co-routed

LSPs

Buffer Equalization

Clock Drift Prevention

Hitless Switching

51

TDM Based 1+1 Protection switching

LER LERLSR LSR

LSR

MPLS-TP Protection

Must be guaranteed < 50ms

LER LERLSR LSR

LSR

Hitless Switching

0 ms, 0 loss, stable delay

SAToP

CESoP

SAToP

CESoP

Adaptive

Differential

SyncE

RS232

4WE&M

G703

E1

C37.94

RS232

4WE&M

G703

E1

C37.94

52

Ethernet Based 1:1 Protection switching – MPLS-TP

LER LERLSR LSR

LSR

Lineair Protection

Must be guaranteed < 50ms

LER LERLSR LSR

LSR

Logical Ring

Must be guaranteed < 50ms

Ethernet

Or IP

based

services

Ethernet

Or IP

based

services

53

Who is using it ?

54

Who is using MPLS-TP ?

• USA & Europe/ANZ: smaller DSO companies

o Mainly used for accurate and deterministic substation automation

o Migration of critical apps like Scottisch power

• LAM: large DSO/TSO like CFE Mexico

o Complete SDH migration and substation automation ongoing

• Middle East: decided to go for MPLS-TP (90%)

o Saudi Electric Company, FEW/SEWA/DEWA UAE, OETC Oman

• Africa & Aisa: still mainly on SDH

o Leading utilities like EETC Egypt have decided for MPLS-TP

o Most still on SDH but increasing maintenance & shortage

• Russia/CIS: Decision for MPLS-TP has been taken

o First pilots ongoing at Rosseti and affiliates

55

XTran: some references

56

Conclusion

• Smart grids and automated substations require an accurate and deterministictelecommunications network

• SDH will not be suitable and not available

• MPLS-TP is objectively the best next generation packet technology

• Vendors focusing on vertical industrial markets develop purpose built equipment

57

XTRAN offers new opportunity's.

Let’s make the change together!