The use of Mobile Data Communications for SCADA

Applications by

Resham Singh eThekwini Electricity Durban, South Africa

Contents

1. Introduction to eThekwini Electricity 2. GPRS SCADA Architecture 3. Performance Bandwidth usage Network latency Protocol considerations

4. Reliability 5. Security 6. Conclusion and Recommendations

Page 2 of 16

eThekwini Electricity : Introduction

• Municipal power utility of the city of Durban, on the east coast of South Africa

• 700 000 customer base • Maximum demand of 1900

MVA • Customer load 30%

residential, 70% industrial/business

• Network covers from 275kV to 230V

Page 3 of 16

eThekwini Electricity : SCADA • ACS PRISM Master

Station • 110 HV RTUs (RTU560,

D400, D20, ERTU) • 400+ MV RTUs (RTU540, BECKHOFF) • 500+ 11kV distributor

substations still require RTUs

• 250+ RMU (Talus T200) • 150+ Nulec Reclosers to

be added to SCADA network

Page 4 of 16

Why use GPRS for SCADA?

"You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is

meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send

signals here, they receive them there. The only difference is that there is no cat."

Albert Einstein

Page 5 of 16

GPRS SCADA Architecture

Page 6 of 16

Performance 1. 3 Distributor Substations selected at

geographically dispersed sites with 3G/GPRS modems installed

2. Traffic captured at the SCADA server 3. Analysis of data traffic for bandwidth

usage, network latency, and data packet loss

4. Comparison between best performing 3G/GPRS link and a fibre link

Page 7 of 16

Bandwidth Usage 1. Are GPRS speeds sufficient or do we need 3G speeds? 2. Let us investigate the data transmitted from our test sites

3. Typical speeds are ~14 kbps on the uplink. GPRS with CS-1 coding scheme delivers 8 kbps (worst case)

4. Therefore bandwidth is not an issue - GPRS speeds are sufficient 5. Be careful about trunk links – between APN and SCADA. This may cause a

bottleneck. The size of this link should be a multiple of the max bit rate per substation.

ICC 100 Woodhurst Zwelibomvu Avg. packet/sec 0.015 (67) 0.014 (71) 0.010 (100)

Avg. packet size 96 bytes (768 bits) 99 bytes (792 bits) 101 bytes (808 bits)

Avg. bps 768 bps 792 bps 808 bps

Max data stream 2 packets totalling 11 160 bits

1 packet totalling 10 648 bits

1 packet totalling 7 032 bits

Max bps ~11 kbps ~11 kbps ~7 kbps

Page 8 of 16

Bandwidth Management 1. Controlling the alarming network is imperative as costs are

incurred per transaction – consider a sustained period of 11 kbps 2. Manage digital inputs

Chatter detection and blocking

3. Manage analogue inputs Threshold supervision What significant change should

an operator detect, 1A?

4. Mode of protocol operation Solicited or unsolicited? Read request = ~ 90 bytes every 10 secs = ~ 24 mB

per month per substation Page 9 of 16

Latency performance (PING test)

Station ICMP request ICMP reply Min response time (milliseconds)

Max response time (milliseconds)

ICC 127756 126016 64 95311

Zwelibomvu 127753 121360 84 33794

100 Woodhurst 127837 116679 58 38118

0

10000

20000

30000

40000

50000

60000

1 2 3 4 5 60

20000

40000

60000

80000

100000

1 2 3 4 5 60

15000

30000

45000

60000

75000

1 2 3 4 5 6

ICC Zwelibomvu 100 Woodhurst

No.

of p

ings

Time (seconds)

No.

of p

ings

No.

of p

ings

Time (seconds) Time (seconds)

Page 10 of 16

0 5 10 15 20 25 30 35 40 450

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

X: 5.5Y: 0.7711

Time (milliseconds)

Pin

g R

espo

nse

Prb

abili

ty

Fibre normalised latenciesX: 33Y: 1

X: 14Y: 0.9999

X: 28Y: 0.9999

X: 11Y: 0.7711

Fibre Latencies

0

20000

40000

60000

80000

100000

120000

10 11 12 13 14 15

Fibre

Fibre - GPRS comparison: • 11ms vs 1s • 28ms vs 28s

Time (ms)

No.

of p

ings

Page 11 of 16

Protocol Considerations 1. Non-deterministic network with

varying latencies poses several challenges for the DNP3 protocol implementer

2. DNP3 has several configurable parameters that should be chosen to match the communication network

– Application layer response timeouts

– Select Before Operate (SBO) Controls

– Time synchronisation

Page 12 of 16

Reliability

Station ICMP requests ICMP replies ICMP lost Packet loss % Longest outage duration (min)

Channel Status Fail

ICC 127756 126016 1740 1.362 28.4 12.0

Zwelibomvu 127753 121360 6393 5.004 34.8 75.0

100 Woodhurst 127837 116679 11158 8.728 43.5 40.0

Station ICMP requests ICMP replies ICMP lost Packet loss % Longest outage duration (min)

Channel Status Fail

Wentworth 127755 127750 5 0.004 0.16 0

GPRS/3G Performance

Performance of a fibre link IEEE 802-2004 requires

less than 0.012%

Page 13 of 16

Reasons for Packet Loss 1. Reasons for high Packet Loss on GPRS/3G

network Corrupted packets FEC Defective network elements Dispersed sites Signal degradation over the air interface e.g., fading Signal Strength High loads on the network/congestion ?

2. Which is the most contributing factor? Station ICMP request ICMP reply ICMP lost Packet loss %

Longest outage duration (min)

100 Woodhurst 30 min before the new year '13 181 176 5 2.762 0.3

100 Woodhurst 30 min into the new year ‘13 181 50 131 72.376 4.3

Congestion follows crisis Page 14 of 16

Security 1. Standards – NERC CIP, IEC62351 2. GPRS employs authentication and ciphering

methods on the Radio Access Network 3. A RADIUS Server is normally deployed by the

service provider to ensure further authentication 4. Best practice is to deploy firewalls at end devices 5. To provide end-to-end security, it is

recommended that site-to-site VPN tunnels be deployed. IEC recommends the use of TLS, therefor an OpenVPN network implemented

Page 15 of 16

Conclusion and Recommendations 1. Data speeds offered on the GPRS network is sufficient for the DNP3

protocol.

2. When engineering protocol parameters the following should be taken into account:

High latency network with a high corresponding variability Cost model is based on data usage (and not duration) so it is

important to design to transmit only what is necessary High packet loss especially during times of congestion. An

efficient transport layer protocol is required.

3. Newer cellular radio standards offer promising capabilities

QoS Load differentiation mechanisms offered by LTE

Page 16 of 16

The end. Any questions?