Development and Testing of Resilience Grid Automation

Carsten Krüger

29.10.2019

Development and Testing of Resilience Grid Automation

using RT simulations

Motivation

10/29/2019 R&D Division Energy 2

Paradigm Shift to Distributed Smart Grids

Motivation

10/29/2019 3

Paradigm shift to Complex Smart Grids

• DER – Clean resource and too some extend flexible, but more uncertainties.

• ICT (i.e. automation sys.) enables better monitoring, operation, decision making and control

• Smart grid services – State estimation, voltage control, unit commitment, etc.

• Strong coupling between the systems

Power ICT

Smart grid services

Data and power supply

R&D Division Energy

Motivation


Analysis of past events

2003 – North AmericaState estimator +

alarm system failure

2005 – SwitzerlandCongestion/

Information overload

2013 – AustriaCongestion caused by

a software bug

2015 – UkraineCyber attack on

control room

2017 – UkraineMost destructive cyber

attack ever

Strong interdependency brings-in new threats and vulnerabilities

Events are caused or aggravated by ICT events Need to consider ICT in power system planning and operation

2003 North American blackout Events

1. State Estimator failure2. Tripping of lines (DP&L)3. Alarm system failure4. State Estimator fixed with incorrect data5. Tripping of lines (Chamberlin)6. Voltage collapse

No

rmal

Pow

er S

yste

m O

per

atio

nal

Sta

tes

Ale

rtEm

erg

Bla

cko

ut

MotivationMapping of 2003 North American blackout

Power system eventsICT status is

missing!

2

5

6

729 October

2019

ICT System States

1

2

3

45

6

ICT system events

Source - NERC Steering Group. (2004). Technical analysis of the August 14, 2003, blackout: What happened, why, and what did we learn. Report to the NERC Board of Trustees, 13.


Qu

elle

: D

aim

ler

„Dri

vin

gSi

mu

lato

r in

Sin

del

fin

gen

“, 2

014

How to start with ICT testing?

6

Appropriate information, communication and automation systems are known from other domains

> But: long-term use in safety critical energy systems mostly untested

> Highly fraught with risk to stakeholders in the energy domain

Rigorous testing necessary!

> Learning from other industrial domains...

> „Hardware in the Loop“

> Operation of a real el. controller hardware or a mechatronic component in a simulation of the real environment

> But: what belongs into this simulated environment?

> How and on what parts of the system?

> Remember: …holistic!

10/29/2019 R&D Division Energy

Smart Energy Simulation and Automation Lab

10/29/2019 7

Automation and Control

Real-time Power Simulator

OPAL-RT eMEGAsim

OPAL-RT ePHASORsim

Virtual OTRTUs – Master/Slave

PMUs

Communication Simulation/Emulation

...VPN Gateway

Power Hardware

Voltage source

type APS 1000

NI CompactRIO

Controller –

PMU and IEC 61850

BECKHOFF

C6920

BECKHOFF

CX2020

Industrial OT Hardware

PxC RTU Master

PxC RTU Station

KoCoS EPPE

CX

Grid Automation

controller

D/A

Remote Access

Anomaly Detection Framework

and ICT Monitoring

Ethernet

Analogue

Time

sync

Big data platform

EtherCAT

Switch

PPCs Smart Meter Ethernet Gateways

Virtualization Server

Gateway administration

CLS Manager

Protocol Switch

Distributed Agents

Co-Simulation framework

NVIDiA DGX-1

Deep learning server / AI

R&D Division Energy

Examples of Testing Projects


ERIGrid vIEDCross validation of virtual vs physical IEDs Use-case: GOOSE messages quality test

CybResLabHow to increase resilience of Smart Grid Services

Use-case: improved state estimation

EneraTesting performance of distributed controllers Use-case: communication round trip testing

ERIGrid vIED


Testing Performance of Virtual IED

> Develop a virtual IEC61850 compliant virtual environment for large-scale simulation / virtualization studies

> Validate the performance of virtual IED in a real-time set-up with the results from a physical set-up

> Test the performance of GOOSE and IEDs for reverse blocking scheme and validate it with physical set-up

> Define KPIs (GOOSE Transmission time, circuit breaker tripping time)

> Use a Holistic Testing Procedure developed at ERIGrid for multi-domain multi-laboratory validation

ERIGrid vIED


Test Platform

Busbar

15 kW Feeder

20kV Distribution grid connection

GOOSE

REF615 sender

REF615 Reciever

UDPcurrent meas.

GOOSE

GOOSE

Vied 1

Vied 2

Research Infrastructure 1 Multipower labVTT Espoo

Research Infrastructure 2SESA LabOFFIS Oldenburg

RT-Lab

Connection point

15 kW load

A

B

20kV Distribution

grid

> Message: Shirt circuit fault - do not trip IED B> PTOC start > XCBR status value> 4ms average time

> The messages are sent via GOOSE communication - Loss of mains protection

> Test case 1: Overcurrent at feeder load

> Fault is cleared and breaker is reclosed

> Relay A would trip and send blocking signals and circuit breaker status information to B

> Test case 2: GOOSE communication failure> Blocking signal not sent

R&D Division Energy 1110/29/2019

1

0

20 6040 80 100 120 140 160 180 200

CB

Po

siti

on

Time (in ms)

CB pos for grid

GOOSE trip block signal generated

Trip is blocked via GOOSE

The grid CB trips1

0

20 6040 80 100 120 140 160 180 200

CB

Po

siti

on

Time (in ms)

CB pos for grid

GOOSE trip block signal failure

Tripping time

Test case 1> GOOSE message trip time

> RI 1 = 1.71ms> RI 2 = still working

on it

Test case 2> GOOSE communication is

impaired > Grid CB trips and grid

connection point is lost

Results

Distributed Controllers Testing


Hardware-in-loop Test

Determine capabilities and characteristics of Physical RTUs

via HIL test.

Determine computational and communication delay

Test Set-up


Modbus Server 2Receives register

Test Values

Modbus Server 1

Sends register

Modbus Client

Modbus Client

IEC 60870-5-104Server

IEC 60870-5-104Client

Statistical Approach


> Data is sent with progressing change frequency rate

> The successful data transmission is determined

> The cut-off change frequency rate is determined

> The experiment is performed 500 times due to stochastic nature of communication.

> The statistical delay is determined

Value characteristic composed by integers

0123456789

101112131415161718192021222324252627282930

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115

Sen

t D

ata

Val

ue

Time [s]

Profile of Data Sent through Modbus Slave to Modbus Unterstation

Highlights


Average Real-Time Roundtrip: 1.028 sStandard deviation: 0.132 s

90.7%

71.6%72.7%

59.6%52.5%

36.1%32.3%

24.7%

12.3%

5.8%3.9%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

0.0

1

0.0

2

0.0

3

0.0

4

0.0

5

0.0

6

0.0

7

0.0

8

0.0

9

0.1

0

0.1

1

0.1

2

0.1

3

0.1

4

0.1

5

0.1

6

0.1

7

0.1

8

0.1

9

0.2

0

0.2

1

0.2

2

0.2

3

0.2

4

0.2

5

0.2

6

0.2

7

0.2

8

0.2

9

0.3

0

0.3

1

0.3

2

0.3

3

0.3

4

Successful Transmission Rate

Successful Transmission Rate

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

11%

12%

13%

14%

15%

16%

0.625 0.675 0.725 0.775 0.825 0.875 0.925 0.975 1.025 1.075 1.125 1.175 1.225 1.275 1.325 1.375 1.425 1.475

Pro

ba

bili

ty

Roundtrip Transmission Time [s]

Data Transmission Roundtrip Time Probability

Data Transmission Normal Distr. Mean: 1.028 StaD: 0.132

Testing Grid Monitoring Applications


CybResLab

HV

MV-12,4 kW-2,3 kVAR

8,2 kW0,7 kVAR

9,4 kW1,2 kVAR

3,4 kW0,3 kVAR

8,1 kW0,6 kVAR

8,4 kW0,9 kVAR

I. Functional Correctness

II. Safety

III. Security

IV. Reliability

V. Credibility

VI. Usability, Understandability

Tru

st F

ace

ts

of Measurement Data

> All conceivable attack vectors manifest themselves in a combination of (violated) trust facets

> What to do with this multivariate assessment?

> E.g. substitute measurements with historical/simulated values?

> Do nothing?

> What is the worst that could happen?

> What information could be used to improve/represent each Facet!

Dynamic Accuracy w.r.t. Operational Margin of Error (State Estimation)

IT Health Monitoring

Intrusion/Anomaly Detection Systems

(Historical) Network/QoS Monitoring

(Historical) Contextual Trust Information

Expert Operator

Inp

ut So

urce

s

0.40.90.80.30.90.7 ?

Integration of OT/IT health status

• ICT monitoring tools (e.g. interface and memory status,...)

• Intrusion Detection System (network malicious, ...)

Creating data reliability index (trustworthiness)

Application

• Improved (integrated) state estimation –identification and consideration of abnormal patterns

• ICT – Power systems state classification

• Selection of proper countermeasure

• Fault detection and management

1710/29/2019

HV

MV

IDS

Alarm

OT Status InformationICT

ICT

Stat

us

Info

rmat

ion

Cyb-PhysAnomalyDetection

TrustIndex

ICT

R&D Division Energy

Anomaly-Aware State Estimation

Set-up in detailed


9

8

10

1

2

3

5 4

6

7

RTU 38

RTU 29RTU 28RTU 26RTU 01

RTU 02

RTU 37

RTU 30

RTU 01

RTU 39

RTU 05

RTU 09

RTU 08

RTU 07

RTU 06

RTU 04 RTU 14

RTU 15

RTU 12

RTU 11 RTU 13

RTU 10RTU 01

RTU 32RTU 34 RTU 33

RTU 20

RTU 19

RTU 03

RTU 18 RTU 17

RTU 27

RTU 16

RTU 21 RTU 22

RTU 23

RTU 36RTU 24

RTU 35

RT-Lab Exata

Anomaly Detector

State Estimator

Highlights


> Estimated state variables:

> outside critical area: close to real values

> inside ciritical area: different from real values

> Uncertainties (as a measure of trust)

> without framework: almost zero everywhere

> with framework: high inside critical area


Thank you!Carsten Krüger

Energy [email protected]

Documents

Development and Testing of Resilience Grid Automation