Cyber-Enabled Smart Distribution Systems and

Micro Grids

Bruce McMillinDepartment of Computer ScienceMissouri University of Science and

Technology(Formerly the University of Missouri-Rolla)

Rolla, MO 65409-0350

Introduction: CPS

• Cyber Physical Systems (CPSs) are integrations of computation with physical processes.– Distributed Control

• Advanced Power Electronics – Finer-grained control over physical entities– Schedulable entities

• Design Issues– Complex and unpredictable interactions between the

cyber and physical processes– Information flow across the cyber-physical boundaries

Schedulable Power Electronics

Under Distributed Control

33

vv

Transmission LineGeneration

FACTS

Wind Power

Energy Storage

Solar Power

Energy Storage

Energy Management Communications Satellite

FACTS device

Distributed Decisions

Power Electronics

Communications

Sensing and monitoring Inputs

Power Electronics

Power Electronics

Distributed controland fault/attack detection

Transmission via Distributed Control Using Power Electronics

A Specific Problem• Prevent Cascading

failures:– 2003 Blackout

• Causes– Physical & Cyber

contingencies– Deliberate disruption

• Hackers• Terrorist Activity

Proposed Solution

Flexible AC Transmission Systems (FACTS) – Power Electronic Controllers– Contain embedded computer and networking– Means to modify the power flow through a

particular transmission corridor - UPFC– Operate under distributed control

Work done with Mariesa Crow at S&T sponsored by NSF & DOE/Sandia: http://filpower.mst.edu

How to Start?

We need a good formalism to work from.

Maximum Flow in a Digraph

Power System

Power system as a graph

Max Flow Predicts Best Power Flow to Set FACTS Devices

Max-Flow

• Assign an initial flow to all arcs• Mark the source and sink• Search for a node that can be labeled. If none is found,

flow is maximum, stop.

• Backtrack the path computing the minimum ij used. Go to previous step.

s a t100 40

=40

60

0

17

28

50

50

22

815

10

30

320

s

a

e

d

c

b

t

100

40

17

28

50

50

22

815

10

30

320

s

a

e

d

c

b

t

100

40

17

28

50

50

22

815

10

30

320

s

a

e

d

c

b

t

s

45

17

28

50

50

22

80

10

30

35

a

e

d

c

b

t

60

17

28

50

50

22

815

10

30

320

a

e

d

c

b

t

s

60

17

28

50

50

22

815

10

30

320

a

e

d

c

b

t

s

28

28

50

50

22

8

10

13

35

a

e

d

c

b

t

s

45

17

28

50

50

22

8

10

30

35

a

e

d

c

b

t

s

45

17

28

50

50

22

8

10

30

35

a

e

d

c

b

t

s

28

0

28

50

50

22

8

10

13

35

a

e

d

c

b

t

s

28

28

50

50

22

8

10

13

35

a

e

d

c

b

t

s

0

0

22

50

22

8

10

13

35

a

e

d

c

b

t

s

22

50

22

8

10

13

35

e

d

c

b

t

s

22

50

22

8

10

13

35

e

d

c

b

s

0

28

0

8

10

13

35

e

d

c

b

t

s

28

8

10

13

35

d

c

b

t

s

28

8

10

13

35

d

c

b

t

s

25

5

10

10

05

d

c

b

t

s

25

5

10

10

5

d

c

b

t

s

25

5

10

10

5

d

c

b

t

s

20

0

10

10

0

d

c

b

t

s

20

10

10

d

c

b

t

0

0

0

c

b

10

s

20

10

10

d

c

b

t

10

s

10

10

10

c

b

t

s

10

10

10

c

b

t

10

10

10

c

b

t

0

s a t60 15

=15d20

s a t45 17

=1730

c

s a t28 28

=2850

e

s b t50 22

=2222

e

s b c28 8

=3

3

d t13

s b t25 5

=55

d

s b20 10

=10t

s b t10 10

=1010

c

• In general, lines are not all maximally loaded. The power flow can then be re-directed to new transmission corridors.– Where re-direct?– How much to re-direct?– How account for KCL?– Control/communication between decision-making

devices?

t

Loss of Line B-D

100

40

17

28

50

50

22

815

18

30

320

s

a

e

d

c

bX

10/20

t

100

40

17

28

50

50

22

15

10

30

320

s

a

e

d

c

b

18

• Power will flow over b-c, overloading it

• Recalculating Flow over b-t removes overload

G

Riversde 1

Pokagon

2HickryCk

3

NwCarlsl

4

11

8

117

12

SouthBnd

Tw

inB

rch

Corey

Olive

Olive

Bequine

Breed

9

10

G

FtWayne15

67

Kankakee

JacksnRd

Concord

14

17

30

Sorenson

Sorenson

13

16

GoshenJt

N. E.

37

34

3536

NwLibrty

39

40 41 42

18

19 43

S.Kenton

38

S.TiffinWest End Howard

WLima

Rockhill

EastLima

Sterling

LincolnMcKinley

Adams20

Jay21

Randolph22

113

31

32

29

28

Grant

Mullin Delaware

DeerCrk

Deer Crk

Outage 37-39a

From 65MuskngumS

Area 25

33Haviland

G

Riversde 1

Pokagon

2HickryCk

3

NwCarlsl

4

11

8

117

12

SouthBnd

Tw

inB

rch

Corey

Olive

Olive

Bequine

Breed

9

10

G

FtWayne15

67

Kankakee

JacksnRd

Concord

14

17

30

Sorenson

Sorenson

13

16

GoshenJt

N. E.

37

34

3536

NwLibrty

39

40 41 42

18

19 43

S.Kenton

38

S.TiffinWest End Howard

WLima

Rockhill

EastLima

Sterling

LincolnMcKinley

Adams20

Jay21

Randolph22

113

31

32

29

28

Grant

Mullin Delaware

DeerCrk

Deer Crk

Outage 37-39b

From 65MuskngumS

Area 25

33Haviland

G

Riversde 1

Pokagon

2HickryCk

3

NwCarlsl

4

11

8

117

12

SouthBnd

Tw

inB

rch

Corey

Olive

Olive

Bequine

Breed

9

10

G

FtWayne15

67

Kankakee

JacksnRd

Concord

14

17

30

Sorenson

Sorenson

13

16

GoshenJt

N. E.

37

34

3536

NwLibrty

39

40 41 42

18

19 43

S.Kenton

38

S.TiffinWest End Howard

WLima

Rockhill

EastLima

Sterling

LincolnMcKinley

Adams20

Jay21

Randolph22

113

31

32

29

28

Grant

Mullin Delaware

DeerCrk

Deer Crk

Outage 37-39c

From 65MuskngumS

Area 25

33Haviland

Add A FACTS Device

• Under Proper Control• Avoids the overload that causes the outage that causes

the cascade

G

Riversde 1

Pokagon

2HickryCk

3

NwCarlsl

4

11

8

117

12

SouthBnd

Tw

inB

rch

Corey

Olive

Olive

Bequine

Breed

9

10

G

FtWayne15

67

Kankakee

JacksnRd

Concord

14

17

30

Sorenson

Sorenson

13

16

GoshenJt

N. E.

37

34

3536

NwLibrty

39

40 41 42

18

19 43

S.Kenton

38

S.TiffinWest End Howard

WLima

Rockhill

EastLima

Sterling

LincolnMcKinley

Adams20

Jay21

Randolph22

113

31

32

29

28

Grant

Mullin Delaware

DeerCrk

Deer Crk

Outage 37-39a

From 65MuskngumS

Area 25

33Haviland

QuestionWhat does this have to do with

Distribution?

Future Renewable Electric Energy Delivery

and Management (FREEDM) – NSF ERCAn efficient and revolutionary power gridIntegrating distributed and scalable alternative energy sources and storage with existing power systems

Shipping 250M pcs/yr.

Ubiquitous ownership

Ubiquitous use

Ubiquitous sharing

Pre-1980sPre-1980s

InternetInternet

Paradigm ShiftParadigm Shift

Distributed ComputingDistributed ComputingCentralized MainframesCentralized Mainframes

Innovation & Industry

Transformation

Ubiquitous sales

Ubiquitous ownership

Ubiquitous use

Ubiquitous sharing

TodayToday

Centralized GenerationCentralized Generation100+ year old technology100+ year old technology

New energy companies based on IT and power

electronics technologies

Paradigm ShiftParadigm Shift

FREEDMFREEDM SystemSystem

Innovation & Industry

Transformation

DistributedDistributed Renewable RenewableEnergy Resources (DRER)Energy Resources (DRER)New technologies

for distributed renewable energy

The FREEDM Concept

• Distributed Intelligence– People share energy

resources– Neighborhood or

industrial level– Where is the

centralized controller?

E SD

U s e r I n te rf a ce

D is t ri bu t ed G ri d In te l lig e n c e (D G I )F RE E D M

S ub s ta tio n

1 2 k V

1 2 0 V

M a rk e t & E co n o m ic s

6 9 k V

IE M

AC

AC

IF M IF M

IF M

L OA D D R ER D ES D

IE M

AC

AC

L OA D D R ER D ES D

IE M

AC

AC

3 Φ 4 8 0V

R SC

Legacy grid

Substation

H2

Substation

Smallturbine

flywheel

PV array

EV

Traditional power grid

Fuel cell car

Renewable hydrogen

PV array

PV array

PV array

Fuel cell car

Remote Wind Farm

Substation

H2

Substation

Smallturbine

flywheel

PV array

EV

Traditional power grid

Fuel cell car

Renewable hydrogen

PV array

PV array

PV array

Fuel cell car

Remote Wind Farm

Substation

H2

Substation

Smallturbine

flywheel

PV array

EV

Traditional power grid

Fuel cell car

Renewable hydrogen

PV array

PV array

PV array

Fuel cell car

Remote Wind Farm

Substation

H2

Substation

Smallturbine

flywheel

PV array

EV

Traditional power grid

Fuel cell car

Renewable hydrogen

PV array

PV array

PV array

Fuel cell car

Remote Wind Farm

• Distributed Intelligence– Spread over components

of a FREEDM node– Components work

together to provide a solution

– Failure of a single component does not cause system failure

– Components are not bound to any specific device or location

– Multiple Points of Vulnerability

The FREEDM System Is Distributed

ESD

User Interface

Distributed Grid Intelligence (DGI)FREEDM

Substation

12kV

120 V

Market & Economics

69kV

IEM

AC

AC

IFM IFM

IFM

LOAD DRER DESD

IEM

AC

AC

LOAD DRER DESD

IEM

AC

AC

3Φ 480V

RSC

Legacy grid• IEM and IFM

nodes each run a portion of the DGI to manage their own resources

• Coordinate to control the whole as a Distributed Algorithm

IEM: Intelligent Energy Management IFM: Intelligent Fault Management

DRER: Distributed Renewable Energy Resource DESD: Distributed Energy Storage Device

Schedulable Entity

The Solid State Transformer

Inside an IEM Node

• Solid State Transformer (SST)– Power Electronics– Schedulable Entity

SH5

SH7

SH6

SH8

S1

S3 S4

SH1

SH3 SH4

SH2 S2

Low Voltage H-Bridge

+

-

+

-

400V DC

High Frequency

Transformer

AC/DC Rectifier DC/DC Converter DC/AC Inverter

High Voltage H-Bridge

High voltage H-Bridge

12kVDC

7.2 kV AC

120V / 240V AC

LLs

Cs

CsLs

Port 1

Port 2

How to use it?

IEM Nodes and Distributed Processes

• Each IEM/IFM node contains a Computer running a Process that sends messages to its peers

• No other sharing of information

Distributed Grid Intelligence• Distributed Long and Short Term Control• Distributed Systems Management

– Distributed Leader– State Maintenance

• Simulation Architectures• Power Economics Models and Control• Fault Tolerance of Cyber-Physical system• Security – Confidentiality, Integrity, and Availability of Cyber-Physical

system• Resilience - Robust Distributed System

– Formal Correctness– Usability as an automomous system

Distributed Algorithm –Load

Balancing• Each IEM node

has an aggregate (S)upply or (D)emand

• Where to get power from or provide power to?

• No centralized picture of the system

Distributed Load Balancing

• Correctness: Keep all IEM nodes’ “balanced” in terms of Supply and Demand

• Pass messages negotiating load changes until the system has stabilized

• Global optimization decomposed into individual processes that cooperate to meet the global correctness.

Satisfy IEM 1’s Demand

IEM 0 D

IEM 1 D

IEM n S

IEM 0 S

IEM 1 D

IEM n S

IEM 0 S

IEM 1 S

IEM n S

IEM 0 IEM 1 IEM n

REQUEST SUPPLYI CAN SUPPLY

IEM n then sends power and IEM 1 receives it

Distributed Leader Election

• System management functions, configuration / reconfiguration on-line, automatic restoration, distributed state maintenance such that each IEM node contributes to DGI.  – In a hierarchical control, a Leader Election is

necessary to dynamically reconfigure higher layers of control

– Dynamic Leader/Coordinator

DGI Leader Election

• A leader is a distinguished dynamically-elected node that may change during operation

DGI Leader Election

• A newly elected leader due to failure of old leader

Merge()

Recovery()

Check()

Timeout()

Ready()

Are you coordinator()

Are you there()

Input: Current Node (inviting coordinator), Coordinator SetSends invitation to merge to the coordinators it knows and to its members using invitation() with current node as leaderAfter a reasonable time, reorganization with the new members of the group is attempted The new members are designated with a task using ready()On time out, calls recovery()

Input: Current Node

Put this node in a singleton group with itself as leader

Subsequently, this leader calls for election to merge

Input: Current Node

Every coordinator checks for other coordinators by calling Are You Coordinator()

It invites the so-found coordinators for a possible merge of groups using Merge()

Input: Coordinator Node, Coordinating Group , Member Node, Yes or No

Calling node wishes to know if the coordinator it knows is still a coordinator and if so, does it still consider it to be its member

Input: Every Node, Yes or No

Calling node wishes to know if node is a coordinator in normal state

Input: New Member Node, Coordinator Node , Group, Task to be assigned

Coordinator of the group assigns a task to the new member node of the group to get it start with its membership

Input: Current Node

Every member that has not heard from its coordinator checks its status using Are you there()

If it yields a NO, recovery() is called

Invitation()

Accept()

Input: Invited node, Inviting node, group to join

Invited node in Normal State forwards invitation from inviting node to its membersCalls Accept() if interested to join

Input: Invited node, Inviting node, group to join

Invited node acknowledges the invitation to join the group coordinated by the inviting node

Invitation Algorithm

f

f

Election

Election

Election

Election f

Election

1 2

3 4

Coordinator node Member node

Group Management and Election

Threats to DGI• Hardware Degradation

– Maintenance required– Rollback and Recovery

• Software Failure– Residual Design Flaws– Rollback and Recover with Alternate Algorithms

• Hackers– Teenager in the basement hacking into an IFM

• Denial of Service Attack• Information Warfare

– Buffer Overflow and Quality of Service (Denial of Service)– Confidentiality of decision making

• Integrity attacks• Confidentiality

– Information flow– Multi-level security model– Less studied aspect in the cyber-physical world – key problems arise from observation of

physical interactions

Confidentiality of CPS• Modern Infrastructures consist of Cyber and Physical

Components– Distributed Energy Resources, Smart Houses, Air

Transport, Vehicle Transport, Smart Structures, Oil and Gas Pipelines

– All have an inherent commonality – Physical Actions

• A Security Leak in a Physical System– Pizzas at the Pentagon

Motivation

• Observable physical changes in cyber-physical systems divulge security related information

• Security Policy defines what level of security

What security do you want?

Information Security (from NERC CIP Standards)

Information Flow Models

• FREEDM contains Power Electronics Devices that perform physical actions that are observable

• Cannot keep these secret – loss of confidentiality/privacy• Some other models

– Non-Interference• High-level events do not interfere with the low level outputs

– Non-Inference• Removing high-level events leaves a valid system trace

– Non-Deducibility• Low-level observation is compatible with any of the high-

level inputs.

Threats & Vulnerabilities?

• Denial of (information) service– Localized power outages

• Privacy– My neighbors can now infer what I’m doing

• Gaming the system– Economic Gains

• Hacker in the Basement– What fun!

Social Aspects

• People Must Use This– Bridging the Cyber, Physical, and Social Worlds

Workshop – May 27-28, Kansas City– Social Scientists, Engineers, Computer People– Linkages between the worlds– Many “a-ha moments”

• Linkage between disciplinary theories• Sociology as a driving force• Enforce correctness, also, through social needs - ethics

Futures

• Understanding what the CPS is truly an integrated system

• Develop widely applicable analysis techniques finding commonality among infrastructures– Theories that can bridge the cyber and physical and

social worlds such that information flow and power flow are uniformly understood.

• Educational programs that cross train computer scientists with engineered domains and social domains

FREEDM DGI Team• The team, Bruce McMillin S&T, Frank Mueller, NCSU,

Mariesa Crow, NCSU, Mo-Yuen Chow, NCSU, Chris Zimmer, NCSU, Derek Ditch, S&T, Ravi Akella S&T, Marfield Meng, S&T, Gerald Heydt, ASU, Alex Huang (Director), NCSU