State estimation and Mean-Field Control with application to demand dispatch

State Estimation and Mean Field Controlwith Application to Demand Dispatch

Yue Chen, Ana Busic, and Sean Meyn

Inria & ENS – Paris, France ECE, UF

Thanks to our sponsors:National Science Foundation & Google

Virtual Energy Storagethrough Distributed Control of Flexible Loads

1 Grid Control Problems

2 Demand Dispatch

3 State Estimation and Demand Dispatch

4 Conclusions

5 References

March 8th 2014: Impact of wind and solar on net-load at CAISO

Ramp limitations cause price-spikes

Price spike due to high net-load rampingneed when solar production ramped out

Negative prices due to highmid-day solar production

120015

0

2

4

19

17

21

23

27

25

800

1000

600

400

0

200

-200

GWGW

Toal Load

Wind and Solar

Load and Net-load

Toal Wind Toal Solar

Net-load: Toal Load, less Wind and Solar

$/MWh

24 hrs

24 hrs

Peak ramp Peak

Peak ramp Peak

Grid Control Problems


Challenges from Renewable EnergyVolatility from solar and wind energy has impacted markets

New “ramping products”

Greater regulation needs

March 8th 2014: Impact of wind and solar on net-load at CAISO

Ramp limitations cause price-spikes

Price spike due to high net-load rampingneed when solar production ramped out

Negative prices due to highmid-day solar production

120015

0

2

4

19

17

21

23

27

25

800

1000

600

400

0

200

-200

GWGW

Toal Load

Wind and Solar

Load and Net-load

Toal Wind Toal Solar

Net-load: Toal Load, less Wind and Solar

$/MWh

24 hrs

24 hrs

Peak ramp Peak

Peak ramp Peak

1 / 18


Frequency DecompositionExample: Serving the Net-Load in Bonneville Power Administration

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

0

1

2

3

4

Net-load curve = G1 +G2 +G3

G1

G2

G3

2 / 18


Frequency DecompositionExample: Serving the Net-Load in Bonneville Power Administration

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

0

1

2

3

4

Net-load curve = G1 +G2 +G3

G1

G2

G3

Low frequency component: traditional generationRemainder: “storage” (batteries, flywheels, ... smart fridges)

2 / 18

Local feedback loop

LocalControl

Load iζt Y i

tU it

Xit

Grid

signa

l

Loca

l dec

ision

Powe

r dev

iation

Demand Dispatch Design

Demand Dispatch

Demand Dispatch

Gr

Gr = G1 +G2 +G3

G1

G2

G3?

Gr

Gr = G1 +G2 +G3

G1

G2

G

Traditional generation

3

Gr

Gr = G1 +G2 +G3

G1

G2

G


Water pumping (e.g. pool pumps)

Fans in commercial HVAC3

Demand Dispatch: Power consumption from loads varies automaticallyand continuously to provide service to the grid, without impacting QoS tothe consumer

3 / 18

Demand Dispatch

Demand Dispatch

Gr

Gr = G1 +G2 +G3

G1

G2

G3?

Gr

Gr = G1 +G2 +G3

G1

G2

G


3

Gr

Gr = G1 +G2 +G3

G1

G2

G





3 / 18

Demand Dispatch

Demand Dispatch

Gr

Gr = G1 +G2 +G3

G1

G2

G3?

Gr

Gr = G1 +G2 +G3

G1

G2

G


3

Gr

Gr = G1 +G2 +G3

G1

G2

G





3 / 18

Demand Dispatch

Demand DispatchResponsive Regulation and desired QoS– A partial list of the needs of the grid operator, and the consumer

High quality AS? (Ancillary Service)

Reliable?

Cost effective?

Customer QoS constraints satisfied?

Virtual energy storage: achieve these goals simultaneouslythrough distributed control

4 / 18

Demand Dispatch

Demand DispatchResponsive Regulation and desired QoS– A partial list of the needs of the grid operator, and the consumer

High quality AS? (Ancillary Service)

Reliable?

Cost effective?

Customer QoS constraints satisfied?

Virtual energy storage: achieve these goals simultaneouslythrough distributed control

4 / 18

Demand Dispatch

General Principles for Design

Two components to local controlLocal feedback loop

LocalControl

Load iζt Y i

tU it Pre�lter Decision

ζt U it

Xit

Xit

Each load monitors its state and a regulation signal from the grid.

Prefilter and decision rules designed to respect needs of load and grid

Randomized policies required for finite-state loads

5 / 18

Demand Dispatch

General Principles for Design


LocalControl

Load iζt Y i


ζt U it

Xit

Xit

Each load monitors its state and a regulation signal from the grid.

Prefilter and decision rules designed to respect needs of load and grid

Randomized policies required for finite-state loads

5 / 18

Demand Dispatch

MDP model

MDP model

The state for a load is modeled as a controlled Markov chain.Controlled transition matrix:

Pζ(x, x′) = P{Xt+1 = x′ | Xt = x, ζt = ζ}


LocalControl

Load iζt Y i


ζt U it

Xit

Xit

Previous work:

• How to design Pζ? • How to analyze aggregate of similar loads?

6 / 18

Demand Dispatch

MDP model

MDP model

The state for a load is modeled as a controlled Markov chain.Controlled transition matrix:

Pζ(x, x′) = P{Xt+1 = x′ | Xt = x, ζt = ζ}


LocalControl

Load iζt Y i


ζt U it

Xit

Xit

Previous work:

• How to design Pζ? • How to analyze aggregate of similar loads?

6 / 18

Demand Dispatch

Aggregate Model≈ Mean field model

Reference Output deviation (MW)

−300

−200

−100

0

100

200

300

0 20 40 60 80 100 120 140 160t/hour

0 20 40 60 80 100 120 140 160

State process:

µNt (x) =1

N

N∑i=1

I{Xit = x}, x ∈ X

Evolution: µNt+1 = µNt Pζt + ∆t

Output (mean power): yt =∑x

µNt (x)U(x)

Nonlinear state space model Linearization useful for control design

7 / 18

Demand Dispatch



−300

−200

−100

0

100

200

300

0 20 40 60 80 100 120 140 160t/hour

0 20 40 60 80 100 120 140 160

State process:

µNt (x) =1

N

N∑i=1

I{Xit = x}, x ∈ X



µNt (x)U(x)


7 / 18

Demand Dispatch



−300

−200

−100

0

100

200

300

0 20 40 60 80 100 120 140 160t/hour

0 20 40 60 80 100 120 140 160

State process:

µNt (x) =1

N

N∑i=1

I{Xit = x}, x ∈ X



µNt (x)U(x)


7 / 18

Demand Dispatch

Nonlinear state space model: µt+1 = µtPζt, yt = 〈µt, U〉Linearization useful for control design

Bode Diagram

Mag

nitu

de (d

B)

-10

0

10

20

30Myopic Passive Optimal

10-410-5 10-3 10-2

Frequency (rad/s)10-1

one h

our n

omina

l cyc

le

Three designs for a refrigerator: transfer function ζt → yt

8 / 18

Demand Dispatch

Grid Control Architecture: ζt = f(?)

ζ = f(∆ω)

ζ = f(y)

ζ

grid freq (Schweppe ...)

load power dev (Inria/UF 2013+)

load histogram (Montreal/Berkeley)= f(µ)

IncreasingInform

ation

ζ = f(∆ω)

ζ = f(y)

ζ = f(y)

ζ


load power (Inria/UF 2013+)

load histogram (Montreal/Berkeley)

This work:

= f(µ)

IncreasingInform

ation

ˆ

ζ = f(∆ω)

ζ = f(y)

ζ = f(y)

ζ




This work:

Linear state space model subject to white noise

State estimation using Kalman Filter

= f(µ)

IncreasingInform

ation

ˆζt ytLoads

µt

Goals: Estimate yt for control and QoS distribution

9 / 18

Demand Dispatch


ζ = f(∆ω)

ζ = f(y)

ζ




IncreasingInform

ation

ζ = f(∆ω)

ζ = f(y)

ζ = f(y)

ζ




This work:

= f(µ)

IncreasingInform

ation

ˆ

ζ = f(∆ω)

ζ = f(y)

ζ = f(y)

ζ




This work:



= f(µ)

IncreasingInform

ation

ˆζt ytLoads

µt


9 / 18

Demand Dispatch


ζ = f(∆ω)

ζ = f(y)

ζ




IncreasingInform

ation

ζ = f(∆ω)

ζ = f(y)

ζ = f(y)

ζ




This work:

= f(µ)

IncreasingInform

ation

ˆ

ζ = f(∆ω)

ζ = f(y)

ζ = f(y)

ζ




This work:



= f(µ)

IncreasingInform

ation

ˆζt ytLoads

µt


9 / 18

State Estimation and Demand Dispatch


Linear State Space Model

State space model:

µNt+1 = µNt Pζt + ∆t

yNt = 〈µNt ,U〉 =1

N

N∑i=1

Y it

Observations: Randomly sample a fixed percentage of {Y it }

Yt =1

m

m∑k=1

Yst(k)t = yNt + Vt

Samples {st} i.i.d. and uniform.

Kalman filter requires second-order statistics of (∆t, Vt).See proceedings

10 / 18



State space model:


yNt = 〈µNt ,U〉 =1

N

N∑i=1

Y it

Observations: Randomly sample a fixed percentage of {Y it }

Yt =1

m

m∑k=1

Yst(k)t = yNt + Vt

Samples {st} i.i.d. and uniform.

Kalman filter requires second-order statistics of (∆t, Vt).See proceedings

10 / 18



State-observation model:


Yt = yNt + Vt

Two versions of the Kalman filter considered,differentiated by Kalman gain Kt

1 Assumption: µNt+1 is conditionally Gaussian given Yt = (Yk, ζk) |tk=0.Under this assumption, the Kalman filter = optimal nonlinear filter.The gain is a nonlinear function of observed variables.

2 The filter that is optimal over all linear estimatorssimilar to [Krylov, Lipster, and Novikov, 1984]

The first is more easily calculated, and worked well in experiments.See proceedings for details

11 / 18





Yt = yNt + Vt





11 / 18





Yt = yNt + Vt





11 / 18





Yt = yNt + Vt





11 / 18


Observability fails?Observed in models of residential pools, HVACs, fridges ...

One example for residential pools:

λ0 λζ

96 Eigenvalues of the Observability Grammian

961

10-10

10-5

100

105

i48 7224

|λi|

In general, all states are not recoverable from observations:

‖µa0 − µb0‖ = 1, yet∞∑t=0

|yat − ybt |2 < 10−12

12 / 18


Key features are observable

1. yNt : total power consumption of loads

-303

Inpu

tζ t

Output deviation Reference

t/hour0 20 40 60 80 100 120 140 160

−100

−50

0

50

100

MW

300,000 residential pools, with 0.1% sampling

2. Discounted QoS (quality of service)

13 / 18



1. yNt : total power consumption of loads2. Discounted QoS (quality of service)

Lit =

t∑k=0

βt−k`(Xik),

for residential pools: `(x) ∝ [power consumption− desired mean]

13 / 18



1. yNt : total power consumption of loads2. Discounted QoS (quality of service)

t/hours

x10 3

−100

−50

0

50

0 100 200 300 400 500 600 7000

2

4

6

Estimate Empirical

Lt

ΣLt

Varian

ceMea

n

13 / 18


Sampling rate, N , and closed-loop performance

Goal is to track reference signal rt.

Normalized error: et =yNt − rt‖r‖2

0

2

4

6

8

10

12

14

16

18

0.1%1.0%10%100%

Sampling Rate

3× 103 3× 1053× 104 3× 106 N

RMS N

orm

alize

d Erro

r (%

)

14 / 18


Un-modeled dynamics and closed-loop performance

Setting: 0.1% sampling, and

1 7th-order reduced-order observer (state is dimension 96)

2 Large uncertainty in heterogeneous population of loads

3 And, load i opts-out when QoS Lit is out of bounds

0

0.5−10

−5

0

5

10

MW

100 120110 130

opt o

ut %

N = 300,000N = 30,000

100 120110 130

Closed-loop tracking

−100

−50

0

50

100

0.5

0


t/hour t/hour

15 / 18


Un-modeled dynamics and closed-loop performance

Setting: 0.1% sampling, and

1 7th-order reduced-order observer (state is dimension 96)

2 Large uncertainty in heterogeneous population of loads

3 And, load i opts-out when QoS Lit is out of bounds

0

0.5−10

−5

0

5

10

MW

100 120110 130

opt o

ut %

N = 300,000N = 30,000

100 120110 130

Closed-loop tracking

−100

−50

0

50

100

0.5

0


t/hour t/hour

15 / 18

Conclusions

Conclusions

Observability provably fails in many cases,yet important features can be estimated in-spite of large modeling error

Much more in the paper:

“Half of the states are unobservable for symmetric models”

Kalman filter for joint ensemble-individual (µt, Xit)

More on pools and fridges

16 / 18

Conclusions

Conclusions

Observability provably fails in many cases,yet important features can be estimated in-spite of large modeling error

Much more in the paper:

“Half of the states are unobservable for symmetric models”

Kalman filter for joint ensemble-individual (µt, Xit)

More on pools and fridges

Outstanding question: What information is needed for successfulapplication of these methods?

ζ = f(∆ω)

ζ = f(y)

ζ




IncreasingInform

ation

ˆ

ˆ

Purely local control may not be effective for primary control, but ...stay tuned

16 / 18

Conclusions

Conclusions

Thank You!

17 / 18

References

Selected References

S. Meyn, P. Barooah, A. Busic, Y. Chen, and J. Ehren. Ancillary service to the grid usingintelligent deferrable loads. IEEE Trans. on Auto. Control, 2015, and Conf. on Dec. &Control, 2013.

P. Barooah, A. Busic, and S. Meyn. Spectral decomposition of demand-side flexibility forreliable ancillary services in a smart grid. In Proc. 48th Annual Hawaii InternationalConference on System Sciences (HICSS), pages 2700–2709, Kauai, Hawaii, 2015.

N. V. Krylov, R. S. Lipster, and A. A. Novikov, Kalman filter for Markov processes, inStatistics and Control of Stochastic Processes. New York: Optimization Software, inc.,1984, pp. 197–213.

J. Mathieu, S. Koch, and D. Callaway, State estimation and control of electric loads tomanage real-time energy imbalance, IEEE Trans. Power Systems, vol. 28, no. 1, pp.430–440, 2013.

P. Caines and A. Kizilkale, Recursive estimation of common partially observed disturbancesin MFG systems with application to large scale power markets, in 52nd IEEE Conferenceon Decision and Control, Dec 2013, pp. 2505–2512.

R. Malhame and C.-Y. Chong, On the statistical properties of a cyclic diffusion processarising in the modeling of thermostat-controlled electric power system loads, SIAM J.Appl. Math., vol. 48, no. 2, pp. 465–480, 1988.

18 / 18

Engineering

State estimation and Mean-Field Control with application to demand dispatch