Master's Thesis Slides

Approximate Dynamic Programming Methods forResidential Water Heating
A thesis submitted in partial fulfillment for the degree of Masters of Sciencein the
Department of Electrical Engineering
byMatthew Motoki
December 3, 2015

Outline
1 Motivation
2 Problem FormulationSystem VariablesState DynamicsObjective FunctionProblem Statement
3 MethodologyFinite-Horizon DPAverage Cost DP for Periodic MDPsApproximate Dynamic Programming
Prescient Lower Bound (PLB)
4 ResultsNumerical Simulations SetupSet-Point MethodsTemperature AggregationUsage History Aggregation
5 Problem ExtensionsSolar Water HeatingAutomated Demand Response
6 Conclusion

Motivation
Why do we need a smarter water heater?
I Energy efficiency is important.
Electricity is expensive. Burning fossil fuels is bad for the environment.
I Can we do better than water heaters with an adjustable set-point?
If so, then are there any provable guarantees that can be made? Theoretically, what is best that we can do?
I The legacy grid is becoming obsolete.
Renewable energy sources are variable and distributed. Energy storage capabilities of water heaters have been fully exploited.
1 / 31

Outline
1 Motivation
6 Conclusion

Problem Formulation
State VariableDefine t {0,t, . . . , (N 1)t}. Define tk = mod(k ,N)t, where k = 0, 1, . . . isthe simulation time stage.
The state x := (T , h) summarizes the information needed to make a decision.
We require T [Tamb,Tmax ]. The temperature at tk is written Tk .The hot water usage history is hk := {(ti ,wi ) | 0 i < mod(k ,N)}, where wk is theintensity of the hot water draw at time tk .
2 / 31

Problem Formulation
Decision VariableThe decision variable is
uk :=
{1, if the water heater is on
0, if the water heater is off.
We assume that the decision uk is constant during the interval [tk , tk+1). A feasibledecision uk u, is one that does not violate T [Tamb,Tmax ]. A policy is amapping from a state into a feasible decision.
3 / 31

Problem Formulation
Hot Water Demand (Disturbance Variable) 1
We model hot water demand as a cyclostationary random process W(t) given by
W(t) := specific heat
Npeoplei=1
Nij=1
F (j),i (T (j),i Tamb
) I{S(j),i t < S
(j),i +D
(j),i
},
where := {shower , bath, . . . , dishwasher} is the set of possible usage events, Npeopleis the number of people in a household, Ni is the number of events of type corresponding to the i th person in the household, and the following are randomvariables:
S(j),i := the start time of E(j),i ,
D(j),i := the duration of E(j),i ,
F (j),i := the flow rate of E(j),i ,
T (j),i := the desired temperature of E(j),i .
4 / 31

Problem Formulation
Hot Water Demand (Disturbance Variable) 2
We can only observe W(t) at pre-specified times t t , therefore, we approximateW(t) using a piecewise linear interpolation
W(t) :=W(tk) +t tk
t[W(tk + t)W(tk)].
for all k = 0, 1, . . . and t [tk , tk + t). The discrete-time analog of W(t) to be theaverage of W(t) over t [tk , tk + t),
Wk :=1
t
tk+ttk
W(t) dt = 12 [W(tk) +W(tk + t)].
We denote particular realizations of W(t) and Wk using w(t) and wk , respectively.We discretize wk {0,w , . . . ,wmax}. We write the conditional probability massfunction of Wk given hk as pWk (wk | hk).
5 / 31

Outline
1 Motivation
6 Conclusion

Problem Formulation
State Equation
The state equation maps the current state xk , current decision uk , and currentdisturbance wk into the next state xk+1 according to
xk+1 = f (xk , uk ,wk) :=(fT (Tk , uk ,wk), fh(tk , hk ,wk)
),
where
Tk+1 = fT (Tk , uk ,wk) := max{Tk rcoolt (Tk Tamb)+ rheatt uk rlosst wk , Tamb
}hk+1 = fh(hk ,wk) :=
{{(tk ,wk)} hk , tk 6= (N 1)t, otherwise,
for all k = 0, 1, . . .
6 / 31

Outline
1 Motivation
6 Conclusion

Problem Formulation
Objective Function
The objective is to minimize over all policies , the following function
J(x0) = limK
EW
[1
K
K1k=0
g(Xk , (Xk),Wk ;
) x0],
= limK
1
K
K1k=0
EW0,W1,...,Wk ,
[g(Xk , (Xk),Wk ;
) x0].where X0 = x0 is given and Xk = f
(Xk1, (Xk1),Wk1
), for all k = 1, 2, . . .
7 / 31

Problem Formulation
Stage Cost
The stage cost is
g (xk , uk ,wk ; ) := gdiscomfort (xk , uk ,wk ;Tmin) + (1 ) goperating (xk , uk) ,
where := {,Tmin} is a customer-defined parameter set, [0, 1] is the relativeweighting of the objectives, and Tmin is the minimum desirable temperature during ahot water use.
Operating Cost
The operating cost is
goperating (uk) :=1
t
tk+ttk
C (t) rating uk dt,
where C (t) is the cost of power and rating is the power rating of the water heater.8 / 31

Problem Formulation
Discomfort CostThe discomfort cost is
gdiscomfort(xk , uk ,wk ;Tmin
):=
1
t
tk+ttk
max{Tmin T (t), 0
} I{w(t) > 0} dt,
where
T (t) := Tk +t tk
t[fT (Tk , uk ,wk) Tk ],
for all k = 0, 1, . . .
9 / 31

Outline
1 Motivation
6 Conclusion

Problem Formulation
Problem StatementFind a feasible on/off policy that minimizes an expected objective cost.
minimize
limK
EW
[1
K
K1k=0
g(Xk , (Xk),Wk ;
) x0]
subject to Xk+1 = f(Xk , (Xk),Wk
), (xk) {0, 1},
Tk [Tamb,Tmax ], for all k = 0, 1, . . .This is a discrete-time, average cost periodic Markov decision problem (MDP).
10 / 31

Outline
1 Motivation
6 Conclusion

Methodology
Finite-Horizon Dynamic Programming
The goal is to minimize over all policies , the following function
J(x0) = EW
[gterminal(XM) +
M1k=0
g(Xk , k(Xk),Wk ;
) x0],
where M is the horizon and gterminal is a terminal cost function.
The optimal policy is the minimizer of Bellmans equations
J(xM) = g

Master's Thesis Slides

Documents

Text of Master's Thesis Slides

Dalkiran-Master's Thesis

Vashist_ Master's Thesis

Master's Thesis Klinkenberg

Master's thesis ppp

Master's Thesis Presentation

Haider master's thesis

Slides for my Master's Thesis

Master's degree thesis

Master's thesis / Masterarbeit

Master's Thesis Report

Master's Thesis Panels

Master's Thesis - Chajae