Slide Lecture4

July 3rd and 15th, 2010 @ Gakushuin

E-mail: [email protected]://sites.google.com/site/masaruinaba/

() 2010/07/3, 15 @ Gakushuin 1 / 67

I. (dynamic programming):

() 2010/07/3, 15 @ Gakushuin 2 / 67

maxct ;kt+1

TXt=0

tu(ct)

subject to

kt+1 kt = f (kt) ct kt for t = 1; 2; ;T;k0 = k0kT+1 0:

0 < < 1 (discount factor)u() f ()

u(0) = 0; u0() > 0; u00() < 0; u0(0) = 1; u0(1) = 0f (0) = 0; f 0() > 0; f 00() < 0; f 0(0) = 1; f 0(1) = 0

() 2010/07/3, 15 @ Gakushuin 3 / 67

(recursive)

.

(Bellmans principle of optimality)

.

.

.

. ..

.

.

((1994)P150)

(backward induction method) t (backwardinduction method)

() 2010/07/3, 15 @ Gakushuin 4 / 67

I.1 (backward inductionmethod)

T kT ucT cT kT+1

maxcT ;kT+1

u(cT )s.t. kT+1 kT = f (kT ) cT kT

kT : givenkT+1 0:

() 2010/07/3, 15 @ Gakushuin 5 / 67

maxkT+1

u

f (kT ) + (1 )kT kT+1

s.t. kT : givenkT+1 0:

kT+1 kT

kT+1 = T (kT ):

policy function

() 2010/07/3, 15 @ Gakushuin 6 / 67

policy functionT

u0(cT ) = TT =

kt+1 0; 0; and kT+1 = 0:

u0(cT ) = kT+1 = 0:

u0(cT ) = > 0kT+1 = T (kT ) = 0

cT = f (kT ) + (1 )kT T (kT ) policy function kT

() 2010/07/3, 15 @ Gakushuin 7 / 67

VT (kT ) =u

f (kT ) + (1 )kT T (kT )

VT (kT ) T kT (state evaluation function) (value function)

() 2010/07/3, 15 @ Gakushuin 8 / 67

T 1T T 1 kT VT (kT )T VT (kT )T 1

maxcT1;kT

u(cT1) + VT (kT )s.t. kT kT1 = f (kT1) cT1 kT1

kT1 : given cT1

maxkT

u

f (kT1) + (1 )kT1 kT+ VT (kT )

s.t. kT1 : given kT1

kT = T1(kT1) () 2010/07/3, 15 @ Gakushuin 9 / 67

value function

VT1(kT1) =uT1(kT1)

+ VT (kT )

VT1(kT1) T 1 kT1 value function

() 2010/07/3, 15 @ Gakushuin 10 / 67

( tt + 1t kt+1Vt+1(kt+1)Vt+1(kt+1)t

maxct ;kt+1

u(ct) + Vt+1(kt+1)s.t. kt+1 kt = f (kt) ct kt

kt : given ct

maxkt+1

u

f (kt) + (1 )kt kt+1+ Vt+1(kt+1)

s.t. kt : given kt+1 = t(kt)

Vt(kt) =u

f (kt) + (1 )kt t (kt)+ Vt+1(kt+1)

() 2010/07/3, 15 @ Gakushuin 11 / 67

( t=0)

maxc0;k1

u(c0) + V1(k1)s.t. k1 k0 = f (k0) c0 k0

k0 : given

c0

maxk1

u

f (k0) + (1 )k0 k1+ V1(k1)

s.t. kt : given

k1 = 0(k0)

V0(k0) =u

f (k0) + (1 )k0 0(k0)+ V1(k1)

() 2010/07/3, 15 @ Gakushuin 12 / 67

.

.

.

1 t = 0 k0 k1 = 0(k0). c0 = f (k0) + (1 )k0 0(k0)

.

.

.

2 t = 1 k1 k2 = 1(k1). c1 = f (k1) + (1 )k1 1(k1):::

.

.

.

3 t kt kt+1 = t(kt). ct = f (kt) + (1 )kt t(kt):::

.

.

.

4 T kT kT+1 = T (kT ) = 0. cT = f (kT ) + (1 )kT T (kT ) kT+1 = 0

() 2010/07/3, 15 @ Gakushuin 13 / 67

T

() 2010/07/3, 15 @ Gakushuin 14 / 67

(fundamental recurrencerelation)

maxct ;kt+1

TXt=0

tu(ct)

subject to

kt+1 kt = f (kt) ct kt for t = 1; 2; ;T;k0 = k0kT+1 0:

() 2010/07/3, 15 @ Gakushuin 15 / 67

T

VT (kT ) = maxcT ;kT+1

u(cT )s.t. kT+1 kT = f (kT ) cT kT

kT : given; kT+1 0:

u0(cT ) = TT =

kt+1 0; 0; and kT+1 = 0:

() 2010/07/3, 15 @ Gakushuin 16 / 67

u0(cT ) = kT+1 = 0:

u0(cT ) = > 0 kT+1 = (kT ) = 0.

cT = f (kT ) + (1 )kT 0T

VT (kT ) = u

f (kT ) + (1 )kT: (1)

(boundary condition)

() 2010/07/3, 15 @ Gakushuin 17 / 67

T 1VT1(kT1) = max

cT1;cT ;kT ;kT+1u(cT1) + u(cT ) (2)

s.t. kT kT1 = f (kT1) cT1 kT1kT+1 kT = f (kT ) cT kTkT1 : givenkT+1 0:

VT1(kT1) = maxcT1;kT

hu(cT1) + max

cT ;kT+1u(cT )

is.t. kT kT1 = f (kT1) cT1 kT1

kT+1 kT = f (kT ) cT kTkT1 : given:

() 2010/07/3, 15 @ Gakushuin 18 / 67

(1) VT (kT )

VT1(kT1) = maxkT

nu

f (kT1) kT + (1 )kT1| {z }cT1

+ VT (kT )

o(3)

() 2010/07/3, 15 @ Gakushuin 19 / 67

T 2VT2(kT2) = max

cT2;cT1;cT ;kT1;kT ;kT+1u(cT2) + u(cT1) + 2u(cT ) (4)

s.t. kT1 kT2 = f (kT2) cT2 kT2kT kT1 = f (kT1) cT1 kT1kT+1 kT = f (kT ) cT kTkT2 : given; kT+1 0:

VT2(kT2) = maxcT2;kT1

u(cT2) +

nmax

cT1;cT ;kT ;kT+1u(cT1) + u(cT )

os.t. kT1 kT2 = f (kT2) cT2 kT2

kT kT1 = f (kT1) cT1 kT1kT+1 kT = f (kT ) cT kTkT2 : given; kT+1 0:

() 2010/07/3, 15 @ Gakushuin 20 / 67

(2) VT1(kT1)

VT2(kT2) = maxkT1

nu

f (kT2) kT1 + (1 )kT2+ VT1(kT1)

o(5)

() 2010/07/3, 15 @ Gakushuin 21 / 67

T 3VT3(kT3) = max

cT3;cT2;cT1;cT ;kT2;kT1;kT ;kT+1u(cT3) + u(cT2) + 2u(cT1)

+ 3u(cT )(6)

s.t. kT2 kT3 = f (kT3) cT3 kT3kT1 kT2 = f (kT2) cT2 kT2kT kT1 = f (kT1) cT1 kT1kT+1 kT = f (kT ) cT kTkT3 : givenkT+1 0:

() 2010/07/3, 15 @ Gakushuin 22 / 67

VT3(kT3) = maxcT3;kT2

u(cT3)

+ n

maxcT2;cT1;cT ;kT1;kT ;kT+1

u(cT2) + u(cT1) + 2u(cT )o

s.t. kT2 kT3 = f (kT3) cT3 kT3kT1 kT2 = f (kT2) cT2 kT2kT kT1 = f (kT1) cT1 kT1kT+1 kT = f (kT ) cT kTkT3 : givenkT+1 0:

() 2010/07/3, 15 @ Gakushuin 23 / 67

(4) VT2(kT2)

VT3(kT3) = maxkT2

nu

f (kT3) kT2 + (1 )kT3+ VT2(kT2)

o(7)

() 2010/07/3, 15 @ Gakushuin 24 / 67

t

Vt(kt) = maxc;k+1

TX=t

tu(c)

s.t. k+1 k = f (k) c k (for = t; ;T 1;)kt : given; kT+1 0:

Vt(kt) =maxct ;kt+1

2666664u(ct) + maxc;k+1

8>>>: TX=t+1

t1u(c)9>>=>>;3777775

s.t. kt+1 kt = f (kt) ct ktk+1 k = f (k) c k (for = t + 1; ;T 1;)kt : given:

() 2010/07/3, 15 @ Gakushuin 25 / 67

t + 1

Vt(kt) = maxkt+1

nu

f (kt) kt+1 + (1 )kt+ Vt+1(kt+1)

o(8)

(Bellmansequation)

() 2010/07/3, 15 @ Gakushuin 26 / 67

k0 = k0

VT (kT ) = u

f (kT ) + (1 )kT

t = 0; 1; 2; ;T 1Vt(kt) = max

kt+1

nu

f (kt) kt+1 + (1 )kt+ Vt+1(kt+1)

o (value function)

VT (kT ) ! VT1(kT1) ! ! V0(k0)policy function

maxkt+1

nu

f (kt) kt+1 + (1 )kt+ Vt+1(kt+1)

o(for t = 0; 1; 2; ;T 1:)

kt kt+1 = t(kt),ct = f (kt) t(kt)|{z}

kt+1

+(1 )kt

() 2010/07/3, 15 @ Gakushuin 27 / 67

(optimal policy function)

maxkt+1

nu

f (kt) kt+1 + (1 )kt+ Vt+1(kt+1)

o(for t = 0; 1; 2; ;T 1:)

kt+1

u0(ct) + V 0t+1(kt+1) = 0 (9)()u0(ct) = V 0t+1(kt+1) (10)()u0

f (kt) kt+1 + (1 )kt

= V 0t+1

kt+1

()

(11)()kt+1 = t(kt) (12)

kt.

() 2010/07/3, 15 @ Gakushuin 28 / 67

kt+1 = t(kt)

Vt(kt) = u

f (kt) t(kt) + (1 )kt+ Vt+1

t(kt)

(for t = 0; 1; 2; ; T 1:)

kt

V 0t (kt) = u0(ct )hf 0(kt) 0t(kt) + (1 )

i+ V 0t+1(kt+1)0t(kt)

()V 0t (kt) = u0(ct )hf 0(kt) + (1 )

i+ 0t(kt)

hu0(ct ) + V 0t+1(kt+1)

i

() 2010/07/3, 15 @ Gakushuin 29 / 67

(10)( (envelope theorem))

()V 0t (kt) = u0(ct )hf 0(kt) + (1 )

i(10)

u0(ct) = u0(ct+1)hf 0(kt+1) + (1 )

i

() 2010/07/3, 15 @ Gakushuin 30 / 67

II (dynamic programming)

() 2010/07/3, 15 @ Gakushuin 31 / 67

II.1

maxct ;kt+1

1Xt=0

tu(ct)

subject to

kt+1 kt = f (kt) ct ktk0 = k0

0 < < 1 (discount factor)u() f ()

u(0) = 0; u0() > 0; u00() < 0; u0(0) = 1; u0(1) = 0f (0) = 0; f 0() > 0; f 00() < 0; f 0(0) = 1; f 0(1) = 0

() 2010/07/3, 15 @ Gakushuin 32 / 67

(value function)

V0(k0) =maxct ;kt+1

1Xt=0

tu(ct) (13)

s.t. kt+1 kt = f (kt) ct ktk0 : given

V0(k0) (indirect utility function)

() 2010/07/3, 15 @ Gakushuin 33 / 67

(13)

V0(k0) =maxc0;k1

nu(c0) + max

c;k+1

1X=1

1u(c)o

(14)

s.t. kt+1 kt = f (kt) ct kt; k0 : givent = 1

V0(k0) =maxc0;k1

nu(c0) + V1(k1)

o(15)

s.t. k1 k0 = f (k0) c0 k0; k0 : given:c0V0(k0)

V0(k0) =maxk1

nu

f (k0) k1 + (1 )k0+ V1(k1)

o(16)

() 2010/07/3, 15 @ Gakushuin 34 / 67

t

Vt(kt) = maxc;k+1

1X=t

tu(c) (17)

s.t. k+1 k = f (k) c k; kt : given

Vt(kt) =maxct ;kt+1

2666664u(ct) + maxc;k+1

8>>>: 1X=t+1

t1u(c)9>>=>>;3777775

s.t. kt+1 kt = f (kt) ct kt; kt : given, ctVt(kt) (Bellman equation)

Vt(kt) =maxkt+1

nu

f (kt) kt+1 + (1 )kt+ Vt+1(kt+1)

o(18)

() 2010/07/3, 15 @ Gakushuin 35 / 67

(17) tt (time-invariant value function)

V() = Vt(): V()

V(kt) =maxkt+1

nu

f (kt) kt+1 + (1 )kt+ V(kt+1)

o:

() 2010/07/3, 15 @ Gakushuin 36 / 67

II.2 (value function) (policy function)

value function policy function T value function V()

V(kt) =maxkt+1

nu

f (kt) kt+1 + (1 )kt+ V(kt+1)

o: (19)

V()

() 2010/07/3, 15 @ Gakushuin 37 / 67

(i) Value function (functional analysis)value function V() value function iteration

.

.

.

1 (19)

.

.

.

2 kt+1 = kkt = k V0 (iteration)j ! 1 V j()

V j+1(k) =maxk

nu

f (k) k + (1 )k+ V j(k)

os.t. k : given:

() 2010/07/3, 15 @ Gakushuin 38 / 67

(ii) Policy function(19) kt+1

u0(ct) + V 0(kt+1) = 0, u0(ct) = V 0(kt+1) (20), u0(ct) = V 0( f (kt) ct + (1 )kt) (21)

u(), f (), V() policy function (time invariant policyfunction)

kt+1 = (kt) (22) t kt tkt t kt ct

ct = f (kt) (kt) + (1 )kt (23) () 2010/07/3, 15 @ Gakushuin 39 / 67

(iii) Euler equation

policy function (23) ct = f (kt) (kt) + (1 )kt

V(kt) = u

f (kt) (kt) + (1 )kt+ V(kt+1)

kt

V 0(kt) = u0(ct)hf 0(kt) 0(kt) + (1 )

i+ V 0(kt+1)0(kt)

() V 0(kt) = u0(ct+1)hf 0(kt) + (1 )

i 0(kt)

hu0(ct) V 0(kt+1)

i

() 2010/07/3, 15 @ Gakushuin 40 / 67

(20)( (envelope theorem))

V 0(kt) = u0(ct)hf 0(kt) + (1 )

i(24)

(20)u0(ct) = V 0(kt+1)

() u0(ct) = u0(ct+1)nf 0(kt+1) + (1 )

o

() 2010/07/3, 15 @ Gakushuin 41 / 67

II.3

value function(i) Value function iteration(ii) Howards improvement algorithm(iii) Guess and verify

() 2010/07/3, 15 @ Gakushuin 42 / 67

(i) Value function iteration

.

. . 1 kt+1 = kkt = k V0

.

.

.

2 (iteration)

V j+1(k) =maxk

nu

f (k) k + (1 )k+ V j(k)

os.t. k : given:

.

.

.

3 j = j + 1

.

.

.

4 V j value function iterationiterating on the Bellmanequation

() 2010/07/3, 15 @ Gakushuin 43 / 67

(ii) Howards improvement algorithm

() 2010/07/3, 15 @ Gakushuin 44 / 67

(iii) Guess and Verify u(c) = log(c) f (k) = Ak 0 < < 1; A > 0 = 1 (19)

V(kt) =maxkt+1

nlog

Akt kt+1

+ V(kt+1)

o(25)

V() V()

V(kt) = E + F log(kt) (26) (guess)E F(undetermined coefficients)

() 2010/07/3, 15 @ Gakushuin 45 / 67

guess policy function(25)

1ct+ V 0(kt+1) = 0

, 1ct= V 0(kt+1) (27)

, 1ct+ F

1kt+1

= 0

, 1ct= F

1Akt ct

, ct = Akt ctF

, 1 +

1F

!ct =

AktF

, ct = Akt

1 + F:

() 2010/07/3, 15 @ Gakushuin 46 / 67

kt+1 policy functionkt+1 = Ak ct

, kt+1 = Ak Akt

1 + F

, kt+1 = Ak Akt

1 + F

, kt+1 = F1 + F Akt

(24)V 0(kt) = V 0(kt+1)Ak1t

, V 0(kt) = 1ctAk1t ((27))

, V 0(kt) = Ak1t

Akt1+F

(ct policy function)

, V 0(kt) = (1 + F)k1t (28) () 2010/07/3, 15 @ Gakushuin 47 / 67

(26) kV 0(kt) = Fk1t (29)

(28) (29)F = (1 + F)

, F = 1

value function policyfunction

V(kt) = E + 1 log(kt) (30)ct = (1 )Akt (31)kt+1 = Akt (32)

() 2010/07/3, 15 @ Gakushuin 48 / 67

kt

t kt kt+1 policy functionkt+1 = Akt kt+1 = Akt

log kt+1 = log(A) + log kt (33)jj < 1t ! 1kt

k = Ak

, k = (A) 11

() 2010/07/3, 15 @ Gakushuin 49 / 67

III

() 2010/07/3, 15 @ Gakushuin 50 / 67

(functional analysis)

() 2010/07/3, 15 @ Gakushuin 51 / 67

0 < < 1r(; )(control variables)futg1t=0

maxfutg1t=0

1Xt=0

tr(xt; ut) (34)

s.t. xt+1 = g(xt; ut); x0 : given.

r(; )xt+1 = g(xt; ut)xt (transition equation) f(xt+1; xt) : xt g(xt; ut)g

() 2010/07/3, 15 @ Gakushuin 52 / 67

(time-invariant) (policy function)hh (state variables)xt(control variables)utmapping

ut = h(xt) (35)xt+1 = g(xt; ut) (36)x0 : given,

futg1t=0recursive

() 2010/07/3, 15 @ Gakushuin 53 / 67

(value function)

V0(x0) = maxfutg1t=01X

t=0tr(xt; ut) (37)


value function

V0(x0) = maxu0

nr(x0; u0) + maxfug1=1

1X=1

1r(x; u)o


(37)

V0(x0) = maxu0

nr(x0; u0) + V1(x1)

os.t. xt+1 = g(xt; ut); x0 : given.

() 2010/07/3, 15 @ Gakushuin 54 / 67

t

Vt(xt) = maxut

nr(xt; ut) + Vt+1(xt+1)

os.t. xt+1 = g(xt; ut); xt : given.

(37) tV0() = V() (time-invariant) x = xt+1x = xtu = utV()

V(x) = maxu

nr(x; u) + V(x)

o(38)

s.t. x = g(x; u); x : given,

() 2010/07/3, 15 @ Gakushuin 55 / 67

value function V()V()policy function

maxu

nr(x; u) + V(x)

os.t. x = g(x; u)

x : given.

() 2010/07/3, 15 @ Gakushuin 56 / 67

value function V()policy functionh()

V(x) = maxu

nr(x; u) + V(x)

o(39)

s.t. x = g(x; u); x : given,

policy function h(x) x = g(x; u)x

V(x) = rx; h(x) + Vgx; h(x): (40) V()h() (functionalequation)

() 2010/07/3, 15 @ Gakushuin 57 / 67

1. (39)2. V0(iteration) j ! 1 V j()

V j+1(x) =maxx

nr(x; u) + V j(x)

os.t. x : given:

3. (39)@r(x; u)

@u+ V 0

g(x; u)@g(x; u)

@u= 0 (41)

(time-invariant) policyfunction h()

() 2010/07/3, 15 @ Gakushuin 58 / 67

4. value function(40)

V 0(x) = @rx; h(x)@x

+ @g

x; h(x)@x

V 0gx; h(x): (42)

Benveniste and Scheinkman x = g(u)@g

@x= 0

V 0(x) = @rx; h(x)@x

: (43)

() 2010/07/3, 15 @ Gakushuin 59 / 67

5. Eulerx = g(u)(41)

@r(x; u)@u

+ V 0(x)@g(u)@u

= 0 (44)

(43)@r(x; u)

@u+

@rx; h(x)@x

@g(u)@u

= 0: (45)

Euler

() 2010/07/3, 15 @ Gakushuin 60 / 67

@r(xt; ut)@ut

+ @rxt+1; h(xt+1)@xt+1

@g(ut)@ut

= 0

@r(xt; ut)@ut

+ @rxt+1; ut+1

@xt+1

g0(ut) = 0

xt = ktut = kt+1r(xt; ut) = u

f (kt) kt+1 + (1 )kt

g(ut) = kt+1

u0(ct) + u0(ct+1) f f 0(kt+1) + (1 )g = 0

() 2010/07/3, 15 @ Gakushuin 61 / 67

(i) (42) (Benveniste and Scheinkman(1979))(1/2)

(40)

V 0(x) = @rx; h(x)@x

+@rx; h(x)@u

@h(x)@x

+ V 0gx; h(x)(@gx; h(x)

@x+@g

x; h(x)@u

@h(x)@x

)

V 0(x) = @rx; h(x)@x

+ V 0gx; h(x)@gx; h(x)

@x

+

"@rx; h(x)@u


@u

#@h(x)@x

:

() 2010/07/3, 15 @ Gakushuin 62 / 67

(42) (2/2)

(41) (envelopetheorem) (42)

V 0(x) = @rx; h(x)@x


@x:

() 2010/07/3, 15 @ Gakushuin 63 / 67

III.2 Value function

value function policy function 3

(i) Value function iteration(ii) Howards improvement algorithm(iii) Guess and verify

() 2010/07/3, 15 @ Gakushuin 64 / 67

(i) Value function iteration

.

..

1 V0

.

.

.

2 (iteration)

V j+1(x) =maxu

nr(u; x) + V j(x)

os.t. x = g(x; u)

x : given,

.

.

.

3 j = j + 1

.

.

.

4 V j value function iterationiterating on the Bellmanequation

() 2010/07/3, 15 @ Gakushuin 65 / 67

(ii) Howards improvement algorithmHowards improvement algorithm

.

. . 1 policy function u = h0(x)value

Vh j(x) =1X

t=0tr

xt; h j(xt); s.t. xt+1 = gxt; h j(xt);x0 : given.

.

.

.

2 policy functionu = h j+1(x)

maxu

nr(x; u) + Vh j

g(x; u)o

.

.

.

3 j = j + 1

.

.

.

4 h j()step 1, 2, 3

() 2010/07/3, 15 @ Gakushuin 66 / 67

(iii) Guess and verify

() 2010/07/3, 15 @ Gakushuin 67 / 67

(dynamic programming):(backward induction method)I.2 (fundamental recurrence relation)

(dynamic programming)(value function)(policy function)

III.1

Documents

Slide Lecture4