The turnpike property - FAU...The turnpike property Emmanuel Trelat´ 1 1Sorbonne Universite (Paris 6), Labo. J.-L. Lions´ Works with Gontran Lance, Can Zhang, Enrique Zuazua FAU,

The turnpike property

Emmanuel Trelat1

1Sorbonne Universite (Paris 6), Labo. J.-L. Lions

Works with Gontran Lance, Can Zhang, Enrique Zuazua

FAU, Kolloquium, 2020 January 7

Turnpike property

The solution of an optimal control problem in large time should spend most of itstime near a steady-state.

In infinite horizon the solution should converge to that steady-state.

Historically: discovered in econometry (Von Neumann points).

The first turnpike result was discovered in 1958 by Dorfman,Samuelson and Solow, in view of deriving efficient programsof capital accumulation, in the context of a Von Neumannmodel in which labor is treated as an intermediate product.

Paul Samuelson (1915–2009)

Nobel Prize in EconomicScience, 1970

Turnpike property



final

turnpike (highway)

initial

finalinitial

Excerpt from: Dorfman - Samuelson - Solow (1958)

It is exactly like a turnpike paralleled by a network of minor roads. There is a fastest routebetween any two points; and if the origin and destination are close together and far from theturnpike, the best route may not touch the turnpike. But if origin and destination are farenough apart, it will always pay to get on to the turnpike and cover distance at the best rateof travel, even if this means adding a little mileage at either end.

Turnpike property



- Turnpike theorems have been derived in the 60’s for discrete-time optimal control prob-lems arising in econometry (Mac Kenzie, 1963).

- Continous versions by Haurie for particular dynamics (economic growth models). Seealso Carlson Haurie Leizarowitz 1991, Zaslavski 2000, Faulwasser Bonvin 2015, Grune2018, Lou Wang 2018.

- More recently, in biology: Rapaport 2005, Coron Gabriel Shang 2014; human locomo-tion: Chitour Jean Mason 2012; MPC: Grune et al. 2012–2019.

- Linear heat and wave equations: Porretta Zuazua 2013.

- Rockafellar 1973, Samuelson 1972: saddle point feature of the extremal equations ofoptimal control.

- Different point of view by Anderson Kokotovic (1987), Wilde Kokotovic (1972):exponential dichotomy property→ hyperbolicity phenomenon.

General nonlinear optimal control problem

f : X × U → X dynamics (in finite or infinite dimension)

R : X × X → Y terminal conditions (example: point-to-point, point-to-free, periodic, ...)

f 0 : X × U → IR instantaneous cost

Dynamical optimal control problem (OCP)T

For T > 0 fixed, find uT (·) ∈ L∞(0,T ; U) such that

x(t) = f (x(t), u(t))

R(x(0), x(T )) = 0

min∫ T

0f 0(x(t), u(t)) dt

x(·): state u(·): control

Intuitive idea

min1T

∫ T

0f 0(x(t), u(t)) dt

x(t) = f (x(t), u(t)), t ∈ [0,T ]

s = t/T−→ε = 1/T

min∫ 1

0f 0(x(s), u(s)) ds

εx ′(s) = f (x(s), u(s)), s ∈ [0, 1]

We expect that, as ε→ 0, there is some convergence to the static optimization problem

min f 0(x , u) subject to f (x , u) = 0


x(t) = f (x(t), u(t))

R(x(0), x(T )) = 0

min∫ T

0f 0(x(t), u(t)) dt

Static optimal control problem

min(x,u)∈X×Uf (x,u)=0

f 0(x , u)

Optimal (assumed) solution (xT (·), uT (·)) Optimal (assumed) solution (x , u)


x(t) = f (x(t), u(t))

R(x(0), x(T )) = 0

min∫ T

0f 0(x(t), u(t)) dt



f 0(x , u)

Optimal (assumed) solution (xT (·), uT (·)) Optimal (assumed) solution (x , u)

First-order optimality conditions:

Pontryagin maximum principle

(generic...) assumption:no abnormal⇒ λ0

T = −1

Lagrange multiplier rule

(generic...) assumption:no abnormal⇒ λ0

T = −1(Mangasarian-Fromowitz)


x(t) = f (x(t), u(t))

R(x(0), x(T )) = 0

min∫ T

0f 0(x(t), u(t)) dt



f 0(x , u)

λT (·): adjoint

xT (t) =∂H∂λ

(xT (t), λT (t),−1, uT (t))

λT (t) = −∂H∂x

(xT (t), λT (t),−1, uT (t))

∂H∂u

(xT (t), λT (t),−1, uT (t)) = 0

+ transversality conditions(−λT (0)λT (T )

)=

k∑i=1

γi∇Ri (xT (0), xT (T ))

λ: adjoint (Lagrange multiplier)

∂H∂λ

(x , λ,−1, u) = 0

−∂H∂x

(x , λ,−1, u) = 0

∂H∂u

(x , λ,−1, u) = 0

H(x , λ, λ0, u) = 〈λ, f (x , u)〉+ λ0f 0(x , u)

(x , λ, u): equilibrium point of the extremal equations

Exponential turnpike

Theorem

Under admissibility, exponential stabilizability and detectability assumptions:

‖xT (t)− x‖+∥∥λT (t)− λ

∥∥+ ‖uT (t)− u‖ 6 C1

(e−νt + e−ν(T−t)

)∀t ∈ [0,T ]

(Finite dim.: Trelat Zuazua, JDE 2015. Infinite dim.: Trelat Zhang Zuazua, SICON 2018)

Moreover:ν = −max

Re(µ) | µ ∈ Spec(A− BH−1

uu B∗E−)> 0

where

E−A + A∗E− − E−BH−1uu B∗E− −W = 0 minimal solution of Riccati

In some sense, this result shows that:

1) (dynamic) control 2) T → +∞ ⇐⇒ 1) T → +∞ 2) (static) control


Theorem


‖xT (t)− x‖+∥∥λT (t)− λ

∥∥+ ‖uT (t)− u‖ 6 C1


)∀t ∈ [0,T ]


Consequence: in large time T , the optimal extremal solution (xT (·), λT (·), uT (·)) of(OCP)T approximately consists of 3 pieces:

1 short-time: (xT (0), λT (0), uT (0))→ (x , λ, u) (transient arc, on [0, ε])

2 long-time, stationary: (x , λ, u) (on [ε,T − ε])

3 short-time: (x , λ, u)→ (xT (T ), λT (T ), uT (T )) (transient arc, on [T − ε,T ])


Theorem


‖xT (t)− x‖+∥∥λT (t)− λ

∥∥+ ‖uT (t)− u‖ 6 C1


)∀t ∈ [0,T ]


Extension to periodic turnpike:Equilibrium point replaced with a periodic trajectory (itself optimal for a periodic optimalcontrol problem). (needs a “periodic Riccati theory”)

Main idea of the proof: hyperbolicity of extremals

Linearizing the first-order optimality system (Pontryagin extremals in the cotangentbundle) at the equilibrium:

z(t) =

(xT (t)− xλT (t)− λ

)⇒ z(t) =

first orderM z(t) with M =

(A BU−1B∗Q −A∗

)

Key lemma

M is Hamiltonian, hence:

µ ∈ Spec(M) ⇒ −µ, µ,−µ ∈ Spec(M)

Moreover, under controllability assumptions:

Re(Spec(M)) 6= 0 (no pure imaginary eigenvalue)

⇒ M is hyperbolic.

Main idea of the proof: hyperbolicity of extremals

Linearizing the first-order optimality system (Pontryagin extremals in the cotangentbundle) at the equilibrium:

z(t) =

(xT (t)− xλT (t)− λ

)⇒ z(t) =

first orderM z(t) with M =

(A BU−1B∗Q −A∗

)

In appropriate coordinates:

v(t) = P−v(t) Re(Spec(P−)) < 0

w(t) = P+w(t) Re(Spec(P+)) > 0

hence

‖v(t)‖ 6 ‖v(0)‖e−νt

‖w(t)‖ 6 ‖w(T )‖e−ν(T−t)

(click on the figure to see time evolution)

Consequences for the numerical computations

Direct methods (full discretization): initialization with the solution of the static problem

⇒ successful convergence

Indirect method (shooting):

solve z(t) = F (z(t)), G(z(0), z(T )) = 0

Usual implementation: z(0) unknown, tuned such that G(z(0), z(T )) = 0.

Here we propose the following variant:

z(0) ←− z(T/2) unknown −→ z(T )

backward forwardintegration integration

↓tuned s.t.

G(z(0), z(T )) = 0

(initialization with the solution of the static problem)

Example

x1(t) = x2(t), x1(0) = 1

x2(t) = 1− x1(t) + x2(t)3 + u(t), x2(0) = 1

min12

∫ T

0

((x1(t)− 1)2 + (x2(t)− 1)2 + (u(t)− 2)2

)dt

Optimal solution of the static problem:

x2 = 0, 1− x1 + x32 + u = 0

minx2=0

1−x1+x32 +u=0

((x1 − 1)2 + (x2 − 1)2 + (u − 2)2

)

whencex = (2, 0) , u = 1, λ = (−1,−1)

Example

x1(t) = x2(t), x1(0) = 1

x2(t) = 1− x1(t) + x2(t)3 + u(t), x2(0) = 1

min12

∫ T

0

((x1(t)− 1)2 + (x2(t)− 1)2 + (u(t)− 2)2

)dt

Oscillation of (x1(·), x2(·)) around

the steady-state (2, 0)

Example

x1(t) = x2(t), x1(0) = 1

x2(t) = 1− x1(t) + x2(t)3 + u(t), x2(0) = 1

min12

∫ T

0

((x1(t)− 1)2 + (x2(t)− 1)2 + (u(t)− 2)2

)dt

Impossible to make converge the usualshooting method if T > 3(explosive term + too high sensitivity)

Easy convergence with the variant, ∀T

Further comments, extensions and open issues

More applications:aerospace (bi-impulse orbit transfers), motion planning and robotics, optimization ofchemotherapies (Pouchol Clairambault Lorz Trelat, JMPA 2018), model of a runner (Aftalion Trelat, ongoing)

Competition between turnpikes (see also Rapaport Cartigny, JOTA 2005)

Periodic turnpike (with a “periodic Riccati theory”) (Trelat Zhang, MCSS 2018)

Extensions to infinite dimension(Gugat Trelat Zuazua, SCL 2016; Trelat Zhang Zuazua, SICON 2018; Grune Schaller Schiela 2019)

“Measure” turnpike results for dissipative systems involving state and/or control constraints(Bonvin Faulwasser 2016; Grune 2018; Trelat Zhang MCSS 2018)

Interaction with discretizations (also discrete-time turnpike) (Grune; Grune Guglielmi 2017)

Shape turnpike: turnpike in optimal design, adiabatic theory (Trelat Zhang Zuazua, PAFA 2018;

ongoing PhD thesis of G. Lance)

Shape turnpike (PhD thesis of Gontran Lance)

Dynamical shape optimization problem:

Ω ⊂ IRn bounded domain

y0, yd ∈ L2(Ω)

0 < L < 1

T > 0

Dirichlet heat equation on Ω with source term:

∂t y = 4y + χω(t) y|∂Ω = 0

y(0) = y0

minω(·)

∫ T

0‖y(t)− yd‖2

L2(Ω)dt

over all ω(t) ⊂ Ω s.t. |ω(t)| = L|Ω|

χω(x) =

1 if x ∈ ω0 otherwise

More generally: parabolic PDE satisfying the maximum principle.

Shape turnpikeTheorem (Lance Trelat Zuazua, 2019):If either yd is convex or yd 6 0 or yd > y1 (with4y1 + 1 = 0, y1

|∂Ω= 0) then:

Dynamical shape optimization problem

∂t y = 4y + χω(t), y|∂Ω = 0

y(0) = y0

minω(·)

∫ T

0‖y(t)− yd‖2

L2(Ω)dt

over all ω(t) ⊂ Ω s.t. |ω(t)| = L|Ω|

has a unique solution ω(t)

Static shape optimization problem

4y + χω = 0, y|∂Ω = 0

minω‖y − yd‖2

L2(Ω)

over all ω ⊂ Ω s.t. |ω| = L|Ω|

has a unique solution ω

and

∃M > 0 | ∀T > 0∫ T

0

(‖y(t)− y‖2

L2(Ω)+ ‖λ(t)− λ‖2

L2(Ω)+ ‖χω(t) − χω‖L1(Ω)

)dt 6 M

⇒1T

∫ T

0y(t) dt L2

−→T→+∞

y ,1T

∫ T

0λ(t) dt L2

−→T→+∞

λ,1T

∫ T

0χω(t) dt L1

−→T→+∞

χω

(integral turnpike) Open question: exponential turnpike

Shape turnpike

Proof:

Relaxation / convexification:

replace χω(t)(x) with a(t , x) s.t.

0 6 a(t , x) 6 1∫Ω a(t , x) dx = L|Ω|

Dynamical shape optimization problem

∂t y = 4y + a, y|∂Ω = 0

y(0) = y0

mina(·)

∫ T

0‖y(t)− yd‖2

L2(Ω)dt

a(t , x) ∈ [0, 1],

∫Ω

a(t , x) dx = L|Ω|

→ ∃! optimal a = a(t , x)

replace χω(x) with a(x) s.t.

0 6 a(x) 6 1∫Ω a(x) dx = L|Ω|

Static shape optimization problem

4y + a = 0, y|∂Ω = 0

mina‖y − yd‖2

L2(Ω)

a(x) ∈ [0, 1],

∫Ω

a(x) dx = L|Ω|

→ ∃! optimal a = a(x)

Shape turnpikeFirst-order optimality conditions:


∂t y = 4y + a, y|∂Ω = 0, y(0) = y0

∂tλ = −4λ+ y − yd , λ|∂Ω = 0, λ(T ) = 0∫Ωλ(t , x)a(t , x) dx = max

∫Ωλ(t , x)u(x) dx


4y + a = 0, y|∂Ω = 0

4λ = y − yd , λ|∂Ω = 0∫Ωλ(x)a(x) dx = max

∫Ωλ(x)u(x) dx

over the set of u s.t. 0 6 u(x) 6 1 and∫

Ω u(x) dx = L|Ω|

(bathtub principle)

⇒ a(x) =

1 if λ(x) > µ0 if λ(x) < µ

with µ s.t.∫

Ω a(x) dx = L|Ω|

and if λ(x) = µ on a subset of positivemeasure then a(x) = −4yd (x) there.



∂t y = 4y + a, y|∂Ω = 0, y(0) = y0




4y + a = 0, y|∂Ω = 0


∫Ωλ(x)u(x) dx


Ω u(x) dx = L|Ω|

(bathtub principle)

⇒ a(x) =

1 if λ(x) > µ0 if λ(x) < µ

with µ s.t.∫

Ω a(x) dx = L|Ω|




∂t y = 4y + a, y|∂Ω = 0, y(0) = y0




4y + a = 0, y|∂Ω = 0


∫Ωλ(x)u(x) dx


Ω u(x) dx = L|Ω|

⇒ a(t , x) =

1 if λ(t , x) > µ(t)0 if λ(t , x) < µ(t)

with µ(t) s.t.∫

Ω a(t , x) dx = L|Ω| for a.e. t

and if λ(t , x) = µ(t) on a subset of positivemeasure then a(t , x) = µ(t)−4yd (x) there.

⇒ a(x) =

1 if λ(x) > µ0 if λ(x) < µ

with µ s.t.∫

Ω a(x) dx = L|Ω|


Shape turnpikeSketch of proof:

Recall that

4y + a = 0 y|∂Ω = 0

4λ = y − yd λ|∂Ω = 0

Let y1 be the solution of4y1 + 1 = 0, y1|∂Ω

= 0.

Since −1 6 4y = −a 6 0, by the maximum principle: 0 6 y 6 y1.

If λ(x) = Cst on a subset E ⊂ Ω of positive measure then4λ = 0 on E and thusy = yd on E .

Contradiction if yd 6 0 or if yd > y1.

Similar arguments for the dynamical problem.

Then, use equivalence with the problem of minimizing

∫ T

0

(‖y(t)− yd‖2

L2(Ω)+ ‖a(t)‖2

L2(Ω)

)dt

which is strictly dissipative (in the sense of Willems 1972), and finally use thatstrict dissipativity implies integral turnpike (Trelat Zhang MCSS 2018).

Shape turnpike

Example in 1D: Ω = [0, 2], y0 ≡ 0, yd ≡ 0.1, L = 0.25, T = 5

(computation with IpOpt + FreeFEM++, G. Lance)

Shape turnpikeExample in 2D: square Ω = [−1, 1]2, y0 ≡ 0, yd ≡ 0.1, L = 0.25, T = 5

t 7→ ω(t)

t = 0 t = 0.5 1 6 t 6 4 t = 4.5 t = T = 5

(computation with IpOpt + FreeFEM++,

G. Lance)

Optimal static shape ω t 7→ ‖yT (t)−y‖L2 +‖pT (t)−p‖L2

+‖χωT (t) − χω‖L2

Shape turnpike

Square Ω = [−1, 1]2, y0 ≡ 0, yd ≡ 0.1, L = 0.25

T = 1

T = 2

Shape turnpike

Square Ω = [−1, 1]2, y0 ≡ 0, yd (x , y) = 120 (xy + 1), L = 3/16, T = 2

Shape turnpikeΩ stadium-shaped, y0 ≡ 0, L = 3/16, T = 2

yd ≡ 0.1

yd = 120 (xy +1)

Shape turnpike

Cube Ω = [0, 1]3, y0 ≡ 0, yd ≡ 0.1, L = 1/40, T = 1

optimal time-evolving shape optimal static shape

Shape turnpike

Relaxation phenomenon:Ω = [−1, 1]2, y0 = 0, yd = − 1

20 (x2 + y2 − 2), L = 0.25, T = 5

t = 0 t = 0.5 1 6 t 6 4 t = 4.5 t = T = 5

(computation with IpOpt + FreeFEM++, G. Lance)

Optimal static solution a

Perspectives

Shape turnpike for general semilinear PDE models

∂t y = Ay + f (y) + χω(t)

Shape-moving bottom pool wave generator (wavemaker)

Nersisyan Dutykh Zuazua 2014

Dalphin Barros 2019

KdV or Saint-Venant equations

Kinetic lift formulation (Perthame)

Competition between two global turnpikes

min∫ T

0

((x(t)− 1)2 + (u(t)− 3.47197)2

)dt

x(t) = −3x(t) + 3x(t)2 + u(t), x(0) = x0, x(T ) = xf or free

The choice of ud = 3.47197 is done so thatthe static optimal control problem

min(x,u) | −3x+3x3+u=0

((x − 1)2 + (u − 3.47197)2

)has two global minima:

x1 = −1.3473, x2 = 0.5939

Plot of x 7→ (x − 1)2 + (3x − 3x3 − 3.47197)2:

Competition between two global turnpikes

min∫ T

0

((x(t)− 1)2 + (u(t)− 3.47197)2

)dt


Global solutions of the static problem: x1 = −1.3473 and x2 = 0.5939.

x0 = −5, xf = −1, T = 10

→ Turnpike around x1 = −1.3473

x0 = 2, xf = 1, T = 10

→ Turnpike around x2 = 0.5939

Local versus global turnpike

min∫ T

0

((x(t)− 1)2 + (u(t)− 1)2

)dt


The static optimal control problem has

a unique globally optimal solution x = 0.7815

a locally optimal solution xloc = −1.1055

Plot of x 7→ (x − 1)2 + (3x − 3x3 − 1)2:


min∫ T

0

((x(t)− 1)2 + (u(t)− 1)2

)dt


Global solution x = 0.7815. Local solution xloc = −1.1055.

x0 = −2, xf = −1, T = 10Initialization with the constant trajectory xloc .

→ Local turnpike around xloc = −1.1055.Cost C = 4.19.

x0 = −2, xf = −1, T = 10Initialization with x .

→ Global turnpike around x = 0.7815.Cost C = 2.61.


min∫ T

0

((x(t)− 1)2 + (u(t)− 1)2

)dt


x0 = −2, xf = −1, T = 2 (quite small)Initialization with the constant trajectory xloc .

Cost C = 8.88.→ Globally optimal!

x0 = −2, xf = −1, T = 2Initialization with x .

Cost C = 9.64.→ Locally but not globally optimal!


min∫ T

0

((x(t)− 1)2 + (u(t)− 1)2

)dt


x0 = −2, xf = −1

T ∈ 2.5, 2.7, 2.9, 3.1, 3.3

Global optimal trajectory:Bifurcation at T = T0 ' 2.9.

T < T0 ⇒ turnpike around xloc

T > T0 ⇒ turnpike around x

→ in accordance with the globalturnpike result for T large enough

Periodic turnpike

X , U, V Hilbert spaces

A : D(A)→ X operator generating a C0 semigroup on X

B ∈ L(U,X) linear bounded control operator

C ∈ L(X ,V ) linear bounded observation operator

Q ∈ L(U,U) positive definite

Tracking trajectory: yd (·) ∈ C([0,+∞); X), ud (·) ∈ L2loc(0,+∞; U), Π-periodic:

yd (t + Π) = yd (t), ud (t + Π) = ud (t) ∀t

Optimal control problem (OCP)T

For T > 0 fixed, find uT (·) ∈ L2(0,T ; U) such that

y(t) = Ay(t) + Bu(t)

y(0) = y0

min12

∫ T

0

(‖C(y(t)− yd (t))‖2

V + 〈Q(u(t)− ud (t)), u(t)− ud (t)〉U)

dt

Periodic turnpike

We replace the steady-state optimal control problem with:

Periodic optimal control problem

min12

∫ Π

0

(‖C(y(t)− yd (t))‖2

V + 〈Q(u(t)− ud (t)), u(t)− ud (t)〉U)

dt

y(t) = Ay(t) + Bu(t)

y(0) = y(Π)

Theorem (Trelat Zhang Zuazua, SICON 2018)

If (A,B) is exponentially stabilizable and (A,C) is exponentially detectable, then:

The periodic optimal control problem has a unique solution (yΠ(·), uΠ(·)), which has aunique extremal lift (yΠ(·), uΠ(·), λΠ(·)) (of which we have explicit expressions).

There exist c > 0, ν > 0 such that ∀T > 0

∥∥∥yT (t)− yΠ(t)∥∥∥

X+∥∥∥uT (t)− uΠ(t)

∥∥∥U

+∥∥∥λT (t)− λΠ(t)

∥∥∥X6 c


)∀t ∈ [0,T ]

ν = exponential stability rate for a C0 semigroup resulting from the (operator) algebraic Riccati equation.

⇒ The optimal extremal is almost Π-periodic (except at the beginning and at the end).

Periodic turnpike

The proof uses in particular a kind of “periodic Riccati theory”:

Lemma

If (A,B) is exponentially stabilizable and (A,C) is exponentially detectable, then the uniquesolution of the periodic optimal control problem is

yΠ(t) = z(t)− Eq(t), λΠ(t) = −Pz(t) + (I + PE)q(t), uΠ(t) = ud (t) + Q−1B∗λΠ(t)

with:

P > 0 unique solution of the algebraic Riccati equation A∗P + PA−PBQ−1B∗P + C∗C = 0

E 6 0 unique solution of the Lyapunov equation 2(A− BQ−1B∗P)E − BQ−1B∗ = 0

z(t) = S(t)(I − S(Π))−1 ∫ Π

0 S(Π− τ)((I + EP)Bud (τ)− EC∗Cyd (τ)

)dτ

+∫ t

0 S(t − τ)((I + EP)Bud (τ)− EC∗Cyd (τ)

)dτ

q(t) = S(Π− t)∗(I − S(Π)∗)−1 ∫ Π

0 S(Π− τ)∗(− PBud (Π− τ) + C∗Cyd (Π− τ))

)dτ

+∫ Π−t

0 S(Π− t − τ)∗(− PBud (Π− τ) + C∗Cyd (Π− τ)

)dτ

Example of periodic turnpike

min12

∫ T

0

((x(t)− cos(2πt))2 + (y(t)− sin(2πt))2 + u(t)2

)dt T = 20, Π = 1

x(t) = y(t), y(t) = u(t), x(0) = 0.1, y(0) = 0

Documents

The turnpike property - FAU...The turnpike property Emmanuel Trelat´ 1 1Sorbonne Universite (Paris 6), Labo. J.-L. Lions´ Works with Gontran Lance, Can Zhang, Enrique Zuazua FAU,