POLYNOMIAL OPTIMIZATION WITH SUMS OF QUARES …

POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARESINTERPOLANTS

Sercan Yıldı[email protected]

in collaboration with

David Papp (NCSU)

OPT Transition WorkshopMay 02, 2017

OUTLINE

• Polynomial optimization and sums of squares– Sums-of-squares hierarchy– Semidefinite representation of sums-of-squares constraints

• Interior-point methods and conic optimization

• Sums-of-squares optimization with interior-point methods– Computational complexity– Preliminary results

POLYNOMIAL OPTIMIZATION

• Polynomial optimization problem:

minz∈Rn

f (z)

s.t. gi (z) ≥ 0 for i = 1, . . . ,m

where f , g1, . . . , gm are n-variate polynomials

Example: minz∈R2

z31 + 3z2

1 z2 − 6z1z22 + 2z3

2

s.t. z21 + z2

2 ≤ 1

• Some applications:– Shape-constrained

estimation– Design of experiments– Control theory– Combinatorial optimization– Computational geometry– Optimal power flow

Source: https://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/Max-cut.svg/200px-Max-cut.

svg.png

Source:http://www.cds.caltech.edu/

˜murray/amwiki/images/thumb/1/19/Doscpp.png/270px-Doscpp.png

Source: https://upload.wikimedia.org/wikipedia/commons/thumb/d/

d2/Kissing-2d.svg/

190px-Kissing-2d.svg.png

https://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/Max-cut.svg/200px-Max-cut.svg.png




http://www.cds.caltech.edu/~murray/amwiki/images/thumb/1/19/Doscpp.png/270px-Doscpp.png



https://upload.wikimedia.org/wikipedia/commons/thumb/d/d2/Kissing-2d.svg/190px-Kissing-2d.svg.png




SUMS-OF-SQUARES RELAXATIONS

• Let f be an n-variate degree-2d polynomial.

• Unconstrained polynomial optimization:

minz∈Rn

f (z)NP-hard already

for d = 2!

• Equivalent “dual” formulation:

maxy∈R

y

s.t. f (z)− y ∈ P

where P = {f : f (z) ≥ 0 ∀z ∈ Rn}

• Sums-of-squares cone:

SOS = {f : f =∑N

j=1f 2j for some degree-d polynomials fj}

f ∈ SOS ⇒ f ∈ P

• SOS relaxation:maxy∈R

y

s.t. f (z)− y ∈ SOS

SUMS-OF-SQUARES RELAXATIONS

• Let f , g1, . . . , gm be n-variate polynomials.

• Constrained polynomial optimization:

minz∈Rn

f (z)

s.t. gi (z) ≥ 0 for i = 1, . . . ,m

• Feasible set: G = {z ∈ Rn : gi (z) ≥ 0 for i = 1, . . . ,m}.• Dual formulation: max

y∈Ry

s.t. f (z)− y ∈ PG

where PG = {f : f (z) ≥ 0 ∀z ∈ G}

• “Weighted” SOS cone of order r :

SOSG,r = {f : f =∑m

i=0gisi , si ∈ SOS, deg(gisi ) ≤ r} where g0 ≡ 1

f ∈ SOSG,r ⇒ f ∈ PG

• SOS relaxation of order r :

maxy∈R

y

s.t. f (z)− y ∈ SOSG,r

WHY DO WE LIKE SOS?

• Polynomial optimization is NP-hard.

• Increasing r produces a hierarchy of SOS relaxations for polynomialoptimization problems.

• Under mild assumptions:– The lower bounds from SOS relaxations converge to the true optimal

value as r ↑ ∞ (follows from Putinar’s Positivstellensatz).

• SOS relaxations can be represented as semidefinite programs of sizeO(mL2) where L =

(n+r/2n

)(follows from the results of Shor, Nesterov,

Parrilo, Lasserre).

• There are efficient and stable numerical methods for solving SDPs.

SOS CONSTRAINTS ARE SEMIDEFINITE REPRESENTABLE

• Consider the cone of degree-2d SOS polynomials:

SOS2d = {f ∈ R[z]2d : f =∑N

j=1f 2j for some fj ∈ R[z]d}

Theorem (Nesterov, 2000)The univariate polynomial f (z) =

∑2du=0 fuzu is SOS iff there exists a

(d + 1)× (d + 1) PSD matrix S such that

fu =∑

k+`=uSk` ∀u = 0, . . . , 2d .


• More generally, in the n-variate case:– Let L := dim(R[z]d ) =

(n+dn

)and U := dim(R[z]2d ) =

(n+2dn

).

– Fix bases {p`}L`=1 and {qu}U

u=1 for the linear spaces R[z]d and R[z]2d .

Theorem (Nesterov, 2000)The polynomial f (z) =

∑Uu=1 fuqu(z) is SOS iff there exists a L× L PSD

matrix S such thatf = Λ∗(S)

where Λ : RU → SL is the linear map satisfying Λ([qu]Uu=1) = [p`]L

`=1[p`]L`=1>

.

• If univariate polynomials are represented in the monomial basis:

Λ(x) =

x0 x1 x2 . . . xd

x1 x2

x2... x2d−1

xd x2d−1 x2d

Λ∗(S) =[∑

k+`=uSk`]2d

u=0.

• These results easily extend to the weighted case.


• To keep things simple, we focus on optimization over a single SOS cone.

• SOS problem:

maxy∈Rk ,S∈SL

b>y

s.t. A>y + Λ∗(S) = c

S � 0

A>y + s = cs ∈ SOS := {s ∈ RU : s = Λ∗(S),S � 0}

• Moment problem:

minx∈RU

c>x

s.t. Ax = b

Λ(x) � 0 x ∈ SOS∗ := {x ∈ RU : Λ(x) � 0}

Disadvantages of existing approaches

• Problem: SDP representation roughly squares the number of variables.

Solution: We solve the SOS and moment problems in their originalspace.

• Problem: Standard basis choices lead to ill-conditioning.

Solution: We use orthogonal polynomials and interpolation bases.

INTERIOR-POINT METHODS FOR CONIC PROGRAMMING

• Primal-dual pair of conic programs:

min c>x

s.t. Ax = b

x ∈ K

max b>y

s.t. A>y + s = c

s ∈ K ∗

• K : closed, convex, pointed cone with nonempty interior

Examples: K = Rn+, Sn

+,SOS,P

• Interior-point methods make use of self-concordant barriers (SCBs).

Examples:– K = Rn

+: F (x) = − log x

– K = Sn+: F (X ) = − log det X


• Given SCBs F and F∗, IPMs converge to the optimal solution by solving asequence of equality-constrained barrier problems for µ ↓ 0:

min c>x + µF (x)

s.t. Ax = b

max b>y − µF∗(s)

s.t. A>y + s = c• Primal IPMs:

– solve only the primal barrier problem,– are not considered to be practical.

• Primal-dual IPMs:– solve both the primal and dual barrier problems simultaneously,– are preferred in practice.


• In principle, any closed convex cone admits a SCB (Nesterov andNemirovski, 1994).

• However, the success of the IPM approach depends on the availability of aSCB whose gradient and Hessian can be computed efficiently.– For the cone P, there are complexity-based reasons for suspecting that

there are no computationally tractable SCBs.– For the cone SOS, there is evidence suggesting that there may not exist

any tractable SCBs.– On the other hand, the cone SOS∗ inherits a tractable SCB from the PSD

cone.

INTERIOR-POINT METHOD OF SKAJAA AND YE

• Until recently:– The practical success of primal-dual IPMs had been limited to

optimization over symmetric cones: LP, SOCP, SDP.– Existing primal-dual IPMs for non-symmetric conic programs required

both the primal and dual SCBs (e.g., Nesterov, Todd, and Ye, 1999;Nesterov, 2012).

• Skajaa and Ye (2015) proposed a primal-dual IPM for non-symmetric conicprograms which– requires a SCB for only the primal cone,– achieves the best-known iteration complexity.

SOLVING SOS PROGRAMS WITH INTERIOR-POINT METHODS

• Using Skajaa and Ye’s IPM with the SCB for SOS∗, the SOS and momentproblems can be solved without recourse to SDP.

• For any ε ∈ (0, 1), the algorithm finds a primal-dual solution that has εtimes the duality gap of an initial solution in O(

√L log(1/ε)) iterations

where L = dim(R[x ]d ).

• Each iteration of the IPM requires– the computation of the Hessian of the SCB for SOS∗,– the solution of a Newton system.

• Solving the Newton system requires O(U3) operations whereU = dim(R[z]2d ).


• The choice of bases {p`}L`=1 and {qu}U

u=1 for R[z]d and R[z]2d has asignificant effect on how efficiently the Newton system can be compiled.– In general, computing the Hessian requires O(L2U2) operations.– If both bases are chosen to be monomial bases, the Hessian can be

computed faster but requires specialized methods such as FFT and the“inversion” of Hankel-like matrices.

– Following Lofberg and Parrilo (2004), we choose• {qu}U

u=1: Lagrange interpolating polynomials,• {p`}L

`=1: orthogonal polynomials.– With this choice, the Hessian can be computed in O(LU2) operations.


• Putting everything together:– The algorithm runs in O(

√L log(1/ε)) iterations.

– At each iteration:• Computing the Hessian requires O(LU2) operations,• Solving the Newton system requires O(U3) operations.

• Overall complexity: O(U3√

L log(1/ε)) operations.

• This matches the best-known complexity bounds for LP!

• In contrast:– Solving the SDP formulation with a primal-dual IPM requires

O(L6.5 log(1/ε)) operations.– For fixed n:

L2

U= Θ(dn).


• The conditioning of the moment problem is directly related to theconditioning of the interpolation problem with {pk p`}L

k,`=1.

• Good interpolation nodes are understood well only in few low-dimensionaldomains:– Chebyshev points in [−1, 1],– Padua points in [−1, 1]2 (Caliari et al., 2005).

• For problems in higher dimensions, we follow a heuristic approach tochoose interpolation nodes.

A TOY EXAMPLE

• Minimizing the Rosenbrock function on the square:

minz∈R2

(z1 − 1)2 + 100(z2 − z21 )2

s.t. z ∈ [−1, 1]2

• We can obtain a lower bound on the optimal value of this problem from themoment relaxation of order r .

• For r = 60:– The moment relaxation has 1891 variables.– The SDP representation requires one 496× 496 and two 494× 494

matrix inequalities.– Sedumi quits with numerical errors after 948 seconds. Primal

infeasibility: 1.2× 10−4.– Our implementation solves the moment relaxation in 482 seconds.

FINAL REMARKS

• We have shown that SOS programs can be solved using a primal-dual IPMin O(U3

√L log(1/ε)) operations where L =

(n+dn

)and U =

(n+2dn

).

• This improves upon the standard SDP-based approach which requiresO(L6.5 log(1/ε)) operations.

• In progress: Numerical stability.– For multivariate problems, the conditioning of the problem formulation

becomes important.– How do we choose interpolation nodes for better-conditioned problems?

Thank you!

REFERENCES

M. Caliari, S. De Marchi, and M. Vianello. Bivariate polynomial interpolationon the square at new nodal sets. Applied Mathematics and Computation,165:261-274, 2005.

J. Lofberg and P. A. Parrilo. From coefficients to samples: a new approachto sos optimization. In 43rd IEEE Conference on Decision and Control,volume 3, pages 3154-3159. IEEE, 2004.

Yu. Nesterov. Squared functional systems and optimization problems. In H.Frenk, K. Roos, T. Terlaky, and S. Zhang, editors, High PerformanceOptimization, volume 33 of Applied Optimization, chapter 17, pages405-440. Springer, Boston, MA, 2000.

Yu. Nesterov. Towards non-symmetric conic optimization. OptimizationMethods & Software, 27(4-5):893-917, 2012.

REFERENCES

Yu. Nesterov and A. Nemirovski. Interior-Point Polynomial Algorithms inConvex Programming, volume 13 of SIAM Studies in Applied Mathematics.Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA,1994.

Yu. Nesterov, M. J. Todd, and Y. Ye. Infeasible-start primal-dual methodsand infeasibility detectors for nonlinear programming problems.Mathematical Programming, 84:227-267, 1999.

A. Skajaa and Y. Ye. A homogeneous interior-point algorithm fornonsymmetric convex conic optimization. Mathematical Programming Ser.A, 150:391-422, 2015.

Documents

POLYNOMIAL OPTIMIZATION WITH SUMS OF QUARES …