Combining ADMM and the Augmented Lagrangian Methodfor Efficiently Handling Many Constraints

Joachim Giesen1 and Soren Laue∗1,21Friedrich-Schiller-Universitat Jena

2Data Assessment Solutions

Abstract

Many machine learning methods entail minimiz-ing a loss-function that is the sum of the lossesfor each data point. The form of the loss func-tion is exploited algorithmically, for instance instochastic gradient descent (SGD) and in the alter-nating direction method of multipliers (ADMM).However, there are also machine learning meth-ods where the entailed optimization problem fea-tures the data points not in the objective functionbut in the form of constraints, typically one con-straint per data point. Here, we address the prob-lem of solving convex optimization problems withmany convex constraints. Our approach is an ex-tension of ADMM. The straightforward implemen-tation of ADMM for solving constrained optimiza-tion problems in a distributed fashion solves con-strained subproblems on different compute nodesthat are aggregated until a consensus solution isreached. Hence, the straightforward approach hasthree nested loops: one for reaching consensus,one for the constraints, and one for the uncon-strained problems. Here, we show that solving thecostly constrained subproblems can be avoided. Inour approach, we combine the ability of ADMMto solve convex optimization problems in a dis-tributed setting with the ability of the augmentedLagrangian method to solve constrained optimiza-tion problems. Consequently, our algorithm onlyneeds two nested loops. We prove that it inher-its the convergence guarantees of both ADMM andthe augmented Lagrangian method. Experimentalresults corroborate our theoretical findings.

1 IntroductionOptimization problems with many constraints typically arisefrom large data sets. An illustrative example is the core vectormachine [Tsang et al., 2007], where the smallest enclosingball for a given set of data points has to be computed. Theobjective function here is the radius of the ball that needs to

∗Contact Author

be minimized, and every data point contributes a convex con-straint, namely the distance of the point from the center mustbe at most the radius.

The increasing availability of distributed hardware sug-gests to address such problems by distributing the constraintson different compute nodes. Unfortunately, to the best of ourknowledge, algorithmic schemes for distributing convex con-straints are only known in a few special cases. Such a schemehas not even been discussed for the well researched smallestenclosing ball (core vector machine) problem. The situationis vastly different when there is not a large number of con-straints but a large number parameters. For instance, the al-ternating direction method of multipliers (ADMM) that wasproposed by [Glowinski and Marroco, 1975] and by [Gabayand Mercier, 1976] already decades ago obtained consider-able attention, because it allows to solve convex optimiza-tion problems with a large number of parameters in a dis-tributed setting [Boyd et al., 2011]. For instance, the parame-ters of any log-likelihood maximization problem like logisticor ordinary least squares regression are just the data points.The loss function of such problems, that is, the negative log-likelihood function, is the sum of the losses for each datapoint. In this case ADMM lends itself to a distributed imple-mentation where the data points are distributed on differentcompute nodes.

Surprisingly, so far no general convex inequality con-straints have been considered directly in the context ofADMM. Although, in principle, standard ADMM can alsobe used for solving constrained optimization problems. Adistributed implementation of the straightforward extensionof ADMM leads to non-trivial constrained optimization sub-problems that have to be solved in every iteration. Solvingconstrained problems is typically transformed into a sequenceof unconstrained problems. Hence, this approach featuresthree nested loops, the outer loop for reaching consensus, oneloop for the constraints, and an inner loop for solving un-constrained problems. Alternatively, one could use the stan-dard augmented Lagrangian method, originally known as themethod of multipliers [Hestenes, 1969], that has been specifi-cally designed for solving constrained optimization problems.Combining the augmented Lagrangian method with ADMMallows to solve general constrained problems in a distributedfashion by running the augmented Lagrangian method in anouter loop and ADMM in an inner loop. Again, we end up

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4525

with three nested loops, the outer loop for the augmentedLagrangian method and the standard two nested inner loopsfor ADMM. Thus, one could assume that any distributedsolver for constrained optimization problems needs at leastthree nested loops: one for reaching consensus, one for theconstraints, and one for the unconstrained problems. Thekey contribution of our paper is showing that this is not thecase. One of the nested loops can be avoided by mergingthe loops for reaching consensus and dealing with the con-straints. Our approach, that only needs two nested loops,combines ADMM with the augmented Lagrangian methoddifferently than the direct approach of running the augmentedLagrangian method in the outer and ADMM in the inner loop.The latter combination, that to our surprise has not been dis-cussed in the literature before, still provides us with a goodbaseline in the experimental section.

Related WorkTo the best of our knowledge, our extension of ADMMis the first distributed algorithm for solving general convexoptimization problems with no restrictions on the type ofconstraints or assumptions on the structure of the problem.The only special case, that we are aware of, are quadrati-cally constrained quadratic problems that has been addressedby [Huang and Sidiropoulos, 2016]. However, their ap-proach, that builds on consensus ADMM, does not scale,since every constraint gives rise to a new subproblem.

[Mosk-Aoyama et al., 2010] have designed and analyzed adistributed algorithm for solving convex optimization prob-lems with separable objective function and linear equalityconstraints. Their algorithm blends a gossip-based informa-tion spreading, iterative gradient ascent method with the bar-rier method from interior-point algorithms. It is similar toADMM and can also handle only linear constraints.

[Zhu and Martınez, 2012] have introduced a distributedmultiagent algorithm for minimizing a convex function thatis the sum of local functions subject to a global equality orinequality constraint. Their algorithm involves projectionsonto local constraint sets that are usually as hard to computeas solving the original problem with general constraints. Forinstance, it is well known via standard duality theory that thefeasibility problem for linear programs is as hard as solvinglinear programs. This holds true for general convex optimiza-tion problems with vanishing duality gap.

In principle, the standard ADMM can also handle convexconstraints by transforming them into indicator functions thatare added to the objective function. However, this leads tosubproblems that need to be solved in each iteration that entailcomputing a projection onto the feasible region. This entailsthe same issues as the method by [Zhu and Martınez, 2012]since computing these projections can be as hard as solvingthe original problem.

The recent literature on ADMM is vast. Most paperson ADMM stay in the standard framework of optimizing afunction or a sum of functions subject to linear constraints.Exemplarily for many others, we just mention [Zhang andKwok, 2014] who provide convergence guarantees for asyn-chronous ADMM and [Ghadimi et al., 2015] who study opti-mal penalty parameter selection.

2 Alternating Direction Method of MultipliersHere, we briefly review the alternating direction method ofmultipliers (ADMM) and discuss how it can be adapted fordealing with distributed data, before we extend it for handlingconvex constraints in the next section.

ADMM is an iterative algorithm that in its most generalform can solve convex optimization problems of the form

minx,z f1(x) + f2(z)s.t. Ax+Bz − c = 0,

(1)

where f1 : Rn1 → R ∪ {∞} and f2 : Rn2 → R ∪ {∞} areconvex functions,A ∈ Rm×n1 andB ∈ Rm×n2 are matrices,and c ∈ Rm.

ADMM can obviously deal with linear equality con-straints, but it can also handle linear inequality constraints.The latter are reduced to linear equality constraints by replac-ing constraints of the form Ax ≤ b by Ax + s = b, addingthe slack variable s to the set of optimization variables, andsetting f2(s) = 1Rm

+(s), where

1Rm+

(s) =

{0, if s ≥ 0

∞, otherwise,

is the indicator function of the set Rm+ = {x ∈ Rm|x ≥ 0}.Note that f1 and f2 are allowed to take the value∞.

Recently, ADMM regained a lot of attention, because itallows to solve problems with separable objective function ina distributed setting. Such problems are typically given as

minx∑i fi(x),

where fi corresponds to the i-th data point (or more generallyi-th data block) and x is a weight vector that describes thedata model. This problem can be transformed into an equiva-lent optimization problem, with individual weight vectors xifor each data point (data block) that are coupled through anequality constraint,

minxi,z

∑i fi(xi)

s.t. xi − z = 0 ∀ i,which is a special case of Problem 1 that can be solved byADMM in a distributed setting by distributing the data.

3 ADMM ExtensionAdding convex inequality constraints to Problem 1 does notdestroy convexity of the problem, but so far ADMM cannotdeal with such constraints. Note that the problem only re-mains convex, if all equality constraints are induced by affinefunctions. That is, we cannot add convex equality constraintsin general without destroying convexity. Hence, here we con-sider convex optimization problems in its most general form

minx,z f1(x) + f2(z)s.t. g0(x) ≤ 0

h1(x) + h2(z) = 0,(2)

where f1 and f2 are as in Problem 1, g0 : Rn1 → Rpis convex in every component, and h1 : Rn1 → Rm andh2 : Rn2 → Rm are affine functions. In the following we as-sume that the problem is feasible, i.e., that a feasible solution

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4526

exists, and that strong duality holds. A sufficient conditionfor strong duality is that the interior of the feasible region isnon-empty. This condition is known as Slater’s condition forconvex optimization problems [Slater, 1950].

Our extension of ADMM for solving Problem 2 and itsconvergence analysis works with an equivalent reformulationof Problem, where we replace g0(x) by

g(x) = max{0, g0(x)}2,with componentwise maximum, and turn the convex inequal-ity constraints into convex equality constraints. Thus, in thefollowing we consider optimization problems of the form

minx,z f1(x) + f2(z)s.t. g(x) = 0

h1(x) + h2(z) = 0,(3)

where g(x) = max{0, g0(x)}2, which by construction isagain convex in every component. Note, though, that the con-straint g(x) = 0 is no longer affine. However, we show in thefollowing that Problem 3 can still be solved efficiently.

Analogously to ADMM our extension builds on the Aug-mented Lagrangian for Problem 3 which is the followingfunction

Lρ(x, z, µ, λ) = f1(x) + f2(z) +ρ

2‖g(x)‖2 + µ>g(x)

+ρ

2‖h1(x) + h2(z)‖2 + λ> (h1(x) + h2(z)) ,

where µ ∈ Rp and λ ∈ Rm are Lagrange multipliers, ρ > 0is some constant, and ‖·‖ denotes the Euclidean norm. TheLagrange multipliers are also referred to as dual variables.

Algorithm 1 is our extension of ADMM for solving in-stances of Problem 3. It runs in iterations. In the (k + 1)-thiteration the primal variables xk and zk as well as the dualvariables µk and λk are updated.

Algorithm 1 ADMM for problems with non-linear con-straints

1: input: instance of Problem 32: output: approximate solution x ∈ Rn1 , z ∈ Rn2 , µ ∈

Rp, λ ∈ Rm3: initialize x0 = 0, z0 = 0, µ0 = 0, λ0 = 0, and ρ to some

constant > 04: repeat5: xk+1 := argminx Lρ(x, z

k, µk, λk)6: zk+1 := argminz Lρ(x

k+1, z, µk, λk)7: µk+1 := µk + ρg(xk+1)8: λk+1 := λk + ρ

(h1(xk+1) + h2(zk+1)

)9: until convergence

10: return xk, zk, µk, λk

4 Convergence AnalysisFrom duality theory we know that for all x ∈ Rn1 and z ∈Rn2

L0(x∗, z∗, µ∗, λ∗) ≤ L0(x, z, µ∗, λ∗), (4)where L0 is the Lagrangian of Problem 3 and x∗, z∗, µ∗,and λ∗ are optimal primal and dual variables. Note, that

x∗, z∗, µ∗, and λ∗ are not necessarily unique. Here, they referjust to one optimal solution. Also note that the Lagrangian isidentical to the Augmented Lagrangian with ρ = 0. Giventhat strong duality holds, the optimal solution to the origi-nal Problem 3 is identical to the optimal solution of the La-grangian dual.

We need a few more definitions. Let fk = f1(xk)+f2(zk)be the objective function value at the k-th iterate (xk, zk) andlet f∗ be the optimal function value. Let rkg = g(xk) be theresidual of the nonlinear equality constraints, i.e., the con-straints originating from the convex inequality constraints,and let rkh = h1(xk) + h2(zk) be the residual of the linearequality constraints in iteration k.

Our goal in this section is to prove the following theorem.

Theorem 1. When Algorithm 1 is applied to an instance ofProblem 3, then

limk→∞

rkg = 0, limk→∞

rkh = 0, and limk→∞

fk = f∗.

The theorem states primal feasibility and convergence ofthe primal objective function value. Note, however, that con-vergence to primal optimal points x∗ and z∗ cannot be guar-anteed. This is the case for the original ADMM as well.Additional assumptions on the problem, like, for instance, aunique optimum, are necessary to guarantee convergence tothe primal optimal points. However, the points xk, zk will beprimal optimal and feasible up to an arbitrarily small error forsufficiently large k.

The proof of Theorem 1 can be found in the full version ofthis paper [Giesen and Laue, 2016].

5 Distributing ConstraintsFinally, we are ready to discuss the main problem that we setout to address in this paper, namely solving general convexoptimization problems with many constraints in a distributedsetting by distributing the constraints. That is, we want toaddress optimization problems of the form

minx f(x)s.t. gi(x) ≤ 0 i = 1 . . . p

hi(x) = 0 i = 1 . . .m,(5)

where f : Rn → R and gi : Rn → Rpi are convex functions,and hi : Rn → Rmi are affine functions. In total, we havep1 + p2 + . . . + pp inequality constraints that are groupedtogether into p batches and m1 + m2 + . . . + mm equalityconstraints that are subdivided into m groups. For distribut-ing the constraints we can assume without loss of generalitythat m = p. That is, we have m batches that each contain piinequality and mi equality constraints.

Again it is easier to work with an equivalent reformulationof Problem 5, where each batch of equality and inequalityconstraints shares the same variables xi, namely problems ofthe form

minxi,z

∑mi=1 f(xi)

s.t. max{0, gi(xi)}2 = 0 i = 1 . . .mhi(xi) = 0 i = 1 . . .mxi = z,

(6)

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4527

where all the variables xi are coupled through the affine con-straints xi = z. To keep our exposition simple, the objectivefunction has been scaled by m in the reformulation.

For specializing our extension of ADMM to instances ofProblem 6 we need the Augmented Lagrangian of this prob-lem, which reads as

Lρ(xi, z, µi,g, µi,h, λ) =m∑i=1

f(xi) +ρ

2

m∑i=1

‖max{0, gi(xi)}2‖2

+m∑i

(µi,g)>max{0, gi(xi)}2

+ρ

2

m∑i=1

‖hi(xi)‖2 +m∑i

(µi,h)>hi(xi)

+ρ

2

m∑i=1

‖xi − z‖2 +m∑i

(λi)>(xi − z),

where µi,g, µi,h, and λi are the Lagrange multipliers (dualvariables).

Note that the Lagrange function is separable. Hence, theupdate of the x variables in Line 5 of Algorithm 1 decom-poses into the following m independent updates

xk+1i = argminxi

f(xi) +ρ

2‖max{0, gi(xi)}2‖2

+ (µki,g)>max{0, gi(xi)}2

+ρ

2‖hi(xi)‖2 + (µki,h)>hi(xi)

+ρ

2‖xi − zk‖2 + (λki )>(xi − zk),

that can be solved in parallel once the constraints gi(xi) andhi(xi) have been distributed on m different, distributed com-pute nodes. Note that each update is an unconstrained, con-vex optimization problem, because the functions that need tobe minimized are sums of convex functions. The only twosummands where this might not be obvious, areρ

2‖max{0, gi(xi)}2‖2 and (µki,g)

>max{0, gi(xi)}2.For the first term note that the squared norm of a non-negative, convex function is always convex again. The secondterm is convex, because it can be shown by induction that theµki,g are always non-negative.

The update of the z variable in Line 6 of Algorithm 1amounts to solving the following unconstrained optimizationproblems

zk+1 = argminz

m∑i=1

ρ

2‖xk+1

i − z‖2 +m∑i=1

(λki )>(xk+1i − z)

=ρ∑mi=1 x

k+1i +

∑mi=1 λ

ki

ρ ·m ,

and the updates of the dual variables µi and λi are as follows

µk+1i,g = µki,g + ρ max{0, gi(xk+1

i )}2,µk+1i,h = µki + ρ hi(x

k+1i ),

λk+1i = λki + ρ

(xk+1i − zk+1

).

That is, in each iteration there are m independent, uncon-strained minimization problems that can be solved in parallelon different compute nodes. The solutions of the independentsubproblems are then combined on a central node through theupdate of the z variables and the Lagrange multipliers. Ac-tually, since the Lagrange multipliers µi,g and µi,h are alsolocal, i.e., involve only the variables xk+1

i for any given indexi, they can also be updated in parallel on the same computenodes where the xki updates take place. Only the variables zand the Lagrange multipliers λi need to be updated centrally.

Looking at the update rules it becomes apparent that Al-gorithm 1 when applied to instances of Problem 6 is basi-cally a combination of the standard Augmented Lagrangianmethod [Hestenes, 1969; Powell, 1969] for solving convex,constrained optimization problems and ADMM for solvingconvex optimization problems in a distributed fashion.

6 ExperimentsWe have implemented our extension of ADMM in Pythonusing the NumPy and SciPy libraries, and tested this im-plementation on the robust SVM problem [Shivaswamy etal., 2006] that has a second order cone constraint for ev-ery data point. In our experiments we distributed theseconstraints onto different compute nodes, where we had tosolve an unconstrained optimization problem in every itera-tion. Since there is no other approach available that coulddeal with a large number of arbitrary constraints in a dis-tributed manner we compare our approach to the baselineapproach of running an Augmented Lagrangian method inan outer loop and standard ADMM in an inner loop. Notethat this approach has three nested loops. The outer loopturns the constrained problem into a sequence of uncon-strained problems (Augmented Lagrangians), the next loopdistributes the problem using distributed ADMM, and thefinal inner loop solves the unconstrained subproblems us-ing the L-BFGS-B algorithm [Morales and Nocedal, 2011;Zhu et al., 1997] in our implementation.

Robust SVMsThe robust SVM problem has been designed to deal with bi-nary classification problems whose input are not just labeleddata points (x(1), y(1)), . . . , (x(n), y(n)), where the x(i) arefeature vectors and the y(i) are binary labels, but a distribu-tion over the feature vectors. That is, the labels are assumed tobe known precisely and the uncertainty is only in the features.The idea behind the robust SVM is replacing the constraints(for feature vectors without uncertainty) of the standard linearsoft-margin SVM by their probabilistic counterparts

Pr[y(i)w>x(i) ≥ 1− ξi

]≥ 1− δi

that require the now random variable x(i) with probability atleast 1 − δi ≥ 0 to be on the correct side of the hyperplanewhose normal vector is w. Shivaswamy et al. show that theprobabilistic constraints can be written as second order coneconstraints

y(i)w>x(i) ≥ 1− ξi +√δi/(1− δi)

∥∥Σ1/2i w

∥∥,under the assumption that the mean of the random variablex(i) is the empirical mean x(i) and the covariance matrix of

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4528

Figure 1: Various statistics for the performance of the distributed ADMM extension on an instance of the robust SVM problem. Theconvergence proof only states that the value V k must be monotonically decreasing. This can be observed also experimentally in the figure onthe bottom right. Neither the primal function value nor the residuals need to be monotonically decreasing, and as can be seen in the figureson the top, they actually do not decrease monotonically.

x(i) is Σi. The robust SVM problem is then the followingSOCP (second order cone program)

minw,ξ

1

2‖w‖2 + c

n∑i=1

ξi

s.t. y(i)w>x(i) ≥ 1− ξi +√δi/(1− δi)

∥∥Σ1/2i w

∥∥ξi ≥ 0, i = 1 . . . n.

This problem can be reformulated into the form of Problem 6and is thus amenable to a distributed implementation of ourextension of ADMM.

Experimental SetupWe generated random data sets similarly to [Andersen et al.,2012], where an interior point solver has been described forsolving the robust SVM problem. The set of feature vectorswas sampled from a uniform distribution on [−1, 1]n. Thecovariance matrices Σi were randomly chosen from the coneof positive semidefinite matrices with entries in the interval[−1, 1] and δi has been set to 1

2 . Each data point contributesexactly one constraint to the problem and is assigned to onlyone of the compute nodes.

In the following, the primal optimization variables are wand ξ, the consensus variables for the primal optimizationvariables w are still denoted as z, and also the dual variablesare still denoted as λ for the consensus constraints and µ forthe convex constraints, respectively.

Convergence ResultsFigure 1 shows the primal objective function value fk, thenorm of the residuals rkg and rkh, the distances ‖zk − z∗‖,‖λk − λ∗‖, and the value V k of one run of our algorithm

for two compute nodes. Note, that only V k must be strictlymonotonically decreasing according to our convergence anal-ysis. The proof does not make any statement about the mono-tonicity of the other values, and as can be seen in Figure 1,such statements would actually not be true. All values de-crease in the long run, but are not necessarily monotonicallydecreasing.

As can be seen in Figure 1 (top-left), the function valuefk is actually increasing for the first few iterations, while theresiduals rkg for the inequality constraints become very small,see Figure 1 (top-middle). That is, within the first iterationseach compute node finds a solution to its share of the datathat is almost feasible but has a higher function value thanthe true optimal solution. This basically means that the errorsξi for the data points are over-estimated. After a few moreiterations the primal function value drops and the inequalityresiduals increase meaning that the error terms ξi as well asthe individual estimators wi converge to their optimal values.

In the long run, the local estimators at the different com-pute nodes converge to the same solution. This is witnessedin Figure 1 (top-right), where one can see that the residuals rkhfor the consensus constraints converge to zero, i.e., consensusamong the compute nodes is reached in the long run.

Finally, it can be seen that the consensus estimator zk con-verges to its unique optimal point z∗. Note that in general wecannot guarantee such a convergence since the optimal pointdoes not need to be unique. In the special case that the opti-mal point is unique we always have convergence to this point.

Scalability ResultsFigure 2 shows the scalability of our extension of ADMMin terms of the number of compute nodes, data points, andapproximation quality, respectively. All running times were

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4529

2 4 6 8 10 12 14 16

compute nodes

0

5

10

15

20

25

30

35

40it

erat

ions

baselinethis paper

103 104 105

data points

0

10

20

30

40

50

60

70

80

iter

atio

ns

baselinethis paper

10−3 10−2 10−1

‖zk − z∗‖∞

0

50

100

150

200

iter

atio

ns

baselinethis paper

Figure 2: Running times of the algorithm on the robust SVM problem. The figure on the left shows that the number of iterations increasesmildly with the number of compute nodes. The middle picture shows that the number of iterations is decreasing with increasing number ofdata points. The figure on the right shows the dependency of the distance of the consensus estimator zk in iteration k to the optimal estimatorz∗. It can be seen that our extension of ADMM outperforms the baseline approach with three nested loops.

measured in terms of iterations and averaged over runs forten randomly generated data sets. For the baseline we usedthe straightforward three nested loops approach.

(1) For measuring the scalability in terms of employedcompute nodes, we generated 10,000 data points with 10,000features. As stopping criterion we used ‖zk − z∗‖∞ ≤5 · 10−3, i.e., the predictor zk had to be close to the opti-mum. Here we use the infinity norm to be independent fromthe number of dimensions. The data set was split into four,eight, twelve, and 16 equal sized batches that were distributedamong the compute nodes. Note that every batch had muchfewer data points than features, and thus the optimal solu-tions to the respective problems at the compute nodes werequite different from each other. Nevertheless, our algorithmconverged very well to the globally optimal solution. Onlythe convergence speed was affected by the diversity of the lo-cal solutions at the different compute nodes. Since we keptthe total number of data points in our experiments fixed, thediversity was increasing with the number of compute nodesthat were assigned fewer data points each. Hence it was ex-pected that the convergence speed decreases, i.e., the numberof iterations increases, with a growing number of computenodes. The expected behavior can be seen in Figure 2 (left).

(2) For measuring the scalability in terms of the numberof data points we increased the number of data points but keptthe number of features fixed at 200. The stopping criterionfor our algorithm was again ‖zk − z∗‖∞ ≤ 5 · 10−3. Weused eight compute nodes to compute the solutions. Again,the points were distributed equally among the compute nodes.This time one would expect a decreasing running time with anincreasing number of data points, because the number of datapoints per machine is increasing and thus also the diversityof the local solutions at the different compute nodes is de-creasing. That is, with an increasing number of data points itshould take fewer iterations to reach an approximate consen-sus about the global solution among the compute nodes. Theresults of the experiment that are shown in Figure 2 (middle)confirm this expectation. The number of iterations indeeddecreases with a growing number of data points. This is sim-ilar to [Shalev-Shwartz and Srebro, 2008] who have also ob-served that an increasing number of data points can decreasethe work required for a good predictor.

(3) For measuring the scalability in terms of the approxi-mation quality, we generated 8000 data points in 200 dimen-sions. Again, eight compute nodes were used for the exper-iments whose results are shown in Figure 2 (right). As ex-pected the number of iterations (running time) increases withincreasing approximation quality that was again measured interms of the infinity norm. In this paper we are not providinga theoretical convergence rate analysis, which we leave forfuture work, but the experimental results shown here alreadyprovide some intuition on the dependency of the number ofiterations in terms of the approximation quality: It seems thatour extension of ADMM can solve problems to a medium ac-curacy within a reasonable number of iterations, but higheraccuracy requires a significant increase in the number of iter-ations. Such a behavior is well known for standard ADMMwithout constraints. In the context of our example applica-tion, robust SVMs, medium accuracy usually is sufficient asoften higher accuracy solutions do not provide better predic-tors, a phenomenon that is also known as regularization byearly stopping.

7 ConclusionsDespite the vast literature on ADMM, to the best of ourknowledge, no scheme for distributing general convex con-straints has been studied before. Here we have closed thisgap by combining ADMM and the augmented Lagrangianmethod for solving general convex optimization problemswith many convex constraints. The straightforward combi-nation of ADMM and the augmented Lagrangian method en-tails three nested loops, an outer loop for reaching consensus,one loop for the constraints, and an inner loop for solving un-constrained problems. Our main contribution is showing thatthe loops for reaching consensus and for handling constraintscan be merged, resulting in a scheme with only two nestedloops. We provide the first convergence proof for such a lazyalgorithmic scheme.

AcknowledgmentsThis work has been supported by the DFG grant GI-711/5-1within the Priority Program 1736 Algorithms for Big Data.

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4530

References[Andersen et al., 2012] Martin Andersen, Joachim Dahl,

Zhang Liu, and Lieven Vandenberghe. Interior-PointMethods for Large-Scale Cone Programming, pages 55–84. MIT Press, 2012.

[Boyd et al., 2011] Stephen P. Boyd, Neal Parikh, Eric Chu,Borja Peleato, and Jonathan Eckstein. Distributed opti-mization and statistical learning via the alternating direc-tion method of multipliers. Foundations and Trends in Ma-chine Learning, 3(1):1–122, 2011.

[Gabay and Mercier, 1976] Daniel Gabay and BertrandMercier. A dual algorithm for the solution of nonlinearvariational problems via finite element approximation.Computers & Mathematics with Applications, 2(1):17 –40, 1976.

[Ghadimi et al., 2015] Euhanna Ghadimi, Andre Teixeira,Iman Shames, and Mikael Johansson. Optimal parameterselection for the alternating direction method of multipli-ers (admm): Quadratic problems. IEEE Trans. Automat.Contr., 60(3):644–658, 2015.

[Giesen and Laue, 2016] Joachim Giesen and Soren Laue.Distributed convex optimization with many convex con-straints. CoRR, abs/1610.02967, 2016.

[Glowinski and Marroco, 1975] R. Glowinski and A. Mar-roco. Sur l’approximation, par elements finis d’ordreun, et la resolution, par penalisation-dualite d’une classede problemes de dirichlet non lineaires. ESAIM: Mathe-matical Modelling and Numerical Analysis - ModelisationMathematique et Analyse Numerique, 9(R2):41–76, 1975.

[Hestenes, 1969] Magnus R. Hestenes. Multiplier and gradi-ent methods. Journal of Optimization Theory and Appli-cations, 4(5):303–320, 1969.

[Huang and Sidiropoulos, 2016] Kejun Huang andNicholas D Sidiropoulos. Consensus-admm for generalquadratically constrained quadratic programming. IEEETransactions on Signal Processing, 64(20):5297–5310,2016.

[Morales and Nocedal, 2011] Jose Luis Morales and JorgeNocedal. Remark on ”algorithm 778: L-BFGS-B: fortransubroutines for large-scale bound constrained optimiza-tion”. ACM Trans. Math. Softw., 38(1):7:1–7:4, 2011.

[Mosk-Aoyama et al., 2010] Damon Mosk-Aoyama, TimRoughgarden, and Devavrat Shah. Fully distributed algo-rithms for convex optimization problems. SIAM Journalon Optimization, 20(6):3260–3279, 2010.

[Powell, 1969] M. J. D. Powell. Algorithms for nonlinearconstraints that use lagrangian functions. MathematicalProgramming, 14(1):224–248, 1969.

[Shalev-Shwartz and Srebro, 2008] Shai Shalev-Shwartzand Nathan Srebro. SVM optimization: inverse depen-dence on training set size. In International Conference onMachine Learning (ICML), pages 928–935, 2008.

[Shivaswamy et al., 2006] Pannagadatta K. Shivaswamy,Chiranjib Bhattacharyya, and Alexander J. Smola. Sec-ond order cone programming approaches for handling

missing and uncertain data. Journal of Machine LearningResearch, 7:1283–1314, 2006.

[Slater, 1950] Morton Slater. Lagrange multipliers revisited.Cowles Foundation Discussion Papers 80, Cowles Foun-dation for Research in Economics, Yale University, 1950.

[Tsang et al., 2007] Ivor W. Tsang, Andras Kocsor, andJames T. Kwok. Simpler core vector machines with en-closing balls. In International Conference on MachineLearning (ICML), pages 911–918, 2007.

[Zhang and Kwok, 2014] Ruiliang Zhang and James TKwok. Asynchronous distributed admm for consensusoptimization. In International Conference on MachineLearning (ICML), pages 1701–1709, 2014.

[Zhu and Martınez, 2012] Minghui Zhu and Sonia Martınez.On distributed convex optimization under inequality andequality constraints. IEEE Trans. Automat. Contr.,57(1):151–164, 2012.

[Zhu et al., 1997] Ciyou Zhu, Richard H. Byrd, PeihuangLu, and Jorge Nocedal. Algorithm 778: L-BFGS-B: for-tran subroutines for large-scale bound-constrained opti-mization. ACM Trans. Math. Softw., 23(4):550–560, 1997.

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4531