Stochastic Newton and quasi-Newton Methods forLarge-Scale Convex and Non-Convex Optimization
Donald Goldfarb
Department of Industrial Engineering and Operations ResearchColumbia University
Joint with Robert Gower, Peter Richtarik, Shiqian Ma, Xiao Wang and Wei Liu
U.S.-Mexico Workshop on Optimization and its Applications2016
Merida, January 4, 2016
1 / 31
Outline
Newton-like and quasi-Newton methods for convex stochasticoptimization problems using limited memory block BFGSupdates.
Quasi-Newton methods for nonconvex stochastic optimizationproblems using damped limited memory BFGS updates.
In both cases the objective functions can be expressed as thesum of a huge number of functions of an extremely largenumber of variables.
We present numerical results on problems from machinelearning.
2 / 31
Related work on L-BFGS for Stochastic Optimization
P1 N.N. Schraudolph, J. Yu and S.Gunter. A stochastic quasi-Newtonmethod for online convex optim. Int’l. Conf. AI & Stat., 2007
P2 A. Bordes, L. Bottou and P. Gallinari. SGD-QN: Carefulquasi-Newton stochastic gradient descent. JMLR vol. 10, 2009
P3 R.H. Byrd, S.L. Hansen, J. Nocedal, and Y. Singer. A stochasticquasi-Newton method for large-scale optim. arXiv1401.7020v2,2014
P4 A. Mokhtari and A. Ribeiro. RES: Regularized stochastic BFGSalgorithm. IEEE Trans. Signal Process., no. 10, 2014.
P5 A. Mokhtari and A. Ribeiro. Global convergence of online limitedmemory BFGS. to appear in J. Mach. Learn. Res., 2015.
P6 P. Moritz, R. Nishihara, M.I. Jordan. A linearly-convergentstochastic L-BFGS Algorithm, 2015 arXiv:1508.02087v1
P7 X. Wang, S. Ma, D. Goldfarb and W. Liu. Stochastic quasi-Newtonmethods for nonconvex stochastic optim. 2015, submitted.
(the first 6 papers are for strongly convex problems, the last one isfor nonconvex problems) 3 / 31
Stochastic optimization
Stochastic optimization
min f (x) = E[f (x , ξ)], ξ is random variable
Or finite sum (with fi (x) ≡ f (x , ξi ) for i = 1, . . . , n and verylarge n)
min f (x) =1
n
n∑i=1
fi (x)
f and ∇f are very expensive to evaluate; e.g., SGD methodsrandomly choose a random subset S ⊂ [n] and evaluate
fS(x) =1
|S|∑i∈S
fi (x) and ∇fS(x) =1
|S|∑i∈S∇fi (x)
Essentially, only noisy info about f , ∇f and ∇2f is available
Challenge: how to design a method that takes advantage ofnoisy 2nd-order information?
4 / 31
Part 1: Using 2nd-order information
Assumption: f (x) = 1n
∑ni=1 fi (x) is strongly convex and twice
continuously differentiable.
Choose (compute) a sketching matrix Sk (the columns of Skare a set of directions).
Following Byrd, Hansen, Nocedal and Singer, we do not usedifferences in noisy gradients to estimate curvature, but rathercompute the action of the sub-sampled Hessian on Sk . i.e.,
compute Yk = 1|T |
∑i∈T ∇2fi (x)Sk , where T ⊂ [n].
We choose T = S
5 / 31
block BFGS
Given Hk = B−1k , the block BFGS method computes a ”least
change” update to the current approximation Hk to the inverseHessian matrix ∇2f (x) at the current point x , by solving
min ‖H − Hk‖s.t., H = H>, HYk = Sk .
This gives the updating formula (analgous to the updates derivedby Broyden, Fletcher, Goldfarb and Shanno).
Hk+1 = (I−Sk [S>k Yk ]−1Y>k )Hk(I−Yk [S>k Yk ]−1S>k )+Sk [S>k Yk ]−1S>k
or, by the Sherman-Morrison-Woodbury formula:
Bk+1 = Bk − BkSk [S>k BkSk ]−1S>k Bk + Yk [S>k Yk ]−1Y>k
6 / 31
Limited Memory Block BFGS
After M block BFGS steps starting from Hk+1−M , one can expressHk+1 as
Hk+1 = VkHkVTk + SkΛkS
Tk
= VkVk−1Hk−1VTk−1Vk + VkSk−1Λk−1S
Tk−1V
Tk + SkΛkS
Tk
...
= Vk:k+1−MHk+1−MV Tk:k+1−M +
k+1−M∑i=k
Vk:i+1SiΛiSTi V T
k:i+1,
whereVk = (I − SkΛkY
Tk ) (1)
and Λk = (STk Yk)−1 and Vk:i = Vk · · ·Vi .
7 / 31
Limited Memory Block BFGS
Hence, when the number of variables d is large, instead ofstoring the d × d matrix Hk , we store the previous M blockcurvature pairs
(Sk+1−M ,Yk+1−M) , . . . , (Sk ,Yk) .
Then, analogously to the standard L-BFGS method, for anyvector v ∈ Rd , Hkv can be computed efficiently using atwo-loop block recursion (in O(Mp(d + p) + p3) operations),if all Si ∈ Rd×p.
Intuition
Limited memory - least change aspect of BFGS is important
Each block update acts like a sketching procedure.
8 / 31
Choices for the Sketching Matrix Sk
We employ one of the following strategies
Gaussian: Sk ∼ N (0, I ) has Gaussian entries sampled i.i.d ateach iteration.
Previous search directions si delayed: Store the previous Lsearch directions Sk = [sk+1−L, . . . , sk ] then update Hk onlyonce every L iterations.
Self-conditioning: Sample the columns of the Cholesky factorsLk of Hk (i.e., LkL
Tk = Hk) uniformly at random. Fortunately
we can maintain and update Lk efficiently with limitedmemory.
The matrix S is a sketching matrix, in the sense that we aresketching the, possibly very large equation ∇2f (x)H = I to whichthe solution is the inverse Hessian. Left multiplying by ST
compresses/sketches the equation yielding ST∇2f (x)H = ST .
9 / 31
Stochastic Variance Reduced Gradients
Stochastic methods converge slowly near the optimum due tothe variance of the gradient estimates ∇fS(x); hence requiringa decreasing step size.
We use the control variates approach of Johnson and Zhang(2013) for a SGD method SVRG.
It uses ∇fS(xt)−∇fS(wk) +∇f (wk , where wk is a referencepoint, in place of ∇fS(xt) .
wk , and the full gradient, are computed after each full pass ofthe data, hence doubling the work of computing stochasticgradients.
Other SGD variance reduction techniques have been recentlyproposes including the methods: SAG, SAGA, SDCA, S2GD.
10 / 31
The Basic Algorithm
11 / 31
Algorithm 0.1: Stochastic Variable Metric Learning with SVRG
Input: H−1 ∈ Rd×d , w0 ∈ Rd , η ∈ R+, s = subsample size, q = sampleaction size and m
1 for k = 0, . . . , max iter do2 µ = ∇f (wk)3 x0 = wk
4 for t = 0, . . . ,m − 1 do5 Sample St , Tt ⊆ [n] i.i.d from a distribution S6 Compute the sketching matrix St ∈ Rd×q
7 Compute ∇2fS(xt)St8 Ht =update metric(Ht−1,St ,∇2fT (xt)St)9 dt = −Ht (∇fS(xt)−∇fS(wk) + µ)
10 xt+1 = xt + ηdt11 end12 Option I: wk+1 = xm13 Option II: wk+1 = xi , i selected uniformly at random from [m];
14 end
11 / 31
Convergence - Assumptions
There exist constants λ,Λ ∈ R+ such that
f is λ–strongly convex
f (w) ≥ f (x) +∇f (x)T (w − x) +λ
2‖w − x‖2
2 , (2)
f is Λ–smooth
f (w) ≤ f (x) +∇f (x)T (w − x) +Λ
2‖w − x‖2
2 , (3)
These assumptions imply that
λI � ∇2fS(w) � ΛI , for all x ∈ Rd ,S ⊆ [n], (4)
from which we can prove that there exist constants γ, Γ ∈ R+
such that for all k we have
γI � Hk � ΓI . (5)
12 / 31
Linear Convergence
Theorem
Suppose that the Assumptions hold. Let w∗ be the uniqueminimizer of f (w). Then in our Algorithm, we have for all k ≥ 0that
Ef (wk)− f (w∗) ≤ ρkEf (w0)− f (w∗),
where the convergence rate is given by
ρ =1/2mη + ηΓ2Λ(Λ− λ)
γλ− ηΓ2Λ2< 1,
assuming we have chosen η < γλ/(2Γ2Λ2) and that we choose mlarge enough to satisfy
m ≥ 1
2η (γλ− ηΓ2Λ(2Λ− λ)),
which is a positive lower bound given our restriction on η.13 / 31
gisette-scale d = 5, 000, n = 6, 000
0 5 10 15 20 25 30
0.01
0.175
0.34
0.505
0.670.835
1
time (s)
err
or
5 10 15 20 25 30
datapasses
LML_gauss_18_M_5
LMLpdd_18_M_5
LMLF_18_M_5
MNJ_b_78_bH_330
SVRG
Figure: limited-gisette
14 / 31
covtype-libsvm-binary d = 54, n = 581, 012
0 10 20 30 40
0.748
0.79
0.832
0.874
0.916
0.958
1
time (s)
err
or
2 4 6 8 10
datapasses
LML_gauss_8_M_5
LMLpdd_8_M_5
LMLF_8_M_5
MNJ_b_763_bH_3815
SVRG
Figure: limited-covtype-libsvm-binary
15 / 31
Higgs d = 28, n = 11, 000, 000
0 50 100 150 200 250
0.933
0.947
0.96
0.973
0.987
1
time (s)
err
or
1 1.2 1.4 1.6 1.8 2
datapasses
LML_gauss_4_M_5
LMLpdd_4_M_5
LMLF_4_M_5
MNJ_b_3317_bH_16585
SVRG
Figure: HIGGS
16 / 31
SUSY d = 18, n = 3, 548, 466
0 50 100 150 200 250
0.723
0.779
0.834
0.889
0.945
1
time (s)
err
or
2 4 6 8
datapasses
MLgauss_3_M_5
MLprev_3_M_5
MLfact_3_M_5
MNJ
SVRG
Figure: SUSY
17 / 31
epsilon-normalized d = 2, 000, n = 400, 000
0 50 100 150 200 250
0.506
0.605
0.704
0.802
0.901
1
time (s)
err
or
5 10 15
datapasses
LML_gauss_45_M_5
LMLpdd_45_M_5
LMLF_45_M_5
MNJ_b_633_bH_3165
SVRG
Figure: epsilon-normaliized
18 / 31
rcv1-training d = 47, 236, n = 20, 242
0 5 10 15 20 25
0.411
0.529
0.647
0.764
0.882
1
time (s)
err
or
50 100 150 200 250
datapasses
MLgauss_15_M_5
MLprev_15_M_5
MLFact_15_M_5
MNJ_b_1000_bH_1000
SVRG
Figure: rcv1-train
19 / 31
url-combined d = 3, 231, 961, n = 2, 396, 130
0 20 40 60 80
0.163
0.302
0.442
0.581
0.721
0.86
1
time (s)
err
or
1 1.2 1.4 1.6 1.8 2
datapasses
MLgauss_10_M_5
MLprev_10_M_5
MLFact_10_M_3
MNJ_b_1548_bH_7740
SVRG
Figure:
20 / 31
zero-real-sim-L2 d = 20, 958, n = 72, 309
0 5 10 15 20 25
0.341
0.473
0.605
0.736
0.868
1
time (s)
err
or
20 40 60 80 100
datapasses
MLgauss_13_M_1
MLgauss_13_M_5
MLprev_13_M_5
MLFact_13_M_5
MNJ_b_1000_bH_1000
SVRG
MLprev_13_M_1
MLFact_13_M_1
MNJ_b_269_bH_1000
SVRG
Figure:
21 / 31
Contributions
New metric learning framework. A block BFGS framework forgradually learning the metric of the underlying function usinga sketched form of the subsampled Hessian matrix
New limited memory block BFGS method. May also be ofinterest for non-stochastic optimization
Several sketching matrix possibilities.
22 / 31
Part 2: Nonconvex stochastic optimization
Most stochastic quasi-Newton optimization methods are forstrongly convex problems; this is needed to ensure a curvaturecondition required for the positive definiteness of Bk (Hk)
This is not possible for nonconvex problem
In deterministic setting, one can do line search to guaranteethe curvature condition, and hence the positive definiteness ofBk (Hk)
Line search is not possible for stochastic optimization
To address these issues we develop a stochastic dampedL-BFGS method:
23 / 31
Stochastic quasi-Newton (SQN) for nonconvex problem
min f (x) ≡ E[F (x , ξ)]
Assumptions
[AS1] f is continuously differentiable; f is bounded below; ∇fis Lipschitz continuous with constant L
[AS2] For any iteration k, we have stochastic gradient satisfies
Eξk [∇f (xk , ξk)] = ∇f (xk)Eξk [‖∇f (xk , ξk)−∇f (xk)‖2] ≤ σ2
[AS3] Exist positive constants Cl , Cu, such that
Cl I � Hk � CuI , for any k
[AS4] Hk depends only on ξ[k−1], i.e., on all the randomsamples in iterations 1, 2, . . . , k − 1.
24 / 31
How to generate Hk to satisfy AS3 and AS4?
Let yk = 1m
∑mi=1(∇f (xk+1, ξk,i )−∇f (xk , ξk,i )) and define
yk = θkyk + (1− θk)Bksk ,
where θk is calculated through:
θk =
{1, if s>k yk ≥ 0.25s>k Bksk ,
(0.75s>k Bksk)/(s>k Bksk − s>k yk), if s>k yk < 0.25s>k Bksk .
Update Hk : (replace yk by yk )
Hk+1 = (I − ρksk y>k )Hk(I − ρk yks>k ) + ρksks>k
where ρk = 1/s>k yk
Implement in a limited memory version
25 / 31
Numerical Experiments
A nonconvex SVM problem with a sigmoid loss function
minx∈Rn
f (x) := Eu,v [1− tanh(v〈x , u〉)] + λ‖x‖22,
u ∈ Rn: feature vector; v ∈ {−1, 1}: corresponding label.
λ = 10−4 in our experiment
RCV1 dataset: Reuters newswire articles from 1996-1997.
A simplified version: 9625 articles classified into fourcategories “C15”, “ECAT”, “GCAT” and “MCAT”, each with2022, 2064, 2901 and 2638 articles, respectively.
Binary classification: predict if an article is in “MCAT” and“ECAT”.
Label: 1 if a given word in “MCAT” or “ECAT”, -1 otherwise.
60% of the articles - training data; 40% - testing data.
Problem dimension: 29992 (number of distinct words)
26 / 31
0 100 200 300 400 500 600 700 800 900 100010
−5
10−4
10−3
10−2
10−1
Iteration
Squ
ared
nor
m o
f gra
dien
t
SGD: β=10
SGD: β=20
SdLBFGS: p=0
SdLBFGS: p=1
SdLBFGS: p=3
SdLBFGS: p=5
SdLBFGS: p=10
SdLBFGS: p=20
0 100 200 300 400 500 600 700 800 900 1000
0.4
0.5
0.6
0.7
0.8
0.9
1
Iteration
Per
cent
cor
rect
ly c
lass
ified
Figure: Comparison of SdLBFGS variants with different memory size onRCV1 dataset. The step size of SdLBFGS is αk = 10/k and the batchsize is m = 100.
27 / 31
0 100 200 300 400 500 600 700 800 900 100010
−5
10−4
10−3
10−2
10−1
100
Iteration
Squ
ared
nor
m o
f gra
dien
t
SGD: m=1
SGD: m=50
SGD: m=100
SdLBFGS: m=1
SdLBFGS: m=50
SdLBFGS: m=75
SdLBFGS: m=100
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
x 104
10−4
10−3
10−2
10−1
100
SFO−calls
Squ
ared
nor
m o
f gra
dien
t
Figure: Comparison of SGD and SdLBFGS with different batch size onRCV1 dataset. For SdLBFGS the step size is αk = 10/k and the memorysize is p = 10. For SGD the step size is αk = 20/k .
28 / 31
0 100 200 300 400 500 600 700 800 900 1000
0.4
0.5
0.6
0.7
0.8
0.9
1
Iteration
Per
cent
cor
ectly
cla
ssifi
ed
SGD: m=50
SGD: m=100
SdLBFGS: m=50
SdLBFGS: m=75
SdLBFGS: m=100
Figure: Comparison of correct classification percentage by SGD andSdLBFGS with different batch size on RCV1 dataset. For SdLBFGS thestep size is αk = 10/k and the memory size is p = 10. For SGD the stepsize is αk = 20/k .
29 / 31
1 3 5 10 200
1
2
3
4
5
6
7
8
Memory size (batch size m=50)
Num
ber
of d
ampe
d st
eps
1 50 75 1000
2
6
10
100
308
1,000
Batch size (memory size p=10)
Num
ber
of d
ampe
d st
eps
Figure: The average number of damped steps over 10 runs of SdLBFGS.Here the maximum number of iterations is set as 1000 and step size is10/k .
30 / 31
Contributions
Our contributions:
A general framework of SQN for nonconvex problemConvergence guaranteeComplexity analysis for random output and constant step sizeStochastic damped L-BFGS falls into the framework
Future work for nonconvex problems:
develop a damped limited memory block BFGS methodVariance reduction techniques?
31 / 31