Upload
others
View
12
Download
0
Embed Size (px)
Citation preview
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Model Correction using a NuclearNorm Constraint
Ning Hao, Lior Horesh, Misha Kilmer
Ragon InstituteIBM Thomas J. Watson Research Center
Tufts UniversityThanks: NSF:CIF:SMALL 1319653, NSF-DMS 0914957
Householder Symposium XIX, June 2014
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
MotivationWe create mathematical models to simulate the operation of areal world or system.
Not surprisingly, the models will contain some error:
• simplifications for computational tractability
• linearization, numerical error
• boundary conditions or geometry
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
MotivationWe create mathematical models to simulate the operation of areal world or system.
Not surprisingly, the models will contain some error:
• simplifications for computational tractability
• linearization, numerical error
• boundary conditions or geometry
“All models are wrong . . . but some are useful “George E. P. Box, 1987
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
MotivationWe create mathematical models to simulate the operation of areal world or system.
Not surprisingly, the models will contain some error:
• simplifications for computational tractability
• linearization, numerical error
• boundary conditions or geometry
“All models are wrong . . . but some are useful “George E. P. Box, 1987
Thinking outside the Box, some models can be made lesswrong / more useful. This is the subject of today’s talk.
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Problem DefinitionLet F : Rn → Rm be “true” operator taking input x ∈ Rn toobservable space. Let d ∈ Rm be an observation obtained as
d = F(x) + ε
where ε represents measurement noise.
Goal: Formulate and solve a model correction optimizationproblem to recover an F that is only partially specified.In this talk, assume A is known and
F(x) = A(x) +B(x).
Model correction problem is to recover B. Need additionalconstraints on B to obtain a well-posed optimization problem.
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Motivating Test ProblemDiscrete ill-posed problem (e.g. deblurring, Xray-CT)
Ax = d+ ε.
If operator not known exactly model should be replaced by
(A+B)x = d+ ε.
While B is unknown, and in general x is unknown, we assumewe can sample input space and have access to correspondingoutput realizations for each of the samples.
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Sample Average ApproximationSAA[1]1: Expectation wrt the input space & the measurementnoise approximated by a sample ave. est. of a random sample.
B̂ = argminB
1
nxnε
nx,nε∑j=1,j=1
D ((A(xi) +B(xi))− dij)
subject to constraint on B, nx is the number of input draws,nε is number of data realizations for each input realization.
Problem (A(x) = Ax,B(x) = Bx,D(y) = ‖y‖22)
B̂ = argminB
1
nxnε
nx,nε∑i=1,j=1
‖(A+B)xi − di,j‖22,
subject to a structural constraint on B .1[1]Shapiro, Dentcheva, Ruszczynski, editors. “Lecture Notes on
Stochastic Programming: Modeling and Theory.” SIAM, 2009.Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Restrictions on Operator StructureMany constraints on B possible. Choose rank-based:
Problem
B̂ =argminB‖(A+B)X −D‖2F
s.t. rank(B) ≤ δ
2
Relax the constraint:
Problem
B̂ =argminB||(A+B)X −D||2F
s.t. ‖B‖∗ ≤δ
2
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Optimization Problem with Nuclear NormRegularization
Several options for solving the optimization problem[1]2
• Interior Point Methods
• Alternating Direction Method of Multipliers (ADMM)
• Projected Sub-Gradient Method
• More recent work by Jaggi and Sulovsky: recast theoptimization problem over positive semidefinite matriceswith unit trace and then apply Hazan’s algorithm.
2[1]Hao, Horesh, & K., “Nuclear norm optimization and its applicationto observation model specification,” in Compressed Sensing and SparseFiltering, 2014
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Hazan’s Algorithm1
Hazan’s Algorithm deals with problems of the form:
minZ∈S
f(Z)
where f is convex and S is the set of all symmetric positivesemidefinite d× d matrices with unit trace.
Each iteration involves the calculation of a single approximateeigenvector of a matrix of size of d× d.
1Sparse approximation solutions to semidefinite programs by Hazan,E.2008
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
AlgorithmFrom Jaggi and Sulovsky “A Simple Algorithm for NuclearNorm Regularized Problem” 2010, for any non-zero matrixB ∈ Rn×m and δ ∈ R:
‖B‖∗ ≤δ
2⇐⇒ ∃ symmetric M,Ns.t.(M BBT N
)� 0
andTr(M) + Tr(N) = δ.
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Steps
Let Z =
(M BBT N
).
• Define f̂(Z) = f(B) = ‖(A+B)X −D‖2F ,
• Want to solveminZf̂(Z)
s.t. Z ∈ S(m+n)×(m+n), Z � 0,Tr(Z) = δ
• Scale all matrix entries by 1δ
• Apply Hazan’s algorithm, then unscale
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
AlgorithmInput: f , v0 ∈ R(m+n)×1 with ||v0|| = 1; scaled matricesInitialize: Z1 = v0v
T0
for k = 1 until convergence doExtract Bk = Zk(1 : m,m+1:m+n)Compute ∇fk := ((A+Bk)X −D)XT
We have ∇f̂k :=(
0 ∇fk∇fTk 0
)Compute vk := eigs(−∇f̂k, 1,′ LA′)Line search for step length akUpdate Zk+1 = Zk + ak(vkv
Tk − Zk)
end forReturn B̂ = δ · Z(1 :m,m+1:m+n)
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Numerical Examples
• Mimic the semi-blind deconvolution problem, only anapproximation to blurring operator is known a-priori.
• 100 MR images from the Auckland MRI Research groupdatabase. 3
• Pre-process the images by cropping the watermark andresizing the cropped images to the size of 55× 55.
• Randomly choose 80 images (sampled integers from1:100 uniformly) to serve as the training set and 20 ofremainder as the test set.
3http://atlas.scmr.org/download.htmlModel Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Example 1
• Let T (bw, σ) be symmetric, doubly block Toeplitz matrixT that models blurring of an image by a Gaussian pointspread function; bw, σ control (block) bandwidth and blur.
• A = T (4, 4)
• B defined from singular triples of T (3, 2) numbered 110to 120.
• Compute D = (A+B)X +N where N ’s columns havezero mean, white Gaussian noise such that the noise tosignal ratio for all measured data is .5 percent.
• Run 120 iterations. Compare the best TSVD regularizedsolutions using the SVDs of A,A+B,A+ B̂.
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
True Blurred Training Images
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
AX vs (A +B)X
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Rel. Errors, TSVD recons, 20 from test set
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Example 1, Test Slice 16
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Rel. Errors after 400 iterations
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Enforcing Different StructureWhat if B(x) 6= Bx, but rather B(x) = K[B]x?
Example [Kamm & Nagy, 1998] Let v be of length n, n odd,
toep(v) creates an n× n banded Toeplitz matrix with v(n+12)
as the diagonal entry:
toep
abc
=
b a 0c b a0 c b
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Additional Structure from B
Given n× n B, make a BTTB matrix from blocks toepB(:, i)in a similar manner.
For B with 3 columns:
K[B] =
toep(B(:, 2)) toep(B(:, 1)) 0toep(B(:, 3)) toep(B(:, 2)) toep(B(:, 1))
0 toep(B(:, 3)) toep(B(:, 2))
.
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Convexity, Constraint RevisitedIt is possible to show thatK[B]x = K[reshape(x, n, n)]vec(B).
So,∑‖Axij +B(xij)− dij‖2F
still convex in entries of B
Significance of the (ideal) rank-based constraint on B:
B =k∑i=1
σiuivTi ⇒ K[B] =
k∑i=1
toep(√σivi)⊗ toep(
√σiui),
Algorithm now cheap - matrix of partials is now n× n,matvecs with FFTs.
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Example 2
• A = T (10, 4)
• B = T (7, 3.1) + T (4, 2.2) + T (5, 3.5)
• Same testing setup as before (random selection of 80images) and testing on the remainder, Gaussian 1
2percent
noise added (once per test image).
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Relative Squared Convergence Error
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Difference in Data Space: |BXtrain|
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Difference in Data Space: |(B − B̂)Xtrain|
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Errors on 20 Test Data
‖AX−(A+B)X‖F‖(A+B)X‖F
vs. ‖(B̂−B)X‖F‖(A+B)X‖F
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
One Extension to TensorsB is order > 2 tensor from which we define K[B](x).
B ∈ Rn×n×n, B =∑r
i=1 ui ◦ vi ◦ wi ⇒ K[B] to a sum of3-way Kronecker products of structured matrices
• Nagy & K., “Kronecker Product Approximation forThree-Dimensional Imaging Applications, IEEE TIP 2006.
• Rezghi & Elden, “A Kronecker Product Approximation ofthe Blurring Operator in the Three Dimensional ImageRestoration Problem,” SIMAX to appear.
But what do we mean by ‖B‖∗?
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
One Extension to TensorsB is order > 2 tensor from which we define K[B](x).
B ∈ Rn×n×n, B =∑r
i=1 ui ◦ vi ◦ wi ⇒ K[B] to a sum of3-way Kronecker products of structured matrices
• Nagy & K., “Kronecker Product Approximation forThree-Dimensional Imaging Applications, IEEE TIP 2006.
• Rezghi & Elden, “A Kronecker Product Approximation ofthe Blurring Operator in the Three Dimensional ImageRestoration Problem,” SIMAX to appear.
But what do we mean by ‖B‖∗?
More thinking outside the Box....
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
One Extension to TensorsB is order > 2 tensor from which we define K[B](x).
B ∈ Rn×n×n, B =∑r
i=1 ui ◦ vi ◦ wi ⇒ K[B] to a sum of3-way Kronecker products of structured matrices
• Nagy & K., “Kronecker Product Approximation forThree-Dimensional Imaging Applications, IEEE TIP 2006.
• Rezghi & Elden, “A Kronecker Product Approximation ofthe Blurring Operator in the Three Dimensional ImageRestoration Problem,” SIMAX to appear.
But what do we mean by ‖B‖∗?
Use ‖B‖TNN definition [Hao, 2014] based on tSVD [K. &Martin, 2011].
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Current & Future WorkOther examples (PDEs) (thanks: Raya Horesh, IBM Watson)
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer
Motivation Operator Correction Problem Algorithm Results:I Additional Constraint Results: II Current Work
Current & Future Work
• Tensors
• Other examples (PDES)
• Other choices for D• Investigate other possible constraints on the operator
(nonlinear correction).
Model Correction using a Nuclear Norm Constraint Ning Hao, Lior Horesh, Misha Kilmer