Linear Least SquaresQR Factorization
Systems of linear equations
Problem to solve: M x = b
Given M x = b : Is there a solution? Is the solution unique?
Systems of linear equations
Find a set of weights x so that the weighted sum of the
columns of the matrix M is equal to the right hand side b
1 1
2 21 2 N
N N
x b
x bM M M
x b
1 1 2 2 N Nx M x M x M b
Systems of linear equations - Existence
A solution exists when b is in the span of the columns of M
A solution exists if:
There exist weights, x1, …., xN, such that:
bMxMxMx NN ...2211
Systems of linear equations - Uniqueness
A solution is unique only if the columns of M are linearly
independent.
Then: Mx = b Mx + My= b M(x+y) = b
Suppose there exist weights, y1, …., yN, not all zero, such that:
0...2211 NN MyMyMy
QR factorization 1
A matrix Q is said to be orthogonal if its columns are orthonormal, i.e. QT·Q=I.
Orthogonal transformations preserve the Euclidean norm since
Orthogonal matrices can transform vectors in various ways, such as rotation or reflections but they do not change the Euclidean length of the vector. Hence, they preserve the solution to a linear least squares problem.
QR factorization 2
Any matrix A(m·n) can be represented as
A = Q·R
,where Q(m·n) is orthonormal and R(n·n) is upper triangular:
nn
n
nn
r
r
rrr
qqaa
000
00
0|...||...| 11
11211
11
QR factorization 2
Given A , let its QR decomposition be given as A=Q·R, where Q is an (m x n) orthonormal matrix and R is upper triangular.
QR factorization transform the linear least square problem into a triangular least squares.
Q·R·x = b
R·x = QT·b
x=R-1·QT·b
Matlab Code:
Normal Equations
Consider the system
It can be a result of some physical measurements, which usually incorporate
some errors. Since, we can not solve it exactly, we would like to minimize the
error:
r=b-Ax
r2=rTr=(b-Ax)T(b-Ax)=bTb-2xTATb+xTATAx
(r2)x=0 - zero derivative is a (necessary) minimum condition
-2ATb+2ATAx=0;
ATAx = ATb; – Normal Equations
5.3
5.2
2
21
12
11
x
Normal Equations 2
ATAx = ATb – Normal Equations
5.3
5.2
2
21
12
11
x
65
56
21
12
11
211
121AAT
...,6
55.105.11
11
6
6
55.105.11
6
510
5.1056
5.1165
5.1056212
xxx
5.11
5.10
5.3
5.2
2
211
121bAT
Least squares via A=QR decomposition
A(m,n)=Q(m,n)R(n,n), Q is orthogonal, therefore QTQ=I.
QRx=b
R(n,n)x=QT(n,m)b(m,1) -well defined linear system
x=R-1QTb
Q is found by Gram=Schmidt orthogonalization of A. How to find R?
QR=A
QTQR=QTA, but Q is orthogonal, therefore QTQ=I:
R=QTA
R is upper triangular, since in orthogonalization procedure only
a1,..ak (without ak+1,…) are used to produce qk
Least squares via A=QR decomposition 2
Let us check the correctness:
QRx=b
Rx=QT
b
x=R-1
QT
b
10
11
01
A
3
20
6
1
2
16
1
2
1
Q
6
1
3
20
2
1
2
2
10
11
01
3
2
6
1
6
1
02
1
2
1
AQR T
10
11
01
18
2
3
20
18
2
6
1
2
11
18
2
6
1
2
11
6
1
3
20
2
1
2
2
3
20
6
1
2
16
1
2
1
QR
4M
3M
2M
1M
12 13 14r r r
1Q
2Q
3Q
4Q
11r
22r 23 24r r
33r
44r
34r
Last lecture reminderQR Factorization – By picture
For i = 1 to N “For each Target Column”
For j = 1 to i-1 “For each Source Column left of target”
end
end
1
ii i
irp p
iii iMpr Mp
Normalize
i ix x v p
i ip e
Tjj
Tiir p M Mp
i ji i jp p pr Orthogonalize Search Direction
QR Factorization – Minimization ViewMinimization Algorithm