1
DSP-CIS
Chapter-13: Least Mean Squares (LMS) Algorithm
Marc Moonen Dept. E.E./ESAT-STADIUS, KU Leuven
[email protected] www.esat.kuleuven.be/stadius/
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 2
Part-III : Optimal & Adaptive Filters : Optimal & Adaptive Filters - Intro
• General Set-Up • Applications • Optimal (Wiener) Filters
– : Least Squares & Recursive Least Squares Estimation • Least Squares Estimation • Recursive Least Squares (RLS) Estimation • Square-Root Algorithms
– : Least Means Squares (LMS) Algorithm • LMS/NLMS : Stochastic Gradient Algorithms • LMS analysis • LMS Family
– : Fast Recursive Least Squares Algorithms : Kalman Filtering
Chapter-11
Chapter-12
Chapter-13
Chapter-14
Chapter-15
2
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 3
Least Mean Squares (LMS) Algorithm
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 4
Least Mean Squares (LMS) Algorithm
3
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 5
Least Mean Squares (LMS) Algorithm
6 6 6 6
b
b
-w
a-bw
w
input signal
6
a
6
µ
6
- - - -
b b
aw
w+ab
u[k] u[k-1] u[k-2] u[k-3]
w0[k-1] w1[k-1] w2[k-1] w3[k-1]
w0[k] w1[k] w2[k] w3[k]
output signal
e[k] d[k]
desired signal
(Widrow 1965 !!)
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 6
Least Mean Squares (LMS) Algorithm
Bernard Widrow https://www.youtube.com/watch?v=hc2Zj55j1zU
4
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 7
Least Mean Squares (LMS) Algorithm
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 8
Least Mean Squares (LMS) Algorithm
5
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 9
Least Mean Squares (LMS) Algorithm
è large λ_max implies a small stepsize
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 10
Least Mean Squares (LMS) Algorithm
h
minh
1<µh
max
-1
1µ=0
µ=2/h
µ
µ=1/h max
opt
max
6
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 11
Least Mean Squares (LMS) Algorithm
error vector projected onto eigenvectors initial error vector projected onto eigenvectors
(=projection on i-th eigenvector)
è small λ_i implies slow convergence è λ_min <<λ_max (hence small µ) implies *very* slow convergence
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 12
Least Mean Squares (LMS) Algorithm
w(4)
w(0)
wopt
w1
w2
q2q1
7
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 13
Least Mean Squares (LMS) Algorithm
w(4)
wopt
w(0)
w1
w2
q1
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 14
Least Mean Squares (LMS) Algorithm
w(0)
w(4)
wopt
w1
w2
q1
8
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 15
Least Mean Squares (LMS) Algorithm
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 16
Least Mean Squares (LMS) Algorithm
9
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 17
Least Mean Squares (LMS) Algorithm
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 18
Least Mean Squares (LMS) Algorithm
10
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 19
Least Mean Squares (LMS) Algorithm
Skip
this
slid
e
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 20
Least Mean Squares (LMS) Algorithm
Skip
this
slid
e
11
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 21
Least Mean Squares (LMS) Algorithm
Skip
this
slid
e
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 22
Least Mean Squares (LMS) Algorithm
Skip
this
slid
e
12
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 23
Least Mean Squares (LMS) Algorithm
Skip
this
slid
e
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 24
Least Mean Squares (LMS) Algorithm
Skip
this
slid
e
13
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 25
Least Mean Squares (LMS) Algorithm
66 6 6 6
b
wb
-w
aa-bw
6
6
6
6
6
6
6
W^1 W^2 W^3 W^4 W^5 W^6 W^7
W^2 W^4 W^6 W^8 W^10 W^12 W^14
6
W^6 W^9 W^12 W^15 W^18 W^21
6 6
W^3
00000000
W^0 W^0 W^0 W^0 W^0 W^0W^0W^0
W^0
W^0
W^0
W^0W^0
W^0
W^0
W^0
w0
- - -
w2 w3
b b
aw
w+ab
--
w1
- - -
w4 w5 w6 w7
desired signal
filter input
error
W^4 W^8 W^12 W^16 W^20 W^24 W^28
W^5
W^6
W^7
W^10 W^15 W^20 W^25 W^30 W^35
W^12 W^18 W^24 W^30 W^36 W^42
W^14 W^21 W^28 W^35 W^42 W^49
µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7
Skip
this
slid
e
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 26
Least Mean Squares (LMS) Algorithm
+
+
+
++
+
+
+
+
+
+
+
+
++ +
+
+
+
+
+
+
+
+
6
6
-1
-1
-1
-1
6 6 6 66 6 6 6
6
b
wb
-w
a
6
6
6
6
a-bw
filter input
W^1
W^2
W^3
-1
-1
-1
-1
-1
-1
-1
-1
W^2
W^2
w0
- - -
w2 w3
b b
aw
w+ab
--
w1
- - -
w4 w5 w6 w7
desired signalerror
µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7
Skip
this
slid
e
14
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 27
Least Mean Squares (LMS) Algorithm
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-1
-1
-1
-1
6 6 66 6 66 6
b
b
-w
aa-bw
6
w
6
6
6
6
6
66
6
6
filter input
-1
-1
-1
-1
W^2
W^2
w0
- - -
w2 w3
b b
aw
w+ab
--
w1
- - -
w4 w5 w6 w7
desired signalerror
µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7
W^1
W^2
W^3
-1
Skip
this
slid
e
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 28
Least Mean Squares (LMS) Algorithm
+
+
+
+
+
+
+
++
+
++
+
+
-1
6
-1
66 66 6 66
b
-w
aa-bw
-1
-1
b
6
6
w
6
6
6
-1
6
-1
6
6
6
6
6
6
filter input
w0
- - -
w2 w3
b b
aw
w+ab
--
w1
- - -
w4 w5 w6 w7
desired signalerror
µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7
-1
W^2
W^2
W^1
W^1.W^2
Skip
this
slid
e
15
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 29
Least Mean Squares (LMS) Algorithm
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 30
Least Mean Squares (LMS) Algorithm
16
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 31
Least Mean Squares (LMS) Algorithm
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 32
Least Mean Squares (LMS) Algorithm
17
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 33
Least Mean Squares (LMS) Algorithm
DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 34
Least Mean Squares (LMS) Algorithm
6 6
6 6 6 6
6 6
6 6 6 6
2 2
HP HP
2 2
LP
2
LP
2
filter input desired signal
xµ
6
w0
- - - -
w1 w2 w3
xµ
6
w0
- - - -
w1 w2 w3
error (HP)
LP
+error
error (LP)
HP