Adaptive Filters Algorithms (Part 2) - Uni Kiel › images › teaching › ... · Adaptive Filters –Algorithms This week and last week: Introductory Remarks Recursive Least Squares

Slide 1

Gerhard Schmidt

Christian-Albrechts-Universität zu KielFaculty of Engineering Electrical Engineering and Information TechnologyDigital Signal Processing and System Theory

Adaptive Filters – Algorithms (Part 2)

Slide 2Slide 2Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 2

Today:

Contents of the Lecture

Adaptive Algorithms:

Introductory Remarks

Recursive Least Squares (RLS) Algorithm

Least Mean Square Algorithm (LMS Algorithm) – Part 1


Affine Projection Algorithm (AP Algorithm)


Basics – Part 1

Least Mean Square (LMS) Algorithm

Optimization criterion:

Minimizing the mean square error

Assumptions:

Real, stationary random processes

Structure:

Unknown system

Adaptivefilter


Derivation – Part 2


Method according to Newton

What we have so far:

Resolving it to leads to:

With the introduction of a step size , the following adaptation rule can be formulated:


Derivation – Part 3


Method according to Newton:

Method of steepest descent:

LMS algorithm

For practical approaches the expectation value is replaced by its instantaneousvalue. This leads to the so-called least mean square (LMS) algorithm:


Upper Bound for the Step Size


A priori error:

A posteriori error:

Consequently:

For large and input processes with zero mean the following approximation is valid:


System Distance


Old system distance New system distance

How LMS adaptation changes system distance:

Target

Current system error vector


Sign Algorithm


Update rule:

with

Early algorithm with very low complexity (even used today in applications that operate at very high frequencies). It can be implemented without any multiplications (step size multiplication can be implemented as a bit shift).


Analysis of the Mean Value


Expectation of the filter coefficients:

If the procedure converges, the coefficients reach stationary end values:

So we have orthogonality:

Wiener solution


Convergence of the Expectations – Part 1


Into the equation for the LMS algorithm

and get:

we insert the equation for the error






Independence assumption:

Difference between means and expectations:

Convergence of the means requires:




= 0 because of Wiener solution

Recursion:

Convergence requires the contraction of the matrix:




Case 1: White input signal

Condition for the convergence of the mean values:

For comparison – condition for the convergence of the filter coefficients:

Convergence requires the contraction of the matrix (result from last slide):




Case 2: Colored input – assumptions




Putting the following results

together, leads to the following notation for the autocorrelation matrix:




Recursion:


Condition for Convergence – Part 1


Condition for the convergence of the expectations of the filter coefficients:

Previous result:


Condition for Convergence – Part 2


A (very rough) estimate for the largest eigenvalue:

Consequently:


Eigenvalues and Power Spectral Density – Part 1


Relation between eigenvalues and power spectral density:

Signal vector:

Autocorrelation matrix:

Fourier transform:

Equation for eigenvalues:

Eigenvalue:




… previous result …

… exchanging the order of the sums and the integral and splitting the exponential term …

… lower bound …

… upper bound …

Computing lower and upper bounds for the eigenvalues – part 1:





… exchanging again the order of the sums and the integral …

… solving the integral first …

… inserting the result und using the orthonormality properties of eigenvectors …




… exchanging again the order of the sums and the integral …


… inserting the result from above to obtain the upper bound …

… inserting the result from above to obtain the lower bound …

… finally we get…


Geometrical Explanation of Convergence – Part 1


System:

System output:

Structure:

Unknown system

Adaptivefilter




Error signal:

Difference vector:

LMS algorithm:




The vector will be split into two components:

It applies to parallel components:

With:




Contraction of the system error vector:

… result obtained two slides before …

… splitting the system error vector …

… using and that is orthogonal to …

… this results in …


NLMS Algorithm – Part 1


Normalized LMS algorithm:

LMS algorithm:

Unknown system

Adaptivefilter




Adaption (in general):

A priori error:

A posteriori error:

A successful adaptation requires

or:




Condition:

Ansatz:

Convergence condition:

Inserting the update equation:




Condition:

Ansatz:

Step size requirement fo the NLMS algorithm (after a few lines …):

For comparison with LMS algorithm:

or




Ansatz:

Adaptation rule for the NLMS algorithm:


Matlab-Demo: Speed of Convergence



Convergence Examples – Part 1


Setup:

White noise:




Setup:

Colored noise:




Setup:

Speech:


Today

Contents of the Lecture

Adaptive Algorithms:







Basics

Affine Projection Algorithm

Signal matrix:

Signal vector:

Filter vector:

Filter output:

M describes the order of the procedure

Unknown system


Signal Matrix


Definition of the signal matrix:


Error Vector – Part 1


Signal matrix:

Desired signal vector:

Filter output vector:

A priori error vector:

Adaption rule:

A posteriori error vector:


Error Vector – Part 2


Requirement:

Requirement:


Ansatz


Requirement:

Ansatz:

Step-size condition:


Geometrical Interpretation


NLMS algorithm AP algorithm


Regularization


Regularised version of the AP algorithm:

Non-regularised version of the AP algorithm:


Convergence of Different Algorithms – Part 1


White noise:




White noise:




Colored noise




Colored noise:




Speech:




Speech:


Summary and Outlook

Adaptive Filters – Algorithms

This week and last week:






Next week:

Control of Adaptive Filters

Documents

Adaptive Filters Algorithms (Part 2) - Uni Kiel › images › teaching › ... · Adaptive Filters –Algorithms This week and last week: Introductory Remarks Recursive Least Squares