With Applications in Image Processing

With Applications in With Applications in Image ProcessingImage Processing

Guest LectureDr. Robi Polikar

OutlineOutline

Review of Fourier based methods Analyzing non-stationary signals

Short-time Fourier transform Continuous wavelet transform Discrete wavelet transform Applications

Discontinuity detectionSignal denoisingCompression

2-D wavelet transformImage analysis and compression

What is a TransformWhat is a Transformand Why Do we Need One ?and Why Do we Need One ?

Transform: A mathematical operation that takes a function or sequence and maps it into another one

Transforms are good things because… The transform of a function may give additional /hidden

information about the original function, which may not be available /obvious otherwise

The transform of an equation may be easier to solve than the original equation (recall Laplace transforms for “Defeques”)

The transform of a function/sequence may require less storage, hence provide data compression / reduction

An operation may be easier to apply on the transformed function, rather than the original function (recall convolution)

Properties of TransformsProperties of Transforms

Most useful transforms are: Linear: we can pull out constants, and apply

superposition

One-to-one: different functions have different transforms Invertible:for each transform T, there is an inverse

transform T-1 using which the original function f can be recovered (kind of – sort of the undo button…)

Continuous transform: map functions to functions Discrete transform: map sequences to sequences Semi-discrete transform: relate functions with sequences

Tf F T-1 f

( ) ( ) ( )T f g T f T f

What Does a Transform What Does a Transform Look Like…?Look Like…?

Complex function representation through simple building blocks

Compressed representation through using only a few blocks (called basis functions / kernels)

Sinusoids as building blocks: Fourier transform Frequency domain representation of the

function

)(),()(

s)(continuou )(),()(

(discrete)

dxxfxKF

dFxKxf

fKFKFf

ionBasisFunctweightFunctionComplex

jjijiij

ii

ii

i

Fourier series

Continuous Fourier transform

Discrete Fourier transform

Laplace transform

Z-transform

What Transforms Are Available?What Transforms Are Available?

1

0

/2/2 ][21)( )(][

N

i

NktjNktj ekFtfdtetfkF

deFtfdtetfF tjtj )(21)( )()(

1

0

/21

0

/2 ][21 ][][

N

k

NkjN

n

Nkj ekFf[n]enfkF

dsesFtfdtetfsF stst )(21)( )()(

1

0

1

0][

21 ][][

N

z

znN

n

zn ezFf[n]enfzF

How Does FT Work Anyway?How Does FT Work Anyway?

Recall that FT uses complex exponentials (sinusoids) as building blocks.

For each frequency of complex exponential, the sinusoid at that frequency is compared to the signal.

If the signal consists of that frequency, the correlation is high large FT coefficients.

If the signal does not have any spectral component at a frequency, the correlation at that frequency is low / zero, small / zero FT coefficient.

deFtfdtetfF tjtj )(21)( )()(

tjte tj sincos

FT At WorkFT At Work

)52cos()(1 ttx

)252cos()(2 ttx

)502cos()(3 ttx

FT At WorkFT At Work

)(1 X

)(2 X

)(3 X

F)(1 tx

F)(2 tx

F)(3 tx

FT At WorkFT At Work

)502cos()252cos(

)52cos()(4

tt

ttx

)(4 XF)(4 tx

Stationary and Stationary and Non-stationary SignalsNon-stationary Signals

FT identifies all spectral components present in the signal, however it does not provide any information regarding the temporal (time) localization of these components. Why?

Stationary signals consist of spectral components that do not change in time all spectral components exist at all timesno need to know any time informationFT works well for stationary signals

However, non-stationary signals consists of time varying spectral componentsHow do we find out which spectral component appears

when?FT only provides what spectral components exist , not

where in time they are located.Need some other ways to determine time localization of

spectral components

Stationary and Stationary and Non-stationary SignalsNon-stationary Signals

Stationary signals’ spectral characteristics do not change with time

Non-stationary signals have time varying spectra

)502cos()252cos(

)52cos()(4

tt

ttx

][)( 3215 xxxtx

Concatenation

Non-stationary Signals Non-stationary Signals

5 Hz 20 Hz 50 Hz

Perfect knowledge of what frequencies exist, but no information about where these frequencies are located in time

FT ShortcomingsFT Shortcomings

Complex exponentials stretch out to infinity in timeThey analyze the signal globally, not locallyHence, FT can only tell what frequencies exist

in the entire signal, but cannot tell, at what time instances these frequencies occur

In order to obtain time localization of the spectral components, the signal need to be analyzed locally

HOW ?

Short Time Short Time Fourier Transform (STFT)Fourier Transform (STFT)

1. Choose a window function of finite length2. Put the window on top of the signal at t=03. Truncate the signal using this window4. Compute the FT of the truncated signal, save.5. Slide the window to the right by a small amount6. Go to step 3, until window reaches the end of the signal For each time location where the window is centered, we

obtain a different FT Hence, each FT provides the spectral

information of a separate time-slice of the signal, providing simultaneous time and frequency information

300 Hz 200 Hz 100Hz 50Hz

STFT ExampleSTFT Example

2/2

)( atet

STFT ExampleSTFT Example

a=0.01

STFT ExampleSTFT Example

a=0.001

STFT ExampleSTFT Example

a=0.0001

STFT ExampleSTFT Example

a=0.00001

STFT ExampleSTFT Example

Time – Frequency Time – Frequency ResolutionResolution

Time – frequency resolution problem with STFTAnalysis window dictates both time and

frequency resolutions, once and for allNarrow window Good time resolutionNarrow band (wide window) Good

frequency resolution When do we need good time resolution, when do we

need good frequency resolution?

Heisenberg PrincipleHeisenberg Principle

41

ft

Time resolution: How well two spikes in time can be separated from each other in the transform domain

Frequency resolution: How well two spectral components can be separated from each other in the transform domain

Both time and frequency resolutions cannot be arbitrarily high!!! We cannot precisely know at what time instance a frequency component is located. We can only know what interval of frequencies are present in which time intervals

The Wavelet TransformThe Wavelet Transform

Overcomes the preset resolution problem of the STFT by using a variable length window

Analysis windows of different lengths are used for different frequencies:Analysis of high frequencies Use narrower

windows for better time resolutionAnalysis of low frequencies Use wider windows

for better frequency resolution This works well, if the signal to be analyzed mainly consists of slowly varying

characteristics with occasional short high frequency bursts. Heisenberg principle still holds!!! The function used to window the signal is called the wavelet

1:44,500,000 1:2,500,000

1:62,5001:375,500

frequencyscale 1

Continuous Wavelet TransformContinuous Wavelet Transform

dtabttx

abaWbaCWTx )(1),(),()(

Normalization factor

Mother wavelet translation

Scaling:Changes the support of the wavelet based on the scale (frequency)

CWT of x(t) at scalea and translation bNote: low scale high frequency

Why Wavelet?Why Wavelet?

We require that the wavelet functions, at a minimum, satisfy the following:

0)( dtt

dtt 2)(

Wave…

…let

Time-Frequency GridTime-Frequency Gridin CWTin CWT

Time

Scal

e

Δf/2

Δf

2Δf

4Δf

Δt

2Δt

4Δt

Δt/2

Δt/4a=1/4

a=1/2

a=1

a=2

Δt * Δf ≥ ½

Matlab Demos on CWTMatlab Demos on CWT

Discretization of Discretization of Time & Scale ParametersTime & Scale Parameters

Recall that, if we use orthonormal basis functions as our mother wavelets, then we can reconstruct the original signal by

where W(a,b) is the CWT of x(t) Q: Can we discretize the mother wavelet a,b(t) in such a way that

a finite number of such discrete wavelets can still form an orthonormal basis (which in turn allows us to reconstruct the original signal)? If yes, how often do we need to sample the translation and scale parameters to be able to reconstruct the signal?

A: Yes, but it depends on the choice of the wavelet!

)(),,()( , tbaWtx ba

Dyadic GridDyadic Grid We do not have to use a uniform sampling rate for the translation

parameters, since we do not need as high time sampling rate when the scale is high (low frequency).

Let’s consider the following sampling grid:

log a

b where a is sampled on a log scale, and b is sampled at a higher rate when a is small,that is,

where a0 and b0 are constants,and j,k are integers.

00

0

bakb

aaj

j

An example on MRAAn example on MRA

dftfk

kjj

j

j

)(21)(

)1(2

2

Zkjktk jjj

,222

0)(lim

)()(lim

tf

tftf

jj

jj

Let’s suppose we wish to approximate the function f(t) using a simpler function that has constant values over set intervals of 2j, such that the error ||f(t)-fj(t)|| is minimum for any j. The best approx. would be the average of the function over the interval length. In general, then…

j=

j=-

j=-2 j=-1

j=0 j=1

j=2 j=4

j=5

f(t)

f0(t)

f2(t)

An Example (cont.)An Example (cont.) As we go from a value of j to a higher value, we obtain a coarser

approximation of the function, hence some detail is lost. Let’s denote the detail lost at each approximation level with a

function g(t):

)()()()( 11 tftftftf jj

)()()( 1 tftftg jjj

Detail functionApproximation function

)()()(

)()()(

010

121

tftftg

tftftg jjj

jj tgtf )()( That is, the original function

can be reconstructed simply by adding all the details (!)

)0( f

Scaling & Wavelet FunctionsScaling & Wavelet Functions

dtttxkja kj )()(),( ,

Approximation coefficientsat scale j

kkjjj tkjatxtf )(),()()( ,

Smoothed, scaling-function-dependentapproximation of x(t) at scale j)2(2)( 2/

, ktt jjkj

)()( ,, tdtgk

kjkjj

kj

kj

d

dtttxbaW

,

, )()(),(

ktt jjkj 22)( 2/

, Detail coefficients

at scale j

Wavelet-function-dependentdetail of x(t) at scale j

Furthermore:

jj tgtf )()( & )()( ,, tdtg

kkjkjj

j kkj tkjdtxtf )(),()()( , Wavelet

Reconstruction

The scaling function

The wavelet function

Putting ItPutting ItAll Together (cont.)All Together (cont.)

We can start reconstruction from any level (scale) in between:

0

0 0, , ,( ) ( ) ( , ) ( )j

j k j k j kk j k

f t x t a d j k t

Approximation coefficientsat some arbitrary scale j0

Smoothed, scaling-function-dependentapproximation of x(t) at the scale j0

Detail coefficientsat scale j0 and below

Wavelet-function-dependentdetails of x(t) at scales j0 and below

0 0j jf t x t 0j

g t

0

0 0

j

j jj

x t x t g t

The essence of wavelet transform: A signal can be obtained from (or decomposed into) an approximation at a specific level, and sum of details lost up to that level.

Discrete Wavelet TransformDiscrete Wavelet TransformImplementationImplementation

G

H

2

2 G

H

2

2

2

2

G

H

+

2

2

G

H

+

x[n]x[n]

Decomposition Reconstruction

~

~ ~

~

n

high kngnxky ]2[][][~

n

low knhnxky ]2[][][~

k

high kngky ]2[][

k

high kngky ]2[][

2-level DWT decomposition. The decomposition can be continues as 2-level DWT decomposition. The decomposition can be continues as long as there are enough samples for down-sampling.long as there are enough samples for down-sampling.

G

H

Half band high pass filter

Half band low pass filter

2

2

Down-sampling

Up-sampling

DWT - DemystifiedDWT - DemystifiedLength: 512B: 0 ~

g[n] h[n]

g[n] h[n]

g[n] h[n]

2

d1: Level 1 DWTCoeff.

Length: 256B: 0 ~ /2 HzLength: 256

B: /2 ~ Hz

Length: 128B: 0 ~ /4 HzLength: 128

B: /4 ~ /2 Hz

d2: Level 2 DWTCoeff.

d3: Level 3 DWTCoeff.

…a3….

Length: 64B: 0 ~ /8 HzLength: 64

B: /8 ~ /4 Hz

2

2 2

22

|H(jw)|

w/2-/2

|G(jw)|

w- /2-/2a2

a1

Level 3 approximation

Coefficients

Matlab ImplementationMatlab ImplementationCommand LineCommand Line

Single level decomposition and reconstruction [cA1 cD1]=dwt(x, ‘db4’) single level decomposition x=idwt(cA1, cD1, ‘db4’):3 single level inverse transform

Direct reconstruction of approximation and detail signals D1=upcoef(‘d’, cD1, ‘db4’, 1, N): one-level reconstruction of detail

signal from detail coefficient A1=upcoef(‘a’, cA1, ‘db4’, 1, N): one level reconstruction of

approximation signal from approximation coefficients Multi-level decomposition & reconstruction

[C L]=wavedec(x, K, ‘db4’): k-level decomposition A0=waverec(C, L, ‘db4’): reconstruct the signal

Extracting coefficients cA3=appcoef(C, L, ‘db4’, 3); cD5=detcoef(C, L, 5);

Single branch reconstruction A3=wrcoef(‘a’, C, L, ‘db4’, 3); Reconstruct level 3 approx. signal D2=wrcoef(‘d’, C, L, ‘db4’, 2); Reconstruct level 2 detail signal

A3 D3 D2 D1

L=[lA3 lD3 lD2 lD1 lsignal]

lA3 lD3 lD2 lD1

C=

Quadrature Mirror FiltersQuadrature Mirror Filters

It can be shown that

that is, h[] and g[] filters are related to each other: in fact, that is, h[] and g[] are mirrors of each

other, with every other coefficient negated. Such filters are called quadrature mirror filters. For example, Daubechies wavelets with 4 vanishing moments…..

1)()(1)()(2~2~22 jGjHjGjH

][)1(]1[ ngnLh n

DB-4 WaveletsDB-4 Wavelets

h = -0.0106 0.0329 0.0308 -0.1870 -0.0280 0.6309 0.7148 0.2304

g = -0.0106 -0.0329 0.0308 0.1870 -0.0280 -0.6309 0.7148 -0.2304

h = 0.2304 0.7148 0.6309 -0.0280 -0.1870 0.0308 0.0329 -0.0106

g = -0.0106 -0.0329 0.0308 0.1870 -0.0280 -0.6309 0.7148 -0.2304

~~

][][ ],[][

][)1(]1[~~

ngngnhnh

ngnLh n

L: filter length (8, in this case)

Implementation of DWT on Implementation of DWT on MATLABMATLAB

Load signal

Choose waveletand numberof levels

Hit Analyze button

Level 1 coeff.Highest freq.

Approx. coef. at level 5

s=a5+d5+…+d1

(Wavedemo_signal1)

Details vs. ApproximationsDetails vs. Approximations

ApplicationsApplications

Detect discontinuities

ApplicationsApplications

Detect hiddendiscontinuities

ApplicationsApplications

Simpledenoising

CompressionCompression

DWT is commonly used for compression, since most DWT are very small, can be zeroed-out!

CompressionCompression

1D-DWT1D-DWT2D-DWT2D-DWT

Recall fundamental concepts of 1D-DWTProvides time-scale (frequency) representation of

non-stationary signalsBased on multiresolution approximation (MRA)

• Approximate a function at various resolutions using a scaling function, (t) • Keep track of details lost using wavelet functions, (t)• Reconstruct the original signal by adding approximation and detail coeff.

Implemented by using a series of lowpass and highpass filters

• Lowpass filters are associated with the scaling function and provide approximation

• Highpass filters are associated with the wavelet function and provide detail lost in approximating the signal

2-D DWT2-D DWT

How do we generalize these concepts to 2D? 2D functions images f(x,y) I[m,n] intensity function What does it mean to take 2D-DWT of an image? How do we

interpret?How can we represent an image as a function?How do we define low frequency / high frequency in

an image?How to we compute it?

Why would we want to take 2D-DWT of an image anyway?CompressionDenoisingFeature extraction

2D Scaling/Wavelet Functions2D Scaling/Wavelet Functions

We start by defining a two-dimensional scaling and wavelet functions:

If (t) is orthogonal to its own translates,

is also orthogonal to its own translates. Then, if fo(x,y) is the projection of f(x,y) on the space Vo generated by s(x,y):

)()(),( yxyxs

Zlklykxs ,:),(

),(),,(),(

),(),(),(

jyixsyxfjia

jyixsjiayxf

o

i joo

)()(),( yxyxs

2D-DWT2D-DWT

Just like in 1D we generated an approximation of the 2D function f(x,y). Now, how do we compute the detail lost in approximating this function?

Unlike 1D case there will be three functions representing the details lost:Details lost along the horizontal directionDetails lost along the vertical directionDetails lost along the diagonal direction

1D Two sets of coeff.; a(k,n) & d (k,n) 2D Four sets of coefficients: a(k,n), b(k, n), c(k, n) & d(k,n)

Four Faces of 2D-DWTFour Faces of 2D-DWT

One level of 2D DWT reconstruction:

)()(),()()(),()()(),(

)()(),(),(1

nxnxpndnxnxpncnxnxpnb

nxnxpnayxf

o

o

o

n po

Approximation coefficients

Detail coefficients along the horizontal directionDetail coefficients along the vertical directionDetail coefficients along the diagonal direction

Implementation of 2D-DWTImplementation of 2D-DWT

INPUTIMAGE…

…

……

ROW

S

COLUMNSH~ 2 1

G~ 2 1

H~ 1 2

G~ 1 2

H~ 1 2

G~ 1 2

ROWS

ROWS

COLUMNS

COLUMNS

COLUMNS

LL

LH

HL

HH

)(1

hkD

)(1

vkD

)(1

dkD

1kA

INPUTIMAGE

LL LH

HL HH

LLLH

HL HH

LHH

LLH

LHL

LLLLH

HL HH

LHH

LLH

LHL

Up and Down … Up and DownUp and Down … Up and Down

2 1Downsample columns along the rows: For each row, keep the even indexed columns, discard the odd indexed columns

1 2Downsample columns along the rows: For each column, keep the even indexed rows, discard the odd indexed rows

2 1

1 2

Upsample columns along the rows: For each row, insert zeros at between every other sample (column)

Upsample rows along the columns: For each column, insert zeros at between every other sample (row)

Implementing 2D-DWTImplementing 2D-DWTDecomposition

ROW i

COLU

MN

j

ReconstructionReconstruction

)(1

hkD

)(1

vkD

)(1

dkD

1kA 1 2

1 2

1 2

1 2

H

G

H

G

2 1

2 1

H

G

ORIGINALIMAGE

LL

LH

HL

HH

2-D DWT ON MATLAB2-D DWT ON MATLAB

Load Image

(must be.mat file)

Choosewavelet type

HitAnalyze

Choosedisplayoptions

2D Wavelets2D Wavelets

Other uses for Wavelets in image processing (as seen in later slides)

1) Image Compression

The Discrete wavelet transform decomposes an image into a set of successfully smaller orthogonal images. Often it is possible to coarsely quantize or eliminate low valued coefficients without sacrificing the integrity of the image

5MB2.5MB

5MB

2D Wavelets2D Wavelets

2) Image Enhancement

Since the DWT decomposes the image into components of different size, position and orientation, you can alter the coefficients before reconstruction so that you can attenuate certain characteristics of the image.

2D Wavelets2D Wavelets

3) Image Fusion

Image fusion combines 2 or more registered images of the same object into a single object, that in most cases is better than the original. This is helpful in the medical imaging field where multiple images from different machines. The images are combined in the transform domain by taking the highest amplitude coefficient then performing the inverse on the new fused image.

Run Image Fusion GUI

Image DenoisingImage Denoising

Project Idea Project Idea

Implement JPEG 2000A much improved form of JPEG compression

that primarily uses the wavelet based compression idea we just discussed.