Fourier Analysis of Video Signals& Frequency Response of the

HVS

(Chapter 2)

Outline

• Fourier transform over multidimensional space– Continuous space FT (CSFT)– Discrete space FT (DSFT)– Sampled space FT (SSFT)

• Frequency domain characterization of video signals– Spatial frequency– Temporal frequency– Temporal frequency caused by motion

• Frequency response of the HVS– Spatial frequency response– Temporal frequency response and flicker– Spatio-temporal response– Smooth pursuit eye movement

Fourier Transform

4

• Fourier: a periodic function can be represented by the sum of sines/cosines of different frequencies, multiplied by a different coefficient (Fourier series)

• Non-periodic functions can also be represented as the integral of sines/cosines multiplied by weighing function (Fourier transform)

Fourier Transform

5

Introduction to the Fourier Transform

• f(x): continuous function of a real variable x• Fourier transform of f(x):

dxuxjxfuFxf ]2exp[)()()(

• (u) is the frequency variable.• The integral shows that F(u) is composed of an

infinite sum of sine and cosine terms and…• Each value of u determines the frequency of its

corresponding sine-cosine pair.

6

Introduction to the Fourier Transform

• Given F(u), f(x) can be obtained by the inverse Fourier transform:

)()}({1 xfuF

duuxjuF ]2exp[)(

• The above two equations are the Fourier transform pair.

7

Introduction to the Fourier Transform

• Fourier transform pair for a function f(x,y) of two variables:

dxdyvyuxjyxfvuFyxf ])(2exp[),(),()},({

dudvvyuxjvuFyxfvuF ])(2exp[),(),()},({1

and

where u,v are the frequency variables.

8

Continuous Space Signals

• K-D Space Signals

• Convolution

• Example function– Delta function

kK Rxxx ],...,,[ ),( 21xx

yyyxxx dhhkR )()()(*)(

9

Continuous Space Signals

• Delta function properties

)fdxx)f2exp(

)x)xxx)

)xx)xx*x)

0T0

00

00

K

K

R

R

j

10

Continuous Space Fourier Transform (CSFT)

• Forward transform

• Inverse transform

• Any signal can be expressed as a linear combination of complex exponential functions

variablefrequency domain continuous - ],...,[fth wi

) 2exp()()(

2,1K

K

R

Tc

Rfff

djk

xxfxf

fxffx djkR

Tc ) 2exp()()(

11

Continuous Space Fourier Transform (CSFT)

• Most properties of 1D and 2D Fourier Transforms can be extended to K-D case.

• Convolution theorem

)(*)()()()()()(*)(ffxxffxx

cc

cc

HhHh

12

Continuous Space Systems

• General system over K-D continuous space (input and output)

• Linear and Space-Shift Invariant (LSI) System

• LSI systems can be completely described by its impulse response

kRT xxx )),(()(

)()()()(*)()(

)()(fffxxx

xx

ccc HhTh

)())(())()(()()(

00

22112211

xxxxxxxx

TT

13

Discrete Space Signals

• K-D Space Signals

• Convolution

• Example function– Delta function

KK Znnn ],...,,[),( 21nn

KZ

hhm

mmnnn )()()(*)(

14

Discrete Space Fourier Transform (DSFT)

• Forward transform

• Inverse transform

• Convolution theorem

)2/1,2/1(, :period lFundamenta

1 of periodwith dimension each in periodic is )(

)2exp()()(

kK

d

R

Td

fI

jK

f

f

nfnfn

KI

Td dj

f

fnffn )2exp()()(

)(*)()()()()()(*)(ffnnffnn

dd

dd

HhHh

Frequency domain characterization of video signals

• Video is 3D signal: x,y,t• Can apply CSFT and DSFT with K=3

• Spatial frequency

• Temporal frequency

• Temporal frequency caused by motion

tfyx ff ,

16

Spatial Frequency

• Spatial frequency measures how fast the image intensity changes in the image plane

• Spatial frequency can be completely characterized by the variation frequencies in two orthogonal directions (e.g horizontal and vertical)– fx: cycles/horizontal unit distance

– fy : cycles/vertical unit distance

• It can also be specified by magnitude and angle of change

)/arctan( ,22xyyxm fffff

17

Illustration of Spatial Frequency

18

Angular Frequency

height-pictureper cyclesin is f whereee)cycle/degr(f180

ff

/2for (degree)180(radian)/22(radian))2/arctan(2

sss

hd

dhdhdhdh

Problem with previous defined spatial frequency:Perceived speed of change depends on the viewing distance.

19

Angular Frequency

• Spatial and angular frequencies are related• Angular frequency increases as the viewing distance increases• Angular frequency can be defined along horizontal direction as well

• Angular frequency depends on the signal and the viewing condition

20

Temporal Frequency

• Temporal frequency measures temporal variation (cycles/s)

• In a video, the temporal frequency is spatial position (2D) dependent, as every point may change differently

• For a fixed (x,y) cycles/sec = Hz• Temporal frequency is caused by camera or object

motion (equivalent)• Maximum temporal frequency (for all points)

21

Temporal Frequency caused by Linear Motion

22

),(),,( 0 tvytvxtyx yx

Relation between Motion, Spatial and Temporal Frequency

)AssumptionIntensity (Constant is at timepattern image the),,( is

0at pattern image theAssume ).,( speed with movingobject an Consider

0 tyx

tvv yx

derive

23

)( :frequency temporaland spatial, motion,between Relation

0at zeronot is ),,( Hence,

)(),(),,( 0

yyxxt

yyxxttyx

yyxxtyxtyx

fvfvf

fvfvffff

fvfvffffff

The temporal frequency of the image of a moving object depends on motion as well as the spatial frequency of the object.

Example: A plane with vertical bar pattern, moving vertically, causes no temporal change; But moving horizontally, it causes fastest temporal change

If scene has multiple objects, then we can subdivide into small regions with uniform motion with constant speed

Relation between Motion, Spatial and Temporal Frequency

24

)( :frequency temporaland spatial, motion,between Relation

yyxxt fvfvf

Relation between Motion, Spatial and Temporal Frequency

The temporal frequency of the image of a moving object depends on motion as well as the spatial frequency of the object.

change) spatialgreatest of direction in the movesobject hen greatest w theisfrequency (temporal

),( toorthogonal is )v,( if 0 (b)

)brightness shomogeneou (e.g.

and of any valuesfor 0 then 0 if (a)

y yxxt

yxtyx

ffvf

vvfff

25

Illustration of the Relation

Frequency response of the HVS

• Temporal frequency response and flicker• Spatial frequency response• Spatio-temporal response• Smooth pursuit eye movement

27

Temporal Response

Critical flicker frequency: The lowest frame rate at which the eye does not perceive flicker.

Provides guideline for determining the frame rate when designing a video system.

Critical flicker frequency depends on the mean brightness of the display:

60 Hz is typically sufficient for watching TV.

Watching a movie needs lower frame rate than TV

28

Temporal Response

• Experiment with human observer• Screen with varied brightness (t)

ysensitivitcontrast - /1noticeablejust flicker -mmidentify tovary

frequencymean - ,brightnessmean - )2cos1()(

min

min

m

mfBftmBt

• Different curves for different B• reduce sensitivity due to “persistence of vision”

29

Temporal Response

• Different curves for different B• “troland” – units of light entering retina• critical frequency 20 – 80 Hz• Computer display 9000 trolands• movies very low ~ 1 troland• note: brightness at the retine, i.e. depends on viewing distance• Hence

• Movies 24fps• TV 50/60fps• SVGA 72fps

30

Spatial Response

• Experiment with human observer• Screen with varied brightness (x)• can be measured in x, y, etc

ysensitivitcontrast - /1noticeablejust changes spatial -m

midentify to vary each for brightnessmean -

)2cos1()(

min

min

m

mfB

fxmBt

• 3 curves – eye movement stabilization• “saccadic” eye movement changes contrast sensitivity• top curve – no eye stabilization

31

Spatiotemporal Response

• Previously: temporal frequency response at zero spatial frequency• Previously: spatial frequency response at zero temporal frequency• Now need to combine

page)next (graph y sensitivitcontrast - /1noticeablejust changes oralspatiotemp -m

midentify to vary and ofset afor brightnessmean -

))2cos()2cos(1(),,(

min

min

m

mffB

tfxfmBtyx

tx

tx

• Graph – next page• Note: experiment under normal saccadic eye movement• much higher sensitivity when tracking moving object compare to a flickering wave• Smooth pursuit eye movement vs. saccadic eye movement

32

Spatiotemporal Response

The reciprocal relation between spatial and temporal sensitivity was used in TV system design:

Interlaced scan provides tradeoff between spatial and temporal resolution.

33

Smooth Pursuit Eye Movement

• Smooth Pursuit: the eye tracks moving objects• Net effect: reduce the velocity of moving objects on the retinal plane, so that

the eye can perceive much higher raw temporal frequencies than indicated by the temporal frequency response.

yyxxtyyxxtt

yx

yyxxt

yyxxtyxtx

yx

yx

vvvvffvfvff

vv

fvfvf

fvfvffffff

ttvytvxtyxtyxtyxvv

~,~ if 0~

and )~~(~

:)~,~(at moving is eye when theretina at thefrequency temporalObserved

)( :onlymotion object by causedfrequency Temporal

)~~,,(),,(~FT takingand ),~,~(),,~

scoordinate retine andscreen - ),,(~ ,),,( and ),(movement eye Assume

y

(a) no smooth persuit

(b) 2 deg/s

(c) 10 deg/s

Outline

• End of Chapter 2