Sampling and Interpolation on Uniform and Non-uniform Gridsdkuva2/Lecture7_SamplingAndInterpolation_v3.pdfSampling and Interpolation on Uniform and Non-uniform Grids Department of

Sampling and Interpolation on Uniform and Non-uniform Grids

Department of Signal Processing

Tampere University of TechnologyTampere University of Technology


Tampere University of TechnologyTampere University of Technology


More about motivation: C ti bj t d di t i lContinuous objects and discrete signals

Most of real-world phenomena are continuous• Continuous in scale• Continuous in time• Continuous in frequencyq y

Computers are digitalSampling and acquisition

• Digital photographySAR i i• SAR imaging

• Medical imaging• Digital holography

Reconstruction (interpolation)• High-quality display• Correction of geometrical distortions• Rescaling, zooming, rotation, etc.


Motivation: DemosMotivation: Demos


NotationsNotations

Hilbert spacesHilbert spaces• The space of measurable, square-integrable functions of continuous 1-D variable denoted by

:)( );(for :)( 222 RLgRLfRL ;)()(,

dxxgxfgf

dxxfff 2)(,f

)(2 RL

For

• The space of measurable, square-sumable functions of discrete 1-D variable denoted by

Inner product Norm

)(2 Zl

:)(b );(afor :)( 222 RlRlZl ;)()(,

k

kbkagf

k

kaaaa 2)(,For

Inner product Norm

• The space of measurable, square-integrable functions of continuous 2-D variable denoted by

:)( );(for :)( 222222 RLgRLfRL ;),(),(,

dxdyyxgyxfgf

)( 22 RL

Inner product


dxdyyxfff 2),(,f Norm

Notations (cont.)Notations (cont.)

Convolutions• Continuos convolution

dtvutvu )()())((

kbkb )()())((• Discrete convolution

• Mixed convolution

k

knbkanba )()())((

k

ktukatua )()())((

• Convolution inverse

k

elsewhere00 if1

)())(( 1 nnnbb


Notations (cont.)Notations (cont.)

Fourier transforms• Fourier transform pair of an (Lebesgue) integrable function

dxexfF xj 2)()(

)()( 1 RLxf

deFxf xj2)()(

• Discrete Fourier transform pair of a discrete sequence

1

0

/2)()(N

n

NknenakA

1

0

/2)()(N

k

NknekAna

Space of band-limited functions (Paley-Wiener space)0n 0k

,supp :)(2 FRLfPV


Part 1: Sampling Theorems and Shift-Invariant Space Formalism


Sampling theoremsSampling theorems

Cauchy’s theorem (1841)

1 )(i)(N m

,)(Let 2/)1(

2/)1(

2

N

Nn

jnxnecxf

Classical Sampling theorem (Whittaker-Shannon-Kotelnikov-Someya)

1

0 )/(sin)(sin)1()/()(Then

N

m

m

NmxNxNNmfxf

),()()( , Then,

.120 and 0,Let

nTxsnTfTxfPVf

TT

n

].,[for 1)( and , where )2/(1

SPVs T

n


Frequency domain interpretation of samplingFrequency domain interpretation of sampling

Th i f tiThe sinc function

• For sake of simplicity assume T = 1

• The sampling function is known as sinc fuction x

xx )sin()(sinc

• Its Fourier transform is rect function

x

otherwise0

21,21for 1)(rect

-1/2 -1/2 x


Frequency domain interpretation of samplingFrequency domain interpretation of sampling

F d iFrequency domain

• Taking samples at integers is equivalent to multiplying by a series of Diracs (comb

function). It allows representing the function samples as impulse train in continuous

domain. In Fourier, it periodizes the original spectrum

• The convolution between the impulse train and sinc (expressed as product in Fourier

domain) ’washes-up’ the unwanted replicas

),()()()()()()( nxnfnxxfxcombxfxfnn

p

)(sinc)()()( xxcombxfxf 1


1/2 1 3/2 2

Shift-invariant space interpretationShift invariant space interpretation

The integer translates of the sampling function form an orthonormal

basis for the space 2/1PV

Shanon theorem for non band-limited functions

]; [ :)()(for )(sinc)()( 21

21

Sxsnxnfxfn

dxfxc )(sinc)()(

kkf )(i)()(~

k

kxkcxf )(sinc)()(

sinc(x-) sinc(x-k)f(x) c(x) c(k) )(~ xf

sampling


A mathematical formalizationA mathematical formalization

Shift i i t f ti d ti b iShift-invariant function space and generating basis• Shift-invariant function space V() being a closed subspace of L2

lcixicxgV 2:)()()()(

• Any function from V() can be represented as a convolution between a discrete set of coefficients and the generating function (continuous)

ilcixicxgV :)()()()(

• Requirements to the generating basis: Riesz basis

22

22

2

2 )()(l

Lil

cBixiccA

• Upon proper choise of the basis functions (t) the sampling problem becomes a problem of finding the model coefficients c(i)


Orthogonal projection on the given spaceOrthogonal projection on the given space

Approximating a function s(x)L2 by a function from V()

• L2 norm defined as

i

ixicxg )()()(~

• L2 norm defined as

• Minimizing the error

dsssssL

)()(,2

2

• Orthogonal projection

22

22min~

LVgLgsss

• Dual function of (t):

d)()()()()( igigic

;)()()()( 1

ii d)()( iip


;)()()()( 1

i

ixipx .d)()( iip

Sampling scenarios: ideal vs non-ideal samplingSampling scenarios: ideal vs non ideal sampling

liIdeal sampling

(t) c(x) c(k) )(~ xg

sampling )( kx

Non-ideal sampling

g(t) c(x) c(k) )(xg)(ˆ xt )( kx

p g

( ) (k)

sampling )( kx

g(t) c1(x) c1(k) )(~ xg)( xt )( nx q(n-k)

c2(n)

11 )()()(H)()()(h))(()( Qkkkk


11 )()()( Hense, .)(),()( where),)(()(

nnnQxkxkakcakq

Part 2: Interpolation, Interpolation Kernels, Error Kernels, B-splines


Image sampling and reconstructionImage sampling and reconstruction

S li d t ti

ga(x)Image

Sampling and reconstruction• The best way is to sample the

continuous finction with a sampling function (device) being dual to the reconstruction one This will

-4 -3 -2 -1 0 1 2 3 4

ga(x)comb(x)SampledImage

reconstruction one. This will minimize the error between the original and reconstructed function

• In most cases we start directly from discrete data and we have no ga( ) ( )Image

-4 -3 -2 -1 0 1 2 3 4

information about the sampling device

• Interpolation: keep the given samples, i.e. generate such a function which has the same ya(x) = [ ga(x) comb(x) ] * h(x)Reconstucted

Imagefunction, which has the same values at the given coordinates

• Smoothing: fit a smoothing function assuming a certain amount of noise


-4 -3 -2 -1 0 1 2 3 4amount of noise

Interpolation problemInterpolation problem

DefinitionGi di t d t [k] if id k 0 1 2 t k f t i ti f tiGiven discrete data g[k] on an uniform grid k=0,1,2,…., taken from certain continuous function ga(x).Fit an approximating continuous function ya(x) to the given discrete data and then resample it along a new (finer) grid determined by the smaller sampling interval .

ha(x) ha(x+1)ha(x-1)

ya(x)g(k)

y(l)

l (l+1)1 nl-1

(l-1)(l-2)nl+1nl( )

l l+1

( )( )



Fitting a continuous model, i.e. choice of the reconstruction function• Classical interpolation

• the given samples g[k] are used as weights of the reconstruction (synthesis) function h(x) – linear model, known also as mixed (digital-continuous) convolution

;)(][)( kxhkgxy

• Interpolation constraint

;)(][)( kkxhkgxy

)()( ][)(][)( 000

kkhkgkkhkgxykkx

1.2

minimax Interpolator, Degree=3 Length=8, Contin

• Example of a classical interpolator

0 6

0.8

1

0.2

0.4

0.6


0 1 2 3 4 5 6 7 8-0.2

0


Fitting a continuous modelFitting a continuous model• Generalized interpolation: building a linear model of mixed convolution type;

reconstruction function weighted by model coefficients rather than the given samples themselves

kkd )(][)( k

kxkdxy )(][)(

How to obtain the model coefficients• Obey the interpolation constraint

T l th i t l t t th di t d th t ti

][])[(h])[(][

)(][ where),)(()(][)(11

000

kkkkd

kkpkpdkkkdkyk

• To sample the interpolant at the same coordinates we need the reconstruction function, sampled at integers; its convolution-inverse is an IIR digital filter

)(/)()()()()( :domain-zin ][])[(where])[(][ 11

zPzYzDzPzDzYkkppkgpkd

p-1: convolution-inverse of p



Generalized (some of them non-interpolating) interpolation kernel examples

0.8

1

0.8

1

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

−2 −1 0 1 20

−2 −1 0 1 20

0.6

0.8

0.6

0.7

0.2

0.4

0.6

0.1

0.2

0.3

0.4

0.5


−2 −1 0 1 20

−2 −1 0 1 20


Fitting a continuous model• Generalized interpolation can be represented as classical interpolation:

kxkdxy )(][)( kkxkdxy )(][)(

)(][)( where)(][)(

)(][][)(])[()(1

11

km

k k n

kxkpxhmxhmgxy

kxnkgnpkxkgpxy

• To regard the interpolation in its explicit form, as sliding kernel being weighted by the given discrete samples, one should use the cardinal function It is infinitely supported due to the recursive part

)(/)()( :domainfrequencyin jePH

function. It is infinitely supported due to the recursive part



Cardinal interpolation kernel examples

1

0.6

0.8

resp

onse

0.2

0.4

Impu

lse

res

−5 −4 −3 −2 −1 0 1 2 3 4 5

−0.2

0

Time in T


Time in Tsx


Resampling• Role of the fractional interval (see Fig. 4)

knhkgxy )(][)(

;lll nx ll xn

k lll knhkgxy )(][)(

k lll knkdxy )(][)( or

;lll nx 10 l

k lll nhkngly )()()(

nhkngly )()()( k lll nhkngly )()()( or


Interpolator’s propertiesInterpolator s properties

Finite supportFinite support• best trade-off between quality and computation complexity

Separability• data to be processed line-by-line, column-by-column

n

Symmetry • to introduce no phase distortions

nn

n

ii Rxxxx

),...,,( )()( 211

xx

Partition of unity• reproduction of the constant

)()( xx

• reproduction of the constant

k

k

kx

kxh

)(1

)(1


k

Interpolation artifactsInterpolation artifacts

1 . 2m i n i m a x I n t e r p o l a t o r , D e g r e e = 3 L e n g t h = 8 , C o n t i n

Ringing• Caused by the oscillatory reconstruction

kernels, known also as Gibbs effect around sharp transitions

Aliasing (imaging)0

0 . 2

0 . 4

0 . 6

0 . 8

1

Aliasing (imaging)• Appearing of unwanted frequencies resulting

from the repetition of the original spectrum around 2k. Known as Moiré patterns

Blocking−10

0

0 1 2 3 4 5 6 7 8- 0 . 2

g• Caused by short kernels, like nearest neighbor

Blurring • Result of the interpolation function non-ideality

in the pass-band. Non-ideal interpolators −50

−40

−30

−20

Ma

gn

itud

e,

[dB

]

suppress some high frequencies. As a result, the interpolated image appeared with no sharp details

0 0.5 1 1.5 2 2.5 3−80

−70

−60

Normalized frequency


Normalized frequency

Image artifactsImage artifacts

Result of image processing systems and algorithms, such as

• Image formation channelg• Acquisition (sampling)• Compression• Resizing• DenoisingDenoising



Blockiness • Caused by coarse quantization of

transform coefficients of block transforms, such as DCT

• Use of low-order reconstruction kernels, such as nearest neighbor



Blurring• Loss of spatial details; reduction

of edge sharpness• Result of

• Low-end optics• Non-ideality of reconstruction

functions in the pass-band• Over-smoothing



Ringing• Decaying waves around edges

and contours • Due to Gibb’s phenomenon: finite

terms approximation of continuous operators, mainly continuous Fourier transform

• Appear in convolution-type interpolation or in JPEG2000 likeinterpolation or in JPEG2000-like coding



Ali iAliasing• Due to improper resampling

• Appearance of unwanted frequencies

• Manifests as Moiré patterns


Evaluation of interpolator’s qualityEvaluation of interpolator s quality

T iTwo main error sources• Non-flat magnitude response in the pass-band (blurring)• Non-sufficient suppresion of the periodical replicas in the stop-band (aliasing)

Sampling and reconstruction blur (SR blur) (Park and Schowengerdt, ’82)• Investigate the influence of the phase of sampling

• The error is a periodic function of the phase u

22

2);()()(

Laa uxyuxgu

p pAverage SR blur

• Regarded as the expectation of the random error, depending on the arbitrary sampling phase

k

aSR

kfHfHf

dfffGE

22

222

)()(Re21)(

;)()(



G li d i t l tGeneralized interpolators

k

fj

kf

feP

ffH))(2(

)2()()2()2( 2

• Error kernelk

0

2

)(2

2

22

)())(2(

)()2(1)2(

kkfjfj ePkf

ePff

E i l t f

22

0

222

)(

))(2()2()(

fj

k

fj

eP

kffeP

• Equivalent form

20

22

02

))(2())(2()2(

kk

kfkffj


))(2(

k

kf


G li d i t l t H t t th k lGeneralized interpolators

k

fj

kf

feP

ffH))(2(

)2()()2()2( 2

How to compute the error kernel• Assume symmetrical and finite-lengh (x)• Its sampled version is finite sequence with finite number of cosine

t i F i d i

kxxkp

)()(

• Error kernelk terms in Fourier domain

• Simplified error kernel form

2/

1)2cos()(2)0()( 2

N

kezkfkzP fj

0

2

)(2

2

22

)())(2(

)()2(1)2(

kkfjfj ePkf

ePff

E i l t f• The infinite sum is the Fourier transform of

22

0

222

)(

))(2()2()(

fj

k

fj

eP

kffeP

.)(

)(2()2().(2)()2( 22

2222

2

fjk

fjfj

eP

kffePePf

• Equivalent form

20

22

02

))(2())(2()2(

kk

kfkffj

the function (x) continuously convolved by itself and subsequently sampled

N

kkkfkkf

1

2 )2cos())((2)0)(())(2(


))(2(

k

kf kk 1


Th l ith H t t th k lThe algorithm• Calculate the Fourier transform of the

kernel (x)• Calculate the Fourier transform of the

sampled kernel

How to compute the error kernel• Assume symmetrical and finite-lengh (x)• Its sampled version is finite sequence with finite number of cosine

t i F i d i

kxxkp

)()(

xkp )()( sampled kernel• Calculate the auto-correlation function at

integer coordinates

terms in Fourier domain

• Simplified error kernel form

2/

1)2cos()(2)0()( 2

N

kezkfkzP fj

kxxkp

)()(

))(()( xx kxxaka

)(][

• Find the Fourier transform of the sequence a(k)

• Calculate the error kernel• The infinite sum is the Fourier transform of

.)(

)(2()2().(2)()2( 22

2222

2

fjk

fjfj

eP

kffePePf

the function (x) continuously convolved by itself and subsequently sampled

N

kkkfkkf

1

2 )2cos())((2)0)(())(2(


kk 1

Approximation theoryApproximation theory

kkd )/()()(Quiatifying the approximation error

• Predictive approximation error

dfffGxyxg

kxkdxy

aLa

k

)()()()()(

);/()()(

2222

2

• Approximation error kernel 2

22

2

)(

)()()(

Zk

ZkZk

kf

kfkff

• Approximation order• Develop the error kernel in McLaurin series• First L-1 would be zero; then we say the

approximation order is L

0

2)2(2

2

)!2()0()2(

n

nn

fn

f

2)2(2

222 )0()2( nn

L ffCf approximation order is L• Neglect the higher order terms (valid for

small , i.e. oversampled signals)• Approximation constant

Predictive approximation error

1 )!2(

)2(Ln

fn

fCf

k

L kL

C2)( )2(

!1


• Predictive approximation error k


Quiatifying the approximation error

• Predictive approximation error

dfffGxyxg

kxkdxy

aLa

kh

)()()()()(

);/()()(

2222

2

• Approximation error kernel 2

22

2

)(

)()()(

Zk

ZkZk

kf

kfkff

• Approximation order• Develop the error kernel in McLaurin series• First L-1 would be zero; then we say the

approximation order is L

0

2)2(2

2

)!2()0()2(

n

nn

fn

f

2)2(2

222 )0()2( nn

L ffCf approximation order is L• Neglect the higher order terms (valid for

small , i.e. oversampled signals)• Approximation constant

Predictive approximation error

1 )!2(

)2(Ln

fn

fCf

k

L kL

C2)( )2(

!1


• Predictive approximation error k

2)(22222222

)2()(L

LLa

LL gCdffGfCe


Factorized approximation error• Predictive approximation error

Strang-Fix conditions• Lth order zeros

1,00)2( ;1)0(

)(

LnZkkn

kxkdxyk

h );/()()(

• Approximation error kernel

• Reproduction of monomials 1,0 0)2( LnZkk

n

nk

nk

xkxc

cLn

)(

1,0

dfffGxyxg aLa

k

)()()()()( 2222

2

22

• Approximation order

• Discrete momentsk

1,0 )()(

lnmkxkxZk

nn

2

2

2

)(

)()()(

Zk

ZkZk

kf

kfkff

2)(2222

)(L

LL gCe


B-splinesB splines

B spline functions of degree nB-spline functions of degree nPiecewise-polynomial functions of degree nwith n-1 cont. derivatives at the nodes

n

nnni

n ixuixn

x 11 )()(1)1()(

Successive convolutions of zero-degree B-spline

i

ixuixin

x0

22 ),()(!

)(

-for1 110 x

;elsewhere0

for 1)( 220

x

x

1

0 0 0 1 0 ...

n

nn

Frequency characteristics1

)() sin()(

nn

fff

x


)( f

B-splinesB splines

B-splines form a Riesz basis• Integer shifts of B-spline function of degree n form a Riesz basis for the spline

space of polynomial degree n, that is, all polynomial splines of degree n with knots at integers can be represented as a linear combination of B-spline basis f ti f th d

0.6

0.7

functions of the same degree

Zk

nn kxkdxs )(][)(

0.3

0.4

0.5

0.1

0.2


−8 −6 −4 −2 0 2 4 6 80

B-splinesB splines

B-spline functions of degree n

- dilated continuous B-spline

.1)(

mx

mx nn

m

m

mm

n

mmmnm uuuuu ]1,....1,1[ ,... 10

1

000

where

- discrete B-spline at integer scale m

- sampled continuous B-spline and corresponding frequency response

; ,1)(

Zk

mk

mkb nn

m

k

nfjnn kfekbf )()()( 2Β k

- m-scale relation

)()(

);()(

1

1

Zkkbukb

xuxnnn

nnm

nm


. ),()( 1 Zkkbukb mm

B-spline interpolatorsB spline interpolators

Direct B-spline transform (Unser, Aldroubi, Eden ’93) • Given discrete data g[k]. Find a B-spline model gn(x) which ensures gn(x) = g[k] for x = k

;)(][))(()( ixidxdxg nnn

)()()()(][

;)(][)(][][)(

)(][))(()(

1

1

zGDkbkd

ikbidikidkgxg

g

n

i

n

i

nkx

n

i

)()()()()(][

1

11 zB

zDkgbkd nn

n(x)1/Bn(z)

g(k) d(k) gn(x)


B-spline InterpolatorsB spline Interpolators

210 x

Nearest neighboor

Linear

101

)()( 1 xxxxh

21

21

210

01)()(

xxxxh

;1)(1 zBLinear

Cubic

10

)()(x

xxh

;61

64)( 13 zzzB

2110

)2()2(

)()( 361

221

32

3 xx

xxx

xx

;1)(zB

66

20

)()()( 6

x

0.6

0.8

1

0.6

0.8

1

n(x)1/Bn(z)

g(k) d(k) gn(x)−2 −1 0 1 20

0.2

0.4

−2 −1 0 1 20

0.2

0.4

0.4

0.6

0.8

0.4

0.5

0.6

0.7


−2 −1 0 1 20

0.2

0.4

−2 −1 0 1 20

0.1

0.2

0.3

B-spline interpolatorsB spline interpolators

Digital pre-filtering• Since the kernel is symmetrical there are pairs of real poles being reciprocal, hence

the IIR filter is not stable. There is a trick to make the filtering stable by applying forward and backward filtering.

E lExamples

)(1)( zBzS nn

32 ),)((61

64)( 113 zzzzzB

f d

)1(1

)1(6

)1)(1(6)(

66

11

zzzzzS

forward

n(x)1/Bn(z)g(k) d(k) gn(x)backward


Efficient RealizationEfficient Realization

Farrow structure• Piecewise-polynomial form of

the basis function

c(k)

G3(z) G0(z)G1(z)G2(z)

• Example: 3rd degree B-spline

x-mμkmkcgZk

,)()()( where

s(x)

3

0

1

2

2361

030303631331

1)(ccc

g

10141 c


Geometric transformation of ImagesGeometric transformation of Images

2-D splines by tensor-product basis functions

2

11

21

1)( )(

;)()(],[),(1

1

1

1

n

nnk

kk

nl

ll

nnn

ylxk

lxkxlkdyxg

Mapping from source image coordinates (x,y) to target image coordinates (u,v): (x,y)=T(u,v)

Algorithm:• Pre-compute the spline coefficients d[k,l] (1-D recursive filtering along rows and

columns)• At each target location (u,v) find the corresponding source (x,y) and interpolate g ( , ) p g ( ,y) p

using the neighbor pixels


Image RotationImage Rotation

• Rotation matrix determines a rotation of the coordinates (x,y) by an angle

cossinsincos

)(R

• Matrix factorization:

cossinsincos

)(

R

• The factorized rotation matrix determines three 1-D translations

102/tan1

1sin01

102/tan1

• Tx=y(-tan(/2)• Ty=xsin• Tx=y(-tan(/2)


Image RotationImage Rotation

Three 1-D translations (interpolations)

T ( (/2)Tx=y(-tan(/2)Ty=xsinTx=y(-tan(/2)


Intermediate summaryIntermediate summary

So far in the lecture: How to reconstruct a continuous function from given uniform samples

• Sampling theorems for the class of band-limited functions (Shannon)• An approximation procedure interpreted as projection on the shift-invariantAn approximation procedure interpreted as projection on the shift invariant

subspace of bandlimited functions; only this projection can be reconstructedGeneralization for other shift-invariant spaces

• Uniform splines a very attractive tool InterpolationInterpolation

• When the sampling procedure is unknown: bandlimitedness assumed• Practical efficient interpolation based on generalized sampling kernels (preferably

B-splines)E l ti f i t l ti k l b d th i f d i h t i ti• Evaluation of interpolation kernels based on their frequency domain characteristics

• Strang-Fix conditions to characterize kernels for their capabilities to reproduce polynomials and to provide good approximation error decay

Next question: can we perfectly reconstruct non-bandlimited functions ith fi it b f t (d f f d )?


with finite number of parameters (degrees of freedom)?

Part 5: Design of Efficient Interpolation KernelsPart 5: Design of Efficient Interpolation Kernels


Optimized kernels based on B-splinesOptimized kernels based on B splines

Modified B-splinesOptimal Lth order functions Modified B splines• А parametric class of splines

(Egiazarian, Saramäki, Chugurian, and Astola, ’96) obtained by a linear combination of B-splines of different degrees

p• Splines of minimal support (Ron’90)

• MOMS (Blu and Unser’99)

n

nk

xdx )()( degrees.

N

n Zm

nnm mxx

0

mod )()(

k

kk xdx

x0

)()(

Example: 3rd + 1st degree splines

)1()1()()()( 111

111

110

3mod xxxxx

3 – blue;

mod red;

Example: 3 + 1 degree splines

od – red;

1 – green.


Piecewise-polynomial kernels of minimal supportPiecewise polynomial kernels of minimal support

Piecewise-polynomial (pp) interpolators

pp basis functions of minimal support: A sub class of pp functions

;)()(1

0

N

kk xPxh 1 ][)(

0

kxkxlaxPM

l

lkk

pp basis functions of minimal support: A sub-class of pp functions with the following properties

• Non-interpolating• Symmetric

Mi i l i h l h h i h l i l d• Minimal support, i.e. the lenght N=M+1, where M is the polynomial degree Splines of minimal support (O-MOMS)

• Preserve the order of approximation L=M+1• Minimize the approximation constant 1L

• In frequency domain• After an analytical optimization one gets

LL ffjf )(sinc)2()2( where

1

0

)(L

k

kk zz

)()14(4

)()( 12

2

1 zLzzz LLL

1(z)= 2(z)=112)!2(

!

LL

LCL


)14(4 L 12)!2( LL

Modified B-splinesModified B splines

Modified B-splines

• Pre-filtering step

N

n Zm

nnm mxx

0

mod )()(

• Example: 3rd + 1st degree splines 3 – blue;

mod – red;

)(1)( modmod zBzS

)1()1()()()( 1113mod xxxxx ;

1 – green.)1()1()()()( 111110 xxxxx



Computational structure

231

mod

1)(

,)()()( x-mμkmkcgZk

where

0

1

2

111110111011

2361

6663666303631331

1)(

ccc

g

c(k)

1

0

111011

111110111011

0616461 c

c(k)

G3(z) G0(z)G1(z)G2(z)


s(x)


Example• combination of fifth-, third-, and first-degree B-splines

))1()1(()(

))1()1(()()()(111

3331

330

5mod

xxx

xxxxx

0 6

0.7

B5B3(x)

• Discrete version in z-domain

. ))2()2((

))1()1(()( 11

12

1110

xx

xxx

0 3

0.4

0.5

0.6 ( )B3(x-1)B3(x+1)B1(x)B1(x-1)B1(x+1)B1(x-2)B1(x+2)

).()(

))(()()()(221

1211

111

10

1331

330

5mod

zzzBzzzBzB

zzzBzBzBzB

0.1

0.2

0.3


-3 -2 -1 0 1 2 30

Modified 5-3-1 caseModified 5 3 1 case

Computational structure for fifth degree cases: regular and modified

2

3

05203020515101051

cc

2

1

0

1012345120

1

01266626105500505010206020100102002010

)(

cccc

g

05203020515101051

2012666261 c

012020112080202612040806620120802612020112012060120512060120501201206012012050120605

06010601202012012060601202060102020601060802060806020202010

12311131301031303011311231

1211311211301011103011121231

3130313031303131

31303130313031313031



What we get?• Same support• Probably the same approximation order• Same computational complexity for the direct B-spline transform

What we gain?What we gain?• Better representation of different signal details• The adjusting parameters offer more degree of freedom and possibilities

to optimizeWhat we loose?

• Regularity• Slightly more calculations


Optimization: a filter design approach

Optimization in minimax sense

Optimization: a filter design approach

Optimization in minimax sense• Assumption: most of the signal energy concentrated at a fraction of the passband

• Preserve the signal in this important passband region

A h d i d i i h b d i

,0pF• Attenuate the undesired images in the stopband regions

• Given , and w, specify D( f )’=’1 and W’( f ) ’=’w for f ’Fp and D( f ) ’=’0 and W ( f ) ’=’1for f Fs and find the unknown coefficients mn to minimize

1

,r

s rrF

Ga(2f);G(ej2f);H(2 f)

)()2( )( max fDfjHfWFf

w


f1/2 1 3/2 2 1- 2 Fs = 1

Optimization problem: ExampleOptimization problem: Example

Fifth-degree piecewise-polynomialg p p y• Cardinal interpolator’s frequency response

)()( 531531 fBffH

46531 i22i)( ffffff 2121110

3130531

sin4cos22cos2

sin2cos2sin)(

ffff

ffffff

fffffB 6/2cos242cos2120/4cos22cos5266)( 3130531

• Unknowns:30, 31, 10, 11, and 12.• DC constant preserving: 30 231 10 211 212 1.

ff 4cos22cos2 121110

• Unconstrained nonlinear optimization: 30 1 231 10 211 212.


Magnitude responses of optimized kernelsMagnitude responses of optimized kernels

3rd degree kernels 5th degree kernels


Error approximation kernelError approximation kernel

dfhfefGh

hxyxg

a

Lha

)()()(

)()()(

222

22

2

2

22

2

)(

)()()(

Zk

ZkZk

kf

kfkffe

)( Zkf


Spline-based interpolators: ConclusionsSpline based interpolators: Conclusions

B iB-spines• Provide separable, symmetric, and compactly supported recosntruction functions. A pre-

filtering step is needed• Provide maximal approximation for a given integer support

M t l f ti ith i d f i ti• Most regular functions with given order of approximationMOMS and modified B-splines

• Preserve all good features of the B-splines except the highest regularity• Allow for additional optimization with well elaborated optimization procedures in Fourier

domain• O-moms are asymptotically optimal approximators• Optimization strategies, based on suffcient suppression of the imaging frequences can

lead to much better interpolation results• The computational complexity remains practically the same, as in the case of regular B-

splines (both classes are in fact piecewise-polynomial functions)


Part 6: Resampling TechniquesPart 6: Resampling Techniques


Decimation problemDecimation problem

• Problem: rescale (resample) the given image to a coarser grid, with possible subsequent upscaling (interpolation). Keep as much information about the original image as possible

s(t)Image

-4 -3 -2 -1 0 1 2 3 4

Sampledpossible • By assuming separable processing, the image

(2-D) problem is downgraded to 1-D decimation problem

• Initial grid

pImage

-4 -3 -2 -1 0 1 2 3 4

h =a =b

x(k)

• Initial grid

ba

h

inL

inkkm

10

110

, , , ,...,

τ Re-sampled

Image

h0=a 1 2 3 Lin-1 b

• Target grid plased at the integers: out?=?1; ?=?in is the decimation ratio

-4 -3 -2 -1 0 1 2 3 4y(t) = (c * (t)Reconstucted

Image


-4 -3 -2 -1 0 1 2 3 4

An intuitive interpolative solutionAn intuitive interpolative solution

Would simple interpolation work?• Fit a continuous model

;)/(][)( kkxkdxy )(][ where][)]([][ 1 kkpikpigkd

• Equivalently

• Resample at integers – aliasing effects are to be encountered since the

;)/(][)( int iixigxy

kkxkpx )(][)()( 1

int

Resample at integers aliasing effects are to be encountered since the reconstruction function in Fourier domain has zeros clustered around multiples of 1/ and 1/> 1

Reconstruction function with nodes at integers• Its frequency response has good antialiasing propertiesIts frequency response has good antialiasing properties• For the generalized model, the filtering is as follows

i

ikigky )(][][ i

ikligkply )(][][)(][ 1


Resampling via Least Squares at a glanceResampling via Least Squares at a glance

2Shift-invariant space formulation

Continuous least squares (L2 error norm minimization)• Fit an initial continuous model (through interpolation)

ilcitictgV 2:)()()()(

• Fit an initial continuous model (through interpolation)• Involve the reconstruction function and its dual (infinitely supported)• Convolution integrals between functions with different scales to be solved• Solutions involving B-splines (You-Pin Wang’98, Munos et al.’99)

Di t l t (l2 i i i ti )Discrete least squares (l2 error norm minimization)• Solution involving splines (de Boor’76)• Recuire a matrix (Grammian) inversion

A near least squares techniquegΦΦ)(Φc TT 1

• Efficient structure: transposed Farrow structure• Avoid matrix inversion through digital recursive filtering and proper scaling• Comparison with the l2 solution

gΦHPc T1)(

TT ΦHPΦ)(ΦΦE 11 )(


ΦHPΦ)(ΦΦE )(

Continuous least squaresContinuous least squares

Minimizing L2 norm• a continuous function needed

kK kxKkgxg )(][)(

• minimizing the error norm

• Solving the convolution integral

22

22min~

LKVgLK gggg

Solving the convolution integral

)()(][)()(][

)()()()(][

d)()(][

1

1

dikKkid

dnignaid

igid

Kn

K

.)()(][)()(][

)()(][)()(][

1

1

dnikKkgnaid

dnikKkgnaid

kn

kn



• An integral involving functions with different scales

• A linear model

dxKx )()()(

• General scheme

k

kxkgxg )(][)(

( l)

(x/) (x) (a)-1 (x)g[k] d[l] y(x)

X

l (x-l)

gK(x) g(x)

continuous modeling orthogonal projection


rescaling


Solution involving splines (Munoz et al., 1999) • Consider an interpolation spline model

i

NN ixibxK )()()()( 1

• An integral involving two splines with different scales • Low-complexity operations: finite differences and running sums

l(t-l)

(bN)-1 (N+1) N+1(x) g[k] d1[k]

X

(b2N+1)-1 bN

d[l] y(l)(N+1)


Discrete least squaresDiscrete least squares

Minimizing l2 norm• Discrete inner product

1

0)()(,

inL

iii vuvu

• Induced semi-norm and corresponding error norm

• Normal equations

1

0

2 )()(2

inL

iiaial

ggg 22

22min~

laVslaa sggg

Normal equations

1,...,0for

][)(][)()(1

0

1

0

1

0

out

L

kk

L

j

L

kkk

Li

kgijdjiinout in

• Matrix form

gΦΦ)d(Φ TT

gΦΦ)(Φd TT 1


gΦΦ)(Φd

A near least squares solutionA near least squares solution

A ’hybrid’ model • Modeling the convoluton kernel by a Gaussian

2)( )4/(2xetK )(2lim )4/(

0

2

xe x

• A simpler form of the integral and an hybrid L2:l2 solution

An implemenation based on pp functions

kkxkgxg )(][)(

)(][)()(][ 1 knikgnaidkn

An implemenation based on pp functions m

N

i

N

m

Nm ixicx )]([)(

0 02

1

k

N NmN

mlkilickgxglg 2

1 )]([][)(][

ki m

mlxggg

0 02 )]([][)(][

N

m

N

i

mNkmlx

kilkgicxglg0 0

21 )(][][)(][


A near least squares solutionA near least squares solution

Transposed Farrow structure

• Fractional interval

N

m

N

i

mNkmlx

kilkgicxglg0 0

21 )(][][)(][

• Preservation of the constant

1for lll kkk x

+ +

x

+

k

g[k]

+

z-1

+

z-1

+

z-1

kl klkglg )(][][

kl kl )(1

• Recursive filteringC0(z) CN(z)C1(z)

g[l]


+ +g[l]l a-1

Near LS versus discrete LSNear LS versus discrete LS

0.7

Near LS in matrix form

gΦ(DA)d T10.4

0.5

0.6

l2 a

nd L

2:l2

sol

utio

n

Norm of the error matrix

TT Φ(DA)Φ)(ΦΦE 11 0.1

0.2

0.3

LS e

rror b

etwe

en l2

( ))(

0 0.2 0.4 0.6 0.8 10

Decimation ratio


Frequency-domain analysisFrequency domain analysis

Comparing different schemes• Interpolative scheme

)()2()2( 2int

fjePff

• Scheme minimizing L2 error norm

)()2(

)()2()2(~

222

fjfjL eP

feA

ff

• Near LS scheme

)()2()2(~

2: 22 fjlL eAff


Frequency-domain analysisFrequency domain analysis

C i diff t k lComparing different kernels• Cubic B-spline

)()( 3 xx

• Modified B-spline

1

11

33

mod )()()(i i ixxx

• MOMS

)()()( 323 xDxxmoms


Quality assessment of least squares techniquesQuality assessment of least squares techniques


Near least squares for different kernelsNear least squares for different kernels

Kernels• Regular cubic B-spline• Optimized MOMS (lowest

approximation constant)• Modified B-spline (minimax

optimized)


Complexity assessmentComplexity assessment

Comparison with the B-spline based L2 methodp p 2


Part 6: Non-Uniform Sampling and ReconstructionPart 6: Non Uniform Sampling and Reconstruction


Non-uniform sampling and reconstructionNon uniform sampling and reconstruction

Scenarios• Combination of multiple displaced uniform grids – appear in the framework of

super-resolution• Sampling in biomedical or astronomic imagery (e.g. spiral sampling)• Sampling within the framework of computer tomography

Theoretical fundamentals – Frame theory• Frames are generalization of bases and provide tools for signal expansions• In general frames provide over-complete signal expansionsIn general, frames provide over complete signal expansions• The theory is interested in the question if a set of functions forms a frame for a

certain space• For the case of non-uniform sampling frame theory provide the conditions

ensuring function reconstruction from given samples, in the flavor of the g g p ,Shannon theorem; the corresponding theorems are those of Paley-Wiener, Kadec and Duffin-Schaeffer


Non-uniform sampling and reconstructionNon uniform sampling and reconstruction

Techniques• The problem is cast as of reconstructing a continuous function belonging to a

function space from its non-uniform samples. Having a function space assumed, the problem is to find the coordinates of the continuous function with

t t th b i ti th trespect to the basis generating that space• Most widely used function spaces are the space of trigonometric polynomials

and splines, as well as those generated by radial basis functions • Iterative methods for solving large systems of equations are employed


Example: Super-resolution reconstructionExample: Super resolution reconstruction

Real sceneOptically blurred image

by camera lenses Image Acquired byCamera Sensor

Final Image(yk)

Sensor Position

PkMCD 1

Blurring(Ck) Downsampling(Dk) Noise(ηk)Sensor Position

According to Scene (Mk)

PknxMCDy kkkkk ,...,1,

Registered Images Non-uniformly SampledImage Data

Restored Scene

Transformationto desired grid

Non-Uniform to Uniform

Data Resampling

Registration

Department of Signal Processing 83

Reconstruction from non-uniform samples in lispline spaces

Iterative method by Aldroubi and Feichtinger• Provides exact reconstruction of spline function from sufficiently dense samples

by successive oblique projections / interpolations • Available samples g(xi,yi) of function g(x,y) V(φ) are interpolated to the desired

grid using a simple interpolator (e.g. nearest neighbour)

• Then the interpolated function is projected to V(φ)

i iiiii yxhgg ),( ),()())(Q( xxxxx

gg PQ1 • The projected function is sampled at the given non-uniform grid and the error

between the resulting and input samples is calculated. Using this error, the same procedure is repeated until convergence

nnn gggggg

)(PQPQ

1

1

I t l ti ProjectorSampling gn gn+1

g(xi)

-

nnn gggg )(1


Interpolatior Projectorp gOperator

g(xi)n

Reconstruction from non-uniform samples in lispline spaces

Local least squares method by Gröchenig and Schwab• Fast local least squares reconstruction () method in spline shift-invariant space• Using compact bases, a function can be reconstructed exactly on an arbitrary

interval solely using samples from that interval. Thus, the problem is to solve small size band systems over image segments

• Extendable to 2d case

y2 y

y

φ1 φ2 φ3 φ4 φn-1vφn

x1 x2 x3 x

y1

y2

k1kn

k1

k2

Arbitraty chosen intervals1

yUb *)( kxU jjk

Define and Compute for chosen interval:

)( )(1

* lxkxT j

J

jjjk

Compute

Solve bTd 1-


2D Non-uniform to uniform resampling: Formal bl t t tproblem statement

Consider P data samples of a certain 2D function, non-uniformly sampled as follows: {g(xi,yi):i=0,1,…,P-1}.

Reconstruct a continuous signal g(x,y) and resample it to uniform 2D grid

lklkd )()(

Assume the function g(.) belongs to shift-invariant space V(2)(φ), spanned by suitable basis φ.

W f h f bl 2D b f d b 1D i i l i l

k k

lykxlkdyxg ),(],[),(

We favor the use of separable 2D bases formed by 1D piecewise-polynomials

1

1 10 0

1( ) ( ) .2

I nm

mi m

nx c i x i

The aim is to obtain unknown discrete sequence d[k,l] which will describe uniquely and entirely g(x,y).There are several available methods in literature that estimate the sequence d[k,l] by different approaches.


Algorithm derivationAlgorithm derivation

The coefficients d[k,l] in desired g(x,y) are given by

Th b i i bi th l(d l) b i f

ddjigjid ш ),(~),(],[ k

kkkk yyxxyxgyxg ),(),(),(ш

)()(~ )2(V )(

, where

The basis is biorthogonal(dual) basis of

The dual basis can be obtained as follows:

)(),(~ )2( Vyx ),( yx

))()(()(~ 1 xax

For a separable basis :

))()(()( xax

diia )()(][

),(~ yx

Substitution of gш (x,y) gives:

qp

ddqjapipagjid )()()()()(),(],[ 11ш


ddqjpiyxayxgqapajid kkp q k

kk )()(),(),()()()()(],[ 11


Using the separable property, we get:

Rearranging the above equation we get:

p q k

kkkk ddpiqjyxayxgqapajid )()()()(),()()()()(],[ 11

Rearranging the above equation we get:

p q k

kkkk dyqjdxpiyxgqapajid )()()()(),()()()()(],[ 11

The latter can be simplified using the replication property of the delta function:

p qqjpigqapajid ),()()()()(],[ 11

where

Sampling uniformly at integers x=s, y = t yields:

k

kkkkk yyxxyxgyxg )()(),(),(


k

kkkkktysx ytxsyxgyxgtsg )]()(][,[|),(],[ ,


4

5

We use compactly supported basis, which is piecewise polynomial of degree n, defined on each interval :

2

3

4

.

I

i

mn

miim inxicx

0 012

1][)(

0 1 2 3 4

0

1

k

n

m

n

i

m

kmk

n

m

n

i

m

kmkkk yinticxinsicyxgtsg0 0

220 0

112 2

2

2

1 1

1

1 21][

21][],[],[

k

Denoting , we get the following form:

12 )()]([][][][ mn n n

mn

yxgicictsg

kk yintv

221

kk xins

121


1 2 2

2

1

1)()](,[][][],[

0 0 02

01 k

m m ikkk

km

imk yxgicictsg

2D version of the transposed Farrow structure

The equation

2D version of the transposed Farrow structure

12 )()]([][][][ mn n n

mn

iit The equation

can be represented by a two dimensional structure based on the Transposed Farrow Structure (TF). We refer to this structure as the 2D Farrow Structure

1

1 2 2

2

2

1

1)()](,[][][],[

0 0 02

01

mk

m m i

mkkk

km

imk yxgicictsg

k

0 ,k k kg x y 1 ,k k kg x y 2 ,k k lg x y 3 ,k k kg x y

X X1[ , ]( )m

k k kg x y X

k

)(0 zC 1( )C z 2 ( )C z 3( )C z

TF TF TF TF +

1z

+

1z

+

1z

+

1z

kblockid

TF =

+++

k +

)(0 zC )(1 zC

+

3( )C z2( )C z

+


,g s t

The 2D Transposed Farrow StructureThe 2D Transposed Farrow Structure

0

k k k k

1 3[ ]( )

1

1 2 2

2

2

1

1)()](,[][][],[

0 0 02

01

mk

n

m

n

m

n

i

mkkk

km

n

imk yxgicictsg

+X

)(0 zC

1z

X

+

)(1 zC

1z

+

3( )C z

1z

0[ , ]( )k k kg x y

+

2 ( )C z

1z

+X

)(0 zC

1z

X

+

)(1 zC

1z

+

3( )C z

1z

+

2 ( )C z

1z

+X

)(0 zC

1z

X

+

)(1 zC

1z

+

3( )C z

1z

+

2 ( )C z

1z

+X

)(0 zC

1z

X

+

)(1 zC

1z

+

3( )C z

1z

+

2 ( )C z

1z

1[ , ]( )k k kg x y 2[ , ]( )k k kg x y 3[ , ]( )k k kg x y

X X X X

2i 2i 2i 2i 2i 2i 2i i 2i 2i 2i i 2i 2i

2 0m 2 1m 2 2m 2 3m 2 0m 2 1m 2 2m 2 3m 2 0m 2 1m 2 2m 2 3m 2 0m 2 1m 2 2m 2 3m

k

2i 2i

kblockid kblockid kblockid kblockid

+

)(0 zC )(1 zC

+

3( )C z

)(0 zC

2 ( )C z

+ +

)(0 zC )(1 zC

+

3( )C z

1( )C z

2 ( )C z

+ +

)(0 zC )(1 zC

+

3( )C z

2 ( )C z

2 ( )C z

+ +

1

+

3( )C z

+

1 0m 1 1m 1 2m 1 3m

2i 2 2 2 2i 2i 2i 2i 2 2i 2i 2i 2 2

1i 1i1i 1i

2 2

+++

k

,g s t


ExperimentsExperiments

We conduct experiments for the applicability of the 2D resampling method to the super-resolution reconstruction from registered images for arbitrary sampling.

The reconstruction results are compared with results from Aldroubi-Feichtinger’s iterativereconstruction method and Gröchenig Schwab’s fast local reconstruction for translational andreconstruction method and Gröchenig-Schwab s fast local reconstruction for translational androtational motions as follows:

• Non-aliased LR images with pure translational motion, and noise added to the coordinates.• Non-aliased LR images with both translational and rotational motion, and noise is added to the

coordinatescoordinatesThese test cases are quite practical as they simulates the real case when we attempt to reconstruct

a SR image from blurred LR images after estimating the motion parameters by registration.Registration is never 100% accurate and by adding small amount of noise to the coordinates we

take into account such registration error. The noise variance has been chosen in accordance with reported results on state-of-the-art image registration methods


Experimental resultsExperimental results

Image MethodAliased Non Aliased

No Noise[dB] Noise[dB] No Noise[dB] Noise[dB]

Lena

Aldroubi >50 (6) 37.07(1) >50 (6) 39.23(1)

Schwab >50 40.51 >50 42.82

Farrow 41.20 41.56 42.61 43.09

Aldroubi >50 (6) 31 16 (1) >50 (6) 35 69 (1)

Barbara

Aldroubi >50 (6) 31.16 (1) >50 (6) 35.69 (1)

Schwab >50 34.45 >50 38.96

Farrow 41.20 37.39 40.47 40.65

Aldroubi >50 (6) 30.24(1) >50 (6) >50 (1)

Camera Schwab >50 35.47 >50 38.82

Farrow 38.71 38.83 40.17 40.39


Average PSNR results from 10 trials for Monte-Carlo simulation of random sampling experiment.


Aldroubi-Feichtinger Gröchenig- 2D resampling g/(iteration)[dB]

gSchwab[dB]

p gmethod[dB]

Barbara 45.36 (6) 50.97 44.27

Lena 45.35 (7) 50.87 46.79

( )Cameraman 42.53 (6) 47.86 42.96

PSNR results for reconstructed SR images using non-aliased LR images with pure translational motion and noise added to the coordinates of the non-uniform samples

Aldroubi-Feichtinger /(iteration)[dB]

Gröchenig-Schwab[dB]

2D resampling method[dB]

Barbara 45.36 (6) 50.97 44.27

Lena 45 35 (7) 50 87 46 79Lena 45.35 (7) 50.87 46.79

Cameraman 42.53 (6) 47.86 42.96

PSNR results for reconstructed SR images using non-aliased LR images with translational and rotational motionand noise added to the coordinates of the non-uniform samples.


p


Cameraman reconstruction results for the aliased random sampling experiment with noise added to the coordinatesnoise added to the coordinates

2D Transposed Farrow Structure Method Aldroubi & Feichtinger Method

(a) (b)

2D Transposed Farrow Structure Method(a)

Aldroubi & Feichtinger Method(b)

(c) (d)( ) ( )


Original ImageSchwab & Gröchenig Method(c)

2D Transposed Farrow Structure MethodWith IIR Filtering

(d)

2D non-uniform resampling: Conclusions2D non uniform resampling: Conclusions

The proposed 2D near least squares resampling method performs well for the non-p p q p g puniform to uniform resampling problem

A very important advantage of the proposed method is that all the processing can bedone by digital filtering and does not require any matrix inversions and hence it iscomputationally very efficient

From the PSNR values this method also seems to be quite immune to typicalregistration errors

The drawback is that large gaps in the non-uniform sampling cannot be handledThis method is very fast and memory requirements are moderatey y qIt gives good, smooth reconstruction results after the 2D resampling structure. The

final reconstruction result after IIR filtering and sampling is sharper, but in somecases it decreases the PSNR, as it is somehow mimicking the true Grammian with abanded matrix

Based on the two sets of experiments, we can conclude, that in cases of perfectalignment, the proposed near least squares 2D resampling technique gives somehowmoderate but still acceptable results compared with state-of-the-art methods

In the case of coordinate noise it is highly competitive, if not better than the other twoh d


methods

Documents

Sampling and Interpolation on Uniform and Non-uniform Gridsdkuva2/Lecture7_SamplingAndInterpolation_v3.pdfSampling and Interpolation on Uniform and Non-uniform Grids Department of