Pawel FORCZMANSKI (West Pomeranian University of Technology) "Advanced digital image processing methods"

Eras

mus

+ s

emin

ar, 18

/04/

2016

1 / 64

Faculty of ComputerScience andInformationTechnology

West Pomeranian University of Technology,Szczecin

Advanced Digital ImageProcessing:

problems, methodsand applications

Paweł ForczmańskiChair of Multimedia Systems, Faculty of Computer Science and Information Tech-

nology, West Pomeranian University of Technology, Szczecin

Vilnius University, Institute of Mathematics and Informatics, 18/04/2016

Eras

mus

+ s

emin

ar, 18

/04/

2016

2 / 64



AgendaAgenda

Introduction (objectives, problems,image classes, acquisition)

Introduction (objectives, problems,image classes, acquisition)

Image filtering methodsImage filtering methods

Image quality estimation (concpets, exemplary metrics)

Image quality estimation (concpets, exemplary metrics)

Simple image features and their applicationSimple image features and their application

Eras

mus

+ s

emin

ar, 18

/04/

2016

3 / 64



Computergraphics

Data processing

Signalprocessing

Digital imageprocessing

Pattern recognition

IntroductionIntroduction

Eras

mus

+ s

emin

ar, 18

/04/

2016

4 / 64



DIP: Application AreasDIP: Application Areas

OCR

CriminalForensic

CAD

Robotics

GIS

Media andEntertainment

CTMRI USG

Barcodes

Textprocessing

Eras

mus

+ s

emin

ar, 18

/04/

2016

5 / 64



ObjectivesObjectives

Imagequality

improvement

compression

Imagerepresentationtransformation

Objective(computer)

transmission

Subjective(human)

coding

storing

Imagequality

improvement

Eras

mus

+ s

emin

ar, 18

/04/

2016

6 / 64



Image classesImage classes

Eras

mus

+ s

emin

ar, 18

/04/

2016

7 / 64



. . .

M

N

K

. . .

Tyical color image is in a raster form which has:M columnsN rowsi K layers:

Sample image with MxNx3 (YUV color-space)

Data representation (1)Data representation (1)

kNMkM

kNkk

NMKk

xx

xx

X

,,,1,

,,1,1,1

Eras

mus

+ s

emin

ar, 18

/04/

2016

8 / 64



Light sensors matrixLight sensors matrix

cones cones

cones

rodsBayer matrix

Human eye

Eras

mus

+ s

emin

ar, 18

/04/

2016

9 / 64



Bryce Bayer - patent (U.S. Patent No. 3,971,065) - 1976

MegaPixels?MegaPixels?

Eras

mus

+ s

emin

ar, 18

/04/

2016

10 / 64



Bayer Matrix vs Foveon X3Bayer Matrix vs Foveon X3

Eras

mus

+ s

emin

ar, 18

/04/

2016

11 / 64



Image acquisitionImage acquisition

quantization

discretization

Digital image

quantization quantization

discretization

discretization

Eras

mus

+ s

emin

ar, 18

/04/

2016

12 / 64



Nadajnik Trans. channel

Signal quality estimation

Source

Reconstruction and presentation

Perception and un-derstanding

processing, storing and transmission

Acquisition and registrationSignal source

Knowlegdeabout distortions

Knowlegde aboutreceiver and application

Knowlwdge aboutsource and transmitter

Receiver

➔ Imaging systems can introduce certain signal distortions or artifacts, there-fore, it is an important issue to be able to evaluate the quality.

Quality estimationQuality estimation

Eras

mus

+ s

emin

ar, 18

/04/

2016

13 / 64



The quality of an image can be reduced during●Image acquisition●Image transmisson●Image processing

Quality measure may be a determinant of quality degradation

Classification of methods I:perceptual (perceptive, subjective)objective (calculative).

Classification of methods II:Scalar-based,Vector-based (sets of scalars)

Classification of methods III:Full-reference,No-reference,Partial-reference

Image QualityImage Quality

Eras

mus

+ s

emin

ar, 18

/04/

2016

14 / 64



• Related works– Pioneering work [Mannos & Sakrison ’74]– Sarnoff model [Lubin ’93]– Visible difference predictor [Daly ’93]– Perceptual image distortion [Teo & Heeger ’94]– DCT-based method [Watson ’93]– Wavelet-based method [Safranek ’89, Watson et al. ’97]

Philosophy:degraded signal = reference signal + error

reference signal → idealquantitive estimation of distortions level

Standard model of IQA:

Image Quality AssessmentImage Quality Assessment

Eras

mus

+ s

emin

ar, 18

/04/

2016

15 / 64



Motivation – simulating elementary characteristics of HVS

Main features:

Channel decomposition linear transformation

Frequency weigthing contrast sensitivity function

Masking intra-channel interactions

Referencesignal

EvaluationChannel

decompositionError

normalization...

AggregationPre-processing

.

.

.

/1

,

l kkleE

Evaluatedsugnal

Standard model of IQAStandard model of IQA(Image Quality Assessment)(Image Quality Assessment)

Eras

mus

+ s

emin

ar, 18

/04/

2016

16 / 64



++

_

= + +...

...

structuraldistortion

+

distortedimage

originalimage

= + +

+

nonstructuraldistortion

cK +1.

c1.

cK +2.

c2.

cM.

cK.+

+

nonstructural distortioncomponents

structural distortioncomponents

Standard model of IQA (Image Quality Standard model of IQA (Image Quality Assessment): Adaptive Linear SystemAssessment): Adaptive Linear System

Eras

mus

+ s

emin

ar, 18

/04/

2016

17 / 64



Eras

mus

+ s

emin

ar, 18

/04/

2016

18 / 64



Structural content

Normalized Cross-Colerraltion

Peak Absolute Error (PAE)

Image Fidelity

Average Difference

Eras

mus

+ s

emin

ar, 18

/04/

2016

19 / 64



Mean Square Error

Zhou Wang and Alan C. Bovik, Mean Squared Error: Love It or Leave It? A New Look at Signal Fidelity Measures, IEEE Signal

Processing Magazine vol. 26, no. 1, pp. 98-117, Jan. 2009

Eras

mus

+ s

emin

ar, 18

/04/

2016

20 / 64



Peak Mean Square Error

Normalized Absolute Error

Normalized MeanSquare Error

Lp norm (Minkowski)

Peak Signal-to-Noise Ratio

Signal-to-Noise Ratio

Eras

mus

+ s

emin

ar, 18

/04/

2016

21 / 64



RMSE 9.5(blurred)(blurred)

RMSE 5.2

Pixel by Pixel ComparisonPixel by Pixel Comparison

Prikryl, 1999

Eras

mus

+ s

emin

ar, 18

/04/

2016

22 / 64



X. Shang, “Structural similarity based image quality assessment: pooling strategies and ap-plications to image compression and digit recognition” M.S. Thesis, EE Department, The University of Texas at Arlington, Aug. 2006.

Structural Similarity (SSIM) IndexStructural Similarity (SSIM) Index

Eras

mus

+ s

emin

ar, 18

/04/

2016

23 / 64



i

k

j

x

xi + xj + xk = 0

x - x

O

luminancechange

contrastchange

structuralchange

xi = xj = xk

),(),(),(),( yxyxyxyx sclSSIM

122

12),(

C

Cl

yx

yx

yx

c ( x , y )=2 σ x σ y+C2

σ x2 + σ y

2+C2

3

3),(C

Cs

yx

xy

yx

[Wang & Bovik, IEEE Signal Processing Letters, ’02]

[Wang et al., IEEE Trans. Image Processing, ’04]

Structural Similarity (SSIM) IndexStructural Similarity (SSIM) Index

Eras

mus

+ s

emin

ar, 18

/04/

2016

24 / 64



MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989

MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723Zhou Wang Image Quality Assessment: From Error Visibility to Structural Similarity

MSE vs mSSIMMSE vs mSSIM

Eras

mus

+ s

emin

ar, 18

/04/

2016

25 / 64



original image

JPEG2000 compres-sed image

absolute error map

SSIM index map

Eras

mus

+ s

emin

ar, 18

/04/

2016

26 / 64



original image

Gaussian noise cor-

rupted image

absolute error map

SSIM index map

Eras

mus

+ s

emin

ar, 18

/04/

2016

27 / 64



original image

JPEG com-pressed image

absolute error map

SSIM index map

Eras

mus

+ s

emin

ar, 18

/04/

2016

28 / 64



Zhou Wang and Alan C. Bovik, Mean Squared Error: Love It or Leave It? A New Look at Signal Fidelity Measures, IEEE Signal Processing Magazine vol. 26, no. 1, pp. 98-117, Jan. 2009

Comparison of quality measuresComparison of quality measures

Eras

mus

+ s

emin

ar, 18

/04/

2016

29 / 64



Image

2

Image

1

Ps

ych

om

etri

cF

un

cti

on

Pro

ba

bili

tyS

um

mat

ion

Vis

ual

isat

ion

of

Dif

fere

nce

s

AmplitudeNonlinear.

AmplitudeNonlinear.

ContrastSensitivityFunction

ContrastSensitivityFunction

+

CortexTransform

CortexTransform

MaskingFunction

MaskingFunction

Unidirectionalor MutualMasking

[Daly ‘93, Myszkowski ‘98]

Visible Differences Predictor (VDP)Visible Differences Predictor (VDP)

➔ Predicts local differences between images ➔ Takes into account important visual charac-

teristics:➔ Amplitude compression➔ Advanced CSF model➔ Masking

➔ Uses the cortex transform, which is a pyra-mid-style, invertible & computationally effi-cient image representation

Eras

mus

+ s

emin

ar, 18

/04/

2016

30 / 64



VDP: ResultsVDP: Results

Reference

Analysed

Pixel differences:Reference - Analysed

Pixel differences

The VDP response:probability ofperceivingthe differences

VDP response

Eras

mus

+ s

emin

ar, 18

/04/

2016

31 / 64



imagef(x,y)

Conversionto digital form

Imagepre-processing

Featuresextraction

Conversion to outputform

Output image

Features

DIP schemeDIP scheme

local transform

point transform

global transform

Eras

mus

+ s

emin

ar, 18

/04/

2016

32 / 64



f(x)

x

b

H(b)

180 200 220 2400

50

100

e

H(e

)

180

200

220

240

0

50

100

Histogram stretching along a defined line changes the distribution of in-tensities in an image by the alterna-tion of intensity assignment in each interval

Each interval changes its width:

whereb –pixel intensity before:e –pixel intensity after stretching;

f(b) –stretching function.

The tangent of an angle of function f(b) is the coeficient that changes the width of each histogram interval

d e= f ' bd b

Histogram modellingHistogram modelling

Eras

mus

+ s

emin

ar, 18

/04/

2016

33 / 64



The most simple is a linear stretching:

Where a can is equal to:

wherex

1, x

2– boundaries of intensity.

E – maximum possible intensity

f (x )={ 0 for x<0ax

E for x>E

a=E

x2−x1

Simple linear caseSimple linear case

50 100 150 200

0

1000

2000

3000

b

H(b)

f(x)

x

5010

015

020

0

0

1000

2000

3000

e

H(e

)

x1

x2

Eras

mus

+ s

emin

ar, 18

/04/

2016

34 / 64



histogramSource image

Non-linear cases (examples)Non-linear cases (examples)

Eras

mus

+ s

emin

ar, 18

/04/

2016

35 / 64



It usually increases the global contrast of images, especially when the usable data of the image is represented by close contrast values.

Through this adjustment, the intensities can be better distributed on the histo-gram. Areas of lower local contrast gain a higher contrast.

Histogram equalizationHistogram equalization

0 2 4 6 80

1

2

3

b

H(b)

mean

0 2 4 6 80

1

2

3

e

H(e)

mean

Eras

mus

+ s

emin

ar, 18

/04/

2016

36 / 64



Work in RGB spaceWork in RGB space

original RGB equalized

Eras

mus

+ s

emin

ar, 18

/04/

2016

37 / 64



Work in HSL spaceWork in HSL space

HSL equalized

Eras

mus

+ s

emin

ar, 18

/04/

2016

38 / 64



RGB and HSL comparisonRGB and HSL comparison

original RGB equalized HSL equalized

Eras

mus

+ s

emin

ar, 18

/04/

2016

39 / 64



One-dimensional histogram if defined by function f :

f : X×Y Zf −1 : Z 2 X×Y

f −1 : {x , y ∣ f x , y=z }

1D vs 2D histogram1D vs 2D histogram

Two-dimensional histogram if defined by functions f and g :

f : X×Y Zg : X ×Y Vf −1 : Z 2 X×Y

g−1 : V 2 X×Y

f −1 : {x , y ∣ f x , y=z }g−1 : {x , y ∣g x , y =v }

Eras

mus

+ s

emin

ar, 18

/04/

2016

40 / 64



There are many 2D histograms! One of the most useful is coocur-rence matrix

M 1=[0 0 0 00 1 1 10 1 2 20 1 2 3

];

z=[0 1 2 3] ;H 1(z )=[7 53 1];

M 2 =[1 3 2 0 2 0 1 0 1 0 2 0 0 0 1 1

] ;

z=[0 1 2 3 ];H 2 z=[7 5 3 1];

Co-occurrence matrixCo-occurrence matrix

r={x , y ,x , y1 };C r=H fg z , v ;

f x , y =g x , y1;

C r1=[

3 3 0 0 0 2 2 0 0 0 1 1 0 0 0 0

]; C r2=[

1 2 1 0 2 1 0 1 3 0 0 0 0 0 1 0

];← 1D Histograms →

← 2D Histograms →

Eras

mus

+ s

emin

ar, 18

/04/

2016

41 / 64



Example of calculation on real image – it helps when we want to tell if the image is crisp or blurred

ExampleExample

Eras

mus

+ s

emin

ar, 18

/04/

2016

42 / 64



exampleexample

Intensity thresholding

for

for

Eras

mus

+ s

emin

ar, 18

/04/

2016

43 / 64



In digital image processing convolutional filtering plays an important role in:

➔ Edge detection and related processes;➔ Sharpening;➔ Blurring;➔ Special effects (motion blur)➔ Etc...

Traditional computing (sequential programming);Parallel computing (mult processors/cores, GPU: „stencil computing”).

Convolutional filteringConvolutional filtering

Eras

mus

+ s

emin

ar, 18

/04/

2016

44 / 64



In practice, f and g are vectors or matrices with discrete values, and integral operator is changed into sum.

Convolutional filteringConvolutional filtering

h [ x ]=∑t=t1

t=t n

f [ x−t ] g [t ]

f1

f2

f3

f4

f5

f6

f7

f8

g3

g2

g1

* * *

h1

h2

h3

h4

h5

h6

norm

(window .*mask)

norm

f∗g=∫−∞

∞

f (x−t)g( t)dt

Eras

mus

+ s

emin

ar, 18

/04/

2016

45 / 64



1

1

1

1 1

1 1

1 1

Norm=9

1

1

1

1 1

2 1

1 1

Norm=10

1

1

1

1 1

3 1

1 1

Norm=11

0

1

0

0 0

1 1

0 0

Norm=3

Averaging filterAveraging filter

1

1

1 1

1

Norm=5

1

1

1

1

1

Norm=5

1

1

1

1 1

1 1

1 1

1

1

1

1 1

1 1

1 1

Norm=21

1

1

1

Eras

mus

+ s

emin

ar, 18

/04/

2016

46 / 64



An image f is filtered with a mask gσ which is a discrete appro-ximation of two-dimensional Gauss function:

Gauss filteringGauss filtering

decides about blurring effect

Eras

mus

+ s

emin

ar, 18

/04/

2016

47 / 64



Edge detectionEdge detection

Edges can be detected using various gradient operators:➔ First derivative of an image shows the edge and its direction➔ Point of sign change of second derivative (zero crossing), can also be

used to detect edges

The main problem is false detection, which comes from the amplification of noise!

Secondderivative

image

Intenstyprojection

Firstderivative

The edge is a local change in image intensity and its vertical (or horizontal) projection can look like that presented above

Eras

mus

+ s

emin

ar, 18

/04/

2016

48 / 64



8 2 222

Horizontal lines Vertical lines+45o -45opoint detection

Line detectionLine detection

Eras

mus

+ s

emin

ar, 18

/04/

2016

49 / 64



ow( j , k)=√[ A4−A8 ]2+[A5−A7 ]

2 0

Roberts vs PrewittRoberts vs Prewitt

A0

A1

A2

A3

A4

A5

A6

A7

A8

ow j , k = X 2 Y 2

X=A2 2 A3 A4 −A0 2 A7 A6 Y=A0 2 A1 A2 − A6 2 A5 A4

ow(j,k)

ow(j,k)

Roberts filtering

Prewitt filtering

Eras

mus

+ s

emin

ar, 18

/04/

2016

50 / 64



Prewitt vs SobelPrewitt vs Sobel

PrewittPrewitt SobelSobel

Eras

mus

+ s

emin

ar, 18

/04/

2016

51 / 64



Laplace operator (Laplasian) is defined as a second derivative of image f at the location (x,y)

Z1

Z2

Z3

Z4

Z5

Z6

Z7

Z8

Z9

Laplace operatorLaplace operator

or

Eras

mus

+ s

emin

ar, 18

/04/

2016

52 / 64



ow( j , k )=max {1 , maxi∈⟨0 ;7⟩

∣5S i−3T i∣}S i=Ai+Ai+1+Ai+2

T i=Ai+3+Ai+4+Ai+ 5+Ai+6+Ai+7

i∈⟨0 ;7⟩

indexes change modulo 8

KirschKirsch

where

Eras

mus

+ s

emin

ar, 18

/04/

2016

53 / 64



Canny edge detectorCanny edge detector

➔ multi-stage algorithm to detect a wide range of edges in images

➔ developed by John F. Canny in 1986➔ Canny also produced a computational theory of edge

detection explaining why the technique works.

An "optimal" edge detector means:

good detection – the algorithm should mark as many real edges in the image as possible.good localization – edges marked should be as close as possible to the edge in the real image.minimal response – a given edge in the image should only be marked once, and where possible, image noise should not create false edges.

Eras

mus

+ s

emin

ar, 18

/04/

2016

54 / 64



1. Image smoothing using Gaussian

2. Derivatives calulation using masks: [-1 0 1] i [-10 1]'.

Canny Edge DetectorCanny Edge Detector

Eras

mus

+ s

emin

ar, 18

/04/

2016

55 / 64



3. Non-maximum suppression as an edge thin-ning technique.

A 3x3 filter is moced over an image and at every lo-cation, it suppresses the edge strength of the center pixel (by setting its value to 0) if its magnitude is not greater than the magnitude of the two neigh-bors in the gradient direction

4. Tracing edges through the image and hy-steresis thresholding

Canny Edge DetectorCanny Edge Detector

Eras

mus

+ s

emin

ar, 18

/04/

2016

56 / 64



Non-linear filteringNon-linear filtering

Output image's pixels result from a nonlinear transform of input image's pixels and a filter maskExample: Media filterInput set: A={9,88,1,15,43,100,2,34,102} Sort elements in A (increasing ➔order): B=sort(A)B={1,2,9,15,34,43,88,100,102} Select median of B (middle element): ➔median(B)=34

Eras

mus

+ s

emin

ar, 18

/04/

2016

57 / 64



Non-linear filteringNon-linear filtering

Eras

mus

+ s

emin

ar, 18

/04/

2016

58 / 64



Adaptive filteringAdaptive filtering

Eras

mus

+ s

emin

ar, 18

/04/

2016

59 / 64



Detecting charactersitic pointsDetecting charactersitic points

Objects/scene detection can be based on detecting charac-teristic points

●Matching point Pij in the image j to the point P

ik in the image k

●Removing false candidates● Certain points P

ij in the image j have no corresponding points P

ik

in the image k ●Ambiguity

● Several points Pij in the image j correspond to a point P

ik

●Noise

Eras

mus

+ s

emin

ar, 18

/04/

2016

60 / 64



How?How?

Corner operator is one solution...

Eras

mus

+ s

emin

ar, 18

/04/

2016

61 / 64



IdeaIdea

It is a possibility that such interesting point may be detected by looking at the image through some small window.By sliding this window over the image we can de-tect significant changes in intensity in a certain di-rection

●Morevec detector●Harris detector

Eras

mus

+ s

emin

ar, 18

/04/

2016

62 / 64



Moravec detectorMoravec detector

There are 3 cases:

●If an area is uniform (flat), the dif-ferences calculated in all directions will be not significant

●If it is an edge, the diferences along its direction will be small, while in the perpendicular direction – large

●If there is an isolated point, the di-ferences in most of directions will be significant

●Finally, the maxima of points with the highest differences are selected

flat edge

cornerisolated point

Eras

mus

+ s

emin

ar, 18

/04/

2016

63 / 64



Harris detectorHarris detector

R(x,y)=det(M) - (trace(M))2

Eras

mus

+ s

emin

ar, 18

/04/

2016

64 / 64



ComparisonComparison

Harris Moravec