In the name of Allah - Sharif

In the name of Allah

the compassionate, the merciful

Kasaei 3

Digital Image Processing

S. KasaeiS. Kasaei

Sharif University of TechnologyRoom: CE 307

E-Mail: [email protected] Page: http://ce.sharif.edu

http://ipl.ce.sharif.eduhttp://sharif.edu/~skasaei

Chapter 2

Two-Dimensional Systems&

Mathematical Preliminaries

Kasaei 5

Notations & Definitions

Kasaei 6

Notations & Definitions1-D continuous signal:

1-D sampled signal:

Continuous image:

Sampled image:

[2 (or higher)-D sequence of real numbers.]

Complex conjugate:

Separable functions:

)(),(),( tsxuxf

nunu ),(

),(),,(),,( yxvyxuyxf

),,(,),,( , kjiuunmu nm

*z)()(),( 21 yfxfyxf =

Kasaei 7


Kasaei 8


Kasaei 9


Kasaei 10


Kasaei 11


Kasaei 12

Linear Systems & Shift Invariance

Kasaei 13


Kasaei 14


Kasaei 15


Kasaei 16


Kasaei 17


Kasaei 18


Kasaei 19


Kasaei 20

The Fourier Transform

Kasaei 21


Kasaei 22


Kasaei 23

Spatial Frequency

Spatial frequency measures how fast the image intensity changes in the image plane.Spatial frequency can be completelycharacterized by the variation frequencies in two orthogonal directions (e.g., horizontal & vertical):

fx: cycles/horizontal unit distance.fy : cycles/vertical unit distance.

It can also be specified by magnitude and angleof change:

)/arctan(,22xyyxm fffff =+= θ

Kasaei 24

Illustration of Spatial Frequency

Kasaei 25

Illustration of Spatial Frequency

Kasaei 26

Angular Frequency

ee)cycle/degr(f180

ff

(degree)180n)h/2d(radia2(radian))2/arctan(2

ss

hd

dhdh

πθ

πθ

θ ==

=≈=

Problem with previous defined spatial frequency:Perceived speed of change depends on the viewing distance.

Kasaei 27


Kasaei 28


Kasaei 29


Kasaei 30


Kasaei 31


Kasaei 32


Kasaei 33


Kasaei 34


Kasaei 35


Kasaei 36


Kasaei 37


Kasaei 38

The Z-Transform

Kasaei 39

The Z-Transform

Kasaei 40

The Z-Transform

Kasaei 41

The Z-Transform

Kasaei 42

The Z-Transform

Kasaei 43

The Z-Transform

Kasaei 44

The Z-Transform

Kasaei 45

Matrix Theory Results

Kasaei 46


Kasaei 47


(a) 2-D Cartesian coordinate representation. (b) Matrix representation.

(a) (b)

Kasaei 48


N

Kasaei 49


Kasaei 50


Kasaei 51


Kasaei 52


Kasaei 53


Kasaei 54


Kasaei 55


Kasaei 56


Kasaei 57


Kasaei 58


Kasaei 59


Kasaei 60


Kasaei 61


Kasaei 62


Red line: direction of the first principal component, Green line: direction of the second principal component.

Data set. Principal axes (eigenvectors). Rotated data set.

Kasaei 63

Block Matrices & Kronecker Products

Kasaei 64


Kasaei 65


Kasaei 66


Kasaei 67


Kasaei 68


Kasaei 69


Kasaei 70


Kasaei 71


Kasaei 72


Kasaei 73


Kasaei 74


Probability, Random Variables, & Random Signal

Processing

A Brief Review

Kasaei 76

References

1. review_of_probability, by Rafael C. Gonzalez and Richard E. Woods, 2002.

2. Lecture Notes on Probability Theory and Random Processes, by Jean Walrand, Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720, 2004.

3. Probability, Random Variables, and Random Signal Principles, by Peyton Z. Peebles, JR., McGraw-Hill, 3rd Edition, 1993, ISBN:0-07-112782-8.

4. Probability, Random Variables, and Stochastic Processes, by Athanasios Papoulis, McGraw-Hill, 1991 (QA 273 .P2 1991).

Famous Theoreticians of the Field

1654 -1987

Kasaei 78

Jacob BERNOULLI1654-1705

Kasaei 79

Abraham DE MOIVRE1667 -1754

Kasaei 80

Thomas BAYES1701-1761

Kasaei 81

Thomas SIMPSON1710-1761

Kasaei 82

Pierre Simon LAPLACE1749-1827

Kasaei 83

Adrien Marie LEGENDRE 1752-1833

Kasaei 84

Carl Friedrich GAUSS1777-1855

Kasaei 85

Andrei Andreyevich MARKOV1856-1922

Kasaei 86

Andrei Nikolaevich KOLMOGOROV1903-1987

Modeling Uncertainty

Kasaei 88

Modeling Uncertainty

In this lecture we introduce the concept of a model of an uncertain physical system and we stress the importance of concepts that justify the structure of the theory.

Kasaei 89

Modeling Uncertainty:Models and Physical Reality

General concept:physical world uncertain outcomes model of uncertaintyProbability Theory

Kasaei 90

Modeling Uncertainty:Models and Physical Reality

“Probability Theory” is a mathematical model of uncertainty.It is important to appreciate the difference between uncertainty in the physical world and the models of “Probability Theory”.

That difference is similar to that between the real world and laws of theoretical physics. Consider flipping a fair coin repeatedly. Designate by 0and 1 the two possible outcomes of a coin flip (say 0for head and 1 for tail). This experiment takes place in the physical world. The outcomes are uncertain.

Here we try to appreciate the probability model of this experiment and to relate it to the physical reality.

Kasaei 91

Modeling Uncertainty:Function of Hidden Variable

One idea is that the uncertainty in the world is fully contained in the selection of some hidden variable.

If this variable was known, then nothing would be uncertain anymore.In other words, everything that is uncertain is a functionof that hidden variable. By function, we mean that if we know the hidden variable, then we know everything else. Everything that is random is some function X of some hidden variable.

If we designate the outcome of the 5th coin flip by X, then we conclude that X is a function of w. We can denote that function by X(w).

Kasaei 92

Some Basic Definitions

Probability Space:Now, we describe the probability model of “choosing an object at random" .We explain that the key idea is to associate a likelihood, which we call probability, to sets of outcomes (not to individual outcomes). These sets are events. The description of the events and of their probability constitute a probability space that completely characterizes a random experiment.

Kasaei 93


Events:

Probability events are modeled as sets.

The sets of outcomes to which one assigns a probability are called events (the event of getting a tail).

It is not necessary (and often not possible) for every set of outcomes to be an event.

Kasaei 94

Some Basic DefinitionsProbability Space:

Putting together the observations of the sections above, we have defined a probability space as follows:

A probability space is a triplet {Q, F, P} where:Q is a nonempty set, called the sample space.F is a collection of subsets - closed under countable set operations. The elements of F are called events.P is a countable additive function from F into [0, 1] such that P(Q) = 1, called a probability measure.

Example:Books at SUT library (sample space)

CE books (events)Probability of existence of a special book (probability measure)

Kasaei 95


Examples will clarify the probability space definition:The main point is that one defines the probability of sets of outcomes (the events).The probability should be countable additive (to be continuous). Accordingly (to be able to write down this property), and also quite intuitively, the collection of events should be closed under countable set operations.

Kasaei 96

Sets & Set Operations

A set is a collection of objects, with each object in a set often referred to as an element or member of the set (e.g., the set of all image processing books).

The set with no elements is called the empty or null set, denoted by the symbol Ø.

The sample space is defined as the set containing all elements of interest in a given situation.

Kasaei 97

Basic Set Operations

The union of two sets A and B is denoted by:

The intersection of two sets A and B is denoted by:

Two sets having no elements in common are said to be disjoint or mutually exclusive, denoted by:

Kasaei 98

Basic Set Operations

Kasaei 99

Basic Set Operations (Venn Diagram)

It is often quite useful to represent sets and sets operations in a so-called Venn diagram, in which:

S is represented as a rectangle.Sets are represented as areas (typically circles).Points are associated with elements.

The following example shows various uses of Venn diagrams.

The shaded areas are the result (sets of points).

Kasaei 100

Basic Set Operations (Venn Diagrams)

Kasaei 101

Basic Set Operations (Venn Diagrams)

The top row are self explanatory. The diagrams in the bottom row are used to prove the validity of the expression

which is used in the proof of some probability relationships.

Kasaei 102

Relative Frequency & ProbabilityA random experiment is an experiment in which it is not possible to predict the outcome (e.g., tossing of a coin).

The probability of the event is denoted by P(event).

For an event A, we have:

where,

Kasaei 103

Relative Frequency & Probability

If A and B are mutually exclusive it follows that the set AB is empty, and consequently, P(AB) = 0.

The conditional probability is denoted by P(A/B), where P(A/B) refers to the probability of A given B.

( ) φ==

∑

==nm

N

nnn

N

nAAifAPAP IU

11

Kasaei 104

Conditional Probability

Assume that we know that the outcome is in B. Given that information, what is the probability that the outcome is in A? This probability is written as P[A|B] and is read “the conditional probability of A given B," or “the probability of A given B", for short.

Kasaei 105

Relative Frequency & ProbabilityRelative Frequency & Probability

A little manipulation of the preceding results yields the following important relationships:

and

The second expression may be written as:

which is known as Bayes' theorem, so named after the 18th century mathematician Thomas Bayes.

a priori

likelihood

a posteriori

Kasaei 106

Bayes' Theorem

The importance of the prior distribution: Bayes' rule.

Bayes understood how to include systematically the information about the prior distribution (a priori) in the calculation of the posterior distribution (a posteriori).

He discovered what we know today as Bayes' rule, a simple but very useful identity.

Kasaei 107

Bayes' Theorem

This formula extends to a finite number of events Bn that partition Q. Think of the Bn as possible “causes” of some effect A.

You know the prior probabilities P(Bn) of the causes and also the probability that each cause provokes the effect A.The formula tells you how to calculate the probability that a given cause has provoked the observed effect.

Applications are abound, as we will see in detection theory. For instance:

You alarm can sound either if there is a burglar or also if there is no burglar (false alarm). Given that the alarm sounds, what is the probability that it is a false alarm?

Kasaei 108

If A and B are statistically independent, then P(B/A) = P(B) and it follows that:

and

For mutually exclusive, A ∩ B = Ø from which it follows that P(AB) = P(A ∩ B) = 0.

So, the two sets are statistically independent if P(AB)=P(A)P(B), which we assume to be nonzero in general. Thus, for two events to be statistically independent, they cannot be mutually exclusive.

Relative Frequency & ProbabilityRelative Frequency & Probability

Kasaei 109

Relative Frequency & Probability

In general, for N events to be statistically independent, it must be true that, for all combinations 1 ≤i ≤ j ≤k ≤ . . . ≤N, we have:

Kasaei 110

Random Variables

The definition is: “A real random variable (RV) is a measurable real-valued function of the outcome of a random experiment”.

Physical examples: Noise voltage at a given time and place.Temperature at a given time and place.Height of the next person to enter the room.

Kasaei 111

Random Variables

A real random variable, x, is a real-valued function defined on the events of a sample space, S (i.e., X(s)).

In words, for each event in S, there is a real number that is the corresponding value of the RV.

Viewed yet another way, a RV maps each eventin S onto the real line.

A complex RV, Z, can be defined in terms of real RVs, X and Y, by: Z=X+jY (the joint density of X and Y must be used.).

Kasaei 112

Random Variables

A random variable mapping of a sample space.

Kasaei 113

Random Variables

Thus far we have been concerned with discreterandom variables.

In the discrete case, the probabilities of events are numbers between 0 and 1.

When dealing with continuous quantities (which are not denumerable) we can no longer talk about the "probability of an event" because that probability is zero.

Kasaei 114

Random Variables

Thus, instead of talking about the probability of a specific value, we talk about the probability that the value of the RV lies in a specified range.

In particular, we are interested in the probability that the RV is less than or equal to a specified constant, a, as:

or

Kasaei 115

Random Variables

• Function F is called the cumulative probability distribution function or simply the cumulative distribution function (CDF).

• If this function is given for all values of a (i.e., −∞ < a < ∞), then the values of RV x have been defined.

Kasaei 116

Random Variables

where x+ = x + ε, with ε being a positive, infinitesimally small number (in words, F(x) is continuous from the right).

•Due to the fact that it is a probability, the CDF has the following properties:

Kasaei 117

Random Variables

•The probability density function (PDF) of the RV x is defined as the derivative of the CDF:

•The PDF satisfies the following properties:

Kasaei 118

Random Variables

(a) CDF and (b) PDF of a discrete random variable.

Kasaei 119

Expected Values & Moments

The expected value of a function g(x) of a continuous RV is defined as:

If the RV is discrete the definition becomes:

Kasaei 120


The expected value of x is equal to its average (or mean) value, defined as:

For discrete RVs as:

Kasaei 121

Of particular importance is the variance of RVs that is normalized by subtracting their mean, as:

and

The square root of the variance is called the standard deviation, and is denoted by σ.


Kasaei 122

The nth central moment of a continuous RV is:

And, for discrete variables as:

where we assume that n ≥ 0. Clearly, µ0=1, µ1=0, and µ2=σ².


Kasaei 123

The central moments: the mean of the RV has been subtracted out.

The moments about the origin: the mean is not subtracted out.

In image processing, moments are used for a variety of purposes, including histogram processing, segmentation, and description.

In general, moments are used to characterize the PDF of an RV.


Kasaei 124

Expected Values & MomentsThe second, third, and fourth central moments are intimately related to the shape of the PDF of an RV.

The second central moment (the variance) is a measure of spread of values of an RV about its mean value.

The third central moment is a measure of skewness(bias to the left or right) of the values of x about the mean value (symmetric PDF 0).

The fourth moment is a relative measure of flatness.

In general, knowing all the moments of a density specifies that density.

Kasaei 125

Gaussian Probability Density Function

A random variable is called Gaussian if it has a probability density of the form:

The CDF corresponding to the Gaussian density is:

Kasaei 126

Gaussian Probability Density Function

(a) PDF and (b) CDF of a Gaussian random variable.

Kasaei 127

Gaussian Random VariablesThe Gaussian distribution is determined by its mean and variance.The sum of independent Gaussian random variables is Gaussian.Random variables are jointly Gaussian if an arbitrary linear combination is Gaussian.Uncorrelated jointly Gaussian random variables are independent.If random variables are jointly Gaussian, then the conditional expectation is linear.

Kasaei 128

Several Random VariablesA collection of random variables is a collection of functions of the outcome of the same random experiment. Here we extend the idea to multiple numerical observations about the same random experiment.Examples include:

We pick a ball randomly from a bag and we note its weight X and its diameter Y.We observe the temperature at a few different locations.We measure the noise voltage at different times.A transmitter sends some signal and the receiver observes the signal it receives and tries to guess which signal the transmitter sent.

Kasaei 129

Several Random Variables

In case, we might need two RVs. This maps our events onto the xy-plane.

Mapping from the sample space S tothe joint sample spaceSJ(xy plane).

Kasaei 130

It is convenient to use vector notation when dealing with several RVs. Thus, we represent a vector random variable x as:

Now, the CDF introduced earlier becomes:


Kasaei 131


As in the single variable case, the PDF of an RV vector is defined in terms of derivatives of the CDF, as:

The expected value of a function of x is defined by:

Kasaei 132

Several Random VariablesThe joint moment becomes:

It is easy to see that ηk0 is the kth moment of x and η0q is the qth moment of y.

The moment η11 = E[xy] is called the correlation of x and y.

Kasaei 133

If the condition:

holds, then the two RVs are said to be uncorrelated.

We know that if x and y are statistically independent, then p(x, y) = p(x)p(y), in which case we write:

Thus, we see that if two RVs are statistically independent then they are also uncorrelated. The converse of this statement is not true in general.


Kasaei 134

The joint central moment of order kq involving RVs x and y is:

where mx = E[x] and my = E[y] are the means of x and y.

Note that:

are the variances of x and y, respectively.

and


Kasaei 135

The moment µ11

is called the covariance of x and y.

As in the case of correlation, the covariance is an important concept, usually given a special symbol such as Cxy.


Kasaei 136

By direct expansion of the terms inside the expected value brackets, and recalling the mx = E[x] and my = E[y], it is straightforward to show that:

From our discussion on correlation, we see that the covariance is zero if the random variables are either uncorrelated or statistically independent.


Kasaei 137

If we divide the covariance by the square root of the product of the variances we obtain:

The quantity γ is called the correlation coefficient of RVs x and y. It can be shown that γ is in the range −1 ≤γ ≤1(the correlation coefficient is used in image processing for matching).


Kasaei 138

The multivariate Gaussian PDF, is defined as:

where n is the dimensionality (number of components) of the random vector x, C is the covariance matrix (to be defined below), |C| is the determinant of matrix C, mis the mean vector (also to be defined below) and Tindicates transposition.

The Multivariate Gaussian Density Function

Kasaei 139

The mean vector is defined as:

and the covariance matrix is defined as:


where:

Kasaei 140


Covariance matrices are real and symmetric.

The elements along the main diagonal of C are the variances of the elements x, such that cii= σxi².

When all the elements of x are uncorrelated or statistically independent, cij = 0, and the covariance matrix becomes a diagonal matrix.

If all the variances are equal, then the covariance matrix becomes proportional to the identity matrix, with the constant of proportionality being the varianceof the elements of x.

Kasaei 141

As an example, consider the bivariate (n = 2) Gaussian PDF of:

with

and


Kasaei 142

Where, because C is known to be symmetric, c12 = c21.

A schematic diagram of this density is shown in Part (a) of the following figure. Part (b) is a horizontal slice of Part (a).

The main directions of data spread are in the directions of the eigenvectors of C.

If the variables are uncorrelated or statistically independent, the covariance matrix will be diagonal and the eigenvectors will be in the same direction as the coordinate axes x1 and x2 (and the ellipse shown would be oriented along the x1 - and x2-axis).


Kasaei 143


(a) Sketch of the joint density function of two Gaussian RVS.(b) A horizontal slice of (a).

Kasaei 144

If, the variances along the main diagonal are equal,the density would be symmetrical in all directions (in the form of a bell) and Part (b) would be a circle.

Note in Parts (a) and (b) that the density is centered at the mean values (m1,m2).


Kasaei 145

Random Processes

We have looked at a finite number of random variables. In many applications, one is interested in the evolution in time of random variables. For instance:

One watches on an oscilloscope the noise across two terminals. One may observe packets that arrive at an Internet router.One may observe cosmic rays hitting a detector.

Kasaei 146

Random Processes

We explained that a collection of random variables is characterized by their joint CDF. Similarly, a random process is characterized by the joint CDF of any finite collection of the random variables. These joint CDF are called the finite dimensional distributions of the random process. Obviously, to correspond to a random process, the finite dimensional distributions must be consistent.

Kasaei 147

Random SignalsThe concept of random process is based on enlarging the RV concept to include time.

A random process represents a family or ensembleof time functions.

Each member time function is called a sample function, ensemble member, or a realization of the process.

A complex discrete random signal or a discrete random process is a sequence of RVs u(n).

Kasaei 148

Random Signals

A continuous random process.

Kasaei 149

Ergodicity Random Processes

Roughly, a stochastic process is ergodic if statistics that do not depend on the initial phase of the process are constant. That is, such statistics do not depend on the realization of the process. For instance, if you simulate an ergodic process, you need only one simulation run; it is representative of all possible runs.

Kasaei 150

Random Signals

Kasaei 151

Random Signals

Kasaei 152

Random Signals

Kasaei 153

Random Signals

Kasaei 154

Random Signals

Kasaei 155

Random Signals

Kasaei 156

Random Signals

Kasaei 157

Random Signals

Kasaei 158

Markov Process

A random process X(t) is Markov if: given X(t), the past and the future are independent.Markov chains are examples of Markov process. A process with independent increments is Markov.Note that a function of a Markov process may not be a Markov process.

Kasaei 159

Random Signals

Kasaei 160

Random Signals

Kasaei 161

Random Signals

Kasaei 162

Random Signals

Kasaei 163

Random Signals

Kasaei 164

Random Signals

Uncorr.

Indep.

Kasaei 165

Discrete Random Fields

In statistical representation of image, each pixel is considered as an RV.

We think of a given image as a sample function of an ensemble of images.

Ensemble of images ordiscrete random field.

Sample function orrandom image.R.V.

Kasaei 166


Such an ensemble would be adequately defined by a joint PDF of the array of RVs.

For practical image sizes, the number of RVs is very large (262,144 for 512x512 images).

Thus, it is difficult to specify a realistic joint PDF.

One possibility is to specify the ensemble by its first -and second-order moments only.

Kasaei 167


Kasaei 168


Kasaei 169


Kasaei 170


Kasaei 171


Kasaei 172


Kasaei 173


Kasaei 174

The Spectral Density Function

Kasaei 175


Kasaei 176


Kasaei 177


Kasaei 178


Kasaei 179


Kasaei 180


Kasaei 181


Kasaei 182


Kasaei 183

Some Results from Information Theory

Information theory gives some important concepts that are useful in digital representation of images.

Some of these concepts will be used in image quantization, image transforms, and image data compression.

The information, entropy, and rate-distortion functionare the main issues concerned in this regard. They will be briefly introduced in the following.

Kasaei 184


Kasaei 185


Kasaei 186


Entropy of a binary source.

Kasaei 187


Kasaei 188


Kasaei 189


Kasaei 190


Rate-distortion function for a Gaussian source.

Kasaei 191


Kasaei 192


Kasaei 193


Kasaei 194


Kasaei 195

DetectionThe detection problem is roughly as follows. We want to guess which of finitely many possible causesproduced an observed effect. For instance:

You have a fever (observed effect); do you think you have the flu or a cold or the malaria? You observe some strange shape on an X-ray; is it a cancer or some infection of the tissues? A receiver gets a particular waveform; did the transmitter send the bit 0 or the bit 1? (Hypothesis testing is similar.) There are two basic formulations: either we know the prior probabilities of the possible causes (Bayesian) or we do not (non-Bayesian). When we do not, we can look for the maximum likelihood (ML) detection or we can formulate a hypothesis-testing problem.

Kasaei 196

Estimation

The estimation problem is similar to the detection problem except that the unobserved random variableX does not take values in a finite set. That is, one observes Y and must compute an estimate of X based on Y that is close to X in some sense.Once again, one has Bayesian and non-Bayesianformulations. The non-Bayesian case typically uses maximum likelihood estimation, MLE[X|Y ], defined as in the discussion of detection.

The End

Documents

In the name of Allah - Sharif