Computer Vision Chapter 1 Introduction. The goal of computer vision is to make useful decisions about real physical objects and scenes based on sensed

Computer VisionComputer Vision

Chapter 1 IntroductionChapter 1 Introduction

The goal of computer vision is to The goal of computer vision is to make useful decisions about real make useful decisions about real physical objects and scenes based on physical objects and scenes based on sensed images.sensed images.

Applications areasApplications areas

Industrial inspectionIndustrial inspection Medical imagingMedical imaging Image database and queryImage database and query Satellite and surveillance imagerySatellite and surveillance imagery EntertainmentEntertainment Handwriting and printed character Handwriting and printed character

recognitionrecognition

Image dimensionalityImage dimensionality

1D1D– audio (sound)audio (sound)

2D2D– digital camera picture, chest x-ray, ultrasounddigital camera picture, chest x-ray, ultrasound

3D3D– video sequence of 2D imagesvideo sequence of 2D images– multispectral 2D imagesmultispectral 2D images– volumetric medical imagery (CT, MRI)volumetric medical imagery (CT, MRI)

4D4D– PET-CTPET-CT– MRIMRI

Image typesImage types

BinaryBinary

GrayscaleGrayscale

ColorColor

MultispectralMultispectral

Operations on imagesOperations on images

Neighborhood (local) operationsNeighborhood (local) operations Enhancing the entire imageEnhancing the entire image Combining multiple imagesCombining multiple images

– Ex. differences, noise reduction, Ex. differences, noise reduction, blendingblending

Feature extractionFeature extraction– Ex. area, centroid (center of mass), Ex. area, centroid (center of mass),

orientation, linesorientation, lines– invariantsinvariants

Extracting featuresExtracting features

Example featuresExample features

General hardware discussionGeneral hardware discussion

General purpose vs. special purpose General purpose vs. special purpose (DSP, GPU)(DSP, GPU)

Uniprocessors vs. parallel processors Uniprocessors vs. parallel processors (COWs, multiprocessors)(COWs, multiprocessors)

Sensors (discussed later)Sensors (discussed later)

General software discussionGeneral software discussion

Android SDKAndroid SDK– Java-basedJava-based– freely available from freely available from

http://developer.android.com/sdk/index.htmlhttp://developer.android.com/sdk/index.html– Albie start appAlbie start app

doxygen for source code documentationdoxygen for source code documentation– freely available from doxygen.orgfreely available from doxygen.org

code formatcode format– http://www.oracle.com/technetwork/java/http://www.oracle.com/technetwork/java/

codeconv-138413.htmlcodeconv-138413.html

General software discussionGeneral software discussion

C#C#– use Visual C# (Express Edition is freely use Visual C# (Express Edition is freely

available from Microsoft)available from Microsoft)– CSImageViewer starter appCSImageViewer starter app– main course web page has linksmain course web page has links

doxygen for source code documentationdoxygen for source code documentation

code formatcode format

INTRODUCTION TO DOXYGENINTRODUCTION TO DOXYGEN

What’s in a program file?What’s in a program file?

1.1. CommentsComments

2.2. CodeCode

What’s a compiler?What’s a compiler?

A programA program– InputInput

– ProcessingProcessing

– OutputOutput

What’s a compiler?What’s a compiler?

A programA program– Input:Input:

Text file (your program)Text file (your program)

– Processing:Processing:Convert HLL statements into machine code Convert HLL statements into machine code

(or similar)(or similar) Ignore commentsIgnore comments

– Output:Output:A binary file of machine code (or similar)A binary file of machine code (or similar)

Traditional documentationTraditional documentation

Code files are separate from design Code files are separate from design documents.documents.

Wouldn’t it be great if we could bring Wouldn’t it be great if we could bring code and documentation together code and documentation together into the same file(s)?into the same file(s)?

Tools like doxygen and javadocTools like doxygen and javadoc

A programA program– Input:Input:

Text file (your program)Text file (your program)

– Processing:Processing:Convert (specially formatted) comments into Convert (specially formatted) comments into

documentationdocumentation Ignore HLL statementsIgnore HLL statements

– Output:Output:Documentation (typically in HTML)Documentation (typically in HTML)

Getting started with doxygen Download from doxyDownload from doxygen.org. Do this only once in directory (folder)

containing your source code: (already done for you)doxygen –g

This creates a doxygen configuration file called Doxyfile which you may edit to change default options.

Edit Doxyfile and make sure all EXTRACTs are YES Then whenever you change your code and

wish to update the documentation:doxygen

which updates all documentation in html subdirectory Demonstrate.

Using doxygen: document every (source code) file

/** * \file ImageData.java * \brief contains ImageData class definition (note that this * class is abstract) * * <more verbose description here> * \author George J. Grevera, Ph.D. */...

Using doxygen: document every class

//----------------------------------------------------------------------/** \brief CSImageViewer class. * * Longer description goes here. */public class CSImageViewer : Form {...

Using doxygen: document every function

//----------------------------------------------------------------/** \brief Given a pixel's row and column location, this * function returns the gray pixel value. * \param row image row * \param col image column * \returns the gray pixel value at that position */public int getGray ( int row, int col ) { int offset = row * mW + col; return mOriginalData[ offset ];}

Using doxygen: document every function (parameters)


Using doxygen: document every function (return value)


Using doxygen: document all class Using doxygen: document all class members (and global and static members (and global and static

variables in C/C++)variables in C/C++)

protected bool mIsColor; ///< true if color (rgb); false if grayprotected bool mImageModified; ///< true if image has been modifiedprotected int mW; ///< image widthprotected int mH; ///< image heightprotected int mMin; ///< overall min image pixel valueprotected int mMax; ///< overall max image pixel valueprotected String mFname; ///< (optional) file name

doxygendoxygen(lengthier example including html)(lengthier example including html)

/** \brief Actual original (unmodified) unpacked (1 component per * array entry) image data. * * If the image data are gray, each entry in this array represents a * gray pixel value. So mImageData[0] is the first pixel's gray * value, mImageData[1] is the second pixel's gray value, and so * on. Each value may be 8 bits or 16 bits. 16 bits allows for * values in the range [0..65535]. * <br> <br> * If the image data are color, triples of entries (i.e., 3) represent * each color rgb value. So each value is in [0..255] for 24-bit * color where each component is 8 bits. So mImageData[0] is the * first pixel's red value, mImageData[1] is the first pixel's green * value, mImageData[2] is the first pixel's blue value, mImageData[3] * is the second pixel's red value, and so on. */protected int[] mOriginalData;

Required documentation rulesRequired documentation rules Each file, class, method, and member Each file, class, method, and member

variable variable mustmust be documented w/ be documented w/ doxygen.doxygen.– Exception is when we follow the one-class-per-Exception is when we follow the one-class-per-

file rule. In that case only the class or file file rule. In that case only the class or file needs to be documented.needs to be documented.

The contents of the body of each method The contents of the body of each method should contain comments, but should contain comments, but nonenone of of these comments should be in the doxygen these comments should be in the doxygen format. (Not every comment is a doxygen format. (Not every comment is a doxygen comment.)comment.)

Not every comment should be a Not every comment should be a doxygen comment.doxygen comment.

Required:Required:1.1. every file/classevery file/class

2.2. every function/methodevery function/method

3.3. every class member (data)every class member (data)

4.4. (in C/C++, every static and/or global (in C/C++, every static and/or global variable)variable)

Use regular, plain comments in the body of a Use regular, plain comments in the body of a function/method. (One exception is the \function/method. (One exception is the \todo.)todo.)

int mColorImageData[][][]; int mColorImageData[][][]; ///< should be mColorImageData[mH][mW][3]///< should be mColorImageData[mH][mW][3] //----------------------------------------------------------------------//---------------------------------------------------------------------- /** \brief Given a buffered image, this ctor reads the image data, stores /** \brief Given a buffered image, this ctor reads the image data, stores * the raw pixel data in an array, and creates a displayable version of* the raw pixel data in an array, and creates a displayable version of * the image. Note that this ctor is protected. The user should only * the image. Note that this ctor is protected. The user should only * use ImageData.load( fileName ) to instantiate an object of this type.* use ImageData.load( fileName ) to instantiate an object of this type. * \param bi buffered image used to construct this class instance* \param bi buffered image used to construct this class instance * \param w width of image* \param w width of image * \param h height of image* \param h height of image * \returns nothing (constructor)* \returns nothing (constructor) */*/ protected ColorImageData ( final BufferedImage bi, final int w, final int h ) {protected ColorImageData ( final BufferedImage bi, final int w, final int h ) { mW = w;mW = w; mH = h;mH = h; mOriginalImage = bi;mOriginalImage = bi; mIsColor = true;mIsColor = true; //format TYPE_INT_ARGB will be saved to mDisplayData//format TYPE_INT_ARGB will be saved to mDisplayData mDisplayData = mOriginalImage.getRGB(0, 0, mW, mH, null, 0, mW);mDisplayData = mOriginalImage.getRGB(0, 0, mW, mH, null, 0, mW); mImageData = new int[ mW * mH * 3 ];mImageData = new int[ mW * mH * 3 ]; mMin = mMax = mDisplayData[0] & 0xff;mMin = mMax = mDisplayData[0] & 0xff; for (int i=0,j=0; i<mDisplayData.length; i++) {for (int i=0,j=0; i<mDisplayData.length; i++) { mDisplayData[i] &= 0xffffff; mDisplayData[i] &= 0xffffff; //just to insure that we only have 24-bit rgb//just to insure that we only have 24-bit rgb final int r = (mDisplayData[i] & 0xff0000) >> 16;final int r = (mDisplayData[i] & 0xff0000) >> 16; final int g = (mDisplayData[i] & 0xff00) >> 8;final int g = (mDisplayData[i] & 0xff00) >> 8; final int b = mDisplayData[i] & 0xff;final int b = mDisplayData[i] & 0xff; if (r<mMin) mMin = r;if (r<mMin) mMin = r; if (g<mMin) mMin = g;if (g<mMin) mMin = g;……

Summary of most useful tagsSummary of most useful tags

\file\file\author\author\brief\brief\param\param\returns\returns\todo (not used in assignments)\todo (not used in assignments)

And many, many others.And many, many others.

THE GOOD, THE BAD, AND THE GOOD, THE BAD, AND THE UGLYTHE UGLY

Back to images and imaging…Back to images and imaging…

The good, the bad, and the ugly.The good, the bad, and the ugly.

Success is usually hard won!Success is usually hard won! Problems:Problems:

1.1. Matching models to realityMatching models to reality2.2. Lighting variationLighting variation3.3. Sensor noiseSensor noise4.4. Occlusion & rotation/translation/scaleOcclusion & rotation/translation/scale5.5. Limited resolutionLimited resolution

An image is a discrete model of an underlying An image is a discrete model of an underlying continuous functioncontinuous function

Spatial discretizationSpatial discretization Sensed values quantizationSensed values quantization

6.6. Levels Of Detail (LOD)Levels Of Detail (LOD)

So let’s try to recognize chairs.So let’s try to recognize chairs.

Task that is trivial for us.Task that is trivial for us.


Problem: Problem: Matching models to realityMatching models to reality
















Problem: Problem: Matching models to realityMatching models to reality– Maybe CAD/CAM models can help!Maybe CAD/CAM models can help!– http://www.3dcadbrowser.com/browse.aspx?category=59http://www.3dcadbrowser.com/browse.aspx?category=59


Problems: Problems: Lighting variationLighting variation


Problem: SProblem: Sensor noiseensor noise


Problem: Problem: OcclusionOcclusion


Problem: Problem: Rotation, reflection, translation, & scaleRotation, reflection, translation, & scale

Problem:Problem:– Limited resolutionLimited resolution


Spatial discretization (above and below)Spatial discretization (above and below) Sensed values quantization (next slide)Sensed values quantization (next slide) Too much of a good thing can be a problem too!Too much of a good thing can be a problem too!



Problem:Problem:– Limited resolutionLimited resolution


Spatial discretization (previous slide)Spatial discretization (previous slide) Sensed values quantization (below: left 16M colors, Sensed values quantization (below: left 16M colors,

right 16 colors very carefully chosen)right 16 colors very carefully chosen)


Problem:Problem:Limited resolution Limited resolution of gray values of gray values (sensed value (sensed value quantization)quantization)


Problem: LProblem: Levels Of Detail (LOD)evels Of Detail (LOD)

EXAMPLE APPLICATION: EXAMPLE APPLICATION: COUNTING BOLT HOLESCOUNTING BOLT HOLES

Example application: counting bolt Example application: counting bolt holesholes

A missing bolt hole is a very costly A missing bolt hole is a very costly defect during assembly.defect during assembly.

Counting bolt holesCounting bolt holes

Pixel = Pixel = (is a contraction for what?)(is a contraction for what?)

Dark = 1 = no lightDark = 1 = no light = no hole= no hole Light = 0 = lightLight = 0 = light = part of hole= part of hole

We could have used other conventions as well.We could have used other conventions as well.– Dark = 0 = no light = no hole; light = 1 = light = Dark = 0 = no light = no hole; light = 1 = light =

part of hole.part of hole.– Dark = 255; light = 0.Dark = 255; light = 0.– Dark > 127; light <= 127.Dark > 127; light <= 127.


External corner (ext)External corner (ext)= 2x2 neighborhood of exactly three 1s = 2x2 neighborhood of exactly three 1s

(and one 0)(and one 0)

Internal corner (int)Internal corner (int)= exactly three 0s (and one 1)= exactly three 0s (and one 1)

Holes = (ext – int) / 4Holes = (ext – int) / 4


External corner = 2x2 neighborhood of External corner = 2x2 neighborhood of exactly 3-1’s (and 1-0)exactly 3-1’s (and 1-0)

Internal corner = exactly 3-0’sInternal corner = exactly 3-0’s

How many (total) possible combinations How many (total) possible combinations (of 4 bits) are there?(of 4 bits) are there?

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0


Ex. 3 objectsEx. 3 objects

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 0 0 0 1 1 1 1 1 0 0 1 1 0 0 11 0 0 0 1 1 1 1 1 1 0 1 1 0 0 11 1 1 1 1 0 0 1 1 1 0 0 1 1 0 11 1 1 1 0 0 0 0 1 1 0 0 0 0 0 11 1 1 1 1 0 0 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

row 0

(y’s)

column 5 (x’s)

(x,y)=(c,r)=(0,0)

[r][c]=[0][0]

(x,y)=(c,r)=(Cols-1,Rows-1)

[r][c]=[Rows-1][Cols-1]

adjacent in

memory (in C++)

Algorithm for counting holes in a Algorithm for counting holes in a binary imagebinary image

Input: a binary image, M, of R rows and C cols Input: a binary image, M, of R rows and C cols (indexed as M[r][c])(indexed as M[r][c])

Output: number of holes M containsOutput: number of holes M contains

int external=0, internal=0;int external=0, internal=0;for (int r=0; r<R-1; r++) {for (int r=0; r<R-1; r++) {

for (int c=0; c<C-1; c++) {for (int c=0; c<C-1; c++) {if ( isExternal(M,r,c) ) external++;if ( isExternal(M,r,c) ) external++;else if ( isInternal(M,r,c) ) internal++;else if ( isInternal(M,r,c) ) internal++;

}}}}return (external-internal) / 4;return (external-internal) / 4;

Hole counting assumptionsHole counting assumptions

All image border pixels must be 1s.All image border pixels must be 1s.

Each region of 0s (holes) must be Each region of 0s (holes) must be 4-4-connectedconnected..

Holes must also be Holes must also be simply connectedsimply connected (not contain any objects).(not contain any objects).



– ExternalExternal

– InternalInternal

ext inte 1 1 e 1 1 1 1 e 1 e 1 e 1 e 1 6 01 0 0 0 1 1 1 1 e i 0 1 1 0 0 1 1 1e 0 0 e e 1 e 1 1 1 i e e i 0 1 6 21 1 1 e i 0 i e 1 1 0 i 1 i 0 1 2 41 1 1 e i 0 i e 1 e 0 0 0 0 e 1 4 21 1 1 1 e 0 e 1 1 1 1 1 1 1 1 1 2 01 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0

totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0

…






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0






totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0


Ex. 3 objects = (21-9) / 4 = 12 / 4Ex. 3 objects = (21-9) / 4 = 12 / 4




totals 21 9

1 1 1 0 0 1 1 11 0 1 1 1 1 0 1

0 0 0 1 1 0 0 00 1 0 0 0 0 1 0

Algorithm for counting holes in a Algorithm for counting holes in a binary imagebinary image

Input: a binary image, M, of R rows and C cols Input: a binary image, M, of R rows and C cols (indexed as M[r][c])(indexed as M[r][c])

Output: number of holes M containsOutput: number of holes M contains

int external=0, internal=0;int external=0, internal=0;for (int r=0; r<R-1; r++) {for (int r=0; r<R-1; r++) {

for (int c=0; c<C-1; c++) {for (int c=0; c<C-1; c++) {if ( isExternal(M,r,c) ) external++;if ( isExternal(M,r,c) ) external++;else if ( isInternal(M,r,c) ) internal++;else if ( isInternal(M,r,c) ) internal++;

}}}}return (external-internal) / 4;return (external-internal) / 4;

Why r<R-1 and c<C-1?

PROBLEMS FOR DISCUSSIONPROBLEMS FOR DISCUSSION

Problems:Problems:

Ex. 1.5Ex. 1.5– Problems can be solved in different ways and a problem Problems can be solved in different ways and a problem

solver should not get trapped early in a specific solver should not get trapped early in a specific approach. Consider the problem of identifying cars in approach. Consider the problem of identifying cars in various situations:various situations:

entering a parking lot or a secured area,entering a parking lot or a secured area, passing through a tollgate,passing through a tollgate, exceeding the speed limit.exceeding the speed limit.

– Several groups are developing or have developed Several groups are developing or have developed machine vision methods to read a car’s license plate. machine vision methods to read a car’s license plate. Suggest an alternative to machine vision. How do the Suggest an alternative to machine vision. How do the economic and social costs compare to the machine economic and social costs compare to the machine vision approach?vision approach?

Problems:Problems:

Ex. 1.6Ex. 1.6

– Identify some defects remaining in the Identify some defects remaining in the right image of Figure 1.8 and describe right image of Figure 1.8 and describe simple neighborhood operations that will simple neighborhood operations that will improve the image.improve the image.

Problems:Problems:

Ex. 1.9Ex. 1.9– Examine the image of Examine the image of

bacteria in Figure 1.8 and bacteria in Figure 1.8 and the sample of the sample of automatically computed automatically computed features in Figure 1.12. features in Figure 1.12. Is there potential for Is there potential for obtaining a count of obtaining a count of bacteria, say within 5% bacteria, say within 5% accuracy (in this accuracy (in this example)? Explain.example)? Explain.

Problems:Problems:

Ex. 1.9 cont’d.Ex. 1.9 cont’d.

Problems:Problems:

Ex. 1.12: On face interpretation.Ex. 1.12: On face interpretation.– Is it easy Is it easy for youfor you to decide the gender and to decide the gender and

approximate age of persons pictured in approximate age of persons pictured in magazine ads?magazine ads?

– Psychologists might tell us that humans have the Psychologists might tell us that humans have the ability to see a face and immediately decide on ability to see a face and immediately decide on the age, sex, and degree of hostility of the the age, sex, and degree of hostility of the person. Assume that this ability exists for person. Assume that this ability exists for humans. If you think it would be based on image humans. If you think it would be based on image features, then what are they? If you think that features, then what are they? If you think that image features are not used, then explain how image features are not used, then explain how humans might make such decisions.humans might make such decisions.

Problems:Problems:Ex. 1.14: Toward the correctness of holecounting.Ex. 1.14: Toward the correctness of holecounting.

(a) How many possible 2x2 neighborhood patterns are (a) How many possible 2x2 neighborhood patterns are there in a binary image? List them all.there in a binary image? List them all.

(b) Which of the patterns of part (a) cannot occur in a (b) Which of the patterns of part (a) cannot occur in a binary image that is 4-connected?binary image that is 4-connected?

Define border point to be the center grid point of a 2x2 Define border point to be the center grid point of a 2x2 neighborhood that contains both 0 and 1 pixels.neighborhood that contains both 0 and 1 pixels.

(c) Argue that a single hole cannot be accounted for by (c) Argue that a single hole cannot be accounted for by just counting the number of just counting the number of ee and and ii patterns along its patterns along its border and that the formula n=(e-i)/4 is correct when border and that the formula n=(e-i)/4 is correct when one hole is present.one hole is present.

(d) Argue that no two holes can have a common border (d) Argue that no two holes can have a common border point.point.

(e) Argue that the formula is correct when an arbitrary (e) Argue that the formula is correct when an arbitrary number of holes is present.number of holes is present.

Documents

Computer Vision Chapter 1 Introduction. The goal of computer vision is to make useful decisions about real physical objects and scenes based on sensed