Computer Vision

Pictures as Functions

Image Function (I) maps the values at pixels

I (x, y) -> Returns value for pixel at x,y position in the image

Color Images: I(x, y,) -> Returns array [r, g, b] for pixel

BW Images I(x,y) -> Returns single value for pixel

Pictures as Functions: Python

Pictures as Functions and numpyarrays: Python Color Images• Mathematical

Expression:

I(100, 50) = {61, 150, 184}

• Python Expression

img[50][100]

# Returns:

array([ 61, 150, 184], dtype=uint8)

NOTE! In Mathematical expressions we write I(x, y)

In programming (most languages) we write image arrays as:img_array[row][column]

-That becomes img_array[y][x]

Pictures as Functions and numpyarrays: Grayscale images• Mathematical

Expression:

I(100, 50) = 150

• Python Expression

img_g[50][100]

# Returns value of 150

NOTE! In Mathematical expressions we write I(x, y)

In programming (most languages) we write image arrays as:img_array[row][column]

-That becomes img_array[y][x]

Numpy Arrays in Python

• Numpy Arrays can be written two ways:

• Nested:• img[50][100][3]

• Means Row Index 50, Column Index 100, Blue Channel

• Vectorised• img[50, 100, 3]

Why Vectors? Can call range of Values without For Loops• Slow Way: Iteration

# Nested For Loop

part_img = [[]]

fill = []

for r in range (50, 100):

fill = []

for c in range(60, 80):

fill.append(img_g[r][c])

part_img.append(fill)

• Fast Way: Vectors and Numpy

part_img = img_g[50:100, 60:80]

# Yes – this one line is equal to

# the nested for loop on the left!

# And, this is much faster to

# to process – needed for real time

# Movie processing

Some Python Examples:

Vectorised Array: Set Green and Blue to 0

Vectorised Array: Divide entire picture by 2

Vectorised Array: Get the Horizontal Derivative

Picture Values

• Usually two choices:• 0 to 255: 0 is dark, 255 is completely bright

• 0 to 1: 0 is dark. 1 is Bright. This is sometimes called “Normalized Value” Usefull in statistical analysis.

• Data Types• Be CAREFUL! Images by default are unsigned 8 bit

numbers (0 to 255 with no decimals)

• When doing processing and calculations – convert the image array to a float 32 bit number before the math. Or, the values will be lost in calculations!

Python Language Guide

• Interpreted: Does not need to be compiled

• Sequential – Define functions before using

• Object Orientated (Like Java)• Both Java and Python are built on C and C++

• Can code short ideas in real time in Interpretter

• Write Longer Text Files and Packages (Called Modules)

Python Language Guide

# Single Line comment

“”” Multi line

Comment “””

# No semicolons in Python!

(But they are OK to use . . .)

Python Variables

• Same datatypes as Java (Boolean, int, float, double, String . . .)

• Do not need to declare type of variable before creating

• Java:int age = 14;

• Pythonage = 14

Python (and Numpy) Lists / Arrays

• Pure Python Arrays are mutable (Like ArrayList or List in Java)

students = [“John”, “Frank”, “George”]

# Note use of brackets for array.

students.append(“Mr. Michaud”)

# Adds “Mr. Michaud” to List

Python Lists(Arrays)

• Python Indexing of Lists/Arrays is like java.

• Zero based Indexing

students = [“John”, “Frank”, “George”]

# Note use of brackets for array.

students[0] # Returns “John”

Python List Functions

• len(array) will return length of array

numbers = [3, 8, 7]

len(numbers) # returns 3

# Numpy: min, max, sum

numbers.min() # Returns 3

numbers.max() # Returns 8

sum(numbers) # Returns 18

Numpy Array Operations

import numpy as np #Import numpy library as np

img = cv2.imread(path, 0)

# Grayscale image as numpy array

len(img) # returns number of rows in np array

len(img[0]) # returns number of columns in np array

np.where(array == value)

# Returns Array object with point location of value

Control Structures in Python

• Iteration:

• Java

for (int i = 0; i < 10: i++) {

System.out.println(i);

}

• Python

for i in range(10):

print (i)

If Statements in Python# Standard If statement

if (x > 200):

pin9.write(120)

# Else Statements

if (x > 200):

pin9.write(190)

else if(x < 150):

pin9.write(60)

else:

pin9.write(90)

While Statements

# While Statement: 100 Frames shown

while(x < 100):

ret, frame = cap.read()

if (ret == true):

cv2.imshow(“Picture”, frame)

x = x + 1

Defining Functions in Python

• Do not need to specify return types

• Keyword is “def”

def addTwoNumbers(x, y):

return x + y

Longer Function Example

# Function to return dx

def getDX(frame):

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

diff = np.float32(gray[:, 2:len(gray[0])]) - np.float32(gray[:, 1:len(gray[0])-1])

out = diff + -1*diff.min()

out = out / out.max()

return out

Python and Tabs

• Python uses Tabs to encapsulate code. (No curly brackets in Python)

• For loops, conditionals, while loops, and functions all use :’s and tabs to encase code.

• Be very careful with tabs in Python. If you get a parse error – your tabs are not consistent.

Open Computer Vision 2

• Library to handle image Processing

• Uses Numpy and Scipy for graphics and computer vision.

• Provides access to cameras

• Reading and writing image files

• Creating Image arrays

• Filters, Harr Cascades, and many other features used in Computer Vision.

• Standard Library for Research at University and Industry applications.

Selected Open CV and numpy API commandsimport cv2 as cv2

import numpy as np

#Imports CV2 Library as object cv2

cv2.imread()

cv2.cvtColor()

cv2.matchTemplate()

cv2.imshow

cv2.rectangle

cv2.VideoCapture()

cv2.waitKey()

cv2.destroyAllWindows()

np.copy()

Open CV File Operationscv2.imread(pathToFile, channels)

pathToFile: Address in file system to image

channels: Number of colors

3: R, G, B Full Color

0: Grayscale (One Channel)

Returns an Image array

Example:

#Path to image

path = "C:\Users\michaudc\OneDrive\BCT 440 Computational Perception\images\lisa.png"

#Open Image and assign to np array. Color Image

img = cv2.imread(path, 3)

Open CV Image Operations

cv2.cvtColor(image, ConversionType)

Converts between color and grayscale images.

Returns an Image array

Example:

frame = cv2.imread(path, 3) # color image

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

Open CV Image Operations

cv2.rectangle(image, (x1,y1), (x2, y2), color, thickness)

Draws a rectangle on an image file. Color is (B, G, R) tuple. x1 and y1 are upper left corner. x2 and y2 are lower right corner.

Example:cv2.rectangle(image, (x, y), (x+50, y+50), (0, 0, 255), 2)

Open CV Image Output

cv2.imshow(‘WindowName’, image)

‘WindowName’: Name of Window

image: numpy array object

Example

cv2.imshow(‘My Picture’, image)

Writing images to files

Example:

• Convert image array to 0-255 np.uint8 datatypeimg = img * 255

img = np.uint8(img)

• Use the cv2.imwrite(path, image) to write to filecv2.imwrite(‘output\name.png’, img)

cv2.imwrite(pathToFile, imageArray)

Template Matching (More on this later . . .)cv2.matchTemplate(img_g, patch, type)

img_g: Grayscale image to be searched

patch: Image patch to search for

type: Type of of search (Normalized correlation, Sum of Squared Differences . . .)

Returns a map of values from the filter

Example:f = cv2.matchTemplate(gray, patch, cv2.TM_CCORR_NORMED)

Using the Video Camera

cv2.VideoCapture(camera)

Returns a Camera object for video image capture.

camera: Number assigned to camera in system. Usually ‘0’ is the internal camera and ‘1’ is an external USB camera.

Example:

cap = cv2.VideoCapture(0)

Using the Video Camera

cameraObject.read()

Returns two objects to program:

Boolean value if the camera is working

Numpy Image array of captured image

Example

ret, frame = cap.read()

Create Numpy Array

np.array(arrayData)

Creates a Numpy array

Example: Creates a 2 x 4 array of integers

img = np.array([[4, 3, 5, 6],

[3, 5, 7, 8]])

Convert Datatype in Numpy Array to 32 bit floating pointnp.float32(arrayData)

Converts incoming data to floating point 32 bit numbers. Returns the an array with the converted and copied data

Example:

patch = np.float32(img)

Copy Numpy Arrays (image data)

np.copy(numpyArray)

Takes input of numpy array and creates a new numpyarray and copies the data.

Example:

# img is an existing numpy image array

img_copy = np.copy(img)

2D Filter Function (Convolution)

cv2.filter2D(img, -1, kernel)

Takes input of grayscale img data filters with kernel.

The kernel is an m x m array used to filter the data. Usually a Gaussian, Laplacian, Sorbel, or box filter.

Example:

# img is an existing numpy image array

kernel = np.ones((5, 5), np.float32)/25

box_blur = cv2.filter2D(img, -1, kernel)

CV2 Canny Edge Function

• Built in Edge detection function in Open CV that uses a more involved algorithm:• Filters image with derivative of Gaussian

• Uses magnitude and orientation of gradients

• Non-Maximum suppression

• Linking and thresholding

• Result is a more refined edge image.

img_edge_canny = cv2.Canny(img_gauss, 50, 100)

Gaussian Blur: Role of Sigma

• We will blur images with the cv2.GaussianBlur() function.

cv2.GaussianBlur(img, (m x m), sigma)

Where:

img = Image Array to Blur

(m x m): size of ‘window’ of Gaussian (odd number)

sigma: The spread of the blur (High for more depth)

2D Gaussian Filter Image

Example:

Why Blur with Gaussian?

• Remove Noise from the Picture.

Code Sample: Blur with Gaussian

Code Sample: Canny EdgeNote: You will have to experiment with the parameters for GaussianBlur() and Canny() to get a good edge image.

Exit Sequences

• Open CV will “lock” your python when showing images. You will need “Exit Sequences” to cleanly exit running programs to preserve data and not crash your python apps.

• Two Kinds: Non-Looping and Looping• Non-Looping: For Still Images

• Looping: when using the Camera Object

Non-Looping Exit Sequence

Looping Exit Sequence

Thank You!

• Many thanks to Charlie Fligg (Class of ’20) for researching Python and Computer Vision and helping gather software, materials, and tinkering with open CV2 for the Perception Class!