Camara for uav jan2012 eas 021

[EAS 499 Capstone Project]

[Development of Object Detection Program for Micro Aerial Vehicles] 1

SIM UNIVERSITY

SCHOOL OF SCIENCE AND TECHNOLOGY

DEVELOPMENT OBJECT DETECTION

PROGRAM FOR A CAMERA FOR MICRO-

AERIAL VEHICLE

STUDENT: AZLI ERWIN AZIZ (PI NO. E0806919)

SUPERVISOR: SUTTHIPHONG SRIGRAROM

PROJECT CODE: JAN2012/EAS/021

A project report submitted to SIM University

In partial fulfilment of the requirements for the degree of

Bachelor of Engineering in Aerospace System

January 2012



Abstract

Today, with the advancements of microprocessor technology, unmanned aerial

vehicles have been increasing in popularity with private agencies and hobby

enthusiast. These unmanned aerial vehicles have also reduced in size and come in

different forms of rotary vehicles. Instead of the conventional fixed wing platform,

unmanned aerial vehicles have taken new forms and unconventional platforms. Tri-

motor and quad-motor copters have gained popularity in the market. These new

platforms make it easier to take-off and land as these aerial vehicles do not need a

long runway and have attributes similar to that of a traditional helicopter. As global

positioning satellites (GPS) are becoming readily accessible to the public,

autonomous aerial vehicles have also sprung out. Attaching GPS receivers and

transmitter, these micro aerial vehicles can fly autonomously to designated way-paints

specified by the user. However, due to limitations of GPS such as weak signal

strength indoors, autonomous flying is restrictive indoors.

The aim of this project is to develop an object detection program for a digital camera

that can be used in conjunction with a microprocessor on a micro aerial vehicle for

autonomous flight in an indoor environment. The object detection program functions

to create object boundaries and edges found in the video recorded by the camera.

Thus creating boundaries of an environment. This project will use video and image

analysis techniques to build colour co-relations and edges of the objects. The results

from the analysis can then be passed to the microprocessor for processing. This then

enables the micro aerial vehicle to be able to avoid and head towards objects in its

surrounding autonomously.

This report will cover on the objectives of the project, literature research on image

analysis and project management aspects. The main focus will be on

conceptualization, development, implementation and testing of the developed object

detection software in MATLAB. Finally, topics such as conclusions,

recommendations, critical reviews and reflections of this project will also be

discussed.



Acknowledgements

I would like to take this opportunity to offer my sincere appreciation to my project

supervisor Dr Sutthiphong Srigrarom (Spot), for his guidance, patience and support

through out this entire capstone project. During this time, Dr Spot showed outmost

patience and support even when there were slow progress and when many challenges

were faced. He never failed to clarify my doubts and assist me in any way possible.

Special thanks to Mr Koh Pak Keng head of the BEHAS program, Ann lee and all

other friends in UNISIM for their support and practical suggestions towards this

project. They have made my learning journey in UNISIM a very fulfilling and

enriching experience.

To all lectures, professors and support staff of the School of science and technology

for help directly or indirectly in helping me complete my project. To my managers,

who have encouraged and supported me throughout my 4.5 years of learning in

UNISIM.

Last of all, I would like to thank my family members, fiancée and loved ones who

have shown great care and concern for me during this period of studies and have

given me moral support to complete this project. With their support and

encouragements, I was able to preserve on despite having other commitments.



Table of Contents

Abstract .................................................................................................................................. 2

Acknowledgements ............................................................................................................ 3

List of figures ........................................................................................................................ 6

Chapter 1: Introduction ............................................................................................... 7 1.1 Background on Desired motivation ............................................................................... 7 1.2 Objectives ...............................................................................................................................10 1.3 Scope ........................................................................................................................................10 1.4 Proposed Approach ............................................................................................................11 1.5 Layout of the project report ............................................................................................12

Chapter 2: Literature review ................................................................................... 13 2.1 Fundamentals of image analysis ....................................................................................13

2.1 .1 Analog Image definition: ........................................................................................................ 13 2.1.2 Digital Image definition:.......................................................................................................... 13 2.1.3: Binary Image Definition ......................................................................................................... 14 2.1.4 Sampling ........................................................................................................................................ 14 2.1.5 Quantisation ................................................................................................................................. 15 2.1.6 Grey level Histogram ................................................................................................................ 15

2.2 Image analysis Methodology ...........................................................................................16 2.1 Image Segmentation ..................................................................................................................... 16 2.2 Point independent thresh holding.......................................................................................... 17 2.3 Neighbour dependant method ................................................................................................. 18 2.4 Edge Detection ................................................................................................................................ 18 2.5 Morphological operations .......................................................................................................... 20 2.6 Representation of objects .......................................................................................................... 20

Chapter 3 Software and Hardware Required ........................................................ 21 3.1 Software ..................................................................................................................................21 3.2 Hardware ...............................................................................................................................22

Chapter 4 Program Design and Development ....................................................... 23 4.1 Project Requirement ..........................................................................................................23 4.2 Object Detection Program Overview ............................................................................23 4.3 Design overview ..................................................................................................................24 4.4 Software Design Procedures ...........................................................................................25

4.4.1 Initialisation of image .............................................................................................................. 25 4.4.2 Image Conversion ...................................................................................................................... 26 4.4.3 Edge Detectors ............................................................................................................................ 28

4.4.4 Display of Data ..................................................................................................................29

Chapter 5 Testing and Evaluation .............................................................................. 30 5.1 Object Detection Program Test (Using Pre-recorded Video) ..............................30 5.2 Object Detection Program Test (Using live acquisition Video) ..........................31 5.3 Field Test of Final design ..................................................................................................32

5.3.1 Object Test .................................................................................................................................... 32 5.3.2 Object Test Evaluation ............................................................................................................. 34 5.3.3 Lighting Test ................................................................................................................................ 34 5.3.4 Lighting Test Evaluation ......................................................................................................... 36

Chapter 6 Recommendations and Conclusion ....................................................... 38



6.1 Summary ................................................................................................................................38 6.2 Overall Conclusion ..............................................................................................................38 6.3 Recommendation ................................................................................................................38

7. Review and Reflection ............................................................................................... 40

References:......................................................................................................................... 42

Appendix A – Gantt chart .............................................................................................. 43

Appendix B – Initialize image code ......................... Error! Bookmark not defined.

Appendix C – Initial Design Simulink Code ............................................................. 44

Appendix D – Final Design Simulink Code .............................................................. 45



List of figures Figure 1: Quad-Copter .............................................................................................................................. 7 Figure 2: UAV camera system ................................................................................................................. 9 Figure 3: Image analysis flowchart ........................................................................................................ 11 Figure 4: Digital Image Sample of a continuous image ......................................................................... 14 Figure 5: Grey-Level Histogram ............................................................................................................ 16 Figure 6: Image Segmentation using Grey-level Histogram .................................................................. 17 Figure 7: Adaptive Thresh holding (non-uniform illumination) ............................................................. 18 Figure 8: Edge Detection ........................................................................................................................ 20 Figure 9: MATLAB Environment .......................................................................................................... 21 Figure 10: MATLAB Simulink Design Environment ............................................................................ 22 Figure 11: Standard Webcam and Digital Camera ................................................................................. 22 Figure 12: Vision Control Motion System ............................................................................................. 23 Figure 13: Object Detection Program Workflow ................................................................................... 24 Figure 14: Image import to MATLAB ................................................................................................... 25 Figure 15: Displaying of Image in MATLAB ........................................................................................ 26 Figure 16: Binary Image Conversion ..................................................................................................... 27 Figure 17: Binary Histogram .................................................................................................................. 27 Figure 18: Grey scaled image and histogram ......................................................................................... 28 Figure 19:Edge Detector ......................................................................................................................... 28 Figure 20: Corridor Video Snapshot ...................................................................................................... 30 Figure 21: Result Display (Pre-recorded video) ..................................................................................... 31 Figure 22: Live feed design output ......................................................................................................... 32 Figure 23: Bottle Test ............................................................................................................................. 33 Figure 24: Table and Chair test .............................................................. Error! Bookmark not defined. Figure 25: Office environment ............................................................................................................... 35 Figure 26: Out door Environment .......................................................................................................... 36 Figure 27: Out door Environment 2 ....................................................................................................... 36



Chapter 1: Introduction

1.1 Background on Desired motivation

Unmanned Aerial vehicles (UAV) have been used with government armed forces and

military applications throughout the world for many years now. UAVs’ served mainly

as surveillance and security methods for these agencies across the world. However,

UAVs’ have grown in popularity among hobby enthusiast and private industries and

agencies in recent years. This growth in UAVs’ occurred with the advancement of

micro processing technology. The cost and size of these processors greatly reduced

and their processing capabilities have increased dramatically. UAVs’ these days are

easier to fly due to microprocessors as they help to control stability and flight of these

vehicles. Programmable microprocessors coupled with other components such as

sensors enable UAVs’ to fly autonomously.

Today, UAVs’ come in different shapes and sizes and conventional flight have been

pushed aside. Aside from the traditional helicopter and fixed wing aeroplanes, multi-

rotor vehicles have been gaining popularity. Some examples of these unconventional

vehicles are the tri-copter and quad-copters. These vehicles unlike a traditional

helicopter use more then one rotary motor to function. An example of a quad-copter

can be seen in figure 1.1 below.

Figure 1: Quad-Copter



Quad-copters have attributes similar to that of a traditional helicopter with additional

stability control. With four motors spinning in opposite directions, speed and attitude

controls are depended solely on power distribution to the motors by the

microprocessor. Flight stability of the quad copter has also increased with four

motors. Programmable microprocessors with auto stabilization function help to

stabilize these vehicles in flight as well.

Currently, UAVs’ are readily accessible in the market and are fully customisable and

configurable to perform all sorts of different functions. These UAVs can be manually

controlled using a traditional radio transmitter or Bluetooth and Wi-Fi enabled

smartphones and tablet computers. The user can also configure them to perform

specified automated task by pre-programming the microprocessor and assigning that

task to a switch on the controller. The UAV will then carry out that automated task

when the switch has been enabled.

In an outdoor environment, with the aid of global positioning satellite (GPS) modules,

UAVs’ can perform automated task such as flying a specific route defined by the user

and circling around a perimeter specified by the user. With the aid of barometers and

sonar-sensors, UAVs’ are able to maintain specific heights to hover and deploy. Thus,

with these sensors and GPS modules, UAVs’ are able to fully function autonomously

in an outdoor environment.

However, in an indoor environment, the scenario is different due to limitations of

GPS triangulations. This is due to weak signal strength of GPS satellite in an indoor

environment. Thus, UAVs’ are not able to function fully autonomously as they cannot

detect their exact location in reference to that of indoor environment. To overcome

this issue in an indoor environment, UAVs have to be installed with additional

sensors that help locate its position in reference to that of the environment. These

additional sensors include a camera system that records real time video of the

environment.

By developing an object detection program that detects key surfaces and references of

the environment will enable the UAV to locate its position in reference to the



surroundings. Used in conjunction with a camera system, results from the program

can then be passed to the main controller module of the UAV thus enabling

autonomous flight in indoor environments. This will then enable automation function

of UAVs’ in an indoor environment. The figure below shows an example of a UAV

camera system.

Figure 2: UAV camera system



1.2 Objectives

The objective of this project is to develop an object detection program for UAVs or

micro aerial vehicle camera systems. The program will take video inputs from the

camera and detect objects in the video allowing the UAV or micro aerial vehicle to be

able to fly autonomously towards or away from obstacles. The information processed

by this program will enable the UAV to detect objects or key surfaces in its

surrounding.

1.3 Scope

This project will consists of the following stages of development:

• Video recording phase (to simulate real time video of the camera system in the

micro aerial vehicle)

• Video rendering phase (splitting the video into still picture frames for

analysis)

• Initial development of the program

• Co-relate each frame of the video using programming software such as mat

lab to find difference in the frame.

• Testing and trail phase (different videos with different objects are tested to test

the robustness of the program)

• Implementation corrections from testing and trial data

A Gantt chart for key milestones and major target has been attached in Appendix

A. This chart shows the detail breakdown of the project in different phases and the

time proposed to accomplish the required phase of the project. This chart helps to

manage the overall progress of the project from the design phase to the final

development of the project. Keeping close attention to this chart will ensure the

project is completed in a timely and organised manner.



1.4 Proposed Approach

The proposed method for tackling this project will be inline with image and video

analysing techniques. There will be a few stages for the development of the object

detection program. The main component to this project will be the detection of object

boundaries in an image or video. The flow chart below in figure 3 will show the

development process of video or image analysing program and this program will

follow closely to this development flow chart. There are 6 major phases in this project

that have been identified and will be explained in further detail in the sub categories

of this report.

Figure 3: Image analysis flowchart



1.5 Layout of the project report

Chapter 1: Introduction and Background

Chapter 2: Literature review on image analysing methods and concepts.

Chapter 3: Hardware and Software Required

Chapter 4: Program Design and Development

Chapter5: Testing and Evaluation

Chapter 6: Recommendation and Conclusions

Chapter 7: Review and Reflections



Chapter 2: Literature review

2.1 Fundamentals of image analysis

Before development of the object detection program can take place, an understanding

of the basics and fundamentals of digital images have to be established.

Image analysis and manipulation can be divided into three stages:

• Image processing (image in => image out)

• Image analysis (image in => measurement out)

• Image Understanding (image => high level description out)

In this topic, concepts of the image processing and image analysis as shown above

will be discussed.

2.1 .1 Analogue Image definition:

An analogue image can be defined as a function of two variables, example A (x, y)

where ‘A’ as the amplitude of the image at a real coordinate position (x, y).

Variable ‘A’ in the function above can represent any parameters of the image such as

brightness, contrast and colours. Some concepts consider images to contain sub-

images and can these sub-images can be referred to as regions of interest (ROI) or

simply regions. With this concept, shows that images frequently contain a collection

of objects each that can be the basis for a region. Specific image processing

operations can be applied individually to these regions. Thus, in an image, different

regions can be processed differently and independently from each other.

2.1.2 Digital Image definition:

A digital image can be defined as a numeric representation of a two-dimensional

image. Digital images are usually represented in binary formats and can be derived in

a 2D space environment through a sampling process of an analogue image. This

sampling process is also referred as digitization. A 2D continuous image A (x, y) is

divided into N rows and M columns and has a finite set of digital values. The

intersection of these rows and columns are also termed as a pixel. Pixels are the

smallest individual element in an image, holding quantized values that represent the



brightness of a given colour at any specific point. An example of a digital image with

rows and columns can be seen in figure 2 below.

As seen in figure 4 above, the image is divided into identical number of rows and

columns. In this case there are 16 rows and 16 columns. Each cell in the image is the

intersection of the columns and rows and can also be referred to as pixels as

mentioned earlier. The value of each pixel is the average brightness in the pixel

rounded to the nearest integer value.

2.1.3: Binary Image Definition

A binary image is an image that has 2 possible values for each pixel. Typically the

two colours showed in a binary image is either white or black. Binary images are also

called bi-level images and have pixel values of 0 or 1. Binary images are typically

used for image analysis and comparisons.

2.1.4 Sampling

Before an analogue image can be processed, it has to be digitized into a 2 dimension

digital image. This process of digitizing an analogue image is called sampling.

Sampling it the process of an image being represented by measurements at regular

spaced intervals. Two important criteria for sampling process are:

• Sampling interval

o Distance between sample points and pixels

• Tessellation

R

O

W

S

COLUMNS

Pixels (Average of

brightness)

Figure 4: Digital Image Sample of a continuous

image



o The pattern of sampling points

The resolution of an image can be expressed as the number of pixels present in the

image. If the number of pixels present in an image is too small, individual pixels can

be seen and other undesired effects can be seen as well. Examples of undesired effects

are aliasing and noise. Image noise is a random variation of brightness and colour

information in digital images and is usually an aspect of electrical noise. Noise can be

generated by circuitry of digital cameras and is usually undesired during image

processing as they add extra information to the image. To reduce noise in an image,

the image has to be smoothen by a smoothening stage, typically Gaussian

smoothening.

2.1.5 Quantisation

Quantisation is a process that compresses a range of values to a single quantum value.

It is mainly used to reduce the number of colours required to display a digital image

thus enabling a reduction of file size of the image. In this process, an analogue to

digital converter is required to transform brightness values into a range of integer

number. The maximum number in that range (e.g. 0 to M) is limited by the analogue

to digital converter and the computer. In the example stated above, M refers to the

number of bits used to represent the value of each pixel. This then determines the

number of grey levels in that image. This process of representing the amplitude of the

2D signal at a given coordinate as an integer value L with different grey levels is

referred to as amplitude quantization. If there are too few bits in that image, steps can

be found in between grey levels.

2.1.6 Grey level Histogram

Grey-level histograms are one of the simplest yet most effective tools used to analyse

images. This is because, a grey level histogram can show faulty setting in an image

digitiser and is almost impossible to achieve without digital hardware. Below is an

example of a typical grey level histogram of an image.



Figure 5: Grey-Level Histogram

A grey- level histogram is a function showing the number of pixels that have grey

level. In this figure, the grey-scaled image is shown as a grey level histogram.

2.2 Image analysis Methodology

Image analysis mainly comprises of different operations and functions. There are 3

main functions commonly used in image analysis and they are:

• Segmentation

1. Thresh holding

2. Edge detection

• Morphological operations

• Representation of objects

2.1 Image Segmentation

To distinguish simple objects from background and determine its area, we can use

grey level histograms to perform this task. Image segmentation is the operation of

distinguishing important objects from the background (or from unimportant objects).

Figure 6 below shows an example of image segmentation using grey-level histogram.



Figure 6: Image Segmentation using Grey-level Histogram

Image segmentation separates the dark object from the bright background as shown in

figure 5 above.

There are two common methods of segmentation and they are:

• Point independent Thresh holding method

o Thresh holding (semi-thresh holding)

o Adaptive thresh holding

• Neighbourhood- dependant method

o Edge detection

o Boundary tracking

o Template matching

2.2 Point independent thresh holding

Points dependent method is a method of thresh holding images. This method operates

by locating groups of pixels with similar properties thresh holding. Thresh holding

assigns the value of 0 to the pixels that have a grey scale less than the threshold and

the value of 255 for pixels that have grey levels higher than the threshold. Thus, this

will segment the image into two regions, one corresponding to the background the

other corresponding to the object. Therefore, thresh holding is used to discriminate

the background from an object in an image. This is a straightforward if the image has

a bimodal grey level histogram. For more complex images, this method of thresh

holding is inaccurate and requires a different method of thresh holding.

Adaptive thresh holding is a more complex method process to discriminate grey

levels from backgrounds. In practical grey level histograms, images rarely have

bimodal grey levels, thus using point independent thresh holding is insufficient. This



is due to random noise captured in the image, varying illumination when the image is

taken and object that have different sizes and shapes found in the image.

Adaptive thresh holding is very useful for images with non-uniform illumination.

2.3 Neighbour dependant method

Neighbour dependent method consists of three different types of operations such as

edge detection; boundary tracking and template matching that have been mention

earlier in this topic. Neighbour dependant operates by generating an “output” pixel by

the basis of the pixel at the corresponding position in the input image and on the basis

of it neighbouring pixels. The size of the neighbourhood may vary in size, however,

several technique uses 3 x 3 and 5 x 5 neighbourhoods centred at the input pixel.

Neighbour operation plays a key role in modern digital image processing.

2.4 Edge Detection

Edge detection is a tool used in image processing that is aimed at identifying points in

a digital image at which the image brightness changes sharply or has discontinuities.

Identifying sharp changes in brightness is to capture important events and changes in

properties of the image. The discontinuities in the image brightness are likely to

correspond to:

• Discontinuities in depth

Figure 7: Adaptive Thresh holding (non-uniform illumination)



• Discontinuities in surface orientation

• Change in material properties

• Variation in scene illumination

The result of edge detection applied to an image may lead to a set of connected curves

that indicate the boundaries of objects, boundaries of surfaces as well as curves that

corresponds to the surface orientation.

There are several methods of edge detection methods commonly used for image

analysis. Edge detection can be classified into two categories. These categories are:

1. Search based (Discrete approximation of gradient and thresh hold the gradient

norm image – First derivative)

2. Zero crossing based (Second derivative)

3. Band pass Filtering

4. Compass operators

A search based methods detects the edges by first computing a measure of strength,

that is usually a first derivative expression such as the gradient magnitude. It then

searches for the local directional maxima of the gradient magnitude using a computed

estimate of the local orientation of the edge. This is usually the gradient direction.

For zero crossing method on the other hand, it searches for zero crossing in a second

order derivative expression computed from the image in order to find edges in that

image. This process is usually a zero crossings of a non-linear differential expression.

A vital and important pre-processing step to edge detection is to reduce noise that was

mentioned in section 2.1.3 commonly using a Gaussian smoothening operation.

Another important operation during edge detection is edge thinning. This operation is

a technique to remove unwanted spurious points on the edges of the image. It is

usually employed after the image has been filtered for noise, edge detector has been

applied to detect the edges of objects in the image and after the image has been

smoothed using an appropriate thresh hold value.

Ann example of edge detection that has been applied to an image can be seen in

figure 7 below.



2.5 Morphological operations

Morphology is a broad set of image processing operations that process an image

based on shapes. This method applies a structured element to an input image, creating

an output image of the same size. In this method, the value of each pixel in the output

image is based on a comparison of the corresponding pixel in the input image with its

neighbours. A morphological operation usually consists of two operations, dilation

and erosion. Dilation adds pixels to the boundaries of objects in an image while

erosion removes pixels from object boundaries in an image.

2.6 Representation of objects

Once thresh holding and edge detectors have been applied to an image, it is necessary

to store information about each objet for use in future extractions. Each object needs

to be assigned a special identifier to it. There are three ways of doing so:

• Object membership map (value of each pixel is encodes the sequence number

of the object)

• Line segment coding (objects represented as collection of chords oriented

parallel to image lines)

• Boundary chain code

(Defines only the position of object boundary)

Figure 8: Edge Detection



Chapter 3 Software and Hardware Required

3.1 Software

One of the requirements stated by the university, is to use proposed software

MATLAB. MATLAB stands for MATrix LABoratory, software developed by Math

works Inc. (www.mathworks.com). MATLAB is a high-level language interactive

numerical computation, visualisation and programming software. It has capabilities to

analyse data, develop algorithms and create models and applications. It uses similar

C++ program languages and object oriented program concept similar to that of C++

programming. The figure below shows MATLAB programming environment.

MATLAB incorporates built in math functions, languages and tools that are faster

than traditional languages such as C++ and java programs. MATLAB has various

toolboxes that enable the user different set of tools for different applications. For our

project, we will be using the image acquisition, image analysis and video analysis

toolbox. MATLAB also has the function of using a graphical interface to create

algorithms. This function in MATLAB is called Simulink. Simulink gives the user to

create programs using pre-defined block set. MATLAB also has different toolboxes

that enhance specific functions within MATLAB. Some examples of MATLAB

toolboxes are the aerospace toolbox and control system toolbox. Figure 10 below

shows Simulink design environment.

Figure 9: MATLAB Environment



Figure 10: MATLAB Simulink Design Environment

With a library of block sets, the user can then create different type of programs by

linking these block sets.

3.2 Hardware

To develop the object detection program, image-capturing devices will be required to

simulate a camera system of a UAV or micro aerial vehicle.

Currently, there are two methods of image capturing available commercially in the

market. They are either by a:

• Digital Cameras

• Analogue cameras

Digital cameras are available in various resolutions and have direct interface with

computer. Analogue cameras on the other hand, require additional suitable grabbing

cards or TV tuner card for interfacing with the computer. Digital cameras give high

quality low noise images unlike analogue cameras that have lower sensitivity

producing lower quality images.

Therefore, we will be using a standard digital camera and a webcam. A webcam is a

video camera that feeds images in real time to a computer via USB, Internet or WI-FI.

A standard webcam and digital camera can be seen in the figure below.

Figure 11: Standard Webcam and Digital Camera



Chapter 4 Program Design and Development

4.1 Project Requirement

To develop an object detection program for a UAV or micro aerial vehicle we need

review the purpose and function of the program.

Requirements

• Grab and capture images and video from an image-recording device

• Co-relate frames of images captured (histogram)

• Detect object boundaries, surfaces and textures.

Based on literature review of image analysing techniques, we can then create a

program on these methodology and concepts to full fill the requirements stated above.

4.2 Object Detection Program Overview

This section will describe the design overview of the object detection program. The

object detection program will come under the Vision Controlled Motion system or

(VCM) in short. VCM is used in robotics and in our case an unmanned aerial vehicle

(UAV) or micro aerial vehicle. VCM as a system can be shown graphically in the

figure below.

VCM is a very important concept and our object detection program is a part of this

system. The object detection program is the image analysis portion of the VCM

system shown in figure 12. The program analyses images and videos acquired from

an image-capturing device. This then enables the UAV the ability to sense it’s

surrounding. User defined inputs will then allow the UAV to avoid obstacles and

Figure 12: Vision Control Motion System



identify key surfaces and track user defined object autonomously. In an indoor

environment, ability to sense the surrounding is the key to autonomous flight.

4.3 Design overview

The primary design objective of the program will be to read and acquire images and

videos from the image-capturing device attached to the microprocessor. The program

shall then need to filter and convert these images and videos to usable data for

analysis. Once these data has been converted, analysis operators can be applied to

these images and videos and the results can be displayed. A detailed workflow of the

program can be found in the figure below.

Figure 13: Object Detection Program Workflow



4.4 Software Design Procedures

The object detection program will consist of critical processes and procedures. The

table below will show the steps involved in creating the program. Program steps are

as follows:

1. Initialise video data to program

2. Convert video and image data for processing

3. Apply edge detectors

4. Display output data

As described in the above table, there are 4 critical steps in our program. The initial

step will be to initialise an image or video that was captured using a digital camera

and then uploaded to the computer. Once initialized, conversions to binary and grey

scale images will be carried out and an edge detector will be applied to the converted

images. Lastly, the images will be displayed with edge detectors. Detailed procedures

of each process will be described in the following sections. For this project, we will

be using the Simulink function of MATLAB mentioned in the previous chapter of this

report.

4.4.1 Initialisation of image

The initial step for our object detection program is to sample images captured by the

webcam. This step involves initializing the image or video to the MATLAB software.

By doing so, we can then read pre-recorded videos or image. Below shows an image

being imported into MATLAB.

Initialized

image

Image

description



In the figure above, an image was imported into MATLAB and we can see the

description of the image. The description of the image includes size, bits and class of

the image. Once the image has been imported, we can then show a preview of the

image in MATLAB. Figure 11 below shows a preview of the imported image.

By importing the image, we can then apply different analysing techniques discussed

in chapter 2 to the image.

4.4.2 Image Conversion

Once we have imported the image, we can then convert the image to binary images

and grey-scaled images. This process is a vital step in image analysis as discussed in

chapter 2 of the fundamentals of image analysis. Figure 16 below shows the above

image in figure 16 converted to a binary image. Converting the images to binary

images, we have changed the values of the each pixel and given them new values of

(0 and 1). The black areas in the picture represent pixels with values 0 while the white

areas in the picture represent pixels with values 1.

Figure 14: Image import to MATLAB

Figure 15: Displaying of Image in MATLAB



Figure 16: Binary Image Conversion

Next we can then display the histogram of the binary image. The figure below shows

the colour histogram of the binary image above. The histogram below shows the

number of pixels that have (0 or 1) values.

Figure 17: Binary Histogram

As explained in chapter 2, binary images have pixel values of either 0 or 1. This can

then be seen in the colour histogram in figure 16. The next step of conversion is to

create a grey scale image. A grey scale image, replaces the colour pixels of the image

with grey shades from white to black. Once the image has been converted, we will

take the grey-level histogram of the image as shown in figure 17 below.



Figure 18: Grey scaled image and histogram

4.4.3 Edge Detectors

Applying edge detectors is the next important step of our object detection program.

The following figure will show the above image being edge segmented. Edge

segmentation creates boundaries in objects of the image. This is an important

component of our program as object boundaries and surfaces are important for a

micro aerial vehicle.

Figure 19:Edge Detector



By applying an edge detector to the image, we can see object boundaries in the image.

The white lines in figure 15 shows the boundaries of the image. The above figure was

applying an edge detector in this case a “sobel” edge detector to a binary image. Edge

detectors can only be applied to a grey scale or binary image. Therefore the

conversion step is very important.

4.4.4 Display of Data

The final step for the object detection program is to display the output image as

useable data for the UAV. The image can then be written as a binary file or written as

a set of codes for the microprocessor. With this data, the microprocessor can then

compute these data and instruct the UAV to fly towards or avoid structures in the

environment. However, for this project we will display the output of the image using

the video display screen of the software. We will also use statistical methods such as

histograms to view the data.



Chapter 5 Testing and Evaluation

5.1 Object Detection Program Test (Using Pre-recorded Video)

The first initial program design takes on the first approach in design flowchart of the

overall object detection program. This approach uses a pre-recorded video that has

been recorded by a digital camera and then uploaded to the computer. A video of an

indoor environment was taken using a digital camera. To simulate structures such as

pillars and ceiling, we used a video of a typical corridor. Some images of this corridor

can be seen in the figure below.

Figure 20: Corridor Video Snapshot

The pre-recorded video will then be initialised to the program and converted to

useable images for analysis. Once this is done, an edge detector will be applied to the

image. For this design, a “Prewitt” edge detector will be used. It calculates the

gradient of the image intensity at each point giving the possible increase from light to



dark and the rate of change in each direction. The output image of this video will be

displayed using video displays and a colour histogram. The figure below shows a

snapshot of this program design and output.

Figure 21: Result Display (Pre-recorded video)

5.2 Object Detection Program Test (Using live acquisition Video)

The second design of the object detection program uses life feed video instead of pre-

recorded video. This approach uses a standard webcam that is connected to the

computer via USB and processes and converts live images instantly. As mentioned in

our flowchart of the design overview, this is the second method of image acquisition.

This process takes place is in real time and any objects in the video are

instantaneously analysed and an output image is displayed. Real time processing

analyses the image at the same speed as the images are occurring. In the context of a

UAV or micro aerial vehicle, real time processing is required to process images

captured by the UAVs camera system. With real time processing, the on board

microprocessor is able to analyse data from the object detection program while the

UAV is in flight. This will then enable UAV autonomous functionality. Thus, this

will be the final design for the object detection program. The first step to this design is

to initialize the webcam to the program. Once that is done, the webcam will grab

frames of images at 30 frames per second. These images will then be converted and



an edge detector will be applied to these images. The output data will be displayed as

images and a colour histogram. A “Prewitt” operator edge detector mentioned in the

first design will also be applied here. The figure below shows a snapshot of this

design output.

Figure 22: Live feed design output

From the output above, real time images were processed and an output was displayed

in the form of video displays. These displays include, the original image, binary

image, grey-scaled image, edged image and lastly an overlay of the edges on the grey-

scaled image. A colour histogram of the original image is also shown.

5.3 Field Test of Final design

To verify the final design of the object detection program, a field test was done. Using

different objects, the output image of the program was recorded and analysed to see

the robustness of the edge detector of the program. To test the design even further, a

test was carried out in different lighting conditions. Different lighting condition is an

important factor for indoor environment.

5.3.1 Object Test

During this test, different object were used to simulate common object found in an

indoor environment. First object used was a common drinking bottle found in an



indoor environment. The figure shows the output of a bottle after being processed by

the object detection program. A histogram of the colours of the video is also

displayed.

Figure 23: Bottle Test

Next, tables and chairs were used to simulate an indoor environment. As the previous

test, tables and chairs were placed in front of the camera and the output data was

being displayed in the video display. The figure below shows the output data of the

chair and table after processing.

Figure 24: Indoor Table and chairs



5.3.2 Object Test Evaluation

The object test satisfies our requirement of an object detection program to detect

object boundaries and edges. From test carried out, we can see in figure 23 of the

bottle test that the full shape of the bottle was being detected and it boundaries

outlined. The colour histogram also reflects an increase in R channel, shows that the

bottle is being detected as it is red in colour. Using the colour histogram, we can

gauge distance of the object from camera. This means that if the amplitude of the

colours increase and the graph of colours widen the object are near the camera. This

test is very satisfactory for the project. From figure 23 above, it is evident that the

surfaces and boundaries of the table and chairs were detected and outlined in the

output display. In the binary image display, it is observed that the walls of the indoor

environment were given a value of 1 and it displays white while the rest of the

environment was given a value of 0 and displayed as black. Boundaries of the table

and chair on the other hand were detected and displayed in the edges display. This

showed that the program performed its desired function. However, the program did

not detect the shadow on wall.

5.3.3 Lighting Test

Lighting test is the next step for verifying the functionality of the object detection

program. For this test, we simulated an indoor environment with two distinct lighting

conditions, low lighting and bright lighting. This test is to simulate typically an office

environment whereby the UAV or micro aerial vehicle maybe placed to do

surveillance and monitoring. The figure below depicts the output data of program in a

normal office environment.



Figure 25: Office environment

The last and final test carried out was a high light environment. This test simulates an

outdoor environment. This test is simulated to captured output data of the object

detection program if it is coupled with a UAV or micro aerial vehicle that has outdoor

autonomous functionality. Out door functionality usually uses GPS triangulation,

however, if the program is able to detect surfaces in the environment, the program can

be used as an object avoidance system for the UAV or micro aerial vehicle. The

figure below show the output data of the object detection program in an outdoor

environment.



Figure 26: Out door Environment

Figure 27: Out door Environment 2

5.3.4 Lighting Test Evaluation

From the lighting test, the object detection program full fills the requirements of our

project. The object detection program is able to detect surfaces and boundaries of

objects in the image. The downside of the program however, is that the program is

unable to detect fully boundaries of objects that have low illumination levels. From

the output display in figure 24 above, it is observed that key structures of the

environment were detected. Wall boundaries and picture frames on the walls were



detected. However, the boundary of the door and the wall was not detected due to

shadow being casted on the door. This shows that in low lighting environments, the

object detection is not robust enough to detect object boundaries. Low light objects

were not detected and edges were not shown in the output display. From the results in

figure 26 and figure 27 of the outdoor environment test, surfaces and object

boundaries were detected. However, as in the case of the indoor environment, object

that have low light levels were not detected. The outdoor environment results shows

that this program can be used in an outdoor application as an object avoidance system.



Chapter 6 Recommendations and Conclusion

6.1 Summary

After every test that was carried out, and after every evaluation that was carried out, it

has been determined that the object detection program is able to detect object

boundaries in the video that was pre-recorded or live video captured by a webcam.

The initial design of the program was able to detect object boundaries from a pre-

recorded video taken from a digital camera. The second and final design of the

program was able to detect object boundaries in real time via a USB webcam attached

to the computer. Therefore in summary, this project is a success whereby; both

designs of program are able to detect object boundaries.

6.2 Overall Conclusion

Overall, the object detection program satisfies the requirements of our project stated

in section 4.1 of this report. The object detection program is able to initialize images

and videos either a pre-recorded video or real time video. The object detection

program is then able to convert these images and video into grey-scaled images and

binary images. Finally the object detection program is able to detect object surfaces

and boundaries in the image and video by image analysing techniques such as

segmentation and applying an edge detector.

6.3 Recommendation

From the data collected from our tests and evaluations, there are many areas in which

this object detection program can be improved on for the project to be more

successful.

Firstly, the object detection program does not have a graphical user interface or GUI

for the user to select different output to be displayed. The program currently displays

all outputs simultaneously when the program starts and to certain users this



information may not be critical to them. I would recommend future work on GUI

interface for easy operation of the program.

Another area, which can be improved on, is the base algorithm of the code. From the

results shown in the test we did, the program is able to detect objects that have

sufficient light levels however, if the object has insufficient light levels the object is

not detected and edge operators cannot be applied to the object. Further exploration in

light analysis in video images should be done to improve robustness of the program.

This would then enable the program to detect objects with different light levels

appropriately. Object tracking can also be improved on and will also be useful for the

program. With object tracking, the user can input pre-defined objects for the UAV to

track.

Lastly, actual implementation of the program to a UAV or micro aerial vehicle should

be done. Our program is just one part of a vision control motion system (VCM) and to

see its true potential it should be implemented and tested with a UAV. As our

program was simulated on a computer, the output displays were video images. On an

actual implementation of a UAV, the result of the output will be different, as it will

need to be analysed by the microprocessor controlling the UAV. Getting the

microprocessor to analyse data from this program and getting it the UAV to react to

these results will be challenging.

Though we are now at the end of the project, testing of the program will still be

carried out till the Poster Presentation day. Some improvements will be implemented

and tested. If another student takes up this project, we hope that the improvements

and suggestions presented in this report will be taken into consideration.



7. Review and Reflection

This project for me has been an interesting and exciting learning adventure. From the

initial stages of research to the actual implementation of image analysing concepts in

the program was a challenge.

In the beginning of the project, I spend countless hours researching on image analysis

techniques and concepts. Understanding how digital images are being processed in

various computer systems was very enriching. As I have no prior experience in digital

image processing, I had to be clear on how digital images were processed and what

kinds of operators were needed to full fill the requirements of the project.

Once that was done, I was faced with another daunting task of writing the program

code and developing this program. As I am not strong in programming algorithms,

this task proved to be a nightmare. I referred to tutorial books and attending additional

lessons on MATLAB. I researched on the full capability of MATLAB and discovered

different toolboxes and functions that are built in to MATLAB. With that, I decided to

take full advantage of the Simulink function in MATLAB that uses block diagrams to

create a program. As I am more of a visual person, programming using Simulink

enabled me to develop the program for this project. When the program was done,

analysing the output data of the program was a challenge. Luckily, with all the

research done on image analysis concepts, I was able to point out the limitation of the

program.

Throughout this entire project, time management was the toughest. As a part time

student, juggling between work commitments and family commitments was very

challenging. Time management was a crucial skill that I developed during the course

of this project. Prioritising important events in the project and having self-discipline

to accomplish these targets was a valuable lesson learnt.

In the course of this project, there were many milestones that were reached through

the use of the Gantt chart. Different goals were set to keep the project on track

difficulties were solved along the way. Problems faced along the way strengthened



my analytical and problem solving skills. One major problem solving skill learnt

during the course of this project was contingency planning. One major event that

happened to me during the duration was a computer crash and I had to quickly rectify

the problem quickly and find alternative plans to continue with the project.

This journey proved to be a very enriching experience for me and despite going

through a lot of challenges and obstacles along the way, I am very glad I am able to

take up this project. I had to pick up concepts of image analysis very quickly as I had

no knowledge of these concepts in the past. At every single stage of he project, there

was something new to learn and everyday I made new discoveries with regards of the

project. Overall, this journey has been a programming adventure for me.



References:

• Segmentation and Boundary Detection Using Multi scale Intensity

Measurements – (Dept. of Computer Science and Applied Math The

Weizmann Inst. of Science Rehovot, 76100, Israel)

• Fundamentals of Image Processing- (Ian T. Young, Jan J. Gerbrands, Lucas J.

van Vliet)

• EdgeFlow: A Technique for Boundary Detection and Image Segmentation –

(Wei-Ying Ma and B. S. Manjunath)

• Computer Vision System Toolbox –(www.Mathworks.com)

• Fundamentals of Image Processing I – (David Holburn University Engineering

Department, Cambridge)

• Fundamentals of Image Processing II – (David Holburn University

Engineering Department, Cambridge)

• Fundamentals of Image Analysis– (David Holburn University Engineering

Department, Cambridge)

• Introduction to Video analysis using MATLAB – (Deepak Malani, REC

Calicut Anant Malewar, IIT Bombay)



Appendix A – Gantt chart



Appendix B – Initial Design Simulink Code



Appendix C – Final Design Simulink Code

Documents

Camara for uav jan2012 eas 021