- VISION-ASSISTED ROBOT KINEMATIC ERROR ES'MMATION:

PRINCIPLE AND APPLICATION

Weidong Tang

A thesis submitted to the Department of Mechanical Engineering

in confodty with the requirernents

for the degree of Master of Scicnce (Engineering)

QueenYs University

Kingston, Ontario, Canada

Jdy, 2001

Copyright 8 Weidong Tang, 2001

National Cibrary 1*1 ,canada Bibliothèque nationale du Canada

* .. and . .. e t = Services B i o g r a p h i q u e s 395 Wellington Street 305, rue Wellington Ottawa ON KfA ON4 ôüawa ON K 1 A W Canada Canada

The author has granteci a non- exclusive licence allowing the National Li* of Canada to reproduce, loan, disîriiute or sel1 copies of this thesis in microform, paper or electronic formats.

The author retains ownership of the copyright in this thesis. Neither the thesis nor substantial extracts fkom it may be printed or otherwise reproduced without the author's pefmissioa.

L'auteur a accordé une licence non exclusive pemettant à la Bibliothèque nationale du Canada de reproduire, prêter, disûiiuer ou vendre des copies de cette thèse sous la forme de microfiche/nIm, de reproduction sar papier ou sur format électronique.

L'auteur conserve la propriété du droit d'auteur qni protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprimés ou autrement reproduits sans son atltorisa~~~.

Abstract

Vision-assisted robot kinematic error estimation is a robot vision application to find out

"where it is" of the robot or the surrounding objects. The vision system demonstrates the

observation of the robot motions in the image form, therefore it cm perfonn the

kinematic error estimation in the image featurr space and then transfomi it into 3D robot

space or camera space if necessary. The intemlated theones for the vision-assisteci

kinematic error estimation are discussed in this thesis.

A direct visual servo control system is an application of robot vision-assisted kinematic

emor estimation. The robot joint displacernent enors are measured in the image featurc

space by the vision system and sent to the joint controller. As a case study, a dirrct visual

servo control method for a SCARA type manipulator is discussed and the comsponding

simulations are presented in this thesis.

The direct visual servo control method could be an alternative control scheme when one

or more internd joint sensors fail. aie possible functions of a vision sysfem in the rhree-

level (servo level, interface level and supe~sor level) fadt tolerant system of a robot are

presented. Simulations demonstrated that a direct visuai servo control system codd be an

alternative controller to fulfill full recovery and partial recovery scherne at servo level.

This application shows a potential method to improve the fault tolerance capability of an

existing robot by employing a vision system.

Acknowledgements

Without the guidance and help of rny supervisor, Dr. Leila Notash, this work would never

have been completed. I would like to express my appreciation for her advice and

financial support throughout the course of my Master's program.

1 am also grateful for the financial support provided by the Department of Mechanical

Engineering, and the School of Graduate Studies and Research.

1 wish to thank ail the students in Robotics Lab of the Department of Mechanical

Engineering for the help on my research.

Finally, 1 hope this effort could be a linle repayment to my family and my fiiends for

their consistent encouragement. Because of hem, 1 had an enjoyable and unforgettable

time during rny study in Queen's University.

Tabfe of Contents

... ................................................................................................................... List of T a b l s viu

................................................................................................................... Nomenclature Ut

Chap ter 1 Introduction ..................................................................................................... 1

1.1 Robot Vision Application .................................................................................. 2 1.2 Vision-Assisted Kinematic Error Estimation ............................. .......................... 2 1.3 Direct Vision Servo Control Method .................................................................. 3 1.4 Improving Robot Fault Tolerance Performance .................................................. 4

2.1 Introduction ........................................................................................................ 6 2.2 Robot Vision Theory and Applications ......................... ..... .......................... ... 7

...................................................... ..... 2.2.1 Human Vision ........................................ 8 2.2.2 Machine Vision ........................................... 10 2.2.3 Robot Vision Application Mode1 .................................................................... 11

2.3 Image Formation and Camera Mode1 ............................................................... 13 2.3.1 Image Formation ............................ .. .......................................................... 13 2.3.2 Carnera Mode1 ................................................................................................. 16

2.3.2.1 Pinhole (Cenaal Perspective) Projective Mode1 .................................... 16 2.3.2.2 Orthogonal Frojective Mode1 .................................................................... 16 2.3.2.3 Affine Projective Mode1 ........................................................................... 17

2.3.3 Camera Placement .......................... ., ............................................................... 18 2.4 Image Features and Feature Extractor ............................................................. 20 2.4.1 Image Features ............................................. ...... ......................................... 20 2.4.2 Feature Extractor ............................................................................................. 24

2.5 Reconstruction, Interpretation and Decision .................................................. .. 26 2.5.1 Reconstxuction ...................................... ,, . . . . 27 2.5.2 Interpretation and Decision ................... ..... ........................................ 30 2.6 Robot Vision Application .......... .. ................................................................ 31

Chapter 5 Simulation of a Dùect Visual Servo Control Method for a Three-DOF SCARA Type Manipulator eeee.ee.......~........................................................... 96

........................................................................................................ 5.1 Introduction 96 ........................................................................ 5.2 Basic Parameters for the Models 97

............................................................................ 5.2.1 Parameters for Manipulator 97 ..................................................................... 5.2.2 Parameten for Projection Mode1 98

......................................................................... 5.2.3 Controi and Other Parameters 99 ....................................................................................... 5.3 Prescribed Trajec tories 99

................................................................................ 5.3.1 Known Joint Trajectory 100 .......................................................................... 5.3.2 Known Endpoint Trajectory 101

5.4 Simulation Program Flow Chart ................... .. ............................................. 101 ........................................................................................... 5.5 Simulation Results 104

5.5.1 Simulation Results with no Error Input ......................................................... 104 ....................................... 5.5.2 Simulation Results with Image Cooordinate Error 118

................... ................. 5.5.3 Simulation Results with Dynamic Disturbance ...... 127 5.5.4 Simulation Results with Link Length Error ................................................ 128 5.5.5 Simulation Results with Joint Velocity Lirnits ............................................. 129

.............................................. 5.5.6 Simulation Results with Torque/Force Limits 130 5.6 Summary .......................................................................................................... 135

...................................................................................................... 6.1 Introduction 137 ............................................................ 6.2 Application in Fault Tolerance S ystem 139

6.2.1 ServoLayer .............................................................................................. 141 6.2.2 Interface Layer ............................................................................................ 149

......................................................................................... 6.2.3 Supervisor Layer 152 6.3 Joint Position Measurement of a Puma Type Manipulator ....................... ..... 153

. . 7.1 Conclusions and Contributions ......................................................................... 159 7.2 Future Works ................................................................................................... 162

Vita ....................... .- ................................................................................................... 170

Figure 2.1 Figure 2.2 Figure 2.3 Figure 2.4 Figure 2.5 Figure 2.6 Figure 2.7 Figure 2.8 Figure 2.9 Figure 2.10 Figure 2.1 1 Figure 3.1 Figure 3.2 Figure 4.1 Figure 4.2 Figure 4.3 Figure 4.4

......................................................... Vision application procedure mode1 12 . . Pinhole projection mode1 ............................................................................. 16

....................................................................... Orthogonal projective mode1 17 Eye-to-hand camera placement .................................................................... 19 Eye-in-hand carnera placement .................................................................... 19 Potential features that can be processed by the image processing ............... 21 Reconstruction is the inverse mapping of projection ................................... 27 Position-based look-and-move(PBLM) mode1 .......................................... 34

................................................ Image-based look-and-move(IBLM) mode1 34 .................................................. Position-based visual servo(PBVS) model 36

Image-based visual servo(R3VS) mode1 .................................................... 36 Task description and observation with vision system ............................... .. 46 Transformation from camera frarne to image frame .................................. 51 SCARA robot (Resource: Adept Inc.). ........................................................ 77 Example of the-DOF SCARA type manipulator ..................... .. ............ 77 Overhead view of the example SCARA type robot ..................................... 85 Joint angles in robot space and camera space under scaied orthogonal

. . .................................................................................................................... projection 87 Figure 4.5 Side view of the third joint from the camera .............................................. 88

....................................... Figure 4.6 Joint PD controller with a known joint trajectory 92 Figure 4.7 A joint PD controller with known Cartesian trajectory . M applied to

. . ............................................................................ estimate the desired joint trajectory 93

Figure 4.8 A joint PD controller with known Cartesian trajectory . FK applied to estimate the real end effector trajectory . ................................... .................... 94

Figure 5.1 The Bow chart of the simulation program ............................................. 103 Figure 5.2-1 TRAJl with no enor input: (a) tmjectory of end point, (b) and (c)

............... trajectories of f i t and second joints, (d) to (f) joint displacernent emrs 106 Figure 5.2-2 M l with no error input: (a) to (c) joint velocity. (d) to (f) joint

............................................. acceleration, (g) to (i) joint forceltorque .. ......... 107 Figure 5.2-3 TRAJl with no e m r input: (a) trajectory of end point. (b) and (c)

trajectones of first and second joints. (d) to (f) joint displacement emrs . Step time = 0.08 ms ....................................................... 108

Figure 5.2-4 TRAJl with no error input: (a) to (c) joint velocity. (d) to (f) joint acceleration, (g) to (i) joint force/torque . Step time = 0.08 rns ................................ 109

Figure 5.2-5 TRkll with no e m r input: (a) to (c) joint velocity, (d) to (f) joint acceleration . (g) to (i) joint torqudforce . Image sampling e m r = 2rnm/pixel ........ 110

figtwe 5.2-6 TRkll with no fiction: f e> tmjeetexy ef end point, (b.> and (c) tnjectoriies of fint and second joints, (d) to ( f ) joint displacement e m n .................................... ... 11 1

Figure 5.3 TRkl2 with no error input: (a) tmjectory of end point, (b) and (c) trajectones ................................. of first and second joints, (d) to (f) joint displacement e r m . 112

Figure 5.4-1 TRAJ3 with no error input: (a) trajectory of end point, (b) and (c) .............. trajectories of fïrst and second joints. (d) to (f) joint displacement errors. 113

Figure 5.4-2 TRAJ3 with no error input &er adjusting the control parameters: (a) trajectory of end point, @) and (c) tmjectories of fint and second joints, (d) to (f)

............................................................... ...................... joint displacement errors. .. 1 14 Figure 5.5 TRAJ3 with no error input after adjusting the mass: (a) trajectory of end point,

(b) and (c) trajectories of f i t and second joints. (d) to ( f ) joint displacement errors

......................................................................................................................................... 115 Figure 5.6- 1 TRAJ3 with random image coordinate emr: (a) trajectory of end point, @)

......... and (c) trajectories of first and second joints, (d) to (f) joint displacemeni errors. 119 Figure 5.6-2 TRAJ3 with adjusted mass and random image coordinate enor: (a)

trajectory of end point, (b) and (c) trajectories of first and second joints. (d) to (f) joint displacement errors. ....................................................................................... 120

Figure 5.7-1 TRAJ3 with dynamic disturbance: (a) trajectory of end point, @) and (c) .............. trajectories of fmt and second joints, (d) to ( f ) joint displacement enors. 121

Figure 5.7-2 TRAJ3 with dynamic disturbance after adjusting the mas: (a) tmjectory of end point, (b) and (c) trajectories of first and second joints, (d) to (f) joint

................................................................................................ displacement enors. 122 Figure 5.8-1 TRAJ3 with iink length enors: (a) trajectory of end point, (b) and (c)

.............. trajectories of fint and second joints, (d) to (f) joint displacement emrs. 123 ................ Figure 5.8-2 TRAJ3 with link length emrs: coordinate e m n of 01 and 0 2 124

Figure 5.9-1 W 3 with link length enors after adjusting the mass: (a) trajectory of end point, @) and (c) trajectones of first and second joints, (d) to (f) joint displacement errors ......................................................................................................................... 125

Figure 5.9-2 TRAJ3 with link length errors a€ter adjusting the mass: coordinate emrs of 01 and 02 ................................................................................................................ 126

Figure 5.10 T M 3 with velocity limits: (a) tnjectory of end point, (b) and (c) trajectones ..................... of first and second joints, (d) to (f) joint displacement errors. ...... 132

Figure 5.1 1 - 1 TRAJ l with force limits: (a) trajectory of end point, (b) and (c) trajectones of fmt and second joints, (d) to ( f ) joint displacement emrs. ................................. 133

Figure 5.1 1-2 TRAJl with force limits after adjusting the control parameters (a) trajectory of end point, (b) and (c) trajectories of £kt and second joints, (d) to (f) joint displacernent emrs ....................... .... ........................................................ 134

Figure 6.1 Vision system in a robot fault tolerance system. ................................... 140

vii

Figure 6.2 Vision system in the seno layer of a fault tnierance mhat syset m. .. 142 Figure 6.3- 1 Scheme without full recovery: displacement tracking performance . The

sensor of the second joint is failed at 1.5 sec ........................................................... 145 Figure 6.3-2 Full recovery scheme: displacement tracking performance . The sensor of the

second joint is failed at 1.5 sec and replaced by the vision system immediately ..... 146 Figure 6.4-1 Scheme without partial recovery: velocity and force . The sensor of the

second joint is failed at 2.5 sec . The system cannot stop smoothly ......................... 147 Figure 6.4-2 Partial recovery scheme: velocity and force . The sensor of the second joint is

failed at 2.5 sec and replaced by the vision system ...................... .. ........................ 148 Figure 6.5 Traditional model-based robot failure anal ysis ......................................... 150 Figure 6.6 Vision-based robot failure andysis ............................................................ 151 Figure 6.7 Vision system in the supervisor layer of the fault tolerance system .......... 153 Figure6.8 PUMArobotwithvisionsystem ............................................................ 154

........................ Figure 6.9 The relationship between the iine segment and its image 155

List of Tables

Table 2.1 Three Ievels of Mm's vision mode1 .............................. ...,.. 11 Table 2.2 Cornparison of different visual control rnethods ............................................. 35 Table 3.1 Comparison of vision-assisted kinematic error estimation methods ............... 65 Table 4.1 D-H parameters of a the-DOF SCARA robot ............................................. 78 Table 5.1 D-H parameten of the example three-DOF rnanipulator ............................. 97

output torque andor force vector of joint actuator (in N m for torque

and N for force)

horizontal and verticai sampling intervals in an image

image coordinates of a point in an image

digital coordinates of an image point

digital coordinate of the center of the image plane

coordinates of the origin of the ith joint frame in base frame

depth from the centroid of the observed point set to the viewpoint

point-to-point tracking error

position error function (in m) and its denvative with respect to time

(in m / s )

Coulomb friction

viscous friction

dynamic term associated with fiction

focal iength of the camera

desired and actual image feature vectors

feature Jacobian matrix

proportionai gain vector (in Nmim for revolute joint and Nlm for

prismatic joint)

derivetive gain veetor fin N m drrtd for revoIute joint and N dm for

prismatic joint)

actual length and image length of a iine segment

inertia matrix

mass (in kg)

dynarnic term associated with the Coriolis force, centrifugai force

and gravity

position vector in 2D image space

position vector in 3D space (robot spacc or camaa space)

Tb..

AIS

CCD

D-H

DOF

FMEA

FTA

IBLM

IBVS

PBLM

PBVS

P D

PUMA

SCARA

TFS

transformation matrix from frarne a to frame b

estimated transformation matrix h m frarne a to frame b

joint force

joint velocity

Augmented Image Space

Charge Coupled Device

Denavi t-Hartenberg

Degree of Freedom

Failure Mode and Effect Analysis

Fault Tree Analysis

Image-Based Look-and-Move

Imaged-Based Visual Servo

Posi tion-Based Look-and-Move

Position-Based Visual Servo

Proportional and Denvative

Proportional. Integral and Derivative

hgrammable Universal Machine for Assembly

Selectively Cornpliant Aaiculated Robot Ann

Transformed Feature S pace

Desired Robot Trajectory #1 @fer to Section 5.3)

Desired Robot Trajectory #2 (refer to Section 5.3)

Desired Robot Trajectory #3 (refer to Section 5.3)

Chapter t

Introduction

The main objective of this project is to improve the performance (such as fault tolerant

capability) of an existing robot by equipping the robot with a vision system. For fault

tolerance, there are generally two tasks: detection of the failures and recovery h m the

failures.

In order to acheve the goal of this project, vision-assisted robot kinematic error

estimation is studied. The purpose of this study is to check the kinematic erron of each

joint of the robot with a suitable vision system. Combined with other advantages of a

vision system and the functions of other sensors, the vision system could become a good

failure inspecter. The vision-assisted robot kinematic error estimation can be applied in a

direct visual servo control method, which has been proposed for a SCARA type

manipulator in this thesis. This method mats the joint position error estimation in terms

of image coordinates as the control e m r signal, which is sent to the joint controller.

A direct visual servo conml system could become an alternative contmller when one or

more sensors fail. From the viewpoint of a fault tolerance theory, this system can play a

role at the servo level of the fault tolemce system of a robot. Further analysis can show

that a vision system can function at different levels of a three-layer fault tolerance

system. The advantage is that an existing robot can gain additional fault tolerant

capabilities without ciramatic structure changes by employing a vision system.

1.1 Robot Vkion Application

Vision technology has k e n a popular topic in mbot research laboratones for years.

Besides the studies on the human/animal vision with the goal of understanding the

operation of biological vision systems, more and more vision resemhers stress on

developing vision systems for the application in an industrial envuonment such as

navigation, and recognizing and tracking objects.

An acnial vision system could not perform the whole functions as weU as the eyes of a

human being do. However, robot vision technology provides a combination of flexibility,

adaptabili ty, precision, non-contact measurement and object recognition for robotics.

Generally, the vision system could answer two questions: what it is and where it is. Here

"it" refers to the observed object in the scenery. A robot vision system could answer both

questions, but often focuses on one of them.

The vision application model introduced in Figure 2.1 shows the information flow among

different function modules. The research on the corresponding theories and applications

will be àiscussed based on the structure of chis model.

1.2 Vision-Assisted Kinematic Error Estimation

Vision-assisted robot kinematic error estimation is a robot vision application ïo find out

"where it is" of the robot or the sumunding objects. The vision system demonstrates the

observation of the robot motion in the image form. Therefore, robot motion description

(expected and actual) c m be defined in the image space and then robot kinemah'c crms d

estimation can be completed in robot space. camera space or image space. Usually the

estimation results are represented in image featue space or msformed into thne

dimensional (3D) robot space or carnera space. However, in some cases the

transformation is not necessary if the required task motion and the actual motion cm be

both described in the image space.

Chapter 3 will present the theories of the vision-assisted Ianematic e m r estimation.

These theories discuss the following topics: transformation among world space, camera

space and image space; camera projection model; reconstruction from the image features;

etc. There are different vision-assisted kinematic e m r estimation methods depending on

the camera placement and the task description space. The different methods wül be

compared with each other. Feature Jacobian matrix, which refiects the relationship

between the feature space and 3D world space, will be presented case by case.

A direct visual servo control system is an application of the vision-assisted kinematic

error estimation. Currently most of the robot tasks are defined by the trajectory of the end

effector, so a vision system is often used to recover the position information of the robot

end effector and the objects around the robot. However, a direct joint position

measurement is necessary if a direct visual servo control system is requii.ed A diract

visual servo control method for a SCARA type manipulator will be discussed and

simdated in Chapters 4 and 5. The simulation results will show that the vision system

could meww the joint position dir;ectly in tenns of the images coordinates of the s@d

geometrical feature points of the robots, and it is feasible to form a visual servo control

system for a SCARA type manipulator with the joint position enor estimation in the

image feature space.

In Chapter 6 a method is discussed to show how to use a senes of line segments as the

features for rneasuring the joint positions of a PUMA type robot. This discussion shows

the flexibility of implementing the vision system for a specific robot. Analysis of the

structure of a robot can help the selection of the suitable features that will lead to a

successful vision application.

1.4 lmproving Robot Fault Tolerance Performance

Robots are being more frequently utilized in inaccessible or hazardous environments to

alleviate some of the risks involved in sending humans to perfom the tasks and to

decrease the respiring time and expense. It is h n g to be a more and more important

issue that a robot should have autonomous fault tolerance without requinng immediate

human intervention for every failure situation. A new robot can always gain fault

tolerance capability frorn the speaal considerations during the design procedure.

However for an existing robot it is quite difficult to improve its fault tolerance

performance wi thout changing its structure.

The discussion in Chapter 6 shows that a vision system can ffexibly and extensively

collect the information in the robot workspace and provide the possibility to improve the

4

fauh tohmce of en existittg robot. A vision system am @fornt di.ffepenk funcéhns in

different iayen (servo layer, interface layer and supervisor layer} of a robot fadt

tolerance system.

The direct visual servo convol system could be the basic module of the robot fault

tolerance system, which can allow the robot to continue perfonning its tasks or to

gracefully degrade if one of the joint sensors is failed. A vision system has the ability to

perform a nontontact measurement and to provide a broad view of the robot workspace

in world space. The results from the vision system could be compared with the

information obtained from other sensors or sources in order to point out the failure of the

joint movernent and supply more information for the rask manager to avoid unsafe

actions. More advantages of a vision system in a fauit tolerant robot are aiso discussed in

Chapter 6.

Chapter 2

Literature Review

2.f Introduction

Robot vision technology has become one of the most popular topics in robotics research.

Currently, vision sensors not only are experimental devices in the laboratones but have

also become standard components for many indusaial robots (such as welding, inspection

of automobile parts), space robots and housework robots, etc.

This chapter is a survey of the related robot vision theories and applications. The

following topics wiil be presented:

Basic vision theories: vision system can be considmd as an infoxmation-processing

model, which is the process of discovenng h m images what is present in the world and

where it is. Based on the different projective assurnptions and mathematical theories,

various vision models have been developed in recent years.

Image processing procedures: an image processiing procedure includes featwe

extraction and pattern recognition. Depending on the featms selected h m the image, a

lot of different methods could be applied. For robot vision applications, selected features

should be understood and detected relatively easily. Matching the comsponding feanires

in two or more images from the same scene is the most difncult part. Some abstract

feanires (such as resuits of wavelet transfomation) have been used in some applications.

Reconstruction approaches: the goal of reconstruction is to obtain information of

the observed o b j a from the image sequemes and rekabty to guide the action of &

robot or related rnachinery. The reconstruction can be in 3D or 2D image space. There is

no general reconstruction method. In order to obtain correct information about targets,

interpretation and decision procedures should be involved. In recent years, considerable

efforts have been devoted to object reconstruction based on image movement and

projective geometry.

Robot vision application: depending on the purpose of the vision system, there are

two kinds of robot vision applications. One is to find "what it is" in the scene. The other

is to find "where it is". Finding "what it is" helps robot to make some decisions for task

planning. A typical applicatiori is obstacle avoidance for mobile robots. Finding "whehen it

is" helps robot to meas= the position of the objects in the scene or the relative position

between objects and carneras. A typical application is visual servo control technology.

The contents of this chapter will be arranged as follows. Section 2.2 discusses the basic

theory and a robot vision application model. Section 2.3 discusses the image formation

and different camera model description. Section 2.4 discusses the image processing

technology. Section 25 intmduces reconstruction methods and necessary description aad

decision making procedure. Section 2.6 intmduces two kinds of robot vision applications

and stresses on the visual servo control technologies.

2.2 Robot Vision Theory and Applications

There are two groups of scientists working on vision. One group includes

neurophysiologists, ps ychophysicists and physicians, who are studying human/animal

vition with the g d of understanding the operation of biofogieitt visim eystem.

including their limitations and diversity. The other group includes cornputer scientists and

engineers doing research in machine vision with the purpose of developing vision

systems that are able to navigate, recognize, track objects and estimate their s p e d and

direction. The first group influences the design of vision systems by the second group. At

the sarne time. the second group helps the first group to mode1 biological vision 1461.

Robot vision is an application of machine vision in robotics. Machine vision simulates

some functions of human vision and therefore it is based on the research on biological

vision. Although some excellent human vision models are available, it is not easy to

apply these models to machine vision. In this section, the structure of robot vision

application procedure will be gi ven.

2.2.1 Human Vision

The efforts to understand how light signals are transuded into neural activity and how that

activity is transfonned through the neuro system has intrigued neurobiologists for many

years. Masland [63] has introduced the research status about the functions of the reha

For many years, the retina was known to be composed of five main classes of neurons.

Now it has become clear that the number of functional elements may be as high as 50.

Each functional element can process different information. Some translate static

information such as color, shape, etc. into human brain; some translate dynamic

information such as subtie changes due to the object movement relative to the

background. According to some researck* the &na has turo sets of n a c o s those that

establish a direct pathway from the source of light to the optic nerve and those that make

laterai connections. It is likely that the human vision system cm collect more information

in the scenery (not just shown on the picture taken by camera) and process them to obtain

a more accurate description of the observed object in the world.

Kondo and Ting [53] presented the main advanrages of the human binocular system:

O Eyes are sensitive to a slight change of brightness or color.

Eyes are able to acquire high-resolution images.

Eyes are able to detect motion parallac with a slight motion of the eyeball.

Brain is able to constnict 3D scenery by combining binocular-vision output with

various dues such as motion parallax and continuity of contours.

However, there is an interesting weakness in human vision system: the computation

speeds of individual neurons in the retina are about a million times slower than electronic

devices (but consume one ten-millionth as much power). They also operate with far less

precision than does a digital computer [6û]. This phenornenon highhghts that

Biological computation must be very different fkom the digital computer.

O Brain has a very different reconstruction model from that used in research.

Insects' vision indicates that they rnainly use selfmotion to sense distances to targets for

navigation and obstacle avoidance. Compared to human vision, insects have relatively

simple, but effective visual systems. This feature implies that robot vision might have

diverse functioaal models but might not necesad- be p e d ~

2.2.2 Machine Vision

Machine vision, which originated from photogrammi=try [64], is thought to have the

ability to sense, store and recover an image that matches the original space as closely as

possible. Photogrammetry concentrates on the recovering of position information of the

world. Based on the photogrammetry, machine vision technologies expanded into new

research areas such as object recognition.

M m [62) presented two questions, one or both of which should be answered by a typical

machine vision system:

What is presented in the world

Where it is

Furthemore, there are four basic questions that should be answered before setting up a

special machine vision system:

What kinds of images are required? This question concentrates on the camera

placement, viewpoint, illumination, etc.

What kinds of features should be found in the images? This question concentrates

on the distinctive characteristics of the objects with specid optical projection.

How to present the features? This question concentrates on the mathematical form

of the features obtained from the images.

How to recover the observed object using the feahires obtained from the images?

The above four questions are interrelated. For example, the selection of the features cm

influence the image processing and reconstruction rnethods.

Table 2.1 Three levels of Mm's vision model [62].

Computational Theory 1 Representation and Algorithm

What is the goal of the

computation?

W h y is it appropriate?

What is the logic of the

strategy by which it c a . be

carried out?

How can this computational

theory be implemented?

What is the representation

for the input and output?

What is the algorithm for the

transformation?

Hardware Irnplementation

How cm the algorithm

and representation be

realized physically?

M m (621 synthesized a vision model. This information processing model inciudes three

leveis as shown in Table 2.1. Marr pointed out that a machine vision system should

perfom two tasks to answer the above four questions:

Representing the information: this is the basis for decisions on how to treat the

observed results. At this step, the types of features and the way to represent the featurrs

should be decided.

Pmcessing the information: this analyzes the usefbl information extracted from the

image to show the various aspects of the world, such as beauty, shape and motion.

2.2.3 Robot Vision Application Mode1

Robot vision, which concentrates on pattern recognition, positioning, inspection and

modeling of objects, is the application of machine vision in robotics. With the

development of maFhins vision techndogy, more and mort robots are equipped with a

vision sensory system. In most of the robot applications, ody a partial description of the

scenery is necessary, which cm help the robot recognize special objects (marks) or

evaluate its movement accuracy relative to the environment.

According to the charactenstics of robot applications, a typical robot vision application

procedure is illustrated in Figure 2.1. Different parts of this model could be investigated

to produce various practical technologies suitable for different applications.

vision S8nSOC - feature

model extractor

Figure 2.1 Vision application procedure model.

The terminologies and concepts used in Figure 2.1 are explained as follows:

Objects: They are observed objects in 3D space. They are also caiied targets.

Images: They are results of the transformation h m 3 0 world space to 2D image

space. They cm be storeci on films or in cornputer memones.

Features: They are usefid image features extracted fiom a 2D image. Usually

they could be geometrical feahues as well as some abstract features.

Interpretation: It is a 3D mode1 description of the objects/targets. It is not the

teal original object but a partial descxiption of the obsenred ohject Sametimcs this

interpretation not only is incomplete, but also produces ambiguities.

O Vision sensor rnodel: It is a mathematical description of the individual vision

sensor. It cm redire the transformation fiom a 3D space to a 2D image space.

Feature extractor: It is a series of image processing methods to extract the useful

features from the images. It should be designed according to the applied features.

Reconstruction model: It is an algorithm for realizing the riansformation from

2D image space to 3D space. Theoretically, it is an inverse procedure of image formation.

Due to information loss during al1 of the previous processes and mathematical

limitations. the reconstruction model is often an independent model and very different

from the vision sensor model.

O Decision model: It is the method to analyze the results of interpretation, to obtain

reasonable information with physical meanings and to decide how to use the results.

2.3 lmage Formation and Camera Model

Image formation is a procedure, which transfers the 3D information into 2D image space.

Image formation has a strong dependency on the camera model (more generally it is

called vision model). In this section, basic image formation theories and three typical

camera models will be discussed.

2.3.1 lmage Formation

The images used in robot vision research are very different fkom the images on the retina

of a human king. The p w p l e of human vision is s t i l l a big question far hidogists even

though some cornplicated descriptions about the retina structure have been set up [63].

Presentiy there is no general vision mode1 for practical vision applications. Depending on

the individual application or research, different image formation techniques can be

selected. Usually only partial information cm be recorded and used from an image.

Physicaliy the image formation is a procedure to process the light flow from the scenmy

or objects and store hem on a special media, which can be film or computer memory.

Mathematically the image formation is a transformation h m a 3D description of the

observed objects into a 2D d e ~ ~ p t i o n .

M m [62] pointed out that the image charactexistics are influenced by many factors:

Geometry of the obsewed objects: this is the basis of an image. Especiaily in a

robot vision application, the geometry is a very important feature.

Reflectance of the visible surfaces: this influences the gray-level of the targets.

IIIumination of the scene: this influences the background of the image or

signdnoise ratio of the image.

Viewpoint: this decides the optic path from the object to the image plane.

For a vision application, charge coupled device (CCD) carnenis have been extmsively

used. CCD cameras can output elecaonic signals, which can be easily üansferred into

digital signals operated by a computer. Furthemore ail photosites are sampled

simultaneously when the photosite charge is transferred to the transport registm [II].

Active vision

Bajcsy [8] pointed out that human vision perception is not passive, but active. This

adaptation is crucial for survival in an uncertain environment. Active vision is the control

of the optics and the mechanical structure of cameras to simpiify the pmcessing of

cornputer vision. Active vision techniques such as active 3D sensors with stnictured

iighting [2] [92] [95] can highlight some features to facilitate the object recognition.

Bajcsy [8] presented two kinds of active vision methods:

Transmit electromagnetic radiation (e.g., radar, sonar) into the environment and

measure the ~flected signal by time-of-fhght sensors.

Employ a passive sensor in an active fashion, purposeNly changing the sensor's state

parameten according to sensing strategy.

Laser measurement s ys tem is an example measurement method that uses tirneof-flight

pnnciple. It is suitable for high accuracy meastuement between two points or two paral1el

planes. If multi dimensional or random directional measurements are quired, then a

high speed scanning mechanism is necessary. It is not easy to use a laser device to foIlow

a point with unpredicted movement. The cost of laser device is another concem.

Passive vision

In passive vision there is no special efforts on the f e a m s of the objects. The objects are

observed in natural iighting. A passive vision system is more suitable for robot

applications in unstructured environments such as space, undemater or missile tracking.

2A2 Camera Mode1

Based on the different projective assurnptions, three carnera models are setup as follows:

23.2.1 Pinhole (Central Perspective) Projective Model

Pinhole model is the most popular and traditional projective model. The center of the lens

is imaged as a point (pin-hole), through which light c m go through. Maybank [64]

introduced the basic principles of the pinhole projection.

3D Object Space

Figure 2.2 Pinhole projective model.

The image surface is taken to be a plane, but sphencal surface with a center at optical

center O shown in Figure 2.2 is also a valid assumption. WO images codd be obtained

on the sphere: 1 and l n . The image formation is a complicated non-linear transformation

because of the physical aberrations and lens distortions of the camera [64].

2.3.23 Orthogonal Projective Model [643

In this model the image is formed by the p d e l beam of rays traveling from the surface

of the object as illustrated in Figure 2.3. When the image is mal1 and the range of depth

is rrseictcd, orrhogonaf projedm is an adeqotite modef. Due to rhe large effect of sm&

emon and conditional restriction, it is not easy to obtain accurate measuring results when

the distance between the camera and objects is too small.

Figure 2.3 Orthogonal projective model.

w

objecî * m * * m

Jang et ai. [45] introduced a revised projective model based on the ideal orthogonal

projection. This model adds one more dimension to the image plane, as the result, a 3D

coordinate system is formed. The direction of the augmented dimension is perpendicular

to the image plane and the coordinates of the points (or pixels) in image planes are

obtained by orthogonal projection.

image

2.3.2.3 Affine Projective Mode1 a91

Many researchen have been working on the affine projective model [19] 1341 [80]. The

linearity of the model (discussed in Chapter 3) allows the camera calibration to be gready

simplified. This mode1 does not correspond to any specific imaging situation. Its

advantage is that it is a gwd local approximation of perspective projection, which

accounts for both the extemal geometry of the camera and the intemal geornetry of the

Iens. Shan [80] gave a practical camera calibration method based on the affine projection.

23.3 Camera Placement

In addition to image formation and carnera model, camera placement is also an important

factor that could influence the result of image formation and feature extraction. There are

three ways of placing cameras in robot vision applications:

Eye-to-hand systern

In this case the camera is fixed in the environment and is calibrated with respect to the

robot base f'arne. As shown in Figure 2.4, eye-to-hand system can give a broad view of

the workspace, but it could lose the target because the components of the robot may

occlude the targets as the robot moves around.

Eye-in-hand system

In this case the carnera is mounted on the robot and calibrated with respect to the end-

effector Frame. As s h o w in Figure 2.5, eye-in-hand system cm focus on the object,

however the target should be located repeatedly at a high rate to obtain reasonable

information in limited time because the camera coordinate h e is changed as the robot

moves around.

Active camera head

In this case the camera is not mounted on the robot but is fîxed on an individual padtilt

control device, which is fixed in the environment. The p d t i l t device can change the pose

of the camera to track the object in the scenery. Gilbert et al. [27] described an automatic

Figure 2.4 Eye-to-hand camera placement.

Figure 2.5 Eye-in-hand camera placement.

tracking canera, which cwld keep the t q e t centeied in the image plane. A conbol

processor received the position information of the target from a track processor and

provided the estimated values of the variables necessary to orientate the tracking optics

(the pose of the carnera, zoom lens setting, etc.) towards the targets for the next field.

2.4 lmage Features and Feature Extractor

The purpose of image processing of robot vision is mainiy to obtain useful features h m

images. Normally image processing includes two procedures:

Feature extraction: the f e a m s related to the identical characteristics of the

observed objects are extracted from the images.

Pattern recognition: the meanings of the featwes to fonn the complete or partial

information of the objects are synthesized.

Most of the feature extraction models concentrate on some special features in the images.

The most difficult part in the feanire extractor is the correspondent feature matching

between two or more images of the same scenery with different viewpoints. The

applicable features and feanire extraction approaches are strongly c o ~ e c t e d to the

application. Currentiy there is no universal approach. More information on image

processing methods can be obtained from the textbooks on image pmessing [74] [89].

2.4.1 lmage Features

GeneralIy, the features are some types of "tokens" that c m be located unambiguously and

rrpeûtedly in differerrt views of the sme and to whidr values of at&ibutes tike

orientation, brightness, size Oength and width) and position c m be assigned [62].

In the machine vision fiterature, an image feature is any structural feanire that can be

extracted from an image [30]. Typically, an image feanire corresponds to the projection of

a physical feauire of an object (e.g., robot end effector) ont0 the camera image plane. A

good feature is the one that c m be located unambiguously in different corresponding

views of the same scene. Image features include:

Straightfomard features

Figure 2.6 Potential features that can be processed by the image processing.

Snaightfonvard features cm be color, line, curve, gay-level, point, edge, etc. Some

potential features are shown in Figure 2.6. If a donkey is given a suitable description that

is easy to extract fiom the image, then a donkey is aiso a very good straightforwad

feature. In robot applications, svaightforward features are always those with physical and

geometrical rneanings such as: points, lines, circles or other regular cuves, edges,

surfaces, areas, etc.

Image movement

Image movement refers to the feature changes in serial image fiames (it is dso called

v i h sueam) or finît ordei- or h i g k o~der &riuatiues of the same image These dynamic

features reflect the object movement, as well as camera movement. Jarvis [46] introduced

two distinct approaches for the computation of motion h m image sequences:

Feature-based approaches require that comspondence be established between a

sparse set of features extracted from one image with those extracted from the next image

in the sequence. The effort of finding f e a m correspondence is often damaged by

occlusion, which may cause features to be hidden or false feanires to be addressed' .

interpretation of motion and structure h m optic flow2 requires the evaluation of

fmt and second partial denvatives of image brightness values and also of the optic flow.

The evaluation of derivatives might drarnatically change the content in an image because

it is a noise enhancing operation.

A bstract features

In some special applications, some abstract features based on mathematics without stmng

physical meaning can be used. These features are the resdts of some mathematical

' In practid cornputer vision. motion analysis requires the storage and manipulation of image sequaas cathe~ than single images. Image sequences can be stored in memory as Ws of -YS, vcctors of mys oc 3D m y s , and on disc as single file or as files with scquentiai numbcring, Each image in a sequenct is ofien called a frame, and the elapsed time for forming each fiame is calleci the frame interval. The inverse of the fhme interval (the number of fiames per second) is the hme rate.

* Optic flow refers to the motion in v ida images. which is representcd by a twoaMcnsiond vcctor field as show below. Each line represents an optic flow vector. The Iength and direction of the line indicate the speed and direction of motion of an image feahue at the position of the dot, Mathematicaiiy, the optic flow

tr;uisformatio11~ (e-g., wavelet transfannatim and Fast Fourier Transfomation ml).

FIusser and Suk [26] classified the features into four types:

Visual features (such as edges, textures and contours)

Transform coefficient featrires (such as Fourier description)

Algebraic features (such as those based on matrix decomposition of an image [37])

Statistical features (such as moment invariants)

The last three types of features are mainly based on the mathematical transformations and

can be categorized into abstract features. The detailed explanation of these feahires can

be found in [26].

Jang and Bien [44] provided a mathematical definition of a feahue as an image function.

They considered an image received by a camera to be a 2D array of iight intemiries. Then

the brighmess intensity of the image at the location (u, v) is 1 (u, v), where

- u, 5 u 5 u,; - v, 5 v 5 v, . (u,, v,) is the origin of the image plane. Then a feature cm

mapping of the locations and brightness intensities of the pixels in the image. The

mapping can be either linear or nonlinear, depending on the nature of the feature under

consideration, and may include Dirac delta functions'.

' Dirac delta fundion 6(1) is not a mie hmction since it cannot k defincd completeiy by giving the

function value for aU values of the argument t. The notation G(t -to) stands for G(t - t g ) = O fw

The selection of features is essential for a swcessfid uision application, G d features are

easy to detect in different noisy backgrounds and could provide effective information to

the application system. Image recognition criteria include feature robustness,

completeness, and cost of feature extraction.

2.4.2 Feature Extractor

Feature extractor is a set of software to explicate some useful features by processing the

image data. Any particular features make certain information explicit at the expense of

some information that is pushed into the background and may be hard to recover [62].

The characteristics of features dictate that the feature extractor must be application

orientated. A typical feature extractor shouid achieve the following three tasks:

Initial processing procedure

This procedure is used to highüght the useful f e a m s in the image. Normaiiy the initial

procedure includes: digitization, thresholding, segmentation, histogram, shape

identification, template match, edge detection, etc. Information on these basic methods

c m be found in any textbooks on image processing, e.g., [74] [89].

Quantization of the feahire parameters

Image feature parameter is any real-valued quantity that can be calculated from one or

more image features [25]. Implementation of robot vision is closely associated with

cornputer calculations, and cornputers cannot work without quantization of feature

parameters. In robot vision applications, common feature parameters include coordinates

and distances between points in image, lm@ d imes or edge4, &r geomeeieel

parameten of lines or curves, etc. Some mathematical transfomations c m be very useful

to obtain distinctive parameters. These mathematical methods include wavelet

transformation. Hough transformation [56], FFI'. etc.

Matching correspondent features

In robot vision applications, obtaining several related images from the stereo or senal

images using the same camera at different times. is very cornmon. Therefore, one of the

important issues is how to tind the corresponding features from the comsponding

images.

This is the most difficult part of feature extraction. Except for sorne special applications

in which extra information about the target (or camera) movement can be obtained h m

other sources, two or more images of the same scenery are needed to recover the target.

The purpose of matching the corresponding features is to find the same features reflected

in different images. Those images are taken from the same scenery but from different

uiewpoint or in different tùne. There are two kinds of methods tu deal with feature

matching problem.

Area-based method: it is a typical method to compute the comlation between two

images in two regions. Fit-order or second-order differential images are used to find the

correlation between two regions. The problem is that sometimes the corresponding points

c m o t be found even if the simiiar areas are found.

F e s t u ~ W methd: Feature-based method is used ta find tbe. same f e â û ~ ~ ~

according to the sarne feanire descriptions such as the same color, or shape, etc. The

features include two categones [33]:

Global fatures such as long edge segments. These features are difficult to detect

in both images. If suitable global features can be found, it will be easy to implement

image matching.

Local fwtures such as edges extracted with local operators. These features are

easy to detect, but it is not easy to be used to match many similar features due to the

effect of noise.

There is no general feature matching method as the technique depends on the

applications. Before choosing a feanire matching method, the analysis of feahires is

required. Appropriate feature analysis could alleviate the workload of feahire matching.

2.5 Reconstruction, lnterpretation and Decision

Interpretation, which refea to the description of the observed object, answers the

question "what is present in the world and where it is". Interpretation is reaüzed by

reconstruction that is an inverse procedure of mapping h m the object to the image.

Norrnally the complete description of the object cannot be realized Sometimes the direct

results from reconsmiction can produce ambiguities. Therefore, a decision mode1 is

needed to process the initial interpretation of the objects.

The goal of robot vision is to obtain information from image sequences rapidly and

reliably to guide the robot to complete its assigned task. As shown in Figure 2.7,

reconstruction can be thought of as an inverse procedure of mapping h m object to

image. Due to the nonlinear and complicated nature of the vision model, the inverse

procedure may generate no solution or redundant solutions. Many researchers have

investigated ways to avoid ambiguities in reconstruction.

projection

target image

reconstruction

Figure 2.7 Reconsmc tion is the inverse mapping of projection.

According to the features used for reconstruction, Maybank [64] discussed two typical

ways to obtain the space information from an image.

Reconstmction fmm static image correspondences

There are many different static features (point, line, c w e , gra wel, color, etc), which

have been used by many researchers to setup image comspondences. In some works.

some vimial and abstract features have been used to recover some special characteristics

of the objects. As the result of this kind of reconstruction, static information of the

observed objects cm be achieved

Refoastmtüon €mm dynamic image corp~sponden~

Image motions have been used in recent decades to deduce the veiocity of the carnera and

the position of one point relative to the camera frame. Analysis of a time-varying image

often involves estimating 2D motion in the image plane. Image motion estimation is very

important in many applications such as object tracking [84].

Huang and Netravali [43] discussed typicd situations of reconstruction from feature

correspondences and categorized them into three types:

3D-3D correspondences: in this case, the locations of N corresponding f e a t m

at two different times are aven, and the 3D motion of the rigid object is estimated

2D-3D correspondences: in this case, some of N comsponding features are

specified in t h e dimensions and the other features are defined by their projections on

the 2D image plane, and the location and orientation of the camera are found.

2D-2D correspondences: in this case, al1 the N corresponding features are given

in the 2D image plane either at two different times for a single canera or at the same

instant of time but for two different cameras. In the former case, the 3D motion and

structure of the rigid object is identifie& and in the lattex case, the relative amh& of the

two cameras is detennined.

Image motions contain information about the motion of the camera and the shapes of the

objects in the field of view. The= are two main types of image motions [64]:

Finite displacement is reconstructed by a finite number of discrete point

correspondences between the two images of one scene iaken at different positions.

* Image v e l e e i k are the veki t ies of rhe points in the image as they mavc over

the projection surface.

In recent years, considerable effort has been devoted to the reconstruction from image

movement. Some of the typical methods are introduced in the following paragraphs.

Spetsakis and Aloimonos [go] introduced reconstruction fmm moving liaes. The

accuracy of the motion estimation results depends critically on the accuracy with which

the location of the image feanires can be extracted and measured. Generally, line feanins

are easier to exmct in a noisy image than point features (e.g., using Hough

transformation). Furthemore, it is easier to determine the position and orientation of a

line to subpixel accuracy than the coordinates of a point.

Faugeras [19] introduced reconstniction from moving curves. Because of the similar

reasons descnbed above moving curves are very wful features in 2D image motion.

Usually more feature parameters are needed to describe the moving curves compared to

the moving lines.

Negahdaripour and Horn [69] introduced reconstruction €rom gray leve1 change. It is

difficult to find suitable features or corners on the image of smwth objects. Furthermore

the feature-matching problrm has to be solved. In this case, the change of gray-level is a

good feature to reaiize the reconstruction.

Wu and Wohn [IO51 inûoduced reconstruction fmm image vdodty estimation. Thc

velocities of points in an image depend an the xmticm af the camera relative ta the

environment and also on the distance from the image plane to the object points.

Theoretically and practicdly image velocities provide a method of reconstruction, which

is an approximation to the more complicated reconstruction. In practical applications, the

estimation of the velocities and positions of objects relative to the camera is often

generated in a short period of tirne.

2.5.2 l nterpretation and Decision

Interpretation is the comprehensive result of reconstruction. In the previous section, it

was mentioned that simple mathematical reconstruction could produce meaningless

results. Sometimes more than one ~constmction procedure is applied in the robot vision

application. The decision procedure can analyze and synthesize al1 the information to

produce meaningf' information about the observed object

In robot vision applications interp~tation can be divided into two categories:

a 3D space interpretation reconstructs the 3D space information h m the

extracted features. Typically 2D image space coordinates and camera models are used to

realize the inverse transformation h m 2D space to 3D space. The geometricd

characteristics of the observed object are dso the main content of description.

a 2D image interpretation is not the simple repeat of the image features but a kind

of synthetic result that c m describe the behavior of the objects in 2D projective space.

Image Jacobian matrix in image space, or the resdts of some puxe mathematical

transformation (wavelet transformation, FFI', etc) can be used to interpret the image.

Decision procedure is ta appiy the infcumîiion obtained finm a vision system to some

practical operations. In robot applications, deàsion output cm be applied to the control

system or for task planning of the system. Though description procedure makes the

reconstruction results very meaningful, complete information recovery of objects is

impossible. Decision procedure decides which part of the information is usehil and

compares it with the information fmm other sensors or other sources. Decision is a very

important step in robot vision application. Because there is no general vision mode1 as

perfect as human vision system. how to apply the results of interpretation in a dynamic

environrnent, as the purpose of a decision procedure, becornes the key point of the

success of a robot vision system.

2.6 Robot Vision Application

Vision sensors have k e n applied extensively in robotic research Iabs, as well as in

indusirial environments. A11 the applications cm be divided into two categones

depending on the purpose of the vision system. Some details about visual servo control

system will be introduced in this section.

2.6.1 Two Types of Robot Vision Applications

It was mentioned in the previous sections that a vision system could answer two

questions: what it is and where it is. A practical robot vision system shodd answer both

of these questions, but often focuses on one of them.

Qurtlity ef Obsewed Objeets (w ha$ ir is)

A typical application for this category is to guide the mobile robot to avoid the obstacles

in its path. Basically the locations of the obstacles should be known, however, the prezise

measurement is not necessary. Recognition of the obstacles should be more helpful for

deciding which kind of obstacles exist in the scenery and then making decisions to allow

the robot to stop or avoid the obstacles by following a specified tmjectory.

Measurement Precision (where it is)

This kind of vision system concentrates on the geometry measurement of objects in the

world. The precision is very important in this situation. Since it will guide the robot

movement, this measurement cares more about some partial geometrical features (such as

points, lines and planes) and ignores the complete quality of the object. A typical

application is visual servo control system that will be discussed in the following section.

2.6.2 Visual S e ~ o Control System

Visual-servoing is a technology that incorporates the vision information directly into the

task control loop of a robot. This kind of control system is not a simple feedback system

but the fusion of results h m many elemental a ~ a s including high-speed image

processing, kinematics, contml theory and rd-time computing 1251. Combined with

sensor fusion technology, a lot of creative visual control techniques have been developed

which encompass manufacturing applications, teleoperation, missile tracking, nuit

picking as weli as robotic table tennis playing 151, jupgiing, etc.

Sanderson and Weiss [76] described the characteristics of different visual semo systems,

which forrn the basis to identify different visual servo applications. According to the

description method of control e m r function, there are two kinds of visual smro

approac hes:

Position-Based Method

In this method, control error function is computed in the 3D space based on an ideal

geometric mode1 of the object. This method requires a calibrated camera to obtain

unbiased results (Position-based look-and-move (PBLM) model as show in Figure 2.8

and Position-based visual servo (PBVS) model as shown in Figure 2.10). The main

advantage of position-based control is that task description in tems of the Cartesian pose

is very cornmon in robotics. The main disadvantage is that this method is sensitive to the

carnera calibration error, and the computation time is longer than image-based method.

Image-Based Method

In this method, control emor function is computed in the 2D space based on the image

Jacobin matrix that shows the characteristics of movement in image space (Image-based

look-and-move @LM) model as shown in Figure 2.9 and Image-based visual servo

(IBVS) mode1 as shown in Figure 2.11 ). Some algorithms provide the camera velocity in

each iteration for minimizing the observed emr in the image. This method is mbust with

respect to the camera and robot calibration e m r s and cornputation time is less than that

in the position-based method. The disadvantage of this method is the presence of posSbIe

Cartesian Robot hntroi Law

-+ Controller

-) Robot -

I I

joint sensors E z l

Figure 2.8 Position-oased look-and-move (PBLM) model.

Figure 2.9 Image-based look-and-move @LM) model.

Based on how to use the visuai information in the robot control system (decision

meihod), two kinâs of visual servo contml systems are defined:

P I jl

- Robot Image space controi iaw

- Robot Controiler

-+

* Dymmit look-and-wve systetn: the h e v d informit€ition is used as the set-point

input for the robot joint-level controller (as shown in Figure 2.8 and Figure 2.9).

Direct visual servo system: the visual servo controller dircctly outputs the

control cornmand to the joint actuator (as shown in Figure 2.10 and Figure 2.1 1).

Considering the description methods and decision methods categorized above, a

comparasion of the four categories is given in Table 2.2.

Table 2.2 Cornparison of different visual control methods

1 ControlType 1 Coordinate System 1 Components of Controller I

The choice of the methods of visual servo control depends on many factors, including the

task description. If a task (e.g., a trajectory tracking) could be described in 2D projective

space without ambiguities, then an image-based system shouid be preferred. Integrating

the vision system into the controller can therefore potentially eliminate the need for joint

sensors, as the location of the end effectm is no longer detennined by applying the

forward kinematic transformation to the joint sensor-derived displacements [106].

PBLM (Fig 2. 8)

IBLM (Fig 2.9)

PBVS (Fig 2.10)

IBVS (Fig 2.1 1)

In industrial applications, look-and-move control system is used more often in robot

controllers compared to visual servo system. The Mons are 1251:

Lower sampling rate of vision system is difncult to provide direct control of the robot

end-effector with cornplex, non-linear dynamics chanicterîstics. Using interna1 feedback

3D

3D

3D

2D

visual controller +joint controller

visual controller +joint controller

visuai controller onIy

visuai controller oniy

can inierndly srabilize the herob.

Simpler construction of robot control system is achieved by look-and-rnove control

system because many robots aIready have an interface for accepting velocity or

incremental position commands.

Figure 2.10 Position-based visual servo (PBVS) model.

ri Camem

J

+

4

Featu re Extraction

Position Estimation

Figure 2.1 1-Image-based visual servo (IBVS) model.

'

Robot Controiier

Cartesian control Law

4

Carnera b

.

1

+

r

Robot ' Image space controi Law

Feature Featu re Location Extraction

Successful robot vision systems are al1 based on the flexible integration of hardware

(such as camera placement, marked object, and illumination) and software (such as

feature selection. task description, and image processing). Some of the typical application

exainples will be discussed in this section.

In 1973 Shirai and houe [84] designed a vision system for a robot to grasp a square

prism and place it in a box. This was one of the earliest visual feedback applications. A

camera was fixed and edge extraction and Iine fitting were used to determine the position

and orientation of the box.

In 1988, Kabuka et al. [47] used Fourier transformation to center a target in the image

plane. An IBM-PCIXT cornputer performed the computation, and the targeting

pmessing took about 30 seconds.

Tracking and grasping a moving target is an important application for robot vision.

Andersson f51 intmduced a d-thne vision-assisted robot ping-pong player. He

developed a 60 Hz, low-latency vision system. Special chips were designed to calculate

the centroid of the ping-pong bal1 accurately in the image plane. The aemdynamics of the

bal1 was used to generate accurate predictions of the position of the ball. Zhang et al.

[107] intmduced a visual servo system for a robot with eye-in-hand placement, which

could pick up items h m a fast moving conveyer. Prediction algorithm was applied to

increase the response of the vision system. M e n et al. [3] used eye-to-hand stereo vision

system to mck a fast mwing t;ngn (2fOmmls). Hoastnmgi f381 presented e viskm

system with a fixed overhead camera for a PUMA 600 to p p a moving target.

Skofteland and Hirzinger [86] discussed capturing of a free-flying polyhedron in space

with a vision-guided robot. Lin et al. [57] introduced an approach for catching moving

targets. The fint step was couse positioning of the robot to approach the target; the

second step was "fine-tuning" to match the robot acceleration and velocity with the

targe t.

Vision system could be more efficient when combined with other sensors. Feddema et al.

[24] and Feddema and Mitchell [25] presented a feature-based trajectory genenitor, which

provided smooth motion between the actual and desired image features. Image features

could be taught off-line either through teach-by-showing techniques or through CAD

simulation. A PUMA robot was used to track a gasket on a circular parts feeder. The robot

was not allowed to travel near a singularity configuration. The authors pointed out that an

important characteristics of the feature-based trajectory generator was that features

provided by joint sensors with different sampling times could also be used, thus ailowing

a unified integration of sensors

Harrell et al. [32] used visual servoing to control a two DOF fruit-picking robot as the

robot reached the fruit. Ultrasonic sensors were used to detennine the distance between

the Fniit and the gripper.

Rives et al. [73] and Castano and Hutchinson [Il] used task function method to position a

target with four circles as the feaairrs Hashimatn et aL €34 apphed position-based and

image-based visual servo systems to a PUMA 560, which tracked a target moving on a

circle at 30mm/s. The results of two visuai-servo rneâhods were compared.

Castano and Hutchinson [ I l ] introduced a visual cornpliance method with the task-level

specification of manipulator goals. The motion estimation was a kind of hybrid

visiodposition measurement structure. The two degrees of freedom, which were paralle1

to the coordinate axes of the image plane of a camera, were decided by image-based

visual measurement, the remaining degree of freedom, which was perpendicular to the

camera image plane, was obtained by the robot joint encoder. The first two rows of the

Jacobian matrix dated to differential changes in the robot's motion to the differential

changes in the image observed by a camera. The third row of the Jacobian ma& related

to differential changes of the robot's motion to differential changes of the perpendicular

distance between the robot end effector and the camera image plane. A PUMA 560 robot

was used to perform various tasks. In order to alleviate the difference in time delay for

the vision processing and joint sensor processing, an LED was attached to the robot end

effector to increase the speed to recognizing the target.

Tendick et al. [94] discussed the possible applications of vision system for tele-robots.

The task was considered to be specified dynamicaiIy by human operators according to the

selected features from the image.

Hager [29] applied the projective geometry to the image-based visual control systern with

eye-ta-hand came= arrangement. Projective geo- was used for &e calibrahm of

robot and camera parameters and for defining the setpoints for visual control that were

independent of the viewing location. The author introduced the mathematical description

and related Jacobin maaix for three typical robot tasks: point-to-point, point-to-line (a

motion that a point moves to any points on a line) and line-to-line (a motion that a line

moves to be digned with another line). As an implemented exarnple, a Zebra Zero robot

a m was used to insert a floppy disk into the floppy disk driver of a compter.

Jang et al. [45] presented the concept of Augmented Image Space (AIS) and Transformed

Feature Space (TFS). AIS is a sensor based space generated from the acquired image and

the depth information extracted through some motion stereo' algorithm. Based on AIS,

TFS is constructed with a finite number of defined features. In the expriment, eye-in-

hand system was used and the desired features in AIS were obtained through "teach by

showing" method. This method avoids an exorbitant amount of on-line computation to

determine the position of objects from their images. The huge amount of computation

could degrade the stability of the system.

Al1 of the above mentioned works are "passive vision" applications. As the application of

active vision, Agin [2] proposed to use hand-held light stripes in order to simplify image

processing procedure. The specified light stripes supplied extra information for image

processing procedure. Venkatesan and Archibald 1971 described a system with two hand-

A major difficuity in the anaiysis of p i c m of a 3D scene is the lack of dWct depth infortnation in a single picture. A progression of views of a scene is available if the observer is moving with respect to the environment, hence the use of the tenn ''motion stereo" [70] [ 1031.

held laser scamiers fur d - t h n e mtd of a 3 EKE robot.

Another control method with active vision is mobile camera head that was introduced by

Gilbert et al. [27]. The purpose of this control was to adjust the pose of the carnera in

order to track the movement of a target. The control system tried to keep the m e t

always at a fixed place in image plane.

2.7 Fault Tolerant Robotic System

In 1942, Asimov [6] adciressed the robot safety issue with his famous "kee laws of

robotics". Nowadays robots are being more fkquently utilized in inaccessible or

hazardous environments [66]. Fault tolerance is highly desirable in these environments

because the results of system failures may be very dangerous and unrecoverable.

Researchers have considered the use of redundancy and safety checks at the joint and

actuator Ievels, and aiso in the software and control architecture [31]. Dhillon 1161

analyzed various factors that influence the robot reliability and safety.

Visinsky et al. [99] divided a robot fault tolerant system into t h e layen: a continuous

servo layer, an interface monitor layer, and a discrete supemisor layer. Servo iayer fadt

tolerance requires the controller to control the robot movement when some failure

happens. Interface layer contains the trajectory planner and provides fault detection and

some basic tolerance functions for the system. S u p e ~ s o r Iayer provides a higher-ievel

fault tolerance and general action cornmands for the robot.

Waiker and &tvdle~o [1@1] us& fad l bees €0 &yze robot fai~u~e6 & d y on the -

sensing failun. Redundant kinematic and actuators design is also an extensively

addressed topic [66]. Huang and Notash investigated the failure modes, causes and

effects of a parallel manipulator [40] and studied the failure situation of a parallel robot

DEXTER (made by MD Robotics Ltd.) by using the failure mode and effect analysis

(FMEA) and fault tree analysis (FTA) [41].

2.8 Summary

Generally, the whole application procedure of robot vision can be treated as a series of

space u?uisformations, which cm be divided into three steps:

Step 1. Transformation from 3D world space to 2D image space by projective mode1

Step 2. Transformation from 2D image space to feature space by image pmcessing

Step 3. Tmsformation from image feature space to 2D space or 3D world space by

reconstmc tion mode1.

The dated technologies of vision application include projection geometry, image

processing technologies, control technologies and suitable decision making methods.

It should be noted that despite the long history of using vision to guide robots, the

applications of this technology remain limite& The robot vision models are far fhm

perfect cornpared with human vision. However many projection models and image

processing methods based on the mathematical development have been successNly used

to simulate the performances of the cameras and obtain useful information from the

images taken from the observed objects.

There are two types of robot vision appicatians based on the purpose of the vision

system. Vision system can be used to find "what it is" in the scenery, as well as "where it

is". Based on the observation of "where it is", visual servo control technology has been

developed in the research labs.

Robot vision technology provides a combination of flexibility, adaptability, precision,

noncontact rneasurement and object recognition for robotics. This technology is

involved with diverse technologies. the development of theories and equipment, this

technology will have a bright future.

Vision-Assisted Robot Kinematic Error Estimation

3.1 Introduction

Robot kinematic error estimation is the cornparison between the required motion

description and the actuai motion description. Robot kinematic error could be produceci

by a sensor or by a part of the mechanism. such as faulty calibration errors or

transmission backlash, respectively. Vision-assisted robot kinematic e m r estimation

realizes the kinematic error estimation using the information obtained from the vision

system. The usual robot motion description is a bbtrajectory" used to describe the sequence

of positions, or velocities that the user of the robot has specified in different reference

frarnes. Task description c m be very different in differemt reference frames, and hence

influences the integration of the vision system and the measurement resuits of the vision

system. Based on the methods of the task description, d i f fe~nt kinematic error estimation

methods will be intmduced in this chapter.

The vision system c m transfomi the observation of the robot motion into the image form

Therefore, robot motion description c m also be defined in the image space, and a series

of methods have been developed to retneve the information of 3D space h m 2D image

space (refer to Section 7-5). These methods simulate the image formation procedure and

the transformation among the different spaces.

According to the robot taslc quiremen& ÿisio~~asktedrnbot îchmtic. ermr e s h a t h

can be completed in robot space, camera space or image space. Usually the estimation

results are represented in or tmsforrned into 3D robot space or camera space. However

in some cases, the transformation is not necessary if the required task motion and the

acnial motion cm be both described in the image space. In order to simpli@ the

kinematic error estimation and lead to a successN vision application, the correct

selection of the reference frame to describe the robot task is quite important. SpeciaI

robot structure can also simplify the procedure of the vision system to obtain kinematic

infornation of the robot.

Vision-assisted kinematic error estimation cm be applied to any applications that need

the motion information of the robot. There an two kinds of basic applications:

Vision is directly incorporated into the task planning and contml of robots, such

as a visual servo control system. A case study will be introduced in Chapters 4 and 5.

Vision is used for the evaluation of the robot performance, such as supervishg the

robot motion to make some decisions related to safcty or integrating the related

machinery harmoniously with the robots. A typicai application is to improve the fadt

tolerance of a robot that will be discussed in Chapter 6.

3.2 Motion Description in Different Reference Frames

One of the ultimate goals of robot vision is to mate autonomous robots that will accept a

high-level description of tasks and wiU execute the motion without human intervention.

Therefore, a proper description of the task in a suitable mathematicd fonn is an

important issue for mbot vision appheatioii. Fm differen~ motion desenptions, differenk

parameters are required to decide the mathematical fom. An observer, such as a vision

system, can obtain those corresponding parameten.

As shown in Figure 3.1, a robot task could be represented in different reference fiames.

These reference frames are usually but not necessarily are in the form of coordinate

frames. The vision system could not only perform direct observation of the image

features, but also perform the role of the recognition observer and joint observer (as

shown in Figure 3.1) through suitable mode1 transformation. Image formation procedure

c m be considered as the result of three lcwls of space transformations, which will be

discussed in the following sections.

I Task Definition

Recognition m I 4 t - Robot Joint

Frame Observer -

Figure 3.1 Task description and observation with vision system.

The observer couid consist of various. which could ped(i17n the 0 b a a t i ~ task

and provide the robot system with the necessary information to carry out the desired

tasks. A vision measuring system provides more choices for robot motion description.

However, sometimes in a complex vision system, traditional joint sensors are still needed

to observe some specific variables of robot motion.

3.2.1 Transformation from Task Space to Robot Space

The task definition for a robot cm be the symbolic representations of the objects and

relations between the robot and the objects, as well as attributes describing the robot

configuration in the world coordinate frarne. The task description could be in the fom of

a simple sentence such as "let the end effector grip a part on the conveyef'. It could aiso

be in the fom of the complicated logic representations, or even some gmphics.

A general robot system cannot ded with diverse representations of the tasks. A typicd

task representation is a tmjectory described in robot coordinate h e s , such as the

motion of specific points on the robots, which are represented in the robot base

coordinate frame. Base and end effector fiames are used very of'ien in robot vision

applications, because most of the time a robot vision system should supervise the motion

of the end effector andior some objects located in the world space. Usually the world

space will be assumed to be the same as the base frame, and then the motion of the end

effector and the objects in the environment can be described in the same robot base

frame.

3.2.2 Transformation from Robot Space to Camera Space

The camera is a necessary cornponent of a vision system. One of the important tasks of

the vision system is to find the relative positions between the objects in the environment'

with respect to the camera. Therefore, if the task could be defined in the camera space, it

would simplify the transformation procedure.

Generally, the position information in the camera frame is required to be msformed into

the robot space. Then the position of the camera with respect to the world f m e should

be detemiined pnor to identibng positions of objects in the camera frame. This

procedure is called carnera calibration. Cmera calibration is the process of determining

and relating the 3D position and orientation of the camera with respect to a certain world

coordinate frame. Carnera calibration is dso referred to the mapping between the 3D

coordinates and the 2D image coordinates.

Three methods for camera placement were introduced in Chapter 2, where each method

has a different effect on the results measured by vision. The camera calibration procedure

for an eye-in-hand system is to detennine the camera position in the moveabie end

effector frarne. Where as the camera caiibration procedure for an eye-to-hand system is to

determine the camera position in the fixed robot base bme (world fiame).

' Environment refers to al1 the space arotmd the robot. The obsmable space for the camera Q -eV. When the camera rnoves, the scenery is variai. The scewry is also influenceci by the change of the physical parameters of camera, such as focal leagh, fieid depth, etc. T ' h a art many objecCs in the environment or sceneq, but ody some of them are necessary to be processed by the vision system, which are calleci targeted objects or simply targek. If ody som specid points on the targeted object art processed by the vision system, those points are caiied targeted pohîs or f'îure points.

For an eye-to-hmct system, the festxlts of the transfonnation of the points m e fu&

object in the environment are more precise than those of the transformation of the points

on the robot, because the relative position between the world frarne and the camera m e

are fixed, but the relative positions between robot joints and the camera are varied.

On the contrary, for an eye-in-hand system, the results of the transformation of the points

on the robot are more precise than those of the points on an object in the environment.

This is because the relative position between the camera and its attachment point on the

robot is fixed, while the relative positions between the objects in the environment and the

camera frame are varied. If the targeted objects in the environment move as weU, then the

result of transformation from the objects in environment to the carnera space could

produce additional errors due to the errors in the camera calibration.

For the active camera head, the situation is similar to the eye-to-hand system, but an

additional problern is that the calibration should be performed dynamically because the

reference point of the camera frame is varied as the camera head moves. Even if some

fixed reference points are assigneci in the scenery, it is sti1.l uery difncult to get d - t i m e

precise measurement results of the points on the robots and the objects in the

environment due to the motion e m r of camera,

3.23 Transformation from Camera Space to Image Space

In a vision system, images are the major sources of information that could describe the

motion of the robot and the objects in the environment. nie transformation h m cameni

space tcr image space is an essentid psocedue. This transfonnatiios is also called --

projective model or camera model. This transformation has been referred to as the

correspondent between 3D features and 2D features by some authors [43]. The image

coordinate frame is set up on the image plane of the camem. Because-an image is formed

by displaying a matrix of pixels in the proper order, the coordinates in image fnime are

digital coordinates.

Several carnera models will be introduced in the following sections. There is no camera

model with absolute precision, and the inverse transfomation to obtain the positional

information in 3D space is often non-linear with multiple solutions.

3.3 Projection Models and Reconstruction

The projection model can develop the corresponding image of the observed object if the

the-dimensional position of the observed object is given. The reconstruction mode1 can

determine the three-dimensional position of the observed object if an image of the

observed object and the relative positions of ail the comsponding image fanires are

given. If error estimation in 3D space is needed, a reconstruction procedure is an essential

process. Reconstruction is the process of detexmining and relating the 3D positions and

orientations of the observed objects from their image spaces. TheoreticaUy,

reconstruction is the inverse process of projection (refer to Figure 2.8). The

reconstruction model can be developed h m the corresponding projection model. The

problem is that the mathematical reconstruction results are nomally not unique, and

more conditions should be searched in order to obtain practical reconstruction results.

Three commonly used projection models are: cenaal perspective pmjection madel, scded . . . . -. . . .-

orthographie projection mode1 and affine projection modei. They are d i simplified from a

general projection model, which cm be represented in homogenous coordinates as [5 11:

where ' p is the 3 x 1 position vector defined in 2D image space, " p is the 4 x 1 position

vector defined in the 3D task space, T is the 3 x 4 transformation matrix projecting the

3D reference frame ont0 the image m e .

view point

, Y . image plane 1 --

Figure 3.2 Transformation h m camera h e to image fiame.

Figure 3.2 shows a coordinate frame denoted by X, Y, 2, for a single camera system. The

observed object is a point in 3D camera space: 'p = Lx, y, z]', and its coordinates are

expressed with respect to the camera coordinate frime, Fc. The point 'p is projeaed onto

the image plane Fi with coordinates 'p = [u, vlT. d is the perpendicular distance between

the observed point and the viewpoint, f is the focal length. The three different projection

models wiU be discussed in the following sections.

3.3.1 Central Perspective Projection

Central perspective projection is the most common model in vision research. This model

is suitable when the center of projection (focal center) is at the origin of the camera frame

and the axes of a camera coordinate frame are aligned with the axes of the world frame.

With necessary error compensation, this model could produce accurate measurement

results.

When the projective geometry of the carnera is modeled by cenaal perspective

projection, the carnera model is represented as:

This transformation correctly models a perfect pin-hole imaging process. Unfortunately,

the resulting solution is not linear due to the dependency on the distance 2, which is the

denominator in equation (3.2). The reconstruction is not unique because the 3D

coordinates [x, y, z]* are decided by two formula in equation (3.2). Physically, a point in

an image cm be the projection result of aoy point on the same ray in 3D space. In order

to recover the depth information, z coordinate, multiple views or knowledge of the

geometric dationships between severai feature points on the target are needed. In

practical applications, this mode1 is sensitive to the subtle changes of camera positions

and to the background noise [95].

In order to calculate the depth (z coordinate of a point on target), it is assurnecl that two

fixect cameras are used and two pictiaes fimm Che same stew rue taken. Tks obsewed

point is O p = [x, y, zjT in the robot base coordinate hune. The transfomation rnatrix h m

the left and right image coordinate frames to the base coordinate f m e are TLo and Tc, ,

respectively. It should be noted that Tl,, and T,, are imown initially from camera

caiibration. Then the coordinates of 'p in the left and right carnera frames are

'P, =[xi Y, z J =T,,(Op)

'P, = Lxr Y, zrr = TrJOp)

The left image of the point 'p is 'pi = Cui, vil ' and its right image is 'p = [ u , WJ '. The

focal Iengths of the left and right cameras ~IX assumed to be equal: fi = f, = f. Using

equation (3.2), equations (3.3) and (3.4) can be cxpressed as:

Equation (3.6) is a function of the coordinates of point Op defined in the robot base fnime.

There are three unknown variables, which are the coordinates of Op, and four equations

(according to the matrices in equations (3.5) and (3.6)), so it is possible to reconstruct the

3D space information. The other aspect of this problem is the calcdation of the rna&ices

TLo and Tc,. Because both TL, and Tc, are 3x3 matrices, there are 18 entities to be

calibrated. When the intemal calibration of camera is also considered, there will be more

unlaiown parameters [95].

In or&r to recouer the deph information (the distance b e e n the abject and the image

plane), human eyes use a 5 a t variety of vision-based depth cues, which include texture,

size perspective (change of size with distance), binocular perspective (inward tuming

eyes muscle feedback), motion parallax, and relative upward location in the visual field,

occlusion effects. outline continuity and surface shading variations [46]. In order to

simulate the function of human vision, some active vision techniques have been

developed [46], including using striped iighting and grid coding with special light effects;

identification of relative range from occlusion cues; calculation of depth from texture

gradient; calculation of the range from focusing; calculation of surface orientation from

image brightness; calculation of the range from stcreo disparity; and calculation of the

time-of-flight range.

3.3.2 Scaled Orthographie Projection

If ail the points could be considered in the same plane with the same average distance d

to the image plane, the previous transformation, equation (3.2), could be approximated by

the scaled orthographie projection:

where s is a fixed scale factor, s = aY; and d is the average depth of the centroid of the

observed point set which are located by the cament Obviously the transformation is

changed to be a linear procedure. This transformation is only valid when the relative

depth of the points in the scene is small enough compared to the distance from the camera

to the scene so that the depth can be assumed to be constant. For a large object close to

the h e m , ih is approximation Wones invaüd. Itemlivelysded odopphic

projections can be used to approximate the real perspective for al1 cases [15], if some

local areas of the objects cari be assumed to satisfy the conditions for the scaled

orthographic projection.

The inverse procedure of the scaled orthogonal projection is easily developed:

An image may be locally well approximated by the scaled orthographic projection (also

called weak perspective), but the centrai perspective projection (full perspective) is

required globally to accurately represent the entire scene. The total depth variation across

the scene can be large, but each individual object may satisQ the weak perspective

approximation. Therefore, it is possible to divide the scene into different areps each with

a different depth for its centroid.

It should be noted that when s = 1, then the model is reduced into a standard orthogonal

projection (as shown in Figure 2.3).

33.3 Affme Projection

An affine projection model is simplifieci by letting t31, t32 and t33 e q d to O, in the

transformation maaùr T given in equation (3.1). The affine projection physically does not

correspond to any specific projection situation. The advantage of the affine projection is

55

that it is a good local epproximarion of the peapective p~ojection that accaunts for: both -A

the extemai geometry of the camera, which relates to the position and orientation of the

camera, and the intemal geometq of the lens, which relates to the physical structure of

the camera. The linearity of the mode1 simplifies the camera calibration greatly. Based on

the affine projective geometry, some abstract geometry invariant characteristics

originated from the transformation between images of the same scene, taken fmm

different viewpoints, have been developed [109].

A general affine transformation f: Rn + Rm is a transformation that can be written as

' I I - , where A is an m x n ma& in R"",

called the defonnation matrix of f, and b is a m x 1 vector in R" , cailed the translation

vector of €.

If it is assumed that the projective geometry of the camera is modeled by the affine

projection, then the aansformation h m Che coordinates in the carnend space to the

coordinates in the image space can be show as:

where Te is a 2x3 transformation rnatrix and b is a 2x1 vector.

U s d y more cameras are quired when affine projection m~dels are used for the . .

reconstruction procedure. The invariant geomeaical characteristics [log] between the

projection procedures of different carneras are used to simplify the calibration procedure

and feature matching between the corresponding images (raken from the same scene).

Similar to the analysis in the centrai perspective projection, in the affine projection it is

assumed that two fmed cameras are used and each camera takes one picture h m the

same scene. The left image of the point is = lur, vil and the right image of the point is

= [u, vJ T. Equation (3.9) can be rewritten as:

In equation (3.10) Taml and Tarnmr are f i n e transfomation matrices for left and nght

carneras. Considenng (3.3) and (3.4), (3.10) can be represented by

In equation (3.11) there are three unloiown variables, which are the coordinates of Op, and

four equations, so it is possible to reconstmct the 3D space information. Here only 16

parameten (12 entities for the 4x3 matrix G and 4 entities for the 4x1 vector b) need to

be calibrated 1521.

Actuatly, a pctÏcal canera modtt has mare catibratiorr parameters ttiair the nititiff iir

those transformation matrices men tioned in equation (3.1 1). Depending on the required

accuracy, different calibration methods cm be applied. Tsai's paper is a guod reference

for camera calibration [95].

3-34 Digitization Error

In addition to the error from the projection model, the other important enor source is the

digitization. The anaiog coordinates [u, v]' of a point on an image plane cannot be

accessed directly when using a standard video system (e.g., CCD camera, digitizers, and

frame grabbers). OnIy digital coordinates that represent the horizontal and vertical entries

of a digital image array stored in the computer c m be accessed. However digitization

error during quantization procedure cannot be avoided. If âu and Av represent the

horizontal and vertical sampling intervals. [1, 4T denote the digital coordinates of an

image point ( in ternis of the position of a

coordinate of the center of the image plane, then

pixel), and (Io, &lT denote the digital

the following relations will hold:

Digitization is an important factor that influences the vision measurement result. Au and

Av are decided by the resolution of the image and the optical distortion of the camera

dong different directions.

3.4 Position Error Estimation in 0)Herent Forrns

If the robot task can be descnbed in different spaces or frames, then the robot kinematic

error estimation can aiso be performed in different spaces (3D robot space, 3D camera

space or 2D image space). This will provide different foms for the robot kinematic error

estimation.

Camera placement (as discussed in Chapter 2) is an important factor that can influence

the result of kinematic error estimation. It changes the relationship between camera space

and robot space. In this section only two kinds of camera placements (eye-to-hand vision

system and eye-in-hand vision system), which have been used cornmonly in robot vision

applications, will be considered for the analysis of position error estimation.

The position e m r estimation is the basis of the robot kinematic error estimation. As an

example, a point-to-point positionhg task is considered in this section. The position error

estimation of points on an end effector is the basis of the foliowing kinematic e m r

analysis. This task requires a point on the robot with the end effector cwrdinate ep to be

rnoved-to a fixed or moving point s with the help of a vision system.

3.4.1 Generai Form for Position Error Estimation

According to the camera placement, two models of position e m r estimation cm be

given. In the general fom, the measurernent results from the vision sysrem are used

direcdy without consicking. the pmcedure ficm the camera fiame to the mbot framc and - -

the reconstruction procedure from the image space to the camera space.

1. Eye-to-hand system: In this model, the camera is fixed in the environment and is

calibrated with respect to the robot base frame, as s h o w in Figure 2.4. The position emr

is calcuiated in the robot base frame as [30]:

Ew(T "S. t ~ ) = " ~ - o ~ = T o , e ( ' ~ ) - o ~ (3.13)

where To,e is the transformation matrix from the end effector frame (e) to the base fiame

(O). Op and Os are the coordinates of points p and s in the base coordinate frame.

II. Eye-in-hand system: In this model, the camera is mounted on the robot and is

calibrated with respect to the end effector frame, as s h o w in Figure 2.5. The position

error is cdculated in the robot end effector frame as [30]:

"E,(T,,. Os, ' p ) = ' ~ - ~ s = ~ p - T , , ( ~ s ) (3.14)

where Te,. is the transformation matrix h m the robot base h e (O) to the end effector

frarne (e).

Equations (3.13) and (3.14) demonstrate the general formula for position e m r calculation

in the robot base frame and end effector frame, respectively. The following sections will

intmduce the position emr estimation forms integrated with the measurement procedure

of the vision system.

3-42 Diffemt Cases for Posigion EPPOF EGfj.nBfjOn

The values of T, s and p in different reference frames, as discussed in Section 3.4.1,

might be estimated from the vision measurement, which cm be represented by ?, 8 and

fi, respectively. f , S and fi cm be thought of as estimation of the parameters and can

be obtained from the robot calibration, camera calibration and reconstruction procedure.

The calibration and reconstmction models should be completed before estimating the

kinematic enon. Al1 the estimated parameten can be identified either statically (before

beginning the task) or dpamicaily (duing the execution of the task). Depending on the

application environment, some of the parameter estination procedures are not required.

Case 1: Error estimation calculated in the robot base €rame or end effixtor €rame

In this case the vision system helps locate the position of the target point s in the world

space. The point on the end effector, 'p represented in the end effector fnune, is assumed

to be known accurately and does not need to be rneasured (estimated) by vision system.

For an eye-to-hand system, the vision-measured position of point s should be transfemd

into the robot base coordinate hune before estimating the position error. The procedure

cm be represented as:

* 1 C A

Em(To.etTo,c, s, e p ) = ~ o , e ( e ~ ) - f " J c ~ ) (3.15)

Here, thRe estunation procedures are needed to decide the values of the following

quantities: fo,,, shows robot calibration (hm robot end effector h e to robot base

frame); shows camera cdibration (fiom carnera h e to robot base fiame);

shows reconstruction results from images.

For an eye-in-hand system. the vision-measured position of point s should be transfemd

into the end effector coordinate frarne More estimating the position error. The procedure

can be ~presented as:

'E, ('Î'eî;, , 'S. 'p) ='p - feî;, ('5) (3.16)

Here two estimation procedures are needed to decide the values of the following

quantities. , shows camera calibration (from camera Frame to mbot end effector

frame); 'î shows reconstruction results h m images.

Case 2: Error estimation caIcuIated in the carnera frame

In this case the same camera can see both targeted point s and the point p on the robot, so

the position information of s and p cm be recovered h m the image information. The

error estimation can be computed in the camera space, which is s h o w as:

= ~ ~ ( ~ g , Cp) (3.17)

Because both s and p can be recovered from the same reconstruction model, only the

parameters of the reconsmiction model should be estimated ActuaiIy the point p can be

anywhere on the robot as long as both points p and s can be watched at the same time by

the camera when the robot or camera moves.

L meriy applications, enw esgrnation in the robot spaee is mxkd and eould be

calculated from as discussed below.

For an eye-to-hand system, the transformation from the camera frame to the robot base

frame is needed:

O E, (TOac, '6, 'fi) = ('fi) - ('s) = (cfi-c~) = CS E~~ (3.18)

Here two estimation procedures are needed to decide the values of the following

A

quantities. T,, shows camera calibration (from camera Frame to robot base m e ) ;

and 'fi show reconshuction results from images.

For an eye-in-hand system, the transformation h m the camera frame to the robot end

effector frame is needed:

Ci

C EPP (Tu 's, = ?eT,,(Cp) - *e,(C~) = T,,,(c+C~) =TUC~, (3.19)

Here two estimation procedures are needed to decide the values of the following

A

quantities. Te, shows carnera calibration (hm camera frame to robot end effector

frame); '6 and 'fi show reconstruction results from images.

Equations (3.18) and (3.19) show that the position enor in the robot space codd be

calculated From the position e m r in the camera space by suitable coordinate

transformation. But ihis transformation matrix, which represents camera calibration, is

unknown and needs parameter estimation. The emrs inaoduced h m the carneni

caübration procedure will affect the resuits of the kinematic ermr analysis.

Case 3: &€OF estiniatiorr in the image spaee

In this case both the targeted point s and the point p can be seen by a camera. The images

of these two points c m be taken at the same time or separately at different times. There

are some cornmon features of the two points that cm be detemÿned from their individual

images and the features can be described by mathematical form.

If fd and fa stand for the desired and actual image feanire vectors' of the observed points.

respectively. then the error estimation in image feature space is

i E,(f,,fd) = f a - fd (3.20)

Obviousl y, reconstruction from the images, robot calibration and camera calibration are

not needed any more. fd can be found directiy using the projection model (which is

calculated from carnera model andor geometry model), or using a "teach by show"

approach in which the robot is moved to a goal position and the corresponding image is

used to compute a vector of desired image feature parameters. Robot tasks are thus

specified in the image space, not in the world space. fa is found by an image processing

procedure on the images taken fmm the scenery.

It is obvious that camera model calibration is still necessary when fd is calculated using

the camera model. However, in this situation the 3D position of the desired point with the

' In cornputer graphics, any characmistic description of an image or a region in an image is d e d a feature. An ordemi set of features that may be used to describe artas or objccts within an image is d e d feature vector. A point in the space can be describeci using many diffcrent features in the image s p e The feaîure of the point can be simply the coordinates in image plane; the brightness or der, or even s ~ m c absolute mathematical quantity. Aii the selected description parameters of the point can fom an image fea- vector of this point

feature vector fd is ~ ~ m a l l y a geomeeical poing Iwated in space that kas no relationskip - -

with a physical point. This situation simplifies the calculation of fd and increases the

accmcy off' SO the result of image-based solution is less sensitive to the emrs fimm the

camera mode1 cdibration.

Another important point about image-based error estimation is the fact that except for the

case that the desired and actud points are on the same ray with different depth, when the

image error estimation is zero, the kinematic emor in 3D space must aiso be zero.

Table 3.1 Cornparison of vision-assisted kinernatic error estimation methods.

As rnentioned in Case 2, the error estimation in robot space is needed in many

applications. The relationship between the position enor estimation in robot space and

the position e m r estimation in the image feature space will be discussed in Section 3.5.

I no

no I no

Eye-in-Hand System

g 2 . œ

8 ' Pi: 5 . CiI

3 . C)

2 ,

Eye-to-Hand S y stem

0 U a 3 a 41 u 3 g 5 8 a

Recons~ction

Y=

Y-

Robot calibration

no

no

calibration

Y=

no

Reconstntction

YeS

YeS

Robot caiibration

YeS

no

îamera cdibration

YeS

no

Table 3.1 summines the different e h ~ t e & i c s og emr &madon procedures in the

robot space, camera space and image space for the eye-to-hand and eye-in-hand camera

placements. Ln the table, "yes" ("no") means calibration andor reconstruction from

image features are (not) necessary theoretically.

3.5 Feature Jacobian Matrix

3.5.1 Definition

In order to achieve the transformation from the image feanire space to the 3D space, it is

necessary to develop a relationship between the feature vector and the coordinates in a

3D space (robot space or camera space). The feature Jacobian matrix J can be used to

describe this kind of transformation.

The relationship between the error estimation in the 3D robot space and 2D image space

could be written as follows:

'EPP = J .EPP (3.2 1)

where shows the emr estimation in the 3D robot space or 3D canent qmce.

Equation (3.21) shows the relationship between the m r in 3D robot space (or camera

space) and the image movement.

If 3 point p is rigidly attached to the end effktor with coordinates of [x, y, zlT defined

with respect to the robot base frame, when the robot moves in the 3D workspace, then the

verocity screw of point p in the base f m e is r=B, i; i. q, q, qfl The ftatmc

Jacobian matrix J is defined by

f = ~ , , i

where the kxl vecior f shows the rates of change of the image features (parameters), k is

the dimension of the feature vector f and rn is the dimension of the task space (it might be

robot space or camera space depending on the space in which the task is described). The

k m Jacobian matrix J is calculated as

Jacobian matrix J is determined by the image feanire types and projection model.

Jacobian matrix performs the mathematicai description of the procedure from the robot

space to the image feature space.

If the pose r of the point in the 3D space is required to be caiculated from the f e a m

vector f, then the calculation of the inverse of Jacobian matrix is necessary. In practicai

applications, there are three cases of the Jacobian matrix cdcuIation:

If the dimension of feature vector is equal to the dimension of robot task space,

Le., k = m; and J is non-singular, then the inverse of J exists and is unique and

'E, = J-' 'E, . This means that the number of featwes is equal to the number of

degrees of &dom in the image-based system [24].

If the dimension of featine vector is larger than the dimension of robot task space,

i.e., k > m, then rank of J is required to be equal to rn in order to obtain a solution for

J+ = (J'J)-' JT . In a practical robot vision system, it is very cornmon to use more

features than the degrees of freedom of task space [35] [Ml .

If the dimension of f e a m vector is less than the dimension of robot task space,

i.e., k c m, this means that the number of features is not suffkient to uniquely determine

the motion of the observed object. 'E,, =J' 'E,, + (1 - JiJ)h may be used to obtain

the solution, where J+ is cdled right pseudoinverse of J and is calculated as

J+ = (J J T ) - l , and h is a vector with m dimension. For a successhil robot vision

application the situations with insuficient feanires should be avoided, except for some

special conditions in which extra consaaints fiom others sensors can be obtained For

example, traditional position sensor supplies the information of one dimension (e.g.,

length or angle) whereas vision could provide the remaining information.

Features can be desaibed in many different formats, and it is not easy to m a t various

features in a unified fashion. In order to obtain a unique inverse for the Jacobian matrix, a

Jacobian matrix should be square and nonsingular, or it should be of full rank for the

pseudo-inverse [24]. If the features are not suffkient, then extra Limitations or extra

information should be imposed on the calculations in order to select a feasible solution

from multiple mathematical solutions.

The detection and robust tracking of the feanires in images is problematic in an

unstructured environment where lighting, background, and a variety of other factors are

not controllable. The most important problem in an unstructured environment is how to

68

go from a gevmeke spificetion of a iask te a vis& specificatb of rhe ta&. This

transformation must include the determination of visual featurcs and invariant projection

properties that c m be used to implement the task, as well as methods to detect and track

these features when presented with images.

3.5.2 Jacobian Matrix Calculation Based on Different Projection Modeis

If p = [x, y, z ] is a point attached to the end effector defined in the robot base frame,

and the angular velocity of the end effector in the robot base frame is o = (a, o,

and the translation velocity of point p is v = [v, v, v , p , the time derivatives of the

coordinates of p, expressed in the base coordinate Frame, are given by

which can be written as

p=oxp+v=-sk (p )o+v

sk(p) is the skew-symmetnc matrix of p = [x, y, z]', which can be expressed as

However i = [vT coTIT, so equation (3.25) cm be written in matrix fom as

In the fokwing sections, the sp&& form of Jimbian J wilL be davefaped for different

projection models. Here the projection models establish the relations between the

coordinates in the 3D space and the coordinates in the 2.D images.

Central perspective model

Equation (3.2) represents the mathematicai form of the central perspective rnodel.

Substituting equation (3.24) in the time derivative of equation (3.2), the following

equation can be obtained:

Then the Jacobian matrix J will be:

Scaled orthographic mode1

Equation (3.7) represents the mathematical form of the scaled orthographic model.

Substituting equation (3.24) in the time derivative of equation (3.7), equation (3.29) can

be obtained

[r] = s[;] = [ sv' + sdru, - va, svr + u q - sdq 1

Then the Jacobian ma& J will be:

Affine model

Equation (3.9) represents the mathematical form of affine model. Substituting equation

(3.24) in the time denvative of equation (3.9), equation (3.3 1) can be obtained

The Jacobian matrix J will be:

J = TarWA(p) (3.32)

Here A should be calculated by camera caiibration based on the affine projection model.

This calibration cm be perforrned statically or dynarnically.

3.6 Summary

Vision measuring procedures make it possible to perforrn the kinematic error estimation

in the robot, camera and image spaces. This provides more flexibility for the position

error estimation, which is the difference between the acttxal and desired robot ûajectories.

Position error estimation is the basis of vision-assisted robot kinematic error estimation.

Position enor estimation can be performed in different spaces and with diffncnt camera

arrangements. The corresponding theones were demonstrated in this chapter. The

characteristics of different methods have been compared in Table 3.1.

Even thou& the image-based position emF estimation has same advantages, in m a q

applications the position error information in 3D space is still wanted, thus a method that

cm reconstnict the 3D position error information from the image feanire space is

required. In this chapter, feature Jacobian matrix was demonsûated for the situation when

the features are image coordinates of the observed points.

The vision-based robot kinematic emor estimation provides more fîexibility to rneasure

joint position errors. The possible applications including robot visual servo control and

robot fault tolerance system will be presented in the following chapters.

Chapter 4

A Direct Visual Servo Control Method for a Three-DOF

SCARA Type Manipulator

4.1 Introduction

Vision-assisted robot kinematic error estimation can be applied to many different robot

applications. One of the possible applications is to use it as an e m r function for a mbot

seao control system. The most influential factors on the design of a visual servo control

system are the specific robot tasks and the observable features on the robot. These factors

not only influence the method of kinematic error estimation but also dictate the choice of

other methodologies (such as the carnera placement, projection model, image processing

method and control strategy), which are also necessary to be taken into account for the

design of a visual servo control system. The correct analysis of specific application

objective is the key point of a successfid robot vision application.

In order to investigate the feasibiiity and the possible application of vision-assisted

kinematic error estimation, a direct visual servo control mode1 for a SCARA type

manipulator will be simulated in Chapter 5. In this chapter, the corresponding methods

will be introduced.

In many visual servo robot control systems, the vision system recovers the position

information of the robot end efiector. However, in the cases discussed in this chapter, the

objective of the vision sysiem is to measore diRctfy the position of each joht of the - ---

robot. The joint position emor will be calculated in image feature space rather than in 3D

robot space. On the other hand, the desired joint trajectories will be transfonned into the

image feature space by ideal mathematical projection mode1 or off-line teaching

(recording the images of joint trajectory in the teaching mode), so the error signal cm be

calculated in image feature space and be directly sent to the joint contruller.

According to the special structure of SCARA type robots, cameras wiIl be instaiied so

that a scaled orthogonal projection mode1 can be applied. A scaled orthogonal projection

model cm significantly reduce the computation load for image processing system, but it

requins the distance between image plane and the object be fixed or the variation of the

distance be ignored while the robot moves.

The simulated robot model is a simplified three d e p e of freedorn @OF) SCARA

manipulator of which only t h e joints (two revolute joints and one prismatic joint) are

studied, while the third revolute joint is fixed. The kinematic and dynamic models

devetoped by Schilling f78] tue presented iuid discussed in this chapter. The mgular

displacements of the revolute joints are measured by one camera and the other camera

measures the linear displacement of the prismatic joint.

The foilowing sections will introduce the mathematical description of the models, which

inciude kinematic model, dynamic model, projection model and control model. The

corresponding theories wiU also be intmduced

4.2 Kinematic Analysis of Exarnple Manipulator

SCARA (Selectively Compliant Articulated Robot Arm) robots have two rotary joints

(shoulder and elbow) that move the f i t two links in a horizontal plane, usually one

additional nvolute joint for the wrist rotation, and one ünear vertical joint as the 1s t

joint. SCARA robots were invented in Japan in the early 1960's and the design has long

existed in the industry domain. Figure 4.1 shows a four-DOF SCARA robot of Adept Inc.

SCARA robots are ideal for a variety of applications, which require fast, repeatable and

articulated point-to-point movements such as palletizing and de-palletking, loading and

unloadinp. and assembly. The rigidity of the robot frame is quite hi& in the vertical axis

and the combination of the rotational motions assures a high cornpliance in the horizontal

plane [IOSI.

In order to simplify the simulation, the wrist rotation will be fixed so that the robot model

is simplified as a the-DOF SCARA type manipulator with two revolute joints (shoulder

and elbow) and a prismatic joint with a vertical motion. Figure 4.2 shows the

configuration of the sirnplified three-DOF SCARA robot m&l. Foward (inverse)

kinematics are terms to describe mapping from the intemal (external) space to the

extemal (intemal) space of a mechanical system 1541. A kinematic model is very

important for simulating the kinematic relationship among the joints and the end effector.

The foUowing sections will present the fonvard and inverse kinematics of the example

manipulator based on the SCARA robot model inaoduced by Schilling[78].

Forward kinematics refers to the direction of the transformation, namely from joint

variables or coordinates to endpoint variables or coordinates [4]. If the desired position,

velocity and acceleration vector of end effector are xd, X, and x, , respectively, and the

desired position, velocity and acceleration vector of joints an q,, q, and q, , then the

forw ard kinematics anal ysis can be represen ted as equation (4.1).

where the (i, 3 e n 0 of matrix J is Jy = afr / aq,, , md hinction f is a nonlinear

transformation from the joint variables to the endpoint position and orientation, and can

be represented by the kinematic parameters of the robot. It is clear that a unique solution

to the fonvard kinematic problem cm be obtained for arbitrary joint positions and any

number of joints [4].

Denavit and Hartenberg (D-H) proposed a systematic notation for assigning right-handed

orthonomal coordinate frames, one attached to each Luik in an open kinematic chah of

links. Once these link-attached coordinate hunes are assigneci, the transfomations

between adjacent coordinate h e s can be represented by a single standard 4x4

homogenous coordinate transfomation ma&. The D-H parameters of the SCARA type

manipulator of Figure 4.2 used in this chapter are listed in Table 4.1, in which ai, a*, a:,

and di, dz d3 represent the length of the links and joint offset. qi, q2 and q3 represent the

joint variables.

Figure 4.1 : SCARA robot (Resource: Adept Inc.).

Figure 4.2 Example of three-DOF SCARA type rnanipulator.

77

Table 4.1 D-H parameters of a drra-DOF SCARA robot. -

According to the D-H parameters, the transformation rnatrix between the -es of joints

i and j can be caiculated as follows. Frame O represents the robot base coordinate frame.

The ongin of the frame O is O, = [O,,, Oo,,Oo,]T = [O,O, O I T . Frame 1 represents the

coordinate frame attached to the shoulder. Frame 2 represents the coordinate fnune

attached to the elbow. Frame 3 represent the coordinate frame attached to the prismatic

joint with a linear vertical motion. T{ shows the transformation From frame i to j. The

notations used in the matrices are: Si = sin qi and Ci = cos q, (i=l, 2); SI-, = sin(q, - q,)

The transformation rnatrix from frame O to frame 1, is:

Then [O,, , O,, , O,JT = [alCl, a,S,, dl]' represents the position of origin 4 of kame 1.

C, -S, O a&

The transformation maaix h m frame 1 to hune 2 is T: =

transformation matrix fiom frame O to h e 2 is:

Then [O,,O,y,O,z]T =[a,C, +a&. alS, +a,S ,-,, dl]' represents the position of the

origin 0, of frame 2.

The transformation rnatrix from frarne 2 to fiame 3 is Ti =

transformation matrix from frame O to frame 3 is:

Then [O,, , O,, , O,, 1' = [a, C, + a,C ,-,, a,S, + a,S ,-,. d, - q, lT represents the position of

the origin O, of frame 3.

O 1 0 0

0 0 1 q 3

The locations of origins of the reference hame of each joint are very important The

objective of the visual servo control system is to control the movement of these origins to

follow up the desired trajectories. These origin points are also the observed targets of the

vision system. In the simulation process, a forward kinematics algorithm will be used to

calculate the real trajectory of the end effector.

. The

Inverse kinematic transformation is required to convert the planned trajectory of the

points on the end effector to joint variables [4]. This is a more difficult problem than

fonvard kinematics. The complexity of this problem arises from the nonlinearity of the

transformation equations. If a real robot is under constrained, for a given goal position,

there rnight be more than one solution. Generally, the inverse kinematics can be

formulated as [4]:

Here f-' is a nonlinear function, which is a one-to-many mapping, and is difficult to

calculate in some situations.

The following paragraphs will discuss the inverse transformation pmcedure for the th=-

DOF SCARA type manipulator. This procedure is based on the foxward kinematic

transformation matrix. In this project, the coordinates of the origin of the third joint are

known and the first and s m d joint Vanables are cornputeci.

The coordinates of the origins of the second and third joints have the following

relationship: 0, = O,, , O,, = O,,. The coordinates of the origin of the second joint are:

Thus the cosine of the second joint angle is determined in terms of the known constant

panmeten. Then the second joint angle cm be recovered as follow:

2 7 oh2 +O,, -a; -4 q 2 = 2 cos-'

*%a,

It should be noted that there are two choices for the second joint angle. In the example

used in ihis thesis, the positive result will be chosen. After calculating the second joint

angie, the fint joint angie can be calculated. After resmuiging equations (4.6) - (4.8), the

sine and cosine of the fiat joint angie can be given as:

Thus the f i t joint angle can be calculated as foilow:

4, = arctg ~ a , s 4 , + (ax + a2c2 IO,, Y( (a1 + a2c2 )O, - a2s202, II (4.1 1)

The displacement of the third pnsmatic joint is calculated as:

4i = dx 4 3 , (4.12)

Now the displacement of every joint has been calculated in tnms of the cooidinates of

the end effector. The origins of the reference h e of each joint can also be calculated by

the method introduced in Section 4.2.1.

An actual joint movement of the robot is govemed by dynamics, taking into account the

mass of each link and joint, the joint velocity and acceleration, and the torqudforce

applied to each joint. The dynamic model is a set of differentid equations, which govem

the motion of the joint variables. According to Lagrange-Euler dynamic mode1 [78], a

general dynamic model of a robot can be represented as follows:

7 = M(q)ii + N(q, q) + F(q)

For the SCARA robot used in this project, the dynamic model is adopted h m the mode1

presented by Schilling [78]. The relating variables in equation (4.13) are introduced as

follows. Joint positions, velocities and accelerations of the robot are represented by

torques of the fmt two revolute joints and the force of the third prismatic joint is

represented by the vector r = [r, s, r, l?

In eqaatim (4.13), the first tcnn on the right side is an amfemion teim, which produees

the inertial forces and torques generated by the motion of the links and joints of the robot.

M is the inertia matrix in tems of the mass and geomeaical structure of the links and

joints:

In order to keep the dynamîc mode1 of the example SCARA type rnaniprùator relaImeiy

simple, the ihree links are assumed to be rods of mass ml, m2 and m3. The center of m a s

of every link is located at the center of the link. The payload is not considered in the

simulation. For the example manipulator, the entries of M are calculated as follows [78]:

m, 2 4, = M,, = - ( ( ~ + r n , ) a , o , ~ +(-+m3)at): 3

In equation (4.13), the second term N(q,Q) is associated with the Coriolis forces,

centrifuga1 forces and loading due to gravity in terms of joint movement. For the example

three-DOF manipulator, the form of N and its entries can be calculated as:

N=[NI N2 N ~ T (4.15) - m, where N, = -ala,S2 ( (m, + 2q)4,q2 - (-2- + m,)<) ; N2 = (2- + rnJo,aJ2q; and

In equation (4.13), the last term is a velocity term representing the friction opposing the

motion of the arm. The friction cm be represented as FI = -F, sgn(v(t)) - F,v(t) + n(t) .

The first term represents Coulomb fnction (a force of constant magnitude (Fc) acting in

the direction opposite to motion); the second terni represents viscous friction (a force

propor&ional to joint velocity v(t)); the thirà tenn represents a static fiction term (a force

of constraint equal to joint force Mt)). For the exampie manipdator, the Coulomb friction

83

is considered in the following fom: F = , where bi is the coefficient

of friction for joint i. Function sgno shows that the direction of friction is opposed to the

motion generated by the actuatoa.

This dynamic model will be used for simulating the dynamic response of the robot under

the control signal. For the simulation purpose in this thesis, the inputs of the dynamic

model are torques (forces) and the outputs are joint accelerations. The joint velocities and

positions are calculated from the joint accelenitions using Euler numencal integration

method ( q, = qi-, + qi& ; qi = qi-, + qi& ; where i represents ith step and & is the step

time). The detailed parameters, such as the mass and length of a h ic , for this dynamic

mode1 will be introduced dong with the simulation results in Chapter 5.

4.4 Projection Model

Al1 of the readily available viewing technologies are 2D (monitors, printers, video and

film), even the visual cortex of human eyes is a 20 environment. In o&r to present the

images of the scenes in the simulation, a 2D representation of the 3D scenes must be

created [110]. As discussed in Chapter 3, the projection model can transform the position

information in 3D robot space into 2D image space. The type of projection model should

be selected by considering the characteristics of the observed objects and the specinc task

of the vision system. Because the top planes of the joints of the SCARA type robots are

parailel to each other, it is possible to use one camera with a scded orthogonal projection

mode1 te capture al1 the planes in one image. Sceled orrhogod projection modeC

simplifies the procedure of the image formation.

Figure 4.3 Overhead view of the example SCARA type robot.

4A.1 Projection of Joint Angles

From overhead, the ideal image of the top planes of the example three-DOF SCARA type

robot mode1 is s h o w as in Figure 4.3. The circles on the top planes can represent the

corresponding joint, which cm be considered as a cylinder. The centers of the circles are

assigned as the origins of the joint coordinate -es.

One stationary camera c m be placed overhead with the image plane parailel to the top

planes of the joints in order to obtain the images as given in Figure 4.3. If the motion

dong the direction of optic axis of the camera is rninor and can be ignoreci, then the

distance between the image plane and the top planes c m be assumed fixed whm the

robot moves in the workspace. In this situation, a scaled orthogonal mode1 can be useci.

The transformation between the robot space and the camera space can be shown as:

whcrcsis afixedsdefaaors=~;dis~~of~ctntrOiéof chtobservtdpoint

set which is located by vision system, and f is the focal iength; [u, vlT are the coordinates

of the observed points in the camera referrnce frame, which are the centers of the circles

on the top planes of the joints; [x. y]T are the coordinates of the observed origin points of

the first or the second joint frame.

In a vision system the analyzed image is usually a digital image, which is easily stored

and processed by a computer. If the digitization emor is considered, according to equation

(3.12), equation (4.16) will be:

where [I, are the digital coordinates of a point in the image plane; [la JO]' are the

digital coordinate of the center of the image plane; Au and Av represent the horizontal

and vertical sampling intervals.

According to equation (4.17), the trajectory P, of the origins of the jth (j= 1, 2) joint can

be represented as

and the projection of the trajectory in the camera plane can be shown as:

P, = P,(u,v) = P,(=,sy) (4.19)

where x, y represent the x and y coordinates of Oj ÿ=I,Z). The angles in the camera

plane, which are the results of the projection of the joint angles from the joint Caaesian

space to the carnera space, are shown in Figure 4.4. It is obvious that

86

8, =arctgf y I x ) = a r e ~ g ( s y l ~ ) = a ~ e i g f v / ~ ) = B , 4.20)

is correct if camera's x-y plane is parallel to the x-y plane of joint Cartesian space.

Equation (4.20) indicates the joint angular displacement in joint Carterian space is equal

to the angular displacement measured in the image feature space. But due to the camera

distortion and digitization enor, equation (4.20) is not held strictly. [u, vJT are actually the

analog coordinates of the image in the camera space. Usually digital coordinates [I,

are obtained directly from the cornputer image fiame. According to equation (3.12).

equation (4.20) can be changed into:

B, = arctg(v l u ) = arctg((J - J,)Av /(l - 1,)Au)

Figure 4.4 Joint angles in robot space and camera space under scaled orthogonal

projection.

For a mal camera, sampling intervals Av and Au are not equal and they are in the form

of non-linear hinctions of the coordinates [K. vJT and can be changed with tempera=

and vibration in the environment [95].

In the simulation, bv wiH be assmnect tcr k eqnat ta &r , chm 8, = &gf LU I M) wiB - - -

be held. This assumption will reduce the complications of the simulation. If the distance

between the camera and the observed objeci is fa. enough, the emr produced by this

assumption cm be ignored. Then the joint angle can be measured by

urctg((J - J O ) / ( [ - I o ) ) .

4.4.2 Projextion of Prismatic Joint

The third joint of the three-DOF SCARA type manipulator is a prismatic joint. The

position variable of the ihird joint is a linear displacement. From the side view, the ideai

image of the third joint from a camera is shown in Figure 4.5.

I

Figure 4.5 Side view of the thirci joint h m the camera.

The displacement of the prismatic joint wiU be recorded by another camera, which will

recognize a mark (shown in Figure 4.5) at the end of the joint. The opticd axis of the

camera is perpendicular to the axis of the pnsmatic joint. The mark could be a circular

iine painted around the prismatic cyünder. The vertical position of this Iine is represented

by d3, which is the third joint variable.

The projection procedure can be developed by the iollavving quaiion, whiçh. presents the

relationship between the coordinates of the observed mark in the robot space and image

featwe space:

w = SO,, = (K - KJAw (4.22)

where w are the comsponding image coordinates of the joint displacernent dong the y

axis; K is the digital coordinate of the joint displacernent in the image space; Ko is the

digital image coordinate of the origin of the image frame; Aw is the sampling intemal

dong the moving direction of the prismatic joint in the comsponding image space.

Because the image formation procedure is closely related to the physical characteristics

of the devices in a vision system it is quite difficult to simulate a real image formation

procedure. The previous projection mode1 shows the relationship between digital image

coordinates and the analog Cartesian coordinates of an observed point. Chapter 5 will

simulate the formation procedure of the digital image coordinates, which wiU be used as

the image features of an observed point.

4.5 Contrat Law

The control strategy for robots is an intensive topic in robotics research. There are

volurninous publications on the robot control theory, ranging from PD (Roportional-

Derivative) control to fuzzy connol. A control system causes the joints or the end effector

of a robot to follow a prescribed trajectory and apply the desired force or toque to an

object that is manipulated by the robot or with which the robot is in contact. PD position

control law will be simulated for the proposed direct Msual servo control system.

89

hdependent-joint PD controt [7ff is su far ttit mo~t papular feedback eon tmk for

robots. A proportional controller has the effect of nducing the rise time and will reduce,

but never eliminate, the steady-state emr. A denvative controller has the effect of

increasing the stability of the system, reducing the overshoot, and improving the transient

response. P D (Proportional-Integral-Derivative) control is also a widely used contml

method. An integral controller has the effect of eliminating the steady-state emr, but it

may make the transient response worse 111 11. The independent-joint PD control means

that every joint has an independent PD controller, which is driven by the error function of

each joint. In this case the position of every joint is measured and sent to the appropriate

PD controller. The structure of the controller is s h o w in Figure 4.6.

The advantages of using a PD contmller include: very simple to implement; suitable for

nal-time control; and the behavior of the system can be controlled by changing the

feedback gains. On the other hand, there are some disadvantages of using a PD controller

such as: it needs high update rate to achieve reasonable accuracy; there is always traàe-

off between control accuracy and the system stability; and it does not consider the

couplings of dynamics between robot ünks

In general, using a PD feedback controller with high update rates cm give an acceptable

accuracy for trajectory tracking applications. In [49], it was proved that using a linear PD

feedback law is useful for positioning and trajajectory tracking.

The desired trajectmy for the robot cm be dtscrikd by the jomt rrajcetay w the end

effector mjectory. The vision sensors may also measure the position of every joint or the

end effector. If the trajectories are given in the end effector fiame, then inverse kinematic

transformation may be necessary. Basic control structure and the corresponding formula

for different trajectones will be discussed in the following sections.

4.5.1 Independent- Joint PD Control with Known Joint Trajectory

A desired joint position q d is compared to an actual joint position q and the difference is

multiplied by a position gain to produce an output for the actuator. In order to remedy the

damping during the fast movement, a velocity referencc is often proposed, which requires

the trajectory planner to specify the desired velocity q, (as show in Figurr 4.6) (41. The

output torque can be represented as the fouowing:

t = K , e + K d è (4.23)

where position gain: K, = diag[k,, k,, k,, 1, derivative gain:

Kd =diag[kdl kd2 kd3], ermrfunction: e=q, -q,andibdenvativewithrespt

to time is e = q, - Q . Because the example manipuhtor has k e e DûF, IE, mi& & are

As mentioned in Section 4.4, the kinematic error wilI be calculateci by cornparhg the

desired and the real image feahires (here are image coordinates), so the desired trajectory

wili be projected onto the image feature space by scaled orthogonal projection, and e will

be the joint m p u k disphcement e m r (for nvdute joint) +x joint ü m displacement

eror (for prismatic joint) caiculated in image space.

Figure 4.6 Joint PD controller with a known joint trajectory. R: Robot Dynamics

45.2 PD Control with Cartesian Trajectory

The position controller discussed in Section 4.5.1 is based on the cornparison of the

desired and red joint trajectories. It is very common that the trajectory of end effector is

planned according to the task variables represented in Cartesian coordinates because the

robot task is usually performed by the end effector. In the simulation, the joint positions

are measured by the vision system directly, so the= is no sensor to measure the position

of the end effector cürectly. If the trajectory is initially specifed in tenns of the

coordinates of the end effector, then kinematic transformation is required to perform the

conversion between the displacement of the end effector in Cartesian coordinaie hime

and the rotational displacement of revolute joints or linear displacement of the prismatic

joint. The key part of this control system is still the independent-joint PD controIIer.

There are two choices to perform the control with Cartesian trajectory.

F l p 4.7 shows that the des ird Cartesian trajectory f i is cunueacd into joint

variables, and then the following control scheme is presented in Figure 4.6. The desired

and real Cartesian trajectories are xd and x , respectively. The desired and real Cartesian

velocities are x, and f , respectively. Considering the possible emr of the kinematic

model of a robot, the estimated desired joint displacement and velocity are assumed to be

a, and 4, . which are calculated by the kinematic model.

Figure 4.7 A joint PD controiler with known Cartesian trajectory. M applied to

estimate the desired joint trajectory.

M: Inverse Kinematic, R: Robot Dynamic Mode1

Figure 4.8 shows that the error between the real and desired trajectories is calculated in

terms of the Cartesian coordinates of the end effector Erame. After inverse transformation,

the error signal is input into the PD controller. Because the vision system does not

measure the position of the end effector dirrctly, it is necessary to transfomi the

measurement results obtained h m the vision system into the Cartesian end effeaor

coordinate frame. The real end effector displacement and velocity P and are

calculated by forward kinematic of the robot.

Figure 4.8 A joint PD controller with known Cartesian tmjectory. FIS applied to

estimate the real end effector trajectory.

M: Inverse Kinematic, FK: forward kinematic, R: Robot Dpamic Mode1

In the situation show in Figure 4.8, the lanematic e m r is evaluated in 3D space, so the

joint position in 3D robot space is reconstructed from the joint position in 2D image

feature space. The method presented in Fi- 4.7 c m avoid fransforming the position

information obtained from the image feature space to the robot space. Actually the '*off-

iine teaching" procedure cm measuring joint motions directly by vision and saving for

reference joint trajectory even if the desired trajectory in robot space is aven in Cartesian

coordinates. Thus the desireci joint rrajectory can be obtained withom inverse bernatic

transformation and projection transformation of the end effector trajectory.

4.6 Summary

In this chapter, a direct visual servo control method for a three-DOF SCARA type

manipulator was proposed With the third revolute joint king fixed, the SCARA robot is

simplified to be a three-DOF manipdator with two revolute joints and one prismatic

joint. me kinematic h-misformatio~ was devefoped ixxof&ng €0 Bs peremetea. Thc

joints and links were represented as rods with mass at the centroid of each rod The

Lagrange-Euler method was used to develop the dynamic mode1 of the manipulator.

Vision-assisted kinematic error estimation was used to obtain the joint position emr.

With the proposed camera placements, the scaled orthogonal projection mode1 can

simulate the projection procedure from the 3D space to the image coordinate frame.

Digitization was also discussed when modeling the digital image coordinates.

The control errors are calculated in image coordinates and an independent PD control

rnethod was used. For the future work of this project, two kinds of image feature

recognition methods were also intmduced.

These models will be used to investigate the feasibiiity of the vision-assisteci kinematic

error estimation for a direct visual servo control method to control the example

manipulator to perform the desired trajectones. Simulation results will be discussed in the

next chapter.

Chapter 5

Simulation of a Direct Visual Servo Control Method for a

Three-DOF SCARA Type Manipulator

5.1 Introduction

In Chapter 4, a direct visual servo control method for a three-DOF SCARA type robot

was introduced. Based on the proposed method, different simulation scenanos wiII be

investigated and the feasibility of implementing an image-based visuai servo joint control

system for a three-DOF SCARA type manipulator will be discussed. The main concem is

to investigate the capability of the proposed method to track specific trajectones. Two

categories of trajectories, known trajectones of the joints and hown trajectories of the

end effector (the endpoint of the third joint), are considered. As discussed in Chapter 4,

the error signals are calculated in the image featwe space. In this simulation the image

feanires are the image coordinates of the points in an image. That means the required and

real trajectories are al1 transformed into the image space and represented in terms of the

image coordinates of the origin of the correspondingioint coordinate fnune.

In order to test the method, several e m r models are intmduced to the simulation process

and the results are analyzed These e m s include robot iink length enors, image

coordinate errors, and robot dynamic disturbances. In a practical application, every joint

actuator has a limited velocity and allowable output forceltorque. These limits codd

influence the ability of the robot to reach the required trajectory and could make the

system unstable. This chapter will alsa investigate the simulations with velocity and

forceltorque limits.

The simulation program is developed using Matlab. The flow chart and the function of

the program will be introduced in this chapter. The corresponding panuneters for the

simulation are also detailed in this chapter.

5.2 Basic Parameters for the Models

The proposed visual servo control rnethod for the three-DOF SCARA type manipulator is

composed of the manipulator model (including kinematic and dynamic parts), projection

model and control model. In this section, the parameten of models will be introduced,

which are used in the simulations presented in Section 5.5. Some panuneters will be

intmduced with specific simulation results.

5.2.1 Parameters of Manipulator

Table 5.1 D-H parameters and mass of the example three-DOF manipulator.

The zem confi~guration of the modeled three-DOF example manipulator is shown in

Figure 4.2. The numerical values of the robot kinematic parameters with D-H parameters

are listed in Table 5.1. The dynamic mode1 was discussed in Section 4.3. The m a s of the

tinks for the simulation is dso s h m in Table 5.h Ttit tmk masses mc? Link lm@s sn

roughly chosen or calculated according to the manual of Adept One robot.

The Coulomb friction F is considered as F =

simulation. The magnitude of fiction force is taken to be about 10% of the average joint

force or torque.

5.2.2 Parameters for Projection Model

One camcra is placed overhead so that the image plane of the camera is paralle1 to the top

planes of the joints in order to obtain the image as shown in Figure 4.3. In the simulation,

the distance between the image plane of the camera and the observed plane of the fint

two joints is assigned to be 1 m. The second camera is placed beside the third joint so that

the optical axis of the camem is perpendicular to the axis of the pnsmatic joint. The

distance between the image plane of the second camera and the axis of the pnsmatic joint

is assigned to be 0.5 m. Depending on particular applications. these distances can be

obtained from camera calibration. In the simulation, the image precision is assumcd to be

0.00 1 mfpixel for the first camera and 0.0005 m/pixel for the second camera.

As discussed in Chapter 4, scded orthogonal projection mode1 is used to simulate the

transformation between the 3D coordinates and the 2D image coordinates of the observed

points. The digi tization transforms the image analog coordinates into digital coordinates.

The PD control method is applied to the joint contmller. The control parameters are nuied

by trial-and-error so that the joint displacement errors can be controlled in a small range

and the joint output torquedforce can be reasonable. Generally, the robot trajectory is a

combination of different curves. It is not easy to dynamically identiw the transition points

among different curves and assign different control gains accordingly. Therefore it is

very helpful if the same control gains could be used for al1 the curves. Extensive test nuis

were performed by trial-and-enor to tune the control parameters such that a unique set of

gains could be used for most caseslajectories. As the tuning result, the conml

parameters for different cases are taken the same (K, = [4000 3000 200001~ and & =

1100 100 2001T) if not specified These numbered values are about the same order of

magnitude as that in Schilling [72]. The units of the proportional gains are N/m (for

prismatic joint) or N &ad (for revolute joints). The units of the derivative gains are N

dm (for prismatic joint) or N m s/rad (for prismatic joint).

The joint velocities and force/torque are assigned according to the specifications of some

commercial robots. In Section 5.5.5, the allowable joint velocities and forcdtorque will

be introduced and their effects will be investigated.

5.3 Prescribed Trajectories

In this section, two types of trajectories are discussed The combination of these

trajectories can simulate more complicaied tracking tasks for the robot.

This type of trajectory is described in the joint space and represented by the joint

variables, and the actual joint trajectory cm be compared with the desired joint tmjectory.

The fiat case for tracking the known joint trajectories (referred to be T W l in the

following discussions) requires every joint move with a constant angular velocity (for the

revolute joints) and constant linear velocity (for the prismatic joint). The desired

trajectory is represented by the following expressions: q, = yt; q, = w2t; q, = v,t. The

numericd values of the angular velocities oi and 02 of the two revolute joints (fmt two

joints) and the linear velocity y of the prismatic joint (third joint) will be assigned with -

the specific simulation results.

The second case for tracking the known joint trajectories (referred to be W 2 in the

following discussions) is that the joint displacements are expnssed as a harmonic

function q, = A, (1 - cos(il,t)) ; q, = a(l- cos(gt)) and q, = d, - A=, cos(%t -xi21 ,

where Al and A2 are the maximum angular displacement of the first and second joints, A3

is the maximum swke of the prismatic joint, q is the angular frequency of the cosine

function (i = 1, 2. 3). Then the joint velocities cm be calculated as ql = qqlsin(qlr) ;

49 - = 4q2sin(q2t) m d q3 = qqisin(q3t - n12) . If the amplitudes of the velocities are

taken to be ml, y and v3 as used in the fint case, then A, = a>, 1 ql 1,; 4 = a>, 111, and

A, = v, I q3. When qi reduces, the fÎequency of the cosine function will be reduced.

This type of trajectory is defined by the trajectory of the end effector in the Cartesian

world coordinate frarne. Because the vision system measures the joint tmjectones in

terms of the image coordinates, the measured joint trajectories by vision system cannot be

compared with the desired tmjectory directly. The desired end effector trajectory should

be transfotmed into joint frame by inverse kinematics.

The third expected trajectory is defined in tem of end effector mjectory (referred to be

W 3 in the following discussions) that requires the end point of the third joint move

along a spiral cuve where the projection of the curve ont0 the xoyo plane is a circle and

the prismatic joint rnoves along a axis (the axis of the pnsmatic joint). The desired

trajectory can be expressed as x, = a, + (a, /2)(l+ cos(a>,t)) ; y, = (a, 1 2) sin@$)

and 2, = dl T v3r ; where is the angular velocity of the end point when moving dong

the circle. Obviously (x, - (a, + a, 12))' + y: = (a, /2)* guarantees that the projection of

the curve ont0 the xozo plane is a circle with a radius of a, / 2 centered at [al + a, 1 2, O].

5.4 Simulation Program Flow Chart

The simulation prograrns are developed in MatLab. The program has a function option

menu, which is easy to be used for the simulations inaoduced in this chapter. The

program flow chart in Figure 5.1 shows the functions of the simulation program, which

include:

a Setdon of the demPd trajedoqr The osrrcarr sekt fourctiffercnt asjtctorics

as discussed in Section 5.3 in an option menu. Joint trajectories are given in t e m of joint

variables. End point trajectories are given in terms of Cartesian coordinates.

O Simulation of the kinematic, dynamic and projection models. This is the main

part of the simulation. The kinematic, dynamic and projection models are based on the

discussions in Chapter 4. Kinematic and dynamic models simulate the robot motion and

the response to the contmi signal. The projection rnodel simulates the fomation of the

image features in terms of image digital coordinates.

O S e l d o n of the error for khematic, dynamic and projection models: The

program can sirnulate the results with link length errors, dynarnic error and image

coordinate errors introduced into the comsponding models.

O Simulation of the control model: The PD control mode1 has two control

parameter vectors, which are proportional to the position gains and derivative gains.

These parameters cm be tuned b y trial and error according to the simulation results.

a Plot of the trajectory: The desired trajectory and real trajectory can be plotted on

the same graph so that the performance of the point-to-point tracking cm be compared

Rot of the errors of the joint rnovement: These Rsalts hdicate the tradairg

performance at each interval, which can supply dues for adjusting the control parameters.

Plot of the actual velocity, acceleration and forcekorque of joints: These

curves of the dynamic performance couid supply compxehensive evaluation of the

tracking performance thou the purpose of the simulation is to obtain reasonable

displacement tracking performance,

l P given joint trajectory given end-point trajectory

plot graphs .

desired joint /I kinematic kinematic displacements \I mode14 errot

image feature projection

projection Cl

dynamic

simulated joint variables

1 kinematic 1 model hP

Figure 5.1 The 80w chart of the simulation program.

This section will present various simulation results to analyze the performance of the

suggested visual servo method. The main concem is the tracking performance, which is

presented by the displacement errors of the first, second and third joints between the real

trajectories and the desired trajectories. The actual velocity, acceleration and torqudforce

of each joint are also investigated for various cases and presented for some cases. Euler

numerical integration method is used. The simulation time is assigned to be 8 seconds.

Uniess otherwise specified, the step time for the numerical calculation is 0.1 m. The step

time is chosen such that the system output can be stable. In Section 5.5.1, the effect of the

step time is also investigated and some results are presented.

5.5.1 Simulation Results with no Error Input

This section analyzes the simulation results with no enor input, i.e., there is no kinematic

emr , image coordinate erron and dynarnic disturbance during the simulations.

Figure 5.24 shows the tracking performance of TRAI1, The two revolute joints

velocities are desired to be oi = = 1 rads, while the prismatic joint velocity is desired

to be v3 = 0.1 d s . The prismatic joint will move upward by 0.2 m and then wiIl change

the direction to move downward. Figures 5.2-l(a) shows the desired and real trajecton

of the endpoint of the third joint. The two trajectories cannot be differentiated because the

e m r of the trajectory is very small and the scale is limited Figures 5.2-1@) and 5.2-l(c)

show the desired and reai trajectories of the origins of the first and second joints,

rcspmively. Figures 5.2- 1 (d) md 5. %Ife) show that dispi~eement emis of ths &w - ---

revolute joints are less than 0.003 rad. Figure 5.2-l(f) shows that the displacement error

of the third joint is limited between -1.5mm and -1 mm. Figure 5.2-2 shows the actual

velocity, acceleration and forcdtorque during the simulation. The average values are in a

reasonable range (except that the spike in Figure 5.2-2(e) is -36 rads2 and the spike in

Figure 5.2-2(h) is S667 Nm).

In the case shown in Figures 5.2-3 and 5.2-4, the robot is required to perform the same

movement as in Figures 5.2-1 and 5.2-2, but the step time for the simulation is changed

from 0.1 ms to 0.08 ms. Except for the initiai spikes in Figures 5.2-l(e) and 5.2-2@), the

other spikes produced by digitization disappear. In applications, the initial spikes could

be reduced by special design of the initial path to reach a stable motion. A more criticai

situation for a robot is that the robot is required to start as quick as possible without initial

path planning. If the robot can go to stable state shortly, then the robot behavior will be

better when a suitable initial path planning is available.

F Ï p 5.2-5 shows the influence of imege resdution. Here every pixel in the hege

represents 2 mm in 3D space. There are more spikes on the curve of the acceleration and

torque output in Figures 5.2-5 (d-h) than in Figure 5.2-2. The results in Figure 5.2-4 and

5.2-5 reflect that the spikes are produced not by the movement states of the joints. These

spikes could be reduced by adjusting the numerical algorithm or parameters. Sharp

changes in velocity (either value or direction) such as those happen in initiai tirne, are

another source for the spikes. The effect of these spikes is discussed in Section 5.5.6.

In Figure 5.2-6 ihe~e i s no frithn tem in the simulate& Qynamic -1. Ths d g . . . - . - - - -

show that the effects of digitization are more obvious. The possible reason is that the

friction in Figure 5.2-1 becomes a damping factor that reduces the effects of digitization.

The average displacement error of the third joint that is shown in Figure 5.2-6(f) is close

to -1 m.

Figure 5.3 shows the tracking performance with TRAJZ. The desired maximum velocities

of the two revolute joints are assigned to be 1 rads and the desired maximum velocity of

the prismatic joint is assigned to be 0.1 mis. Compared with Figure 5.2-1, there is more

fluctuation on the c w e shown in Figure 5.3. change of the velocity causes these

fluctuations. When the frequency of the cosine lunction, which describes the

displacement of each joint, reduces, the magnitude and frequency of the output reduce

too. On the other hand, if the changes are very dramatic, the response time may be longer

for the system to reach a stable state. Because the robot joints are started harmonically,

the initial fluctuation is not as serious as those cases for TRAJ1 and W 2 .

Figure 5.4-1 shows the tracking perfonnance with TRAJ3. The endpoint of the third joint

is required to move dong a circle with a constant velocity (W2 radls) in a plane pardel

to plane x3y3 while it moves dong the 23 axis with a constant velocity (0.1 ds). In 8

seconds, the end ef5ector performs two cycles that means it will draw two circles and the

prismatic joint will move downward and upward twice. The spiral movement of the end

effector should be transformed to the joint space by inverse kinematic transformation,

and then projected into the image space. The PD control parameters are assigned the

same as TRAJl and TRAJ2 cases The dispiacement ecrors of the two cevolute joints are

between kû.0024 rad. The displacement error of the pnsmatic joint is between -1.5 mm

and -1 mm.

By adjusting the control parameters of the joint controllers, the average error cm be

further reduced. Figure 5.4-2 shows the same simulation result as that in Figure 5.4-1

except that the parameters of the joint controllers have been changed to 4 = [20000

5000 20000]~; & = [100 150 5001T. Obviously the output c w e of displacement emrs

are better than that in the simulation of Figure 5.4-1. The results of other cases cm also

be improved by adjusting the control panuneters.

In Figures 5.2-1 (0, 5.3(f), 5.4- 1 (f) and 5.4-2(f), the maximum emrs of the third joint

displacements are al1 multiplied by 0.5 mm. Actually every pixel in an image of the third

joint represents 0.5 mm, so after digitization, the maximum emr of the third joint is

always the multiple of 0.5 mm. This phenornenon proves the influence of the digitization.

In TRkll and TRAJ3 cases. the third joint is reqyired to move at a constant velocity.

However in TRAJZ case, the third joint is required to move at a variable velocity,

therefore there are more fluctuation on the output as s h o w in Figure 5.3.

The previous results show that the static displacement emrs of the third joint are more

stable than the displacement emrs of the other joints. The first two rotary joint

displacements are in t e m of the angles in image space, which rn calculateci by

arctangent function of the digital coordinates dong tuw image coordinate axes The

coordinate of the third joint is the digital coordinate dong one image coordinate axis.

Thqs more digitization enon are produced than the calculation of the linear displacement

of the third joint.

TRAJ1, 2 and 3 are based on the link masses of 10 kg for the first link, 5 kg for the

second link and 2.5 kg for the third link, respectively. The order of magnitude of these

masses is the same as that in Schilling [72]. As this mass was too large for the slender bar

assumption, the masses of links 1, 2, 3 were reduced to be 1 kg, 0.8 kg and 0.5 kg, and

the results of simulation are included as TRAJ3 with adjusted mass case as shown in

Figure 5.5. The magnitudes of friction force are changed to be 0.2 N m, 0.2 N m and 0.8

N for the three joints, respectively. Because of reduced inertia of each iink, the control

gains iue reduced as well, which are & = 1600 600 60001T and & = [60 50 100~~.

5.5.2 Simulation Results with Image Coordinate Error

This section will analyze the simulation results with image coordinate emrs. Because it

is not easy to simulate various emon in a vision system, the image coordinate emrs can

be considered as the comprehensive results of the lens emrs, the projection mode1 emrs

and the errors generated during image processing procedure. During the simulation, the

image coordinate errors are represented by the value of sarnpling error (dp ixe1) in an

image and produced by a randorn f'unction with a nomial disaibution. The maximum

error is assigned to be 0.5 rndpixel.

Figure 5.6-2 TRAJ3 with adjusted mass and random image coordinate emr: (a) trajectory of end point, @) and (c) trajectones of first and second joints, (d) to (f) joint displacement eifors.

Figure 5.6-1 shows the tracking performances when the robot is required to ûzck TRA13

and image coordinate errors are produced randomly. The comsponding kinematic and

control parameten are assigned to be the same as those in Figure 5.4-1 in order to

observe the differences between the results with random image coordinate emrs and the

results without image coordinate errors. In Figure 5.6-2. the masses of links are reduced

as in Figure 5.5. The control parameters in Figure 5.6-2 are dso the same as those in

Figure 5.5. The other trajectories are dso simulated but due to space limitations are not

shown here. A11 the results show that image coordinate emrs generate more digital noise

on the output cwes. By adjusting the control parameters and robot joint velocities, the

average error and the frequency of the fluctuation can be reduced.

5.5.3 Simulation Results with Dynamic Disturbance

This section will analyze the simulation results with dynamic disturbances. Dynamic

disnubance is simulated as the forces or toques of the joints and represented by a cosine

function. r,,, = [50 50 50r cos(5t) in N (for prismatic joint) or N m (for revolute

joint). The dynamic disturbance c m be thought as the comprehensive simulation resuits

of the disturbances produced by the robot itself andlor the environment. The source of

disturbance might be vibration, irnbalance distribution of mass. electrical disturbance, etc.

In a practical situation the dynamic disturbance would be a cosine function with a d t i

frequency and amplitude.

Figure 5.7-1 shows the tracking performance with TRAJ3 and dynamic disturbance. The

control gains in Figure 5.7- 1 are I(, = 11200 1600 400001~ and I(a = [50 80 ZOO]. The

figute shows that the eRict of the dynamic disturbmce is much stronger ttran that of&=

digitization e m r , even though the frequency of the fluctuation is lower than that in the

case with image coordinate enors show in Figure 5.6-1.

In Figure 5.7-2 the masses of Links are reduced as in Figure 5.5. The expected tmjectory

is TRAJ3 as well. Dynamic disturbance is represented by a function

tdirrvn = [5 S 5r cos(%) in N (for prismatic joint) or N m (for revolute joint). The

control gains are IC, = [ 1 6 0 1600 60001T and &= (50 50 100IT.

5.5.4 Simulation Results with Link Length Error

The link Iength erroa may corne from various sources, such as the manufacturïng emrs,

link defornation, mechanical Wear, transmission and backlash, etc. Instead of choosing

some specid e m r models, this section will consider the link length emrs as random

enors with a nomal distribution. The comsponding kinematic and control parameters

are assigned to be the same as those in Section 5.5.1 in order to observe the differences

between the results with link errors and the results without link erron. The maximum hnk

emor is assmed to be 0.002 m.

Figure 5.8- 1 shows the displacement mcking performances with T W 3 and link mors.

The figures show that the iink hgth emm have strong effects on the first two joints. The

amplitude of the joint displacement error is similar to the prevîous simulation cases. The

third joint displacement e m n in the entire four situations are between -1.5 mm and

In order to observe the effect of link Iength errors clearly, the coordinate emrs of Oi and

Oz have been s h o w in Figure 5.8-2. In Figure 5.8-2, the effect of Link emrs is more

obvious. The introduction of the link enors enlarges the erroa dong the coordinate axes.

When the link length changes during the robot movement, the controi emn in terms of

the errors between the desired and actuai image coordinates will be influenced. The

controller cannot figure out the source of this error and simply adjusts the joint

torquelforce to reduce the control emrs that are produced by iink length emrs. This

behavior may produce more errors due to joint movement.

In Figure 5.9-1 and 5.9-2 the masses of links are reduced as in Figure 5.5. The expected

trajectory is TRAJ3. The control parameters are the same as in Figure 5.5. The input link

emn are the same as in Figure 5.8-1. The results show that displacement errors of the

revolute joints are in the same range as in Figure 5.5 and the displacement emrs of the

third joint are between -1 mm to 4.5 mm.

5.5.5 Simulation Results with Joint Velocity Limits

The previous simulations al1 required that the joint velocities be much smaller than the

maximum velocities the joints can achieve and no velocity limit was set during the

simulation. In this section the desired joint velocities are assigned to be close to the

maximum velocities in order to observe the tracking perfo~mance of the suggested

method at maximum joint vehcities. According io the specifications of some commefciaf

robots, the maximum velocity lirnits for this simulation are chosen to be q, = & = 6

rack and v3- = 5 1x11s.

In Figure 5.10, the expected trajectory is TRAJJ and the desired velocities of the first two

joints are 1 .Sn rad/s. The desired veloci ty of the third joint is 1 m/s. Even &ter adjusting

the control parameters, the emor output and the frequency of the fluctuation are still much

higher than that in Figure 5.5-1. The PD control parameters are &,=[15000 15000

20000jT; &=[lO 10 1001' This result shows that higher velocity triggers higher

digitization emrs during image processing. The desired joint velocities should be

designed relatively lower than the maximum velocity.

5.5.6 Simulation Results with forqueForce Limits

In an actual environment, the joint actuators (motors or cylinders) have limited power

that may not provide enough acceleration in order to follow the desired trajectory at the

corresponding velocity. This practical limitation may produce strong fluctuations of the

output torquefforce when dramatic changes of the velocities are required-

According to the data of some commercial robots, in the simulation the maximum toques

for the fmt two revolute joints are assumed to be 100 Nm while the maximum force for

the prismatic joint is assumed to be 100 N. The following figures show the simulation

results for the different desired tnjectories. There is no kinematic, dynamic or image

coordinate e m r in the following simulations. The kinematic and control parameters are

assigned to ùe the s a m t as hose in Section 5.5.1 in orda io observe the

between the results with and without forcdtorque limits.

Figure 5.11-1 shows the tracking perfomance with TRAJI. Because of the limited

torques of the first two joints, it takes about 2.5 sec for the actual joint movement to

approach the desired state. The maximum displacement errors for the fint and second

joints are 0.02 rad and 0.008 rad, respective1 y. These errors are both produced during the

initial penod. For the third joint, there is stronger fluctuation when the joint changes its

direction of motion.

By adjusting the control parameten, the tracking performance can be improved Figure

5.1 1-2 shows the simulation results when the control parameters are changed to be &, =

[50000 20000 25000]' and Kd = [150 150 3001T. The initial time of the first joint is

reduced to be 1- 1.5 sec. After that the displacement erron are limited beh~een M.003

rad. For the second joint, the initial time and the amplitude of the fluctuation are both

reduced. For the third joint, the response to the change of the velocity is faster than in

Figure 5.11-1.

The other cases with TRAJ2, TRAJ3 and TRAJ3 with adjusted masses are also

simulated. Ail the results prove that the suggested servo control system can be convergent

but the= is not a universal set of conml parameters that can be used in al l the situations.

Figure 5.1 1-2 TRN l with force limits after adjusting the control parameters. (a) trajectory of end point, @) and (c) trajectofies of first and second joints, (d) to ( f ) joint displacement emrs.

Based on the robot kinematic and dynamic models. projection model and control model

presented in Chapter 4. a three-DOF SCARA type manipulator with a visual servo

conwl system has k e n simulated in this Chapter. The control error is obtained by

cornparing the desired and acnial digital image coordinates, which are calculated by the

scaled orthogonal projection model. A PD joint controller was simulated to investigate

the tracking performance (displacement errors) of the robot. The trajectories of the robot

include two categories, known joint trajectories and known end effector trajectories

where the joint trajectories are calculated by invrrw kinematics.

Generally speaking, the outputs show strong digital noises. Meastuement emrs of image

coordinates have strong effects on the output when they are produced randomly in every

step of the whole simulation procedure. D ynamic disturbances as additional

torques/forces to the joint are simulated as a hannonic function. According to the

simulation results, dynamic disturbances might have stronger effects on the outputs than

the image coordinate e m a . The errors of the link lengths increase the displacernent

errors but could be controlled in an acceptable range.

The response is also investigated when the system cannot provide enough power to meet

the acceleration requirement. The results show that the system is convergent.

No cornparison was made with published data, as to the best knowledge of the author,

there is no work on visual servo control of a SCARA robot. However, extensive test runs

for different cases were performed (most incïuded in the thesis) to make sure the program

was correct and the results were vaiidated. The simulation results demonsirated that

vision-assisted robot kinematic error estimation could be used in a visual servo control

system for a three-DOF SCARA type manipulator.

Chapter 6

Discussion: Vision-Assisted Robot Kinematic Error

Estimation in a Fault Tolerance Robot

6.1 Introduction

Robot vision technology has become one of the most popular topics in robotics research.

Besides the studies on the humanhimai vision with the goal of understanding the

operation of biological vision systems, robot vision researchers stress on developing

vision systerns that are able to narigate, recognize, track objects and estimate their speed

and direction.

The vision application mode1 introduced in Chapter 2 shows the basic structure and the

information flow among different function modules. Generally, the vision system could

answer two questions: "What it is" and "Where it is" (621. These two questions are also

the standards for categorizing the vision applications. A practical robot vision system

could answer both questions, but often focuses on one of them. One of the important

steps in a vision application is how to make decisions based on the information obtained

from the vision system.

Vision-assisted robot kinematic emr estimation reaIizes the kinematic error estimation

based on the "where it is" information obtained h m the vision system. A dKea visual

servo control system for a SCARA type manipulator discussed in Chapters 4 and 5 is a

typicd appficatim of visim-assisteck robot kRrematic em>r estimation. The smmfmion

results demonstrate that it is feasible to form a visual servo control system with the joint

position error estimation in the image feature space. For the SCARA type manipulator,

the features could be the image coordinates of the special geometrical feiiture points of

the joints.

A new robot can always obtain fault tolerant capability h m special considerations

during the design procedure. However for an existing robot it is quite difficult to improve

its fault tolerance performance without structure changes. The direct visual servo control

rnethod currently could not replace the traditional robot control system due to the

shortcomings of the vision system, but it could be an alternative choice when some

senson fail. Furthemore because a vision system can flexibly and extensively collect the

information in the robot workspace, it cm perfonn high-level analysis of the joint

movements to make decisions as well as low-level servo control to fulfill requested tasks.

The interesting point is that drarnatic stmcture changes are not necessary for an existing

robot to improve its fadt tolerance by employing a vision system. As a case study of this

chapter, the vision system is required to replace a failed joint sensor of a threeDOF

SCARA manipulator so that the robot can follow a fidl or a partial recovery scheme.

Currenily there is no universal method for implementing robot vision application. The

selectiun of the feanires and the placement of the cametas are both dependant on the

actual robot task and the specific structure of the robot. At the end of this chapter, a

pssibie methad to m m the joint positions of a PUMA type maniputatm with e vision

system will be discussed.

6.2 Application in Fault Tolerance System

The essence of the vision-assisted kinematic emor estimation discussed in the previous

chapters is to measure the joint positions with a vision system. Cumntly the joint sensor-

based servo control technology is still the paramount choice in an industrial environment

because the rd-time response, accuracy and reliability of the traditional control system

are still better than a vision-based servo control method. However the flexibility and

broad view of a vision system contribute to the unique advantages in sorne applications.

The following sections will discuss the possible roles that a vision system can play in a

fault tolerant robot.

A robot is a rnix of mechanical, electrical and electronic components. The situation is

compounded by the fact that robot systerns are highly dynamic, resulting in very rapid

and wild responses of robots to subsystem failure [101]. Robots are king more

frequently utilized in inaccessible or hazardous environments to alleviate some of the

nsks involved in sending humans to perform the tash and to decrease the time and

expense. However, the dangerous, long distance nahm of these environs also makes it

difficult, if not impossible, to send humans to repair a faulty robot. Typically human

operaton are integral in monitoring such a system for problems, but they are not

completely reliable in detecting failures. The robots need to independently detect and

isolate internai failures and utiiize the remaining functiond capabilities to automatically

overcome the limitations imposed by the failures. If robots could be provided with

autonomous fault tolerance without requinng immediate human intervention for every

fai l u e situation their working life can be lengthened [99].

Figure 6.1: Vision system in a robot fault tolerance system.

-

To develop and realue a trdy fault tolerance system, many issues including mechanical

design, redundancy management, fault detection, system reconfiguration and recovery,

intelligent decision-making, real time control, operational software and task prioritizatioa

must be taken into consideration and stuclied. Fault tolerant control of a robot system cm

be divided into three layers: a continuous servo layer, an interface monitor layer, and a

discrete supervisor Iayer [99].

Process for controller b Sem tayer -

The vision-assisted kinematic error estimation can improve the performance of an

existing robot besides forming a visual servo control system. The discussion in the

following sections shows that a vision system can play a role in ai l the three be l s of the

-

+- Robot and Environment

P m for Interface fauit check W'3r

Image Process for Supervlsor safe check LaFr

+

robot fanit tolerant system as shown in Figure 6. t. Thepint sensor-baJ& eontrot system . A

lacks these comprehensive functions.

6.2.1 Servo Layer

Servo layer consists of the normal robot controller and the robot. In this layer fault

tolerance requires the controller to control the robot movement when a failure happens.

Usually an alternative controller must be installed for the robot. The alternative controller

cm be a totally new controller or a special control scheme for the same controiler. The

alternative controller must allow the robot to resume performing its tasks (full recovery)

or to gracefully degrade (partial recovery) if a failure is detected. One important type of

this kind of failure is sensor failure. Traditionally redundant sensors are used to increase

the fault tolerance capability of the sensor-based controller. As will be discussed in the

following sections, a vision system not only can work as a sensor to measure the

positions of some joints of a robot, estimate the kinematic error and form a visual control

system, but also can help with the failure detection of the robot. Through some special

processing on the image, the vision system cm even rneasure the velocity of the observed

object, which means one camera c m huiction as both position and velocity sensor at the

same time.

The previous chapter discussed a direct visual servo control method for a SCARA type

manipulator, which proved that the vision system could replace the traditional joint

sensors when some or ail of them have failed. The main weakness of the vision system

cornes from the computing load of the image processing. Currently the measmment

accuracy of a vision system stiH b i t s the appKcatim of a M m system ta be d as a

normal position sensor. However, when some failures happen, the accuracy and fast

response of the control system may not be very important in some situations. A safe path

planning can be designed so that the trajectory and the velocities of the joints are suitable

for the vision system. Figure 6.2 shows that a vision system can perfom different

commands given fiom the higher layer to recover the robot h m a failure state.

Depending on the robot task, the vision system can replace some of the joint sensors to

form a multi-sensor control system to perform some special tasks.

Figure 6.2: Vision system in the servo layer of a fault tolerance robot system.

Case Study

The following paragraphs will discuss a case study to demonstrate the functions of a

vision system at the servo Ievel of the robot fault tolerance system. The simulations

demonstrate that the vision system can replace the failed joint sensors to recover the robot

task completely or partidly. A three-DOF SCARA type manipulator with the saw

configuration as ttiat in Chapter 4 ancf 5 is us& in the simd*. H o w m , it is asseme& --

that the traditional joint sensors are used to supply information for the PD controller

duxing the normal operation. At some time, which is input by the user, the second joint

sensor is assumed to fail and the task manager requires the vision system replace the

second joint sensor. Now the position information of the second joint is supplied by the

vision system and the position information of the remaining joints is stiil supplied by the

traditional joint sensors. With the mixed sensor control system the robot is required to

achieve full recovery scheme (in Figure 6.3-2) or partial recovery scheme (in Figure 6.4-

2) in the simulation.

The full recovery procedure requires that the mixed sensor control system finish the

cumnt task after the information from the failed sensors is replaced by the information

from the vision system. The partial recovery scheme requires that the mixed sensor

control system reduce the velocity of each joint harmonically to zero after the

infurnation from the failed joint sensor is replaced by the information h m the vision

system. In both schemes, the transitions are assumed to be immediate and seamIess,

which means the vision servo conttoUer is dways feady ta w&

The simulation program pennits the user to input the failure time. Before a failure

happens, joint sensors measure ail the joint positions and the control emrs are caicuiated

in the 3D space. After the failure of second joint happens, the joint position of the sccond

joint is measured by the vision system, and the remaining joint positions are still

measured by the original joint sensors. When calculating the control exmr of the second

joint, the desired and actuai joint trajectories are both represented in the image coordinate - A

frame after the failure happens. In the full recovery scheme, the desired trajectory is kept

to be the original one. In partial recovery scheme, the desired trajectory is changed to be

TRAJ2 (refer to Figure 5.3) with the amplitude of the velocity equal to the velocity at the

failure time. When the velocity of a joint is reduced to 0.001 rack (for the revolute joint)

or 0.001 m/s (for the pnsmatic joint), the control system wili stop the joint.

Figure 6.3-1 shows the simulation results without full recovery scheme. In this simulation

the task requires the end point of the third joint move dong a spiral curve (cded TRAJ3

in Chapters 4 and 5) with control gains as &,=[20000 5000 2 0 0 0 0 ~ ~ and &=[IO 150

5001'. Origindly dl the joint displacements are measured by joint sensors. At thne 1.5

sec the controller cannot receive response from the sensor of the second joint. The results

show that the robot lost control and cannot hilfill the original task. Figure 6.3-2 shows the

simulation results with full recovery scheme. The sensor of the second joint is failed and

replaced by the vision system at time 1.5 sec, the mixed sensor control system is nquked

to finish the current task. The transition is assumed to be smooth and the comsponding

controt parameters are not changed. Arrordmg to the d t s , thm are some spikm with

the angular errors of the second joint that also influence the output of the first joint.

Adjusting the control parameters can M e r improve the results. The joint displacement

emor of the third joint shown in Figure 6.3-2(f) is smaller than that in figure 5.4-2(f), in

which al1 the joint position are measured by vision system. This simulation shows that the

original task can be fulfilied even though the control performance is degraded afta the

sensor of the second joint is replaced.

Figure 6.44 shows tht sirnatation resnits wittiout pmtist m v c r y sckxm. hr this

simulation the task requires every joint to move with constant angular velocity (for

rewolute joints) and constant iinear velocity (for the prismatic joint). This trajectory is

called T W l (refer to Section 5.3.1) and its parameters are the same as the trajectory

shown in Figure 5.2-1, except for the assigned original angular velocities of the fmt and

second joints are 4 rack instead of 1 rads while the control parameters are assigned to be

Kp=[300ûû 35000 700001~ and &=[200 200 5001T. Again dl the joint displacements are

measured originally by joint sensors. The sensor of the second joint is failed and the task

manager requires the robot stop every joint immediately. The resuits show that it is quite

difficult to stop the robot when it moves at a high speed. Figure 6.4-2 shows the

simulation results of the partial recovery scheme. The mixed sensor control system is

required to reduce the velocity of each joint hannonically to zero. The transition is

assumed to be smooth and the correspondhg control parameters are not changed. The

velocity output after 2.5 sec shows a smooth velocity reducing procedure even though

there are some spikes with the displacement and torque output of the second joint. This

simulation shows that it is possible to stop the robot graceNly by the mixed sensor based

controller when the sensor failure happe= This partial recovery process uui avoid

damage caused by the sharp braking of the joint actuators.

6.22 Interface Layer

The interface layer contains the tmjectory planner and provides fault detection and some

basic tolerance functions for the system. The fault detection routines continually monitor

the state of the robot in order to catch any failures in the system. The joint sensors can

mfy obtaiir the rotary or hmr dispi-t of a jomt k is diffieutt tv measurr thejoint

position in the world space with an intemal sensor. However the vision measurement has

the ability to perform a nonîontact measurement. Therefore the vision system cm be

used as a redundant sensor to check the actual position of the joints and end effector in

the world space. The joint displacements in the corresponding joint frame can be

calculated from the position descn'bed in the world m e , and compared with the output

of the joint senson. The actual joint trajectories in the world space can be compared with

the trajectories expected by the robot tasks. Furthemore the position information

obtained from the vision system and joint sensors can be transformed into the

coordinate frame in order to identify possible failures.

Actual robot dynamics in dynamk environment

Figure 6.5: Traditional model-based robot failure analysis.

output -

Normaily a fault detection algorithm monitors systems faîlures by cornparhg measured

system outputs with expected values of sensors [IO11 or the correspondhg expected

output derived from a mode1 of the system behavior [99] as shown in Figure 6.5. Thm is

Robot M ~ d d

Contrd Failure Task signai + + IC,

Actual Output

- Actuai Robot + Sensors 1

m g e r

-b

parameters or the mode1 itself could change. It is difficult to request the mode1 to adjust

its parameten dynamically according to the state of the mbot movement. Even if the

mode1 is perfect, it is still quite dificult to decide the tolerance threshold for the failure

detection. A vision system can obtain the expected trajectory ihrough a teaching

procedure whenever a new task is requested (as s h o w in Figure 6.6). Then the ideal

output is straightforward and easy to operate. If the kinematic error can be estimated by

comparing the features in an ideal image and an actual image, then the kinematic and

dynarnic parameters will not be necessary to form the standard outputs (expected

images). The intemal position information obtained fmm the joint sensors is still useful

for the failure detection when they are cornbined with the information extracted h m the

images. When the failure is found in the joint sensors, special arrangements with the

vision system could replace the failed sensors to resume the robot's task.

Standard Conditions

Figure 6.6: Vision-based robot faiIure analysis.

I

t

Indirect ~ p a r l s o r i

in joint space

W e d . Images

4

1 " Ackial Achral a

Conditians images

Indirect campariscm in warld space

+ + Task Failure ,

-€Pr +

~ ~ l y r b Direct image Cornpansan

A sensing faihire is any event teading to ctefective perception. These amts rnigtir k

sensor hardware malfunction and changes in the environment that negatively impact

sensing output [67]. With the assistance of the vision system, it might be possible to point

out the causes of the failure and supply more information for the task manager to avoid

unsafe actions. For example, the wrong readings of the sensors, comparing with the ideal

values, rnight corne from the malfunction of the encoders or the defonnation lbreakage of

links. A vision system cm check the imgular changes of the images of the links.

6.2.3 Supervisor Layer

The supervisor layer provides a higher-level fadt tolerance and general action comrnands

for the robot. It could aiso enable easier integration of research such as long-tem analysis

of failure situations, which cm be performed as the background processes so as not to

interfere with the fast fault detection and fadt tolerance algorithms [99]. The supervisor

layer can be fiuther enhanced to monitor the reachable workspace of the robot throughout

its operation 1721. If a human king is needed when a failure happens, the supervisor

layer can give the operator suggestions and choices to cope with the failure.

A vision system can supply an o v e ~ e w of the robot itseif and its environment,

especidly existence of an unexpected object or even hurnan beings in the robot

workspace. The information obtained from the vision system not only cm lead to making

an emergency deàsion, but also can be applied to do off-line performance andysis of the

robot movement in order to predict possible failuses and supply necessary information for

the servo con&oUer to adjust the parametas of the conhm1 schemes. In many situations

the openitors are invoived w i h he decision =let& t e fa& tolerani:e. The vision systeiil

can supply a real-time display to give the operators direct observation to make suitable

decisions. The continuous images of the robot movement obtained nom the vision system

can also be recorded to do statistical analysis of the robot behavior in order to improve

the future structure design and control strategies. Figure 6.7 shows the basic functions

that a vision system can have at the supervisor layer of a fault tolerance system.

Links shape and relative pasitioris

tong.Urne Analysis

- ( Record of images 4

Fi- 6.7: Vision system in the s u p e ~ s o r layer of the fault tolerance system.

4 Human lnterfaco

6.3 Joint Position Measurement of a PUMA Type Manipulator

Rd-Ume dispiay

PUMA (Programmable Universal Machine for Assembly) type robot has been one of the

most commonly used research and assembly robots. The structure of a PUMA robot is

shown in Figure 6.8. If the eye-to-hand camera placement, which was pmsented in the

previous chapters, is used to perform joint position measurement for a PUMA robot, it is

difficult to use a scaled orthographie projection mode1 because the distance betweai the

image plane and the objects (some marks on the arms) wiil change dramaticdlywtrm the

arms move. The general solution is to form a stereovision sptem with two or more

cameras. The s t em vision is aiways applicable to measure the position of the points in

3D space. However, this kind of camera placement will increase the computation cost

because of the increased number of images. Feature matching among the images of the

observed objects, which are taken from different cameras, is also a challenge. Accunite

caiibration of the relative positions among different carneras is necessary.

Figure 6.8: PUMA robot with vision system.

Figure 6.8 shows that the second and third joint axes of a PUMA robot are paraIlel to

each other, which ensures that the two links (inner link and outer iink as shown in Figure

6.8) will move in two parallei planes. In Figue 6.8, three iine segments with lengths of

LI. L2 and& are m d e d on the anm. Tti t tmtscgmcntsaredt~fektothemes~

and 02Z2 so that they will always be pardel to the robot base plane &Yo while the

rnanipulator moves. If the camera cm be placed overhead with its optical axis

perpendicular to the robot base plane &Yo, then the marks will always be parallel to the

image plane while the rnanipulator moves. It will be verified in the following paragraphs

that the image length of a line segment is decided by the distance between the line

segment and the image plane if the line segment is parallel to the image plane.

t . image

Figure 6.9: The relationship between the line segment and its image.

Figure 6.9 illustrates the ~iationship between a line segment with length L and its image

length 1. The line segment L is parallel to the image plane. The two ends of the Iine

segment are represented by the two points VI (x,, yl, a) and (xt y2 22) in the camera

coordinate frarne xac&. The corresponding images of the two points are pl (uI, vr) and pz

(UA v2). The distance between the line and the viewpoint is z and the focal length is f.

Because line L is paralle1 to the image plane, = a = z is held. According to the central

perspective projection mode1 (equation (3.2)), the folIowing equation can be developed:

Then the length of 2 is

When the robot stays in a known original position, the line segment length and its

corresponding image length are assumed to be known as L and I,,. The original position

of line L is assumed to be at b, in the camera coordinate frame and thus the following

equation can be developed:

when the line moves to a new position, the z coordinate cm be calculated by equations

(6.2) and (6.3):

Equation (6.4) shows that the z coordinate is only decided by the length of the image of

the fine. Because the line segment is always parailel to the image plane, the coordinates

dong r, axis (see Figure 6.9) of the points on the Line segment marks are dl equal to z. As

long as is known, then the coordinates of those points on the h e segment cm be

calculated by equation (3.2). The feature vector for this vision systern can be defined as f

= [u v @, where (u, v ) is the coordinate of the cen td point of the image o f the mafi -

(line segment) , q = lu& is the ratio of the actual image length 1 and the onginal image

length lorg of the observed line segment. In this way the relationship between the feature

vector f = [u v vlT and the point T ( x . y, Z) in 3D camera space can be setup as:

q o o u

[]=:[O . O][.) O O f ?

The essence of the measurement of the joint position by a vision system is how to obtain

the coordinates of the origins of each joint frame either in the joint coordinate fnune or in

the robot base frame. Afthough the marks Li, L2 and L3 are not located at the origins of

the comsponding joint frames, the relative positions of the line segments to the origins of

the corresponding joint frames cm ôe calibrated. The position of the central point of the

line segment must be found initially by the vision system, and then the origin of the

corresponding joint frame can be calculated by a suitable transformation. The relative

positions of the different marks can be used to calculate the displacement of each joint.

It should be noted that the movement of the fint joint (trunk) can only change the

positions of the marks but not the image length of the line segments. But the rotation of

the arms (second and third joints) can change both the positions and the image lengths of

the line segments. For example, if the image length h of the line segment L2 changes, it is

certain that the second joint (upper a m ) is moving; if the image length Iz is fixed but the

orientation of the image of the iine segment L2 changes, it is certain that the h t joint

(tnink) is rotating. Therefore, the orientation of the line can also be an entry of the fe8ture

vector. A direction indicator could 6e added to the line segment in order fo i d e n m tne

changes of the orientation.

If the fint joint (trunk) is fixed. the upper a m and forearm will move in paralle1 planes.

The camera c m be put beside the robot with its optical axis parallel to the axes of the

second and third joints. The axis of the camera will be perpendicular to the planes in

which the two a m s move, and then the orthopphical projection model can be used.

Furthemore the camera c m be fixed to the fint joint in order to let the cameni move with

the fint joint. In this way, the relative position between the camera plane and the planes

in which the arms move when the trunk rotates c m be kept constant.

The discussion in this section demonstrated that many useful geometrical feanires could

help a vision system to obtain the joint positions. The physical constraints of the robot

movement can also supply some information for the vision system. There is no universai

method for designing a vision system because a vision system is flexible and application-

oriented On the other hand, when more and more robots are required to be equipped with

it vision systern, the robot designer shodd consiâer the requirements of the viaoit

application and design structures with good features that cm be processed by a vision

sys tem.

Chapter 7

Conclusions and Future Work

7.1 Conclusions and Contributions

The vision application model introduced in Chapter 2 demonstrated a new task onented

viewpoint for appiying a vision system to a robot. From the traditional viewpoint, the

vision system is simply divided into two procedures: image formation and reconstruction

from the images. Now interpretation procedure and decision procedure are added to the

vision application model in Chapter 2. These two procedures illustrate that the selection

of image features and the application of extracted features are decided by a specified

application requirement.

Robot kinematic emor estimation is the cornparison between the required and the actuai

robot task motion description. Vision-assisted robot kinematic e m r estimation realizes

the kinematic emor estimation based on the "where it is" information obtained fn>m the

vision system. The vision system demonstrates the observation of the robot motian in the

image form. If robot motion description is defined in the image space, robot kinematic

error estimation c m be completed in the robot space, c a m m space or image space. In

Chapter 3 the corresponding theories were presented systematically, the characteristics of

different cases were analyzed and compared, and the corresponding formulas were

developed. Jacobian feature matrices were developed when the features are chosen to be

image coordinates of the observed points.

The key point of the implementation of vision-assisteci Rinematic errar estimation is the

method to describe the robot task with suitable feature selection based on the specific

structure of the robot. Actually most of the robot tasks can be described in the image

feature space if the feature selection is suitable. The particular rnovement of each joint

cm also supply useful information for searching the correspondents between the feahire

parameten and the joint position in 3D space.

In Chapter 4, the image coordinates of the centers of these circles on the top plane of the

f h t and second joints of the YCARA type robot were selected to be the image featuns to

describe the robot task in the image feature space. Because of the different stnichire of a

PUMA type robot, a senes of line segments were chosen to be the features for measuring

the joint positions in Section 6.3. The image feature vector for the PUMA type robot has

three entries instead of two entries for the SCARA type robot. The analysis of the

PUMA case in Chapter 6 further proved that the anaiysis of the robot structure and its

task could help with a successful vision application.

Vision-assistai kinematic emr estimation is o#en used to Rcover Lhe position

information of the robot end effector. However, in this thesis, the vision system was

required to measure directly the position of each joint of a robot. The difficulty of the

direct visual servo control method is how to find suitable feanires of the joints to describe

the robot task in image space. The direct visual servo control method for a SCARA type

manipulator discussed in Chapters 4 and 5 showed that the vision systern could measure

the joint position directly in terms of the image coordinates of the specîal geometncal

-- feature points of the joints. The simulation resuIts démonstrated that it was feasibk to

f o m a direct visual servo control system with the joint position e m r estimation in the

image space.

Because of the simplification of robot models and the vision system in the simulation, the

simulation results could not guarantee high performance of a direct visual control method

in a real-time application. However. the research in this thesis provides a new appkation

ana for the vision system. The direct visual servo control method provides a basis for a

three-level fault tolerant robot with a vision system. The study in this thesis proved that

the vision-assisted kinematic e m r estimation could help robot recover h m the sensor

failures. Furthemore, in Chapter 6 a relatively complete analysis of the possible

functions of a vision system in a fault tolerant robotic system was presented. As an

alternative controller in the servo layer, the direct visual servo controller completes the

recovery procedure after the failure happens. In the interface layer the vision system can

continually monitor the state of the robot to identify some failmes in the system. In the

supervisor layer the vision system cm supply information for the emergency decisions

for safety Rasons and off-he perfmmce andysrs of the robot movement.

An important point is that a vision system can be integrated with an existing robot

without dramatic changes of the robot structure. This advantage can be used to improve

the fault tolerance capabilities of an existing robot by a vision system.

7.2 Future Work

With more and more vision systems used in robot applications, establishing a standard

would be necessary for the design of the arms and joints of the robots in order to optimize

the performance of the vision system. It is a common sense that a designer of the

automobile should consider the requirement of the road; on the conaary a designer of a

road should consider the requirement of the automobile. Vision-fnendly robot design is

an interesting topic for the future work. Basically the standard could include two aspects:

easily recognizable features on the robot; easy calibration of the relationship between the

feature parameten and the joint or end effector position or movement.

In Chapter 6, the feature vector for the PUMA robot included the image coordinates of

the central point and the length of the rnarked line segment in the image. This is another

possible topic for the future work. The feature vector can include the image coordinates

of a point plus one feanire parameter associated with the depth of the view. The image

length of a line is not a unique selection. If a rrlationship between the depth of the view

and the feanires couid be found, those feature parameters (such as area of a circle) in

image space could also be used If necessary, even four or more feature parameters couid

be included in the feature vector that could be used to describe the robot task.

In order to have a high performance of a direct visual sewo control methoci, further

investigations should be done on the following topics:

* More afctmte d y n d c robot mode1 should be deve10ped f o ~ the pptnpose OC the

simulation. Some experiments should be done in order to irnprove the dynamic models

developed by physics and mathematics.

The image feature recognition procedure has strong influences on the static precision

and the dynamic response of a visual servo control system. It is difficult to simulate the

precise image formation procedure. The images for the simulations should corne from the

real pictures taken when the robot moves.

The method to describe the robot rask in the image f e a m space needs mon

investigation. Another important aspect for this research is to find out the limitations of

some speaal image feaiures when they are used to describe the robot task.

Red-time experiments would show the real characteristics of machine vision

dynamics, such as the time delay due to pixel transport. According to the expriment

results, better control architecture could be employed.

The case study in Chapter 6 only showed the procedure in which vision system could

help robots recover from the sensor failure. Further research on how to complete the

inspection of various failures including sensoi f a i 1 ~ 1 ~ ~ should be done in the future.

Methodologies for the cornparison of information obtained h m the vision system and

the information obtained from joint sensors could be developed.

References

[1] Agganval, J. K. and Nandhakumar, N. (1988) On the computation of motion h m sequences of images-a review, Proceeding of IEEE, v76, n8, pp917-932

[2] Agin, G (1985) Calibration and use of a light stripe range sensor mounted on the hand of a robot, Pmc. IEEE Con$ Robotics and Automcztl'on. pp680-685

[3] Allen, P. K. Yoshimi, B. and Timcenko, A. (1991) Real-time visual servoing, Pm. IEEE Int. Con5 Robotics Automation, pp85 1-856

[4] An, C. FI., Atkeson, C. G and Hollerbach, J. M. (1988) Model-bczsed Control of a Robot Manipu lator, MIT

[SI Andersson, R. L. (1988) A Robot Ping-pong Playec meriment in Real-lime Intelligenr Contml, MIT

[6] Asimov, 1. (199 1) 1, Robot, Bantarn Books [7] Astron, K. J. and Wihenmark, B. (1984) Computer Contmlled Systems, Prentice Hail [8] Bajcsy, R. (1988) Active perception, Proceeding of ZEEE, v76, n8, pp996-1005 [9] Berry, F. Martinet, P. and Gallice, J. (1997). Trajectory generation by visual sexvoing,

Int. Conf. on Iwelligent Robots and Systems, v2, pp1066- [IO] Birkhoff, G and Rota, G C. (1989) Ordinary Diferentiul Equations (4& edition),

John Wiley [I l ] Castano, A. and Hutchinson, S. (1994) Visual cornpliance: task-directed visual servo

control, IEEE Tram. Robotics Automafion, v10, n3, pp334-342 [12] Chaumette, F. Rives, P. and Espiau, B. (1991) Positioning of a robot with respect to

an object, tracking it and estimating its velocity by visual servoing, Pmc. IEEE Int. Conf: Robotics Automation, pp2248-2253

[13] Corke, P. 1. (1993) Visuai control of robot manipulators - a review, V s u l Servohg, K. Hashimoto ed., Worid Scientific, pp 1-3 1

[14] Corke, P. 1. (1996) Ksual Contnd of Robots: High-pevormmice V i l Setvoing, Research Studies Press Ltd. and John wley & Sons Inc.

[lS] Dementhon, D. and Davis, L. (1995) Model-based objects pose in 25 lines of code, Int. J. Computer Vision. v15, nl, ppl23-141

[la] Dhillon, B. S. (199 1) Robot Relidiliq ami Scrfeery, Springer-Verlag [17] Dietrich, J. Hirzinger, G Gombert, B. and Schott, J. (1989) On a unified concept for

a new generation of light-weight robots, Experirnentd Robotics 1. i l Ha-yward und O. Khatib, eh., Springer-Verlag. pp287-295 (v139 of Lecm Notes in Control and Information Sciences)

[18] Espiau, B. Chaumette. F. and Rives, P. (1992) A new approach to visual sexvoing in robotics, IEEE T r m Robotics und Aurommion, v8, n3, pp313-326

[19]Faugeras, O. D. (1990) On the motion of 3D c w e s and its relation to optic flow. Ploc. 2" ECCV Lecture Notes in CS 1990t, v427, ppIO7-ll7

[ZOIFaugeras, O. D. (1992) What can be seen in three dimensions with an ubcalibmted stereo rig? Computer Vision-ECCV'92, Lemm Notes in Computer Science, Springer Verlag, n588, pp563-578

[2 11 Faugeras, O. D. Luong, Q.T. and Maybank, S. J. (1992) Camera self-calibration: theory and experiences, Computer MFion-ECCV'92,Lecture Notes in Computer Science, Spnnger Verlag, n588, pp321-334

f 22~Fangrras, 0. D. and Maybmk, S. f. t 1990) Motion from point matches: mdtipficisr A

of solutions, Int. J. Computer Vision, v6, n4, pp225-246 [23]Fedderna, J. T. and Lee, C. S. G (1990) Adaptive image feature prediction and

control for visual tracking with a hand-eye coordinated camera, IEEE Trans. Systemî. Man and Cybemetics, v20, n5, pp1172-1183

[24] Feddema, J. T. Lee, C. S. G and Mitchell, O. R. (1 99 1) Weighted selection of image features for resolved rate visud feedback control, IEEE Trans. Robotics Automa~ion., ~ 7 , pp3 1-47

[25] Feddema, J. T. and Mitchell, O. R. (1989) Vision-guided servoing with feature-based trajectory genera tion, IEEE Trans. Robotics Automation, v5. n5, pp69 1-700

[26] Flusser, J. and Suk, T. (1993) Pattern recognition by affine moment invariants, Panern Recognition, v26, n 1, pp 167- 174

[27]Gilbert, A. L. Giles, M. K. Fiachs, G M. Rogm, R. B. and Yee, H. U. (1980) A d - time video tracking system, IEEE Trans. Patîem Analysis and Machine Intelligence, v2, nl

[28] Golub, G H. and Low, C. F. (1983) Matrix Compurations, North Oxford Publishing [29] Hager, G D. (1995) Calibration-fRe visual control using projective invariance, P m

ZCCV, ~~1009-1015 [30] Hager, G Hutchinson, S. and Corkc, P. 1. (1996) A nitonal on visual Servo Control,

IEEE T m . Robotics and Automation, vl2, n5, pp6S 1-670 [3 11 Hamilton, D. L. Walker, 1. D. and Bennett, J. K. (1996) Fault tolerance versus

performance metrics for robot systems, Reliabilily Engineering and System Safety, v53, pp309-3 18

[32]Harrell, R. C. Slaoghter, D. C. and Adist, P. D. (1989) A fruit-tracking system for ro botic harves ting, Machine Vision and Applications, pp69-80.

[33]Harris C. (1990) Resolution of the bas-relief ambiguity in structure-hm-motion under orthogonal projection, BWC90 - Pmc. British Vision conf., pp. 67-77

[34] Hartley, R. 1. Rajiv, R. and Tom, C. (1992) Stereo h m uncalibrated cameras, Proc. of IEEE ConJ Cornputer Vision and Paîîem Recognition, pp. 761-764.

[35] Hashimoto, K. Kimoto, T. Ebine, T. and Kimura, H. (199 1) Manipulator control with image-based visud servo, Pmc. IEEE Int. Conf: Robotics Az~omczt~on, pp2267-2272

[36]Hollinghurst, N. and Cipolla, R. (1994) Uncahbrated stereo hand-eye coordination, Image and Visibn Computing, u12, n3, pp187-192

[37] Hong, 2. Q. (1991) Algebraic feature extraction of image for recognition, Pattem Recogni tien, v24, pp2 1 1-2 19.

[38] Hom, B. K. P. (1986) Robot Vision, M ï î Press [39]Houshagi, N. (1990) Control of a robotic manipulator to gnisp a moving target using

vision, Pmc. IEEE Znt. ConJ Robotics Automation, pp6W-609 [40]Huang, L. and Notash, L. (1999) Failure Andysis of Parallel Manipulators, Pmc. of

lûth World Congress on the Theory of Machines and Mechanisms, v3, pp1027-1032 [41]Huang, L. Notash, L and Gagnon, E. (2000) Failure Analysis of DEXTER Paralle1

Manipulator, Pmc. of CSME Fonun, 10 pages [42] Huang, T. S. (1 98 1) Image Sequence Analysis, Springer-Verlag* [43]Huang, T. S. and Netravali, A. N. (1994) Motion and structure fiam feattrre

correspondences: a review, Proceeding of lEEE, 1182, n2, pp252-267 [44] Jang. W. and Bien, 2. (1991) Feature-based visual servoing of an eye-in-hand robot

with improved tracking perfmance, Pmc. of fEEE Conf Robotics Awundm, pp2254-2260

[45] Jang, W. Kim, K. Chung, M and Bien, 2. (1991) Concepts of augmented image space and transfonned feature space for efficient visual servoing of an "eye-in-hand robot", Robotica, v9, pp203-2 12

[46] Jarvis, R. A. (1983) A perspective on range finding techniques for computer vision, IEEE Tram. Pattern Analysis and Machine Intelligence, v5, n2, pp122-139

[47]Kabuka, M. Desto, J. and Miranda, J. (1988) Robot vision tracking system, IEEE h s . Industrial Electronics. v3 5, n 1, pp40-5 1

[48] Kailath, T. (1 980) Linear Systems, Prentice Hall [49]Kawamura, S. Miyazaki, F. and Arimoto, S. (1988) 1s a local linear PD feedback

control law effictive for trajectory tracking of robot motion? Pmc. IEEE Int. Con$ Robotics and Automation. pp 1335- 1340.

[SOI Kim, D. Ri&, A. Hager, G D. and Koditschek (1995) A "robust" convergent visual servoing system, Proc. ZEEECRSJ Int. Con$ on Intelligent Robots and Systems, vl pp348-353

[SlIKim, W. (1999) Cornputer vision assisted vimial ~a l i t y calibration, IEEE Tram Robotics Automation, v 1 5, n3, pp450-464

[52] Koenderink, J. J. and Van Doom, A. J. (1991) Affine structure from motion, J. Opt. Soc. Am., v8, n2, pp374-382

[53] Kondo, N. and Ting, K. C. (1998) Roborics for Biopmduction Systm, ASAE [Ml Lacaze, A.' Tasoluk, C. and Meystel, A. (1997) Solving the forward kinematics

problem for the Stewart platform by focusing attention and searching, http://www.ee. umd edu/medab/papers/robau~urse~7/mbuu~orse~7.h~l

[55] Latombe, J. C. (199 1) Robot Motion Planning, Kluwer Academic Pubiishers [56]Leavers, V. F. (1992) Shape Detection in Cornputer M o n Using the Hough

Transfom, Springer- Verlag [57] Lin, 2. Zeman, V. and Patel, R. V. On-Iine robot tmjectory planning for catching a

moving object. Proc. IEEE Int. Con$ Robotics Autommion, ppl726- 173 1 [58]Liu, Y. and Huang, T. S. (1988) Estimation of ngid body motion using saaight line

correspondence, CVGIP. v44 , pp 35-57 [59] Lu, N. (1 997) Fractal Zmaging, Academic Press [W]Mrthowôld, M. A. md Mead, C. (1991) The silicon retina, Science's Visiox the

Mechanics of Sight, Special Issues of Scientific American 1998, pp 107- 1 13 [6 11 Maiis, E., Chaumette, F., and Boudet, S. (1998) Positioning a coarsecalibrated

carnera with respect to an unknown object by 2D 41 visual servoing Pmc. IEEE 1 , . Con$ Robotics ond Automation

[62]Marr, D. (1982) Vision. a Computational Invesîigartrartron inro the Hummt Representation and Pmcessing of fisual Infonnatlgon, WB. Freeman Co.

[63] Masland, R. H. (1986) The functional architecture of the retina, Science's Vision: the Mechanics of Sight, Special Issues of Scientiic Ametican I998, pp4-13

[64] Maybank, S. (1993). nieory of Recollstmction from Image Motion, Spnnger-Verlag [65] Moffitt, F. H. and Mikhait, E. M. (1980) Phtogtcanme?ry (3"' edition) (66]Monteverde, V. and Tosunoglu, S. (1997) Effect of kinematic structure and dual

actuation on fault tolerance of robot manipulators, P m Int. Con$ Robotics rmd Automation, pp2902-2907

[67]Mqky, RR. R. and Heshbergerf R (1996) cl- and recoverinp ficm sen@& - -

Mures in autonomous mobile robots, MAI-96, Portland OR. , pp922-929 [68]Murray, D. W. and Buxton, B. F. (1990) Erperiments in the Machine Interpretation

of Visual Motion, M ï î Ress [69]Negahdaripour, S. and Hom, B. K. P. (1987) Direct passive navigation, IEEE T m .

Panem Analysis and Machine Intelligence, v9, pp 169- 176 [IO] Nevatia, R. (1976) Depth measurement by motion stereo, Cornputer Graphics mid

Image Processing, v5, pp203-2 14 [7 11 Paul, R. (198 1) Robot Monipulators: Mathemtics. Pmgramming, und Contml, ha

Press [72] Pradeep, A. K. Yoder, P. J. M u ~ u I I ~ ~ , R. and Schilling, R. J. (1988) Crippled motion

in robots, ZEEE Trans. Aerospace and Electmnic Systems, v24, nl, pp2-13 [73] Rives, P. Chaumette, F. and Espiau, B. (1989) Positionhg of a robot with respect to

object, tracking it and estimating its velocity by visual servoing, Erperimntal Robotics 1, Hoyward, C! and Khatib. O. (Ed), Spnnger-Verlag, pp412-428

[74] Russ, J. C. (1992) Image Pmcessing Hmdbook, CRC Press [75] Samson, C. Le Borgne, M. and Espiau, B. (199 1) Robot Control: the Task Function

Appmach, Oxford Science Publications [76] Sanderson, A. C. and Weiss, L E. (1980) Image-based visuai servo control using

relationai graph error signds, Proceeding of ZEEE, pp 1074- 1077 [77] Sanderson, A. C. and Weiss, L. E. (1983) Adaptive visual servo contml of robots,

Robot Vision, IFS Ltd. [78] Schilling, R. J. (1990) Fundamentais of mbotics analysis and conml, FVentice Hall [79] Semple, J. G and Kneebone, G T. (1953) Algebraic Projective Geometry, Clarendon

Press [80] Shan, J. (1996) An aigorithm for object reconstruction without intenor orientation,

ISPRS J. Photogrummetry and Remote Sensing, n5 1, pp299-307 [8 11 ShariatJI. and Price, ICE. (1990) Motion estimation with more than 2 -es, IEEE

Trarts. Pattern Analysis and Machine Intelligence. v12, pp 417434 [82] Shaman, R. (1998) Visual Servoing with independently controlled cametas using a

learned invariant representation, Pmc. 3 9 IEEE Conj Decision and Control, pp 3263-3268

[83]Sharma, R. (1992) Active vision in robot navigation: Monitoring tirne-tocoIlision while tracking, Pmc. IEEURTJ Conj On Intelligent Robots and Systems. pp2203- 2208

[84]Shirai, Y. and houe, H. (1973) Guiding a robot by visual feedback in assembüng task, Pattern Recognition, v5, pp99-108

[85]Singh, R., Voyles, R. M. Littan, D. and Papawkdopobe, NP. (1998) Grasping =al objects using Wtual images, Proc. 3p ZEEE Con$ Decision and Conmt, pp3269- 3274

[86] Skaar, S. B. Brockman, W. H. and Hanson, R. (1987) Camera-space manipulation. Zn?. J. Robotics Research, v6, n4, pp20-32

[87] Skofteland, G and Himnger, G Computing position and orientation of a k f l y i n g polyhedron fmm 3D data, Pmc. IEEE Znr. Con$ Robotics Automation, pp150-155

[88] Sokolnikoff, 1. S. and RedheEer, R. M. (1966) Mathematics of Physics mrd Mo&m Engineehg USA, McGraw-HiIl

[89]Sonka, M. (1993) hnage hcessing Anassis, wtdMacfIirre Wsim? Chapmw & tfall -a

Computing [90]Spetsakis M. E. and Aloimonos J. (1990) Structure h m motion using line

conespondences, Int. J. Compuier Vision, v4, ppl7 1-1 83 [91] Spivak, M. (1970) A Comprehensive Introduction tu Diffe~ntial Geomeiry, Publish

or Perish [92] Stange, G Srinivasan, M. V. and Dalczynski, J. (1991) Rangfinder based on intensity

gradient measurement, Applied Optics, v30, n 13, pp 1695- 1700 [93]Sznaier, U. M. Lamps, O. L. (1998) Control issues in active vision: open problems

and some answea, Proc. 3p lEEE Con$ Decision and Contml, pp3238-3244 [94]Tendick, F. Voichick, J. Tharp. G and Stark, L. (1991) A supervisory telerobotic

conwl system using model-based vision feedback, Proc. IEEE In?. Con& Robotics Automation, pp2280-2285

[95]Tsai, R. Y. (1986) An efficient and accurate camera calibration technique for 3D machine vision, Pmc. IEEE Computer Sociey Con$ Computer Ksion and Pattern Recognition? pp364-374

(961Ullrnan (1979) The interpretation of structure from motion, Pmc. Royal Society London, R203, pp405-426

[97] Venkatesan, S. and Archibad, C. (1990) Realtime tracking in five degrees of freedom using two wrist-mounted laser range finders, Pmc. IEEE Con& Robotics Automatz'on, pp200Q-20 10

[98]Vemon, D. and TistareIIi, M. (1990) Using camera motion to e s h a t e range for robotic parts manipulation, IEEE Trans. Robotics Automation, v7, n5, pp509-521

[99]Visinsky, M. L. Cavailaro, J. R. and Walker, 1. D. (1993) Dynamic sensor-based fault detection for robots, Pmceedings of SPIE, ~2057, pp3 85-396

[IO01 Visinsky, M. L. Cavallaro, J. R. and Walker, 1. D. (1995) A dynamic fault tolerance ffamework for remote robots, IEEE Trm. Robotics Automution, v l 1 , n4, pp477-490

(1011 Walker, 1. D. and Cavallaro, J. R. (1996) The use of fault trees for the design of robots for hazardous environments, Annual Reliabiliiy rmd maintainability Symposium: 1996 pmceedings, pp229-235

[102] Waxman, A. M. Kangar-Pan, B. and Subbaraw, M. (1987) closed f o m solutions to image-flow quations for 3D stmctu~ and motion, I n t J. Cornputer V i s h , vl, pp239-258

[103] William, T. D. (1980) Depth from camera motion in a na1 world scene, IEEE Trans. P~anem Annlysis and Machine Intelligence, v2, n6, pp5 11-5 16

[ 1041 Wolf, P. R. (1 983) Elements of Phrogrmetry (;yld edition), McGraw-Hill [IOSI Wu, J. and Wohn, K. (1991) On the deformation of image intensity and zero-

crossing contours under motion. CVGIP: Image Understanding, v53, pp66-75 [106] Wijesoma, S. W. WoIfe, D. F. H. and Richards, R. J. (1993) Eye-to-hand

coordination for vision-guide robot control applications, Znt. 3. Robotics Research, v12, nl, pp65-77

[107] Zhang. D. B. Van Gool, L. and Oosterlinck (1990) Stochastic predictive robot tracking systems with dynamic visual feedback, Pmc IEEE Int. Robotics Automation, pp6 10-6 15

Web Addresses

[108] htt~://www.nl ue.umd.edu/-~diana/surve~. h , University of Maryland, College Park

[IO91 htt~://cui .unige.chJ-startchi/PhD/FORMAT m c h a ~ t e d 0 3 Geomeûichvar iants/GeornenicInv~ant~.hbnl#41223, University of Geneva

[110] htt~:llwww.cs.csustan.edu/-~~cISDSUNiewindf, California State University, S tanislaus

[ I l l ] h~:llwww.enein.urnich.ed~ou~/ctm/PID/PID.htm, University of Michigan University, Ann Arbor