64-424 Intelligent Robotics - uni-hamburg.de · I Inspection during assembly I Mounting tasks I Welding, screwing, attaching, gluing, etc. 410. ... 9.1 Vision systems - Motivation

University of Hamburg Department of Informatics

9 Vision systems 64-424 Intelligent Robotics

Outline1. Introduction2. Fundamentals3. Rotation / Motion4. Force / Pressure5. Frame transformations6. Distance7. Scan processing8. State estimation9. Vision systems

MotivationApplication exampleCamera calibration

408


9.1 Vision systems - Motivation 64-424 Intelligent Robotics

Available vision systems

I Linear camera (e.g. barcode scanner)I Analog CCD camera (black/white)I Analog CCD color camera (1 chip or 3 chips)I High-Dynamic-Range CMOS cameraI Digital web camera (USB or Firewire)I Patterned/Structured light camera (Infrared, Laser, RGB)I Stereo vision systemsI Omnidirectional vision systems (catadioptrical systems)

409



Vision systems in industrial applications

I Object grasping tasksI Objects with predetermined position (e.g. production line)I Randomly positioned objects (e.g. ’bin-picking’)

I Object handling tasksI Cutting, tying, wrapping, sealing, etc.I Inspection during assembly

I Mounting tasksI Welding, screwing, attaching, gluing, etc.

410



Vision systems in industrial applications (cont.)

411



Vision systems in robotics

I Perception of objectsI Static: Recognition, searching, indexing, . . .I Dynamic: Tracking, manipulation, . . .

I Perception of humansI Face recognitionI Gaze trackingI Gesture recognitionI . . .

412



Vision systems in robotics (cont.)

Navigation and modeling:I Object recognition and localizationI Robot localization

I relativeI absoluteI in relation to various coordinate systems

I 3D reconstructionI Allocation of the environment

413



Vision systems in robotics (cont.)

Visual motion control:I Visual-Servoing

I Coarse and fine positioningI Tracking of movable objectsI Swinging, juggling, balancing, . . .

I Collision avoidanceI Based on the principle of optical flowI 3D-based distance measurement

I Coordination with other robots and/or humansI Intention recognitionI Motion estimation

414


9.2 Vision systems - Application example 64-424 Intelligent Robotics

Outline9. Vision systems


415



Coordinate frames of a manipulator

416



Manipulator - Object - Camera

417



Transformations

418



Transformations (cont.)

Points inside one coordinate frame can be transformed intoanother coordinate frame using suitable transformations:Z : Transformation from world coordinates to manipulator base

coordinatesT6: Complete kinematic transformation from the base of the

manipulator to the end of the manipulatorE : Transformation from the end of the manipulator to the gripperB: Transformation from world coordinates to object coordinatesG: Specification of the grip position and grip orientation using

object coordinates

419




If the robot grasps an object, the world coordinates of the grasppoint can be determined in two ways and results in:

ZT6E = BG

420




If the manipulator needs to be moved, the following equation needsto be solved:

T6 = Z�1BGE�1

In order to determine the location of the object after manipulation,one has to solve:

B = ZT6EG�1

421




A camera included in the system yields two additional matrices:C : Transformation of camera coordinates into world coordinates

(O�-line determination of the transformation through cameracalibration)

I: Transformation of grasp point coordinates into the cameracoordinate system(Grasp point is determined using image processing techniques)

422




The transformation P from the grasp point to the world coordinatesystem is:

P = I C

The camera to world transformation is determined through thefollowing equation:

C = I�1 P

423


9.3 Vision systems - Camera calibration 64-424 Intelligent Robotics

Outline9. Vision systems


424



Camera calibration

Camera calibration in the context of three-dimensional imageprocessing is the determination of the intrinsic and/or extrinsiccamera parameters

Intrinsic parametersInternal geometrical structure and optical features of the camera

Extrinsic parametersThree-dimensional position and orientation of the camera’scoordinate system in relation to a world coordinate system

425



Camera calibration (cont.)

What information does one get?I In order to reconstruct 3D information of the environment from

two or more images, it is necessary to know the relationbetween the coordinate systems of the 2D image and theobjects of the 3D environment

I The relation between 2D and 3D can be described using twotransformationsI Perspective projection (3D ! 2D)I Backprojection (2D ! 3D)

426




1. Perspective projection (3D point ! 2D point)I Knowing the camera projection matrix the 2D projection of a 3D

point can be determined (approximated)2. Backprojection (2D point ! 3D beam)

I If a 2D image point is known, there is a ray in 3D space on whichthe corresponding 3D point lies

I If there are two or more views of the 3D point, its coordinate canbe determined using triangulation

427




I The perspective projection is useful in order to reduce thesearch space during scene analysis

I The backprojection is helpful for deriving 3D information basedon features in 2D images

I These transformations can be used in various application fields:

I Automatic assemblyI 3D metrologyI Robot calibrationI TrackingI Trajectory analysisI . . .

428



Calibration techniques

I Camera calibration can be done on-line or o�-lineI Using a calibration object:

I Identification of the camera parametersI Determination of coordinate transformation between camera

coordinates and world coordinatesI Using self-calibration approachesI Using machine learning methods

429


9.3.1 Vision systems - Camera calibration - Pinhole camera model 64-424 Intelligent Robotics

Model of a camera without distortion

Pinhole camera without lens distortion

430



Model of a camera without distortion (cont.)

I (xw , yw , zw ): 3D world coordinate system with the origin Ow

I (x , y , z): 3D coordinate system of the camera with the originO (optical center)

I (X ,Y ): 2D image coordinate system with the origin O1

I f : Focal length of the camera

431



Transformation from world to camera coordinates

I Let P(xw , yw , zw ) be a point in the world coordinate systemI Its projection onto the image plane can be determined as

follows:2

4

xyz

3

5 = R

2

4

xwywzw

3

5+ t

having R =

2

4

r1 r2 r3r4 r5 r6r7 r8 r9

3

5 and t =

2

4

txtytz

3

5

I The parameters R and t are the extrinsic parameters

432



Projection of camera coordinates onto image coordinates

I Point P is projected onto the corresponding (ideal) imagecoordinate (u, v)

I Perspective projection with focal length f :

u = f xz v = f y

zI The image coordinate (X ,Y ) is calculated from (u, v) as

follows:X = suu Y = sv v

I The scaling factors su and sv are used to convert the imagecoordinates from meters to pixels

I su, sv and f are the intrinsic camera parameters

433



Projection of world coordinates on image coordinates

I Since only two independent intrinsic parameters exist, onedefines:

fx ⌘ fsu and fy ⌘ fsv

I These equations yield the distortion-free camera model:

Xf = fxr1xw + r2yw + r3zw + txr7xw + r8yw + r9zw + tz

Yf = fyr4xw + r5yw + r6zw + tyr7xw + r8yw + r9zw + tz

434



Pixel coordinates

I The coordinates (Cx ,Cy ) of the image center are subtractedfrom the image coordinates (Xf ,Yf ) determined duringperspective projection

I Due to the above, one has:

X = Xf � Cx

Y = Yf � Cy

435


9.3.2 Vision systems - Camera calibration - Basic concept of camera calibration 64-424 Intelligent Robotics

Main calibration parameters

The pinhole camera model contains the following calibrationparameters:I The three independent extrinsic parameters of RI The three independent extrinsic parameters of tI The intrinsic parameters fx , fy , Cx and Cy

436


9.3.2 Vision systems - Camera calibration - Basic concept of camera calibration 64-424 Intelligent Robotics

Calibration

I The main problem during camera calibration is theidentification of the unknown parameters of the camera model

I The determination of these parameters for the distortion-freecamera model yields the position of the camera in worldcoordinates

Calibration requires a set of m object points, which1. have known world coordinates {xw ,i , yw ,i , zw ,i}, i = 1, . . . ,m

with su�ciently accurate precision2. lie within the camera’s field of viewThese calibration points are detected in the camera image withtheir respective camera coordinates {Xi ,Yi}

437


9.3.3 Vision systems - Camera calibration - Camera model with lens distortion 64-424 Intelligent Robotics

Camera model with lens distortion

Real cameras and lenses produce a variety of imaging errors and donot satisfy constraints of the pinhole camera modelThe main error sources are:I Low spatial resolution due to low resolution of the camera

device being usedI Most (cheap) lenses are asymmetrical and generate distortionsI Imprecision during assembly (e.g. center of the image sensor

does not lie on the optical axis, sensor is not parallel to thelens, etc.)

I Timing errors between camera hardware and grabber hardware

438



Lens distortion

I Distortion by the lens system results in a changed position ofthe image pixels on the image plane

I The pinhole camera model is no longer su�cientI It is replaced by the following model:

u0 = u + Du(u, v)v 0 = v + Dv (u, v)

where u and v are the non-observable, distortion-free imagecoordinates, and u0 and v 0 the corresponding distortedcoordinates

439



Lens distortion (cont.)

440



Types of distortion

I There are two primary types of distortions:I radialI tangential

I Radial distortion causes an o�set of the ideal position inwards(barrel distortion) or outwards (pincushion distortion)Possible cause: Flawed radial bend of the lens

I Tangential distortion shifts the ideal position along a tangentialcurvePossible cause: Non-parallel sensor/lens

441



Radial distortion

Straight lines ! no distortion

442



Tangential distortion

Straight lines ! no distortion

443



Modeling of the lens distortion

I According to a literature (Weng et. al.), three kinds ofdistortion are distinguished:

1. Radial distortion2. Decentering distortion3. Thin prism distortion

I Decentering distortion and thin prism distortion are both radialand tangential

I In the case of decentering distortion, optical centers of thelenses are not colinear

444



Model: Lens distortion

Lens distortion

Dur = ku(u2 + v 2) + O[(u, v)5]

Dvr = kv(u2 + v 2) + O[(u, v)5]

with k being the first radial distortion coe�cient and (u2 + v2)being the squared radius

445



Simplified model

Since radial lens distortion is the dominating e�ect, the followingequation system can be used as simplified camera model:

Simplified camera model with distortion:

u0 = u(1 + k 0r 02)v 0 = v(1 + k 0r 02)

with r 02 = u2 + v2

446



Radial distortion coe�cient

Since u and v are unknown, they are replaced by the measurableimage coordinates X and Y and one has

r 02 = (X/su)2 + (Y /sv )

2

and withµ ⌘ fy

fx=

svsu

one getsr2 ⌘ µ2X 2 + Y 2

447



Model for small radial distortions

With the previously mentioned modifications, one gets thefollowing camera model for small radial distortions

X (1 + kr2) ⇠= fxr1xw + r2yw + r3zw + txr7xw + r8yw + r9zw + tz

,

Y (1 + kr2) ⇠= fyr4xw + r5yw + r6zw + tyr7xw + r8yw + r9zw + tz

448

Documents

64-424 Intelligent Robotics - uni-hamburg.de · I Inspection during assembly I Mounting tasks I Welding, screwing, attaching, gluing, etc. 410. ... 9.1 Vision systems - Motivation