SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Guided Robotics

and Applications in Industry and Medicine

Matthias Rüther

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Contents

Robotics in General

Industrial Robotics

Medical Robotics

What can Computer Vision do for Robotics?

Vision Sensors

Issues / Problems

Visual Servoing

Application Examples

Summary

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Robotics

What is a robot?"A reprogrammable, multifunctional manipulator designed to move

material, parts, tools, or specialized devices through various programmed motions for the performance of a variety of tasks"

Robot Institute of America, 1979

Industrial– Mostly automatic manipulation of rigid parts with well-known shape in a

specially prepared environment.

Medical– Mostly semi-automatic manipulation of deformable objects in a

naturally created, space limited environment.

Field Robotics– Autonomous control and navigation of a mobile vehicle in an arbitrary

environment.

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Robot vs Human

Robot Advantages:

– Strength

– Accuracy

– Speed

– Does not tire

– Does repetitive tasks

– Can Measure

Human advantages:

– Intelligence

– Flexibility

– Adaptability

– Skill

– Can Learn

– Can Estimate

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Requirements:

– Accuracy– Tool Quality– Robustness– Strength– Speed – Price Production Cost– Maintenance

Industrial Robot

Production Quality

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Medical (Surgical) Robot

Requirements

– Safety– Accuracy– Reliability– Tool Quality– Price– Maintenance– Man-Machine Interface

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

What can Computer Vision do for Robotics?

Accurate Robot-Object Positioning

Keeping Relative Position under Movement

Visualization / Teaching / Telerobotics

Performing measurements

Object Recognition

Registration

Visual Servoing

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Single Perspective Camera

Multiple Perspective Cameras (e.g. Stereo Camera Pair)

Laser Scanner

Omnidirectional Camera

Structured Light Sensor

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Single Perspective Camera

XPx x43

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Multiple Perspective Cameras (e.g. Stereo Camera Pair)

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Multiple Perspective Cameras (e.g. Stereo Camera Pair)

0Fxx'T Fxl'

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Laser Scanner

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Laser Scanner

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Omnidirectional Camera

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Omnidirectional Camera

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Vision Sensors

Structured Light Sensor

                                                                             

                                                             

                                                         

Figures from PRIP, TU Vienna

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Issues/Problems of Vision Guided Robotics

Measurement Frequency

Measurement Uncertainty

Occlusion, Camera Positioning

Sensor dimensions

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Vision System operates in a closed control loop.

Better Accuracy than „Look and Move“ systems

Figures from S.Hutchinson: A Tutorial on Visual Servo Control

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Example: Maintaining relative Object Position

Figures from P. Wunsch and G. Hirzinger. Real-Time Visual Tracking of 3-D Objects with Dynamic Handling of Occlusion

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Camera Configurations:

End-Effector Mounted Fixed

Figures from S.Hutchinson: A Tutorial on Visual Servo Control

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Servoing Architectures

Figures from S.Hutchinson: A Tutorial on Visual Servo Control

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Position-based and Image Based control

– Position based: • Alignment in target coordinate system• The 3D structure of the target is rconstructed• The end-effector is tracked• Sensitive to calibration errors• Sensitive to reconstruction errors

– Image based:• Alignment in image coordinates• No explicit reconstruction necessary• Insensitive to calibration errors• Only special problems solvable• Depends on initial pose• Depends on selected features

target

End-effector

Image of target

Image of end effector

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

EOL and ECL control

– EOL: endpoint open-loop; only the target is observed by the camera

– ECL: endpoint closed-loop; target as well as end-effector are observed by the camera

EOL ECL

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Position Based Algorithm:1. Estimation of relative pose

2. Computation of error between current pose and target pose

3. Movement of robot

Example: point alignment

p1

p2

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Position based point alignment

Goal: bring e to 0 by moving p1

e = |p2m – p1m|

u = k*(p2m – p1m)

pxm is subject to the following measurement errors: sensor position, sensor calibration, sensor measurement error

pxm is independent of the following errors: end effector position, target position

p1m p2m

d

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing Image based point alignment

Goal: bring e to 0 by moving p1

e = |u1m – v1m| + |u2m – v2m|

uxm, vxm is subject only to sensor measurement error

uxm, vxm is independent of the following measurement errors: sensor position, end effector position, sensor calibration, target position

p1 p2

c1 c2

u1

u2

v1 v2

d1d2

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Example Laparoscopy

Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Visual Servoing

Example Laparoscopy

Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Registration

Registration of CAD models to scene features:

Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Registration

Registration of CAD models to scene features:

Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Tracking

Instrument tracking in laparoscopy

Figures from Wei: A Real-time Visual Servoing System for Laparoscopic Surgery

SSIP 2004 Graz © Inst. For Computer Graphics and Vision, 2004

Summary

Computer Vision provides accurate and versatile measurements for robotic manipulators

With current general purpose hardware, depth and pose measurements can be performed in real time

In industrial robotics, vision systems are deployed in a fully automated way.

In medicine, computer vision can make more intelligent „surgical assistants“ possible.