Robots that see like humans

Robots that see likeRobots that see likehumanshumans

Real Time Stereo Vision With ZED Camera on Nvidia Jetson TX1

Walter [email protected]

Summary

21/05/2016Officine Robotiche 2016

2

Walt

er

L uce

tti –

myzh

ar@

rob

ot-

hom

e.it

Live Demo

Conclusions

Introduction

Stereo “Binocular” Vision

Robotic Stereo Vision

“Simplified” Stereo Vision

Pin Hole Camera model

Stereo Vision camera model

Stereo Vision process

A Robotic Stereo Vision System

Nvidia Jetson TX1

Stereolabs ZED camera

Stereolabs ZED SDK

Stereolabs ZED engine

Simple Code example

Introduction3

Introduction4

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Can a robot replicate the complicatedhuman binocular vision system?


Stereo “Binocular” Vision 5

[from Wikipedia: Binocular vision]

Vision made with TWO “EYES”

It gives a “creature” a spare eye in case one is damaged

It gives a wider field of view

It can give stereopsis in which binocular disparity (or parallax) provided by the two eyes' different positions on the head gives precise depth perceptionIt allows a creature to see more of, or all of, an object behind an obstacle

It gives binocular summation in which the ability to detect faint objects is enhancedW

alte

r L u

cett

i – m

yzh

ar@

rob

ot -

hom

e.it


Robotic Stereo Vision6

How to replicate Human Stereo Vision? Two synchronized camera sensors

A solid mechanical structure

A powerful elaboration system

and if we want to be more precise, but we want to complicate the system:

Two motors to replicate vergence capability

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it


“Simplified” Stereo Vision7

“Simplified” Stereo Vision8

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

A complete Stereo Vision system has vergence capability, but is too complicated

We will study a Simplified Stereo Vision system where cameras cannot rotate


9

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Pin hole camera model

Camera Plane

Optical Axis

Image coordinatesWorld coordinates

World to image

Image to world

Without “Z” we cannot estimate the real dimension of an object using a single camera!

“f” can be estimatedcalibrating thecamera system

This is a simplifiedcamera model thatdoes not take care

of Optical Distorsion

andSensor

Imperfections


u=f ∗x

z

v=f ∗y

z

x=u∗z

f

y=v∗z

f

10

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Stereo Vision model

Very important parameter:● Baseline: b

From triangles similarity:

Very important value:● Disparity: D = u

l - u

r

From disparity we can calculate howmuch a point is far from a camera:THE “Z” VALUE!!!


fZ P

=ul

X P

fZ P

=−ur

b−X P

Z P=b∗ f

u l−ur

Calculate “Disparity” (D) for each point of the image

11

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Stereo Vision process

Calculate the depth (Z) ofeach point of the image

Calculate X and Y of eachpoint of the image

Algorithm

MATCHING

REQUIRES A LOT OF

COMPUTATIONALPOWER

D = ul - u

r


12

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Stereo Vision summary

Allows to calculate real world coordinate of an objectRequires a solid mechanical structureRequires a precise calibration of the optical systemRequires a lot of computational power…Is more complicated than what seen until now


A Robotic Stereo Vision system13

14

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

A Robotic Stereo Vision System

Nvidia Jetson TX1 www.nvidia.com/object/jetson-embedded-

systems.html

Stereolabs ZEDwww.stereolabs.com


15

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Nvidia Jetson TK1

Main features

Nvidia Tegra X1 SOC

NVIDIA Maxwell GPU with 256 CUDA cores

NVIDIA 4-Plus-1™ 2.3 Ghz quad-core ARM® Cortex™-A15 CPU

4 GB LPDDR4 Memory

16 GB eMMC 5.1 Flash Storage

1 full-size SD/MMC connector

1 full-size HDMI port

1 USB 2.0 port, micro AB

1 USB 3.0 port, A

1 RTL8111GS Realtek GigE LAN

GPIO, SPI, I2C, UART, CAN

WiFi & Bluetooth connection

...

Native OpenCV support with GPU module for Real Time computer vision

ROS enabled

Ubuntu powered

and more...

Android ...


16

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Stereolabs ZED cameraMain features

Auto calibrating and auto sync

USB 3.0

High Res at high FPS2208x1242 15 Hz

1920x1080 30 Hz

1280x720 60 Hz

640x480 120 Hz

Simple and well documented C++ SDK

Automatic Depth Map generation

3D Range: 1 m to 20 m(12 cm baseline)

Outdoor and indoor

Jetson TX1 driver

ROS driver

OpenCV support


17

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Stereolabs ZED SDK

C++ Classes and APIs for Linux and Windows

2 main classes and 5 support classes

Main classes:

Camera: the main class to use the ZED camera

Mat: the class to store image data in GPU and CPU memory spaces

Support classes:

CamParameters: calibration params for the two sensors

StereoParameters: calibration params for stereo processing

resolution: dimensions of the images

float3Struct: 3D point

uchar3Struct: RGB pixel

Current version:

0.9.3-beta for Linux and Windows, 0.9.3-alpha for Jetson TX1


18

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Stereolabs ZED engine

Stereo results are availablein GPU memory

Direct processing by CUDAor download to CPU memoryfor CPU processing


19

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Stereolabs ZED tools

ZED Explorer ZED Depth Viewer


20

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Simple code example

//ZED Includes#include <zed/Camera.hpp>

int main(int argc, char **argv){ sl::zed::SENSING_MODE dm_type = sl::zed::RAW; sl::zed::Camera* zed;

// Camera object creation zed = new sl::zed::Camera(sl::zed::HD2K); // We can use different resolutions: // HD2K HD1080 HD720 VGA

// Sizes of the images int width = zed->getImageSize().width; int height = zed->getImageSize().height;

// Initialization of the camera sl::zed::ERRCODE err = zed->init( sl::zed::MODE::PERFORMANCE );

if (err != sl::zed::SUCCESS) { delete zed; return 1; }

// Stereo processing zed->grab(dm_type);

// CPU Data sl::zed::Mat depth_cpu; sl::zed::Mat left_cpu; sl::zed::Mat right_cpu;

// Data acquiring on CPU memory depth_cpu = zed->retrieveMeasure(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage(sl::SIDE::LEFT); right_cpu = zed->retrieveImage(sl::SIDE::RIGHT);

// GPU Data sl::zed::Mat depth_gpu; sl::zed::Mat left_gpu; sl::zed::Mat right_gpu;

// Data acquiring on GPU memory depth_cpu = zed->retrieveMeasure_gpu(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage_gpu(sl::SIDE::LEFT); right_cpu = zed->retrieveImage_gpu(sl::SIDE::RIGHT);

return 0;}


Live demo21

Conclusions22

23

Wal

ter

L uce

tti –

myzh

ar@

rob

ot -

hom

e.it

Conclusions

With Stereo VisionA robot can see obstaclesA robot can measure obstaclesA robot can localize obstacles

Stereo Vision requiresa lot of mathematical elaborations a lot of computational powera solid mechanical structure

so

Nvidia Jetson TX1 with Stereolabs ZED is a really amazing solution for Robotic Stereo Vision!


The End!

OR

THANK YOU FOR YOUR ATTENTION!

Walter Lucettihttp://myzharbot.robot-home.it

GitHub@myzhar

YoutubeMyzhar

Twitter@myzhar

Facebookrobothome

24


Technology

Robots that see like humans