Robotic Vision: Sensing, Localization and Controlranger.uta.edu/~gianluca/teaching/CSE4392-5369_F12/1_CSE4392-5369... · Robotic Vision: Sensing, Localization and Control ... (Scuola

CSE 4392/5369

Robotic Vision:Robotic Vision: Sensing, Sensing, Localization and ControlLocalization and Control

University of Texas at Arlington

Dr. Gian Luca Mariottini, Ph.D.Department of Computer Science and Engineering

University of Texas at Arlington

WEB : http://ranger.uta.edu/~gianluca

Gian Luca Mariottini, Univ. Texas at Arlington

“ The most beautiful thing that we can experience is the Mystery. It is the true source of all art and all science.”

- Albert Einstein


Robotic Vision: Sensing...Robotic Vision: Sensing...

No. of objects; Area; Center;Object Orientation;


... Localization ...... Localization ...

No. of objects; Area; Center;Object Orientation;

Camera orientation w.r.t. Table;Distance to each object;


and Controland Control

FANUC robot vision system


ApplicationsApplicationsStair-Climbing & Descending for Rescue Operations


ApplicationsApplicationsPersonal and Health-care Robotics

Omnivision sensing 3-D object locations Robot control

Courtesy from Charlie Kemp, director of the Healthcare Robotics Center at Georgia Tech;


ApplicationsApplicationsMulti-robot Navigation

Team of Robots

Formation Control


ApplicationsApplicationsLaparoscopic SurgeryCourtesy from Univ. Strasbourg


ApplicationsApplications

Blue Team (U.Berkley, TA&M) Carnegie Mellon

Autonomous Navigation



NAO (Aldebaran Robotics) http://www.youtube.com/watch?v=HS2QfpGAfRU

Sensing, Cognition and Humanoid-Robot Control

http://www.youtube.com/watch?v=HS2QfpGAfRU


ApplicationsApplicationsSpace Exploration



MOVAID Project to aid disabled people (Scuola Superiore St.Anna, Pisa, ITALY)

PAM-AID (Trinity College, Ireland)

Health-care RObotics


ApplicationsApplicationsVisual ServoingCourtesy from SIRS Lab, University of Siena


ApplicationsApplicationsAutonomous Aerial-Vehicle Sensing and Navigation

http://www.youtube.com/watch?v=WoMrPKCpWXwhttp://www.youtube.com/watch?v=WoMrPKCpWXw

http://www.youtube.com/watch?v=WoMrPKCpWXw

http://www.youtube.com/watch?v=WoMrPKCpWXw


ApplicationsApplications3-D Dense Reconstruction with Commodity Cameras

Video Kinect Fusion


• Determine user’s location• Identify surroundings (parks, bench, trees, etc.)

• Represent objects with unique sounds• Audio waypoints (e.g. metasounds) to destination

path3D (top)

...

•MAP localization (RANSAC/LM)•N-views Structure-from-Motion

Goal:

System for Wearable and Audio Navigation @ GaTech



Am I a robot ? (Yes/No)Am I a robot ? (Yes/No)

A B C

D E F


Course TopicsCourse TopicsRobotics and Vision-sensing: Basics

Vision-based Localization

Probabilistic Robotics

- Introduction to Robotics;- The human eye and image formation;- Camera models;

– Pinhole, Stereo and Omnidirectional- Feature detection

– Edge, Corner, Color, SIFT- Feature tracking

- Reference frames and rigid body transformations- Full pinhole and stereo camera models- Camera Calibration- Camera Pose Estimation- Two-view geometry

- Position- and Image-based Visual Servoing- Epipole-based Visual Servoing

Vision-based Control

- Basics of probability reasoning in robotics- Bayesian filtering for localization


Class Schedule and Office HoursClass Schedule and Office Hours

Class Schedule: Mon. & Wed., 5:30-6:50pm in COBA 141

Office Hours: Mon. & Wed., 1:30-3:30pm

I am generally available before or after class and by appointment, as well as at the office hours scheduled above.

Since this is a special-topic course in robotics, and since each presented topic contains many exciting sub-fields, the interested students who want to know more about a specific problem are encouraged to schedule an appointment with Dr. Mariottini for additional material


Course PrerequisitesCourse Prerequisites

CSE 4392/5369 is self-contained and does not need special prerequisites.

MATLAB programming experience is required. If you are unfamiliar, please check this: http://www.math.ucsd.edu/~bdriver/21d-s99/matlab-primer.html.

During the course we will also make use of the MATLAB's Epipolar Geometry Toolbox, which can be efficiently used for creating vision-based robot simulations.

http://egt.dii.unisi.it/

http://egt.dii.unisi.it/#tth_sEc5


Course TextCourse Text

There is no required textbook for this course.

Suggested references:

B. Siciliano, L. Sciavicco, L. Villani, G. Oriolo "Robotics: Modelling, Planning and Control", 3rd Edition, Springer, 2009

E. Trucco, A. Verri "Introductory Techniques for 3-D Computer Vision", Prentice Hall, 1998

R. Hartley, A. Zisserman, "Multiple View Geometry", 2nd Edition, Cambridge Univ. Press 2004

Y. Ma, S. Soatto, J. Kosecka, S. Sastry "An Invitation to 3-D Vision. From Images to Geometric Models", Springer 2000

Roberto Cipolla, “Robot and Computer Vision Course”, Cambridge University, UK


Homeworks and Final Project (1)Homeworks and Final Project (1)Students will be graded based on homework assignments and a major course project.

Homeworks: theoretical questions, programming assignments (in MATLAB), etc. related to the

topics of the course. Dates will be announced in class.

Tentatively, course grades will based on the following:

Assignments % of final grade Percent Grade LetterHomeworks (total) 60 % 90 %-100 % ACourse Project 30 % 80 %-89.9 % B - Proj. Presentation 10 % 70 %-79.9 % C

60 %-69.9 % D<60 % F

Note 1: For students enrolled in CSE 5369 the homeworks will contain additional problems (not required for CSE 4392).

Note 2: All work turned in for grading must be the student's own work. (see also the course's webpage)

Note 3: MATLAB (with some toolboxes) is available also in NH 231


Homeworks and Final Project (2)Homeworks and Final Project (2)Course project: designing and implementing a particular algorithm from a list provided in class (check the webpage!)

Additional projects proposed by the students and related to the course topics can be added to the list.

The course project can be done either individually or in a team (~max 2 students).

Students will report their findings in a 8-10 page research report, and an in-class oral presentation (with Power Point slides) during last day of class.

I recommend using OpenCV (a C/C++ library by Intel) for the Image Processing part (LK tracker, Color tracker, Optical Flow, etc.)

Epipolar Geometry ToolboxEpipolar Geometry Toolboxhttp://egt.dii.unisi.it

Structure-from-Motion Multiple View Geometryfor Omnidirectional Cameras

epipole

• Texture mapping• Structure from motion from lines• Detailed manual • Many examples• and many others...


Intel's OpenCVIntel's OpenCV



List of Projects (1) List of Projects (1) Note that this list is temporary (e.g., you can propose new projects)

Project 1: Design a robotic cane that guides visually impaired in an indoor environment (e.g., indoor a corridor).

Project 2: Design an algorithm that allows a camera mounted on the robot to detect a person and follow him.

Project 3: Design a control algorithm to move a robot with your gestures


List of Projects (2)List of Projects (2)Note that this list is temporary (e.g., you can propose new projects)

Project 4: Create an omnidirectional camera and design an obstacle avoidance algorithm that uses it.

Project 5: Stair detection, localization and approach

Project 6: Vision-Based Robot Localization Sytems



Project 7: Augmented reality system for surgery

Project 8: Visual Hull Reconstruction system

Project 9: Building recognition with your cellphone



Project 10: Multi-camera calibration

Project 11:Time-of-Flight camera system

I will show more applications in the next classes...you'll find many fascinating topics!

Project 12: Cane Localization from on/off-the-shelf camera


What is a Robot ?What is a Robot ?


A more formal definition of RobotA more formal definition of RobotHuman being always looked for substitutes that would mimic their behaviours.

Great ambition: give life to human's artifacts (e.g.,Talus, the bronze giant forged by Hephaestus, protected the island of Crete)

“Robot” (from the czech “robota”=slave) was coined in 1920 by Karel Capek in the play Rossum's Universal Robots.

RIA Definition: A robot is a reprogrammable, multifunctional manipulator designed to move material, parts, tools, or specialized devices through variable programmable motions for the performance of a variety of tasks. [Robot Institute of America]

A more recent definition, defines robotics as the science studying the intelligent connection between perception and action

= PERCEPTION + COGNITION + CONTROL


Intelligent...action...perceptionIntelligent...action...perception

Actuation system: – Provides the capability to exert an action (locomotion or manipulation).

Sensory system: – “Proprioceptive sensors” = acquire data on the status of the robot.

(e.g., position, joint angles, etc.)– “Exteroceptive sensors” = acquire data on the external status of environment.

Control system: – It commands the execution of the action with respect to the goals.

Components of a robotic system:Mechanical system:

– Locomotion apparatus (wheels, tracks, legs,...)– Manipulation apparatus (mechanical arm, end-effector, artificial hand).


Am I a robot ? (Solutions)Am I a robot ? (Solutions)

A=Yes B=No C = No --> Yes

D=No E=Yes F=No

Documents

Robotic Vision: Sensing, Localization and Controlranger.uta.edu/~gianluca/teaching/CSE4392-5369_F12/1_CSE4392-5369... · Robotic Vision: Sensing, Localization and Control ... (Scuola