Jryde Robotics Pr

7/28/2019 Jryde Robotics Pr

1/24

PR in

Robotics1/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Pattern Recognition in Mobile Robotics

CSE 455/555 Introduction to Pattern Recognition

Dr. Julian Ryde

SUNY at Buffalo

January 21, 2011


2/24

PR in

Robotics2/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Abstract/Lecture Overview

A more realistic set of motivating application examplesbased in Mobile Robotics

Motivations for mobile roboticsChallenges of mobile roboticsExamples involving pattern recognitionNot going to talk about Industrial/static robotics

Even thought there are many applications of patternrecognition in this area of robotics


3/24

PR in

Robotics3/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

What is Robotics?

DenitionsIntelligent connection of perception to action

Perception - Acquisition and interpretation of sensordataIntelligence?

Decision makingLearning

Actuation - Effecting changes in the physical world

LocomotionManipulationRoles of pattern recognition in robotics

Pattern recognition also has a role to play in intelligencePattern recognition forms part of the interpretation


4/24

PR in

Robotics4/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Robotics Areas

StaticIndustrial roboticsControlled environmentsMore commercially successful

Mobile

Indoor RoboticsPersonal roboticsOfce robots

Remote telepresenceDomestic robots

Roomba - most successful consumer mobile robot?

Field RoboticsMiningMilitary

Health careHome care for the elderly (Especially Japan)

Wherever it is Dirty, Dull or Dangerous


5/24

PR in

Robotics5/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Willow Garage

Long term funded company that does not need to makea prot

Adheres to the open source development modelRobot Operating System (ROS)OpenCV

Computer VisionMachine Learning algorithms

PR2 RobotPR2 Beta Overview Video
http://www.youtube.com/?v=UyLq4lfBsI0http://www.youtube.com/?v=UyLq4lfBsI0


6/24

PR in

Robotics6/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Sensors

Internal sensorsGlobal position system - GPSInertial Measurement Unit - IMU

Vision basedStandard Cameras

OmnidirectionalRange sensors

Time of ightTriangulation based

Structured light

Stereo visionUltrasonic

KinectStructured light 3D depth sensorReturns RGBD - Red, Green, Blue and DepthInexpensive at 150$


7/24

PR in

Robotics7/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Pattern Recognition in 3D Data versus Pictures

More difcult for humansMore suitable for computers?

3D data is more independent of observer positionMany objects are rigid or articulated so 3D shape is

important and invariantFor features invariant is goodimplies reliable/repeatable extraction

Camera images susceptible toillumination changesloss of information in conversion from 3D to 2D


8/24

PR in

Robotics8/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Considerations specic to mobile robotics

AutonomyThere can be no guarantee of continuous connection.Therefore a certain level of autonomy is required forreliable continuous operation.

Power consumptionRestricted processing power

Especially for indoor and smaller robots that run onbatteries

Real-time constraintsPedestrian recognition has to be fast and reliable forautomotive applicationsBatch algorithms not really suitable

For long term operation robot ultimately has to processdata as fast as it receives it.


9/24

PR in

Robotics9/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Localisation and Mapping

LocalisationFor most applications need to have an awareness oflocationAugment the robots workspace with features

Possible but expensive for factories and other controlledenvironments

MappingLearning a model of the operatingworkspace/environmentAids localisationAids path planning

Simultaneous Localisation and Mapping (SLAM)The process of both inferring location from a map whilstsimultaneously updating the map with new observedinformation.


10/24

PR in

Robotics10/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Simultaneous Localisation and Mapping(SLAM)

SLAM is vital for indoor robotsInterestingly not used for the iRobot roomba

Outside not as necessary where GPS coverage isavailableVarious avours of SLAM

MetricTopological

Matching2D/3D scan matchingImage


11/24

PR in

Robotics11/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Loop Closing

Closing of an open loop in a large (80m by 25m) cyclicenvironment from Konolige and Gutmann (1999)Improving map accuracy through loop closing

Have I been here before?Recognising previously visited locations


12/24

PR in

Robotics12/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Scan matching

Two scans taken from different posesFind features present in each scan

Determine the correspondence between featuresUse this information to align the scansThis gives the pose change between scansMuch research done with 2D range scans

Example with 3D scans or an indoor environmentScan matching Ilustration
http://jcr/csiro/introseminar/presentation/msmpeg4v2/scanmatching.avihttp://jcr/csiro/introseminar/presentation/msmpeg4v2/scanmatching.avi


13/24

PR in

Robotics13/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

3D mapping indoors

3D map y around video
http://jcr/csiro/introseminar/presentation/msmpeg4v2/flyaround-lowres.avihttp://jcr/csiro/introseminar/presentation/msmpeg4v2/flyaround-lowres.avi


14/24

PR in

Robotics14/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Probabilistic robotics

Return not only an answer but some idea of theprobability distribution

For example the pose distribution can be multimodalMany ways of representing these probablity densityfunctionsThus you have some idea of how likely the result is tobe correct

Algorithms in robotics need to be robustRobust statistics e.g. median

contrast with sufcient statisticsInsensitive to outliers

Sufcient statistics e.g. meanMathematically analyticComputational convenientSensitive to outliers


15/24

PR in

Robotics15/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

DARPA Grand and Urban Challenges

Grand Challenge$2 million Prize awarded to Stanford Racing Team132 mile desert course in just under 7 hours

Urban challengeFollow on from the easier grand challenge

Autonomous vehicle drive through an urbanenvironmentDrive in trafcManeuvers merging, passing, parking, negotiatingintersections

Featured the rst autonomous car crashReliable perception and interpretation vitalFinal highlights

DARPA Urban challenge highlights videoKey Enablers

Velodyne 3D laser scannerNear to far learning
http://www.darpa.mil/grandchallenge/video/DARPA_highlight_preview3_low.wmvhttp://www.darpa.mil/grandchallenge/video/DARPA_highlight_preview3_low.wmv


16/24

PR in

Robotics16/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Learning for mobile autonomous systems

Pattern recognition enables systems to learn andgeneralise from new data.Interaction with humans provides plenty of opportunityfor supervised learning.Amazons mechanical turk

Humans working for the robots!DARPA urban challenge videos

near to far learning in images e.g. using laser data andcorresponding vision results to train and so classify far

image pixels.Self supervised learning

Training data might arrives at a later time for somecasesTraining data provided by others sensors

e.g. Near to far learning


17/24

PR in

Robotics17/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Near to Far Learning

Image and research from Raia HadsellOnline machine learning

Either laser or stereo provides reliable near range 5mclassication (driveable or not based on smoothness)Then used as training data for appearance based

classier for pixels beyond 5m


18/24

PR in

Robotics18/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Solutions in Perception Challenge

GoalRigid objects recognized and their 6DOF posedetermined with the Kinect sensor

Kinect sensorBest solutions likely to combine conventional imageinformation with depth informationIdeal opportunity to put into practice what you learn inthis course


19/24

PR inRobotics

19/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Field robotics

Work undertaken at the Australian CommonwealthScientic and Industrial Research Organisation(CSIRO)Autonomous earth moving
http://home/jryde/demo/bobcatdemo150210.avihttp://home/jryde/demo/bobcatdemo150210.avi


20/24

PR inRobotics

20/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Autonomous Skid Steer loader (Bobcat)

Earth moving taskMove soil material from one location to another

Ultimate goal is the design the terrain you want in aCAD package and have it replicated in the physicalworld by autonomous excavation and earth movingmachines


21/24

PR inRobotics

21/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Reliable Human/Novelty Detection

Learn the operational environmentHighlight those areas that have changedEssential for safety reasons

If unknown objects detected or the environment hasbeen tampered with need to potentially stop operation

Handle noise, how much variation constitutes change?

Some active research questions related to


22/24

PR inRobotics

22/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Some active research questions related toPattern Recognition

SLAMMonocular camera based visual SLAMSLAM with RGBD sensors

Correspondence problem

Finding corresponding points in multiple imagesFor stereo vision

Object recognition in 3D data3D shape based retrieve and lookupView and occlusion invariant hashingLocality sensitive hashing

Object recognition from a moving camera sensorDifferent to object recognition in random imagesBackground subtraction for moving cameras


23/24

PR inRobotics

23/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Recap

What is (mobile) robotics?Example applications involving pattern recognition

LocalisationMapping

Scan matchingLoop closing

Terrain classication for path planningNear to far learningSelf supervised machine learning

Field roboticsAutonomous Earth moving

Current work in the research communitySolutions in Perception Challenge


24/24

PR inRobotics

24/24

Dr. JulianRyde

Introduction

Challenges

ExampleApplications

Summary

Questions for consideration?

What is holding back mobile robotics?Was the lack of low cost depth sensors, but now withthe Kinect?Low cost manipulator platforms? The PR2 is $400,000

Motivated by the recent debut of the Kinect how canexisting computer vision pattern recognition algorithmsbe applied to RGBD data?

Documents

Jryde Robotics Pr