From Vision to Actions towards adaptive & autonomous robots

Jürgen ‘Juxi’ Leitner i'stituto dalle molle di studi sull’intelligenza artificiale

idsia / supsi / usi

humanoidiCub

projectIM-CLeVeR

http://robotics.idsia.ch/im-clever/

integration

http://robotics.idsia.ch/

overview

a novel way of object segmentation (CGPIP)

learning and teaching spatial perception

integration action-perception sidereactive reaching/graspingimproving perception with (inter-)actions

!

perceptionvisual

thanks to G. Metta and IIT for this picture

object manipulationtowards learning

http://robotics.idsia.ch/

!

objectsdetecting

!

objectsdetecting

Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013

+ min dilate avg INP INP INP diff

using building blocksOpenCV

+ min dilate avg INP INP INP thresh+ min dilate avg INP INP INP blur+ min dilate avg INP INP INP normalize+ min dilate avg INP INP INP input

icVision

! icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); ! icImage* out = node49->threshold(81.532f); return out; }

framework

embedding domain knowledge

+ min dilate avg INP INP INP OpenCV functions

full images

cartesian genetic programming

+ min dilate avg INP INP INP

Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013

learningapproach

[Matthews, 1975]

TP… true positive, TN .. true negative!FP… false positive, FN .. false negative

learningapproach

detection

! icImage GreenTeaBoxDetector::runFilter() { icImage node0 = InputImages[6]; icImage node1 = InputImages[1]; icImage node2 = node0.absdiff(node1); icImage node5 = node2.SmoothBilateral(11); icImage node12 = InputImages[0]; icImage node16 = node12.Sqrt(); icImage node33 = node16.erode(6); icImage node34 = node33.log(); icImage node36 = node34.min(node5); icImage node49 = node36.Normalize(); ! //cleanup ... icImage out = node49.threshold(230.7218f); return out; }

detect

detect

detection

approachcgp

[Leitner et al, iSAIRAS 2012]

visualhand detection

[Leitner et al, CEC 2013]

handsdetecting

[Leitner et al, CEC 2013]

approachsupervised learning

BUT

clusteringfeature

saliencymap

Autonomous Approach

[Leitner et al, ICDL/EpiRob 2012]

presegmentation

resultscomparing

[Leitner et al, ICDL/EpiRob 2012]

features

CGP-IP

object manipulationtowards learning

http://robotics.idsia.ch/

localisation approachescurrent

[P2,1] * F * [P1,1]' = 0

!

learningspatial perception

trainingset

9DOF!

iCubpositio

n in the frame!

2/6 per eye

Carte

sian!

Coor

dinate

s

!

transferringspatial perception

[Leitner et al, IROS 2012]

setuplearning

trainingset

9DOF!

iCubpositio

n in the frame!

2/6 per eye

Carte

sian!

Coor

dinate

s

.

.

.~1000

spatial perception neural network

.!.!.

9DO

F!iC

ubpo

sitio

n in

the

imag

e!2/

6 pe

r eye

Cart

esian!

Coor

dina

tes

!fu

lly c

onne

cted!

!fu

lly c

onne

cted!

.!.!.

results ANN

-700

-600

-500

-400

-300

-200

-100

0

100

200

300

400

0 200 400 600 800 1000

Y (

mm

)

Sample Index

PredictedExpected

100

150

200

250

300

350

400

450

500

550

600

0 200 400 600 800 1000

X (

mm

)

Sample Index

-1300

-1200

-1100

-1000

-900

-800

-700

-600

-500

-400

-300

-200

0 200 400 600 800 1000

Z (

mm

)

Sample Index

-700

-600

-500

-400

-300

-200

-100

0

100

200

300

400

0 200 400 600 800 1000

Y (

mm

)

Sample Index

PredictedExpected

100

150

200

250

300

350

400

450

500

550

600

0 200 400 600 800 1000

X (

mm

)

Sample Index

-1300

-1200

-1100

-1000

-900

-800

-700

-600

-500

-400

-300

-200

0 200 400 600 800 1000

Z (

mm

)

Sample Index

[Leitner et al, IJCNN 2013]

results

[Leitner et al, IJARS 2012]

x"="17.81"−"0.0191"v1"+"0.1527"v4"+"0.1378"v7"+"0.0111"v10""""""−"0.0296"v11"−"0.1207"v12"!

y"="1.1242"+"0.1296"v10"+"0.1156"v8"+"0.0170"v0

GPv0 … left camera x coordinate,v1 … left camera y coordinate,

v2 … right camera x coordinate,v3 … right camera y coordinate,

v4-6 … neckv7-9 … eyes v10-12 … torso

simplified caseon the table

object manipulationtowards learning

http://robotics.idsia.ch/

MoBeE[Frank, Leitner et al., ICINCO, 2012]

MoBeEframework [Frank, Leitner et al., ICINCO, 2012]

[Frank, Leitner et al., ICINCO, 2012]

motion

robot[Stollenga, Leitner et al, IROS 2013]

generationmotion

Shak

ey 2

013

Win

ner

MoBeEv2[Frank, Leitner et al. 2012, 2013]

hand/armop-space forcing

CSWorld

CSHand

CSR/CSL

[Leitner et al, in prep]

-

repertoireaction

[Leitner et al, in prep]

object manipulationtowards learning

http://robotics.idsia.ch/

operationtele

hand/armop-space forcing

CSWorld

CSHand

CSR/CSL

[Leitner et al, in prep]

- CSHand’

teleoperation

coordinationhand-eye

model

http://robotics.idsia.ch/

manipulation for improved perception

manipulation actions

extracting information

improveddetection

detection

! icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); ! icImage* out = node49->threshold(81.532f); return out; }

detection

! icImage* BlueCupFilter::runFilter() { icImage* node0 = InputImages[4].Exp(); icImage* node5 = InputImages[0]; icImage* node16 = node0->Gabor(-8,14,1,13); icImage* node17 = InputImages[4]->LocalAvg(6); icImage* node18 = node16->Laplace(5); icImage* node19 = node5->Sobel(13,9); icImage* node24 = node17->Erode(5); icImage* node28 = node19->Min(node18); icImage* node29 = node28->Min(node24); icImage* node41 = node29->LocalAvg(7); icImage* node49 = node41->LocalMax(7); ! icImage* out = node49->threshold(68.032f); return out; }

resulting

conclusions

a novel way of object segmentation

learning and teaching spatial perception

integration action-perception sidereactive reaching/graspingimproving perception with (inter-)actions

S. Harding!M. Frank!

A. Förster!M. Stollenga!

L. Pape!

http://dilbert.com/strips/comic/2013-10-24/

thanks to

for listeningthank you

juxi@idsia.ch !http://Juxi.net/projects

1/2referencesReactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub. Jürgen Leitner, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. ICINCO 2014. !Improving Robot Vision Models for Object Detection Through Interaction. Jürgen Leitner, Alexander Förster, Jürgen Schmidhuber. WCCI - IJCNN 2014. !Teleoperation of a 7 DOF Humanoid Robot Arm Using Human Arm Accelerations and EMG Signals.. J Leitner, M Luciw, A Förster, J Schmidhuber. iSAIRAS 2014. !Curiosity driven reinforcement learning for motion planning on humanoids. Mikhail Frank, Jürgen Leitner, Alexander Förster, Jürgen Schmidhuber. Frontiers in Neurorobotics, 7:25, 2014. !Task-Relevant Roadmaps: A Framework for Humanoid Motion Planning. M Stollenga, L Pape, M Frank, J Leitner, A Förster, J Schmidhuber. IROS 2013. !ALife in Humanoids: Developing a Framework to Employ Artificial Life Techniques for High-Level Perception and Cognition Tasks on Humanoid Robots. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. Alife WS @ ECAL 2013. !Artificial Neural Networks For Spatial Perception: Towards Visual Object Localisation in Humanoid Robots. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. IJCNN 2013. !Learning Visual Object Detection and Localisation Using icVision. Jürgen Leitner, Simon Harding, Pramod Chandrashekhariah, Mikhail Frank, Alexander Förster, Jochen Triesch, Jürgen Schmidhuber. Biologically Inspired Cognitive Architectures, Vol. 5, 2013. !Humanoid Learns to Detect Its Own Hands. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. CEC 2013. !Cartesian Genetic Programming for Image Processing (CGP-IP). Simon Harding, Jürgen Leitner, Jürgen Schmidhuber. In Genetic Programming Theory and Practice X, pp 31-44. ISBN: 978-1-4614-6845-5. Springer, 2013.

http://Juxi.net/papers

referencesCartesian Genetic Programming for Image Processing and Robot Vision. J. Leitner, S. Harding, A. Förster, J. Schmidhuber. IEEE RAS Summer School on "Robot Vision and Applications". Santiago, Chile. December 2012. !Learning Spatial Object Localization from Vision on a Humanoid Robot. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. International Journal of Advanced Robotic Systems, Vol. 9, 2012. !Autonomous Learning Of Robust Visual Object Detection And Identification On A Humanoid. J. Leitner, P. Chandrashekhariah, S. Harding, M. Frank, G. Spina, A. Förster, J. Triesch, J. Schmidhuber. ICDL/EpiRob 2012. Paper of Excellence Award !An Integrated, Modular Framework for Computer Vision and Cognitive Robotics Research (icVision). Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. BICA 2012. !Transferring Spatial Perception Between Robots Operating In A Shared Workspace. J Leitner, S Harding, M Frank, A Förster, J Schmidhuber. IROS 2012. !Towards Spatial Perception: Learning to Locate Objects From Vision. J Leitner, S Harding, M Frank, A Förster, J Schmidhuber. RobotDoC-PhD 2012. !Mars Terrain Image Classification using Cartesian Genetic Programming. J Leitner, S Harding, A Förster, J Schmidhuber. i-SAIRAS 2012. !The Modular Behavioral Environment for Humanoids & other Robots (MoBeE). Mikhail Frank, Jürgen Leitner, Marijn Stollenga, Gregor Kaufmann, Simon Harding, Alexander Förster, Jürgen Schmidhuber. ICINCO 2012. !MT-CGP: Mixed Type Cartesian Genetic Programming. S. Harding, V. Graziano, J. Leitner, Jürgen Schmidhuber. GECCO 2012. !icVision: A Modular Vision System for Cognitive Robotics Research. J. Leitner, S. Harding, M. Frank, A. Förster, J. Schmidhuber. CogSys 2012. !Artificial Curiosity for Autonomous Space Exploration. Vincent Graziano, Tobias Glasmachers, Tom Schaul, Leo Pape, Giuseppe Cuccu, Jürgen Leitner and Jürgen Schmidhuber. Acta Futura, 4, pp. 41-52, 2011.

http://Juxi.net/papers2/2