Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Presentation ‘Benchmarking Autonomous Robotics Controllers’ by Pablo Muñoz (University of Alcalá)
07/12/2016
Introduction by TO (M. van Winnendael, TECMMA) Title of the contract: Cooperative Systems for Autonomous Exploration MissionsBudget + Program: 90 k€, NPI (Networking/Partnering Initiative) Duration: 3 yearsContractor: University of Alcalá, UAH (Spain)Subject of the activity: Onboard autonomy of space robotics systems, in the first place planetary roversMain Objectives: Metrics for measuring planning and execution performance Path planning algorithms to define navigation routes Algorithms for controlling the reactive behavior of ESA’s GoalOriented Autonomous Controller (GOAC) in uncertain situations
ESA UNCLASSIFIED -For Official Use
Benchmarking Autonomous Robotics Controllers
Pablo Muñoz Martínez TEC-ED & TEC-SW Final Presentation Days ESA/ESTEC December 7, 2016
PhD funded by ESA NPITechnical officer : Mr. Michel van Winnendael
PhD program on space researchPhD advisor :
Dra. María Dolores Rodríguez Moreno
PhD co-advisor: Dr. Amedeo Cesta Dr. Andrea Orlandini
Contents
1. Introduction
2. Objectives
3. Planning for planetary rovers
4. Evaluating autonomous controllers
5. OGATE Demo
5/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Motivation
Photos: ESA
Space exploration is a challengeNew missions carry more science instrumentsGo deeper into the spaceOn-board autonomy can be helpful to
Maximize science returnMinimize risksReduce operators workload
6/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Autonomy levels
Decision making capability (AI)According to ESA standards
E1 On-ground control (teleoperation)E2 On-board scheduler for on-ground commandsE3 Event based autonomous operationE4 Goal oriented and mission replanning
7/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Autonomous controllers
Integration of P&S (Planning & Scheduling) in roboticsP&S is used for on-ground operationsMost common schema: 3T (3-Tired)Wide research field
ESA: GOACNASA: IDEA, RAUAH: MoBAr
Deliberation
Execution
Functional
8/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Open issues
No appropriate software to facilitate operationLack of experimentationHard to extract meaningful dataApproaches are validated with few scenariosDifficult to generate reproducible experiments
→ Tests can be seen as proof of concept → Difficult to compare approaches → Hard to trust in autonomy
Contents
1. Introduction
2. Objectives
3. Planning for planetary rovers
4. Evaluating autonomous controllers
5. OGATE Demo
10/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Objectives
Create a framework for autonomous controllers characterization and evaluation
Testbench methodologyPerformance metricsCompare at least two controllersEase deployment and operationImprove trust in autonomy for robotics
Contents
1. Introduction
2. Objectives
3. Planning for planetary rovers
4. Evaluating autonomous controllers
5. OGATE Demo
12/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Planetary exploration case study
Perform scientific tasks in different locationsLimited communication opportunitiesE4 autonomy: user provides initial goalsCommon scenario for autonomous controllers testingRequires
Task planning
Path planning
13/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Path planning
Obtain feasible and optimal paths2D binary grid map
Used in real missionsDeterministic algorithms
A*A*PSTheta*
Path planning
Navigation
More about 2D path planning: P. Muñoz and M.D. R-Moreno. On Heading Change Measurement: Improvements for Any Angle Path Planning. Novel Applications of Intelligent Systems, ch. 6 (2015)
14/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Path planningPlanetary robotics require more realistic terrains with
Terrain characteristics (cost maps)Altitude (DTM)
Field D* (MER/MSL operations) exploits cost mapsModified A* can use different DTM representations
15/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Path planning – 3Dana
3Dana: path planning on realistic surfacesAccurate lineal interpolationAllow DTM and/or cost mapPath cost = path length × cell costHeading changes heuristicTerrain slope Known altitude
Interpolated altitude
cell cost
16/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Path planning – 3Dana
Evaluation on Mars DTM3Dana provides safer pathsSlope limitationReachability map
More results: P. Muñoz, M.D. R-Moreno and Bonifacio Castaño. 3Dana: Path Planning on 3D Surfaces. In procs. of the 36th SGAI International Conference on Artificial Intelligence (2016)
17/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Task planning – The UP2TA planner
Exploration requires to achieve multiple goalsUP2TA (Unified Path planning & Task planning Architecture)
PDDL planner : FFPath planner : 3Dana
Objective: optimize the planMinimize travelled distanceBest task ordering
18/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Task planning – The UP2TA planner
More results: P. Muñoz, M.D. R-Moreno and D. F. Barrero. Unified Framework for Path Planning and Task Planning for Autonomous Robots. In Robotics and Autonomous Systems, vol. 82, pp. 1—14 (2016)
Contents
1. Introduction
2. Objectives
3. Planning for planetary rovers
4. Evaluating autonomous controllers
5. OGATE Demo
20/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
An evaluation framework
Hard to compare autonomous controllersObjective
Generation of meaningful execution dataAnalysis integration of P&S in roboticsGeneration of reproducible/comparable testbenchs
Based onA methodology to guide the testing phaseA set of metrics to assess P&S integrationA software tool to
Automate testbenchsImprove autonomy support for users
21/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Methodology
Evaluation consists on 3 sequential phases
1. Evaluation design
2. Tests execution
3. Performance report
22/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Methodology
1. Evaluation design
Objective of the testApplication scenario and robotic platformControllers configurationsOperative conditions
NominalContingencyGoal injection
Normalized performance metrics
23/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Methodology
2. Tests execution
Scenario and controller instantiationControllers executionMonitoring of metrics dataCollection of average behaviours
24/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Methodology
3. Performance report
Data synthesisGeneration of objective evaluationsMeaningful data representation
25/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
General metrics
Characterize autonomous behavioursEnable different controllers comparisonFour groups
Model
Controller
Plan accuracyPlanner model adequacyPlanner performancePlanning & Execution (P&E) integration
These groups fit in the graphical reportEasy to focus on particular characteristics
26/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
General metrics
Characterization of plan accuracy
Planning performanceTimeMemory
Lower diff
Higher diff
Execution ends
Minimum duration
Maximum duration
27/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
General metrics
Integration of P&S with underlying layers of the controllerAnalysis of information flow Characterization of reaction time to unexpected events
Deliberation
Execution
Functional
Plan
Sensor dataCommands
Unexpected event
28/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
The OGATE software
OGATE (On-Ground Autonomy Test Environment)Designed to support autonomy
Simplify controllers deployment/operationMonitor P&S featuresPerform automatic testbenchsPlug-ins
Implements the methodologyand metrics collection
Allows user-defined metrics
More info: P. Muñoz, A. Cesta, A. Orlandini and M.D. R-Moreno. First Steps on an On-Ground Autonomy Test Environment. In procs. of the 5th IEEE Conference on Space Mission Challenges for Information Technology (2014)
29/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
The OGATE software
GOAC (GMV)
GOAC (PST) OGATE Beta
No data available
OGATE Alpha Automated tests, data gathering
30/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
The OGATE software
Current deployment: mission planning
31/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
The OGATE software
Current deployment: mission monitoringCurrent statusPast status
Map with rover and goals positions
Goals status
32/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
The OGATE software
Current deployment: performance assessment
33/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
A comparison testbench
Two controllersGOAC (Goal Oriented Autonomous Controller)
ESA effort for autonomy in space (E1-E4)Timelines based planning (APSI-TRF)
MoBAr (Model Based Architecture)UAH E4 controllerPredicate based planning (UP2TA)
Exploration domain, simulated TurtleBot 2OGATE carries out automated tests
Nominal / Goal injection / Execution failureReports summarize average behaviours
34/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
A comparison testbench
Report after 30 executions MoBAr GOAC
More results: P. Muñoz, A. Cesta, A. Orlandini and M.D. R-Moreno. A Framework for Performance Assessment of Autonomous Robotic Controllers. In procs. of the 2nd Workshop on Planning and Robotics – ICAPS (2015)
Contents
1. Introduction
2. Objectives
3. Planning for planetary rovers
4. Evaluating autonomous controllers
5. OGATE Demo
36/37Benchmarking Autonomous Robotics Controllers – ESA/ESTEC Dec. 7th 2016
Introduction · Objectives · Plan · Eval. Controllers · Demo
Demo
Thanks for your attention
Pablo Muñoz Martíneze-mail: [email protected]