National Aeronautics and Space Administration Practices for Improving Robotic Software Reliability in Flight and Research Projects Khaled S. Ali and Issa

National Aeronautics and

Space Administration

Practices for Improving Robotic Software Reliability in Flight and

Research Projects

Khaled S. Ali and Issa A.D. Nesnas

Jet Propulsion Laboratory, California Institute of Technology

The Sixth IARP-IEEE/RAS-EURON Joint Workshop on Technical Challenges for Dependable Robots in Human Environments, Pasadena, California, May 17-18, 2008



Workshop on Robot Dependability 2May 17-18, 2008

Presentation Overview

• The Need for Dependability in Robots for Space Applications

• The Flight Side: Mars Exploration Rovers– MER FSW Architecture Design Decisions for Reliability– MER FSW Development Procedures for Reliability

• The Research Side: CLARAty– What is CLARAty?– Techniques used by CLARAty to Improve Software Reliability



Workshop on Robot Dependability 3

The Need for Dependability in Space Applications

• If a robot fails after leaving Earth, repair may be difficult or impossible

• If a robot fails during a critical time, such as descent and landing, the results can be disastrous

• Research efforts need to consider dependability to address scalability and facilitate integration into flight applications

May 17-18, 2008



The Flight Side: Mars Exploration Rovers (MER)




MER Flight Software Design Decisions Overview

• Software Decomposition– Object Oriented– Hierarchical state machines

• Runtime Model– Multiple tasks– Asynchronous message passing– Multi-layered fault protection– Automation of critical behaviors

• Software Implementation Principles– Coding standards– Static memory allocation– Design patterns

May 17-18, 2008




MER Flight Software Decomposition

• Object-oriented style design, with emphasis placed on interfaces, encapsulation, and modularity

• Objects implemented as hierarchical state machines

May 17-18, 2008




MER Flight Software Runtime Model (Part 1)

• Multiple tasks on a RTOS, with task priorities reflecting criticality and required response time

• Asynchronous message passing as the principle means of communication between objects, and the messages drive the heierarchical state machines

• Multi-layered fault protection – Hardware– Software handling

• Minor faults handled locally• Major faults handled by separate Health and Fault Protection tasks

– Command sequences

May 17-18, 2008




MER Flight Software Runtime Model (Part 2)

• Critical mission behaviors are automated– Entry, Descent, and

Landing– Communication– Fault Handling– Attitude Acquisition– Wakeup and Shutdown

• Allows faster handling than ground-in-the-loop and richer handling than either hardware or command sequences

May 17-18, 2008




MER Flight Software Implementation Principles

• Dynamic memory allocation is severely limited to prevent heap fragmentation and allow memory problems to be discovered during development

• Coding conventions, rules, and guidelines.

• Design patterns (message passing, command handling, hardware device objects)

• Reference:– Glenn E. Reeves & Joseph F. Snyder "A Overview of the Mars Exploration Rovers' Flight Software" 2005

IEEE International Conference on Systems, Man and Cybernetics Waikoloa, Hawaii, October 10-12, 2005

May 17-18, 2008




Flight Software Development Procedures

• Use only mission-proven or thoroughly tested technologies• Formal design reviews

– Inheritance reviews– Design reviews– Implementation reviews

• Code reviews– Independent developer reviews code– Static code analyzers used




Rigorous Flight Software Testing

• Unit Testing:– Extensive testing of each module

in isolation by the developer

• Regression Testing:– Integrated module testing by a

dedicated test team after new modules are integrated

• System Testing: – Project wide rehearsals of

expected mission scenarios– Can last several days where

several different activities would be tested in the manner they would be used in the mission

– All communication is done during communication passes

May 17-18, 2008



The Research Side: CLARAty

Coupled Layer Architecture for Robotic Autonomy




What is CLARAty?

CLARAty is a unified and reusable software that provides robotic functionality and simplifies the integration of new technologies on robotic platforms

A research tool for technology development and maturation




Problem and Approach

• Problem:– Difficult to share software/algorithms across systems– Different hardware/software infrastructure– No standard protocols and APIs– No flexible code base of robotic capabilities

• Objectives– Improve software reliability by enabling and encouraging reuse– Provide mature and dependable robotic infrastructure

• Simplifies creation of robotic applications• Simplified integration of new technologies• Enables research efforts to concentrate on the algorithm being investigated

– Mature software used on heterogeneous robots, enabling more reuse• Same interfaces to high-level components




Interoperability: Software & Hardware

May 17-18, 2008

• Rover

• Acquire Image• Goto Target 1

• Swappable Algorithm orRobot Adaptation

• ATRV Jr.• Rocky 7

• ROAMS

• Functional• Layer

• Decision• Layer

• Declarative Activity

• Functional Abstraction

• Rocky 8

• Explore Site

• Goto Target 3• Deploy

Instrument

• Acquire &• Analyze

• Navigator• Morphin

• Locomotor

• R8_Model

• Motor• R8_Motor

• Pose Estimator• SAPP

• Pt Cloud

• IMU• ISIS

• Target Tracker

• Falcon

• Camera• 1394 Cam

• Stereovision• JPLV




Technology Tasks

Technology Development, Integration and Validation

CLARAty

Jet Propulsion Lab

CMU

NASA ARC

U. Minnesota

R&TD, MDS, DRDF

Competed Mars TechnologyProgram

Other NASA Programs

Rover SimulationROAMS

Rover Hardware

JPL Internal Programs

Flight FocusedTechnology Programs

Science InstrumentsSimulation

Operator Interface

Legacy AlgorithmsFlight Algorithms

NASA Centers and

UniversitiesTechnology Tasks

NASA Centers and


NASA Centers and


NASA Centers and


TechnologyValidation Tasks

TechnologyValidation Tasks

Technology Tasks

Technology Tasks




Techniques We Use in CLARAty

• Some of the techniques that we have explored to improving software reliability are:– Improved processes and

procedures for software development

– Unified coding conventions– Static code analysis and

validation tools– Increased software reliability

through reuse– Formal technology validation– Automated nightly regression

testing (to a limited extent)– Fault-tolerant software

May 17-18, 2008



Back-up Slides




Acknowledgements

CLARAty Team (multi-center)

Jet Propulsion Laboratory

Ames Research Center

Carnegie Mellon University

University of Minnesota



20May 17-18, 2008 Workshop on Robot Dependability

Current CLARAty Core Team

• NASA Ames Research Center– Lorenzo Flueckiger– Hans Utz

• Carnegie Mellon University– Reid Simmons– David Apelfaum– Nick Melchior

• University of Minnesota– Stergios Roumeliotis– Nikolas Trawny– Anastasios I. Mourikis

• Jet Propulsion Laboratory– Issa A.D. Nesnas– Hari Das Nayar– Tara Estlin– Richard Petras– Daniel Gaines– Robert Steele– Daniel Clouse– Michael McHenry– Khaled S. Ali– Mihail Pivtoraiko– Kelly Breed

• Affiliates– Jeffrey Edlund

Full Credits for all Developers and Contributors at:http://claraty.jpl.nasa.gov/man/project/team/index.php

Documents

National Aeronautics and Space Administration Practices for Improving Robotic Software Reliability in Flight and Research Projects Khaled S. Ali and Issa