Model Based Test Planning for Autonomous Systems...Example Results: Position Response 18-400-300-200-100 0 100 200 300 0 500 1000 1500 2000 2500 0 20 40 60 80 100 120 Crosstrack (ft)

Innovations in Engineering

UNCLASSIFIEDCopyright 2011 by the Charles Stark Draper Laboratory, Inc. all rights reserved

Model Based Test Planning for Autonomous

Systems

Merging Technology Acquisition and T&E

September 15, 2011

1

Team MembersTroy Jones, George Sass, Melissa Durfee, Sean Buckley, Nick Borer,

Stephen York, Eric Nelson, Mike Ricard, Glenn Ogrodnik, Scott Ingleton


Implementation

Concept of

Operations

Architecture

Requirements

Design

Integration, Test

& Verification

System

Verification &

Validation

Operations &

Maintenance

Behavioral

Model

Project Vision

2

Research Markov Reliability Analysis and DOE for System-Level test planning

Complementary to Model-Based Engineering

Performance Based Evaluation

Inject failure conditions

Force off-nominal decisions

Continuous Model Refinement

Feedback performance data over time

Improve predictions of future reliability

Implementation

Concept of

Operations

Architecture

Requirements

DesignIntegration, Test

& Verification

System

Verification &

Validation

Operations &

Maintenance


Lifecycle Autonomous System Test Design

3

0 10 20 30 40 50 60 70 80

propulsion Omission

Att Psi Omission

Att theta Omission

pressure Omission

fin7 Omission

fin5 Omission

fin6 Omission

Att r Omission

fin4 Omission

sonar fls Omission

sonar bathy Omission

Att P Omission

sonar rsls Omission

sonar lsls Omission

0 nominal

Att q Omission

Att Phi Omission

Position Error: CurrentMag

|T|

current only rank

Current Magnitude

Markov Reliability Sensitivity

Ma

p

Mis

ma

tch

Current

Direct

ion

100%

1(0,0,0)

Priorities?

50%

Mismatch

Environmental

Conditions

B

Current

TemperatureWeather

A

C E

D

Design of Experiments

Mission

Conditions

Map

Quality

Objectives

Failure

Conditions

Sensors

AcutatorsSoftware

180°

Test Condition Ranking

Mission

Simulations

& Analysis

G(z)

K


Behavioral Markov Reliability Analysis

Draper developed PARADyM Tool

System Markov Model

Component connections & logical dependencies

Component reliability values (MTBF)

Model Outputs

Probabilities

– Any failure condition over system life

– System Loss

Reliability Metrics

– System Reliability

– Sensitivity of Reliability to Component Failures

4

1

2

3

4

Failure

rate = a

Failure

rate = b

Failure

rate = c

a

b

c

Operational State

System Loss State

0 FL 1 FL

P(System Loss) = Σ(System Loss States)

Reliability = Σ(Operational States)

Motor

Pu

mp

Tank


Case Study: Unmanned Underwater Vehicle (UUV)

Based on Typical Small UUV

Non-redundant

architecture

ASTM F41 Software

Architecture

Component Reliability

Estimates

Many hardware item

values (motors, valves)

from U.S. Army service

data

Computer & Sonar values

from vendors of similar

hardware

Simulated Failure Nodes

5

PCControls

Vehicle

Payload

Sensors

ACPerforms Mission

Planning, Commands

Steering & Speed,

Payload Use

Scheduling, SAVC

Performs Low-

Level Vehicle

Control and

Management

Simulation Models

(Ocean + Vehicle)

Sink 1

Sink 2

Injected

FailureSource


Predominantly equivalent sensitivities (2 Fault Level Analysis)

Non-fault tolerant system design

Sonars and Battery Scanners stand out

Cascading failures to other components

Markov Reliability Analysis Results: UUV

6


Sensitivity


SensitivityAttitude Sensor H Bathy Looking Sonar

Attitude Sensor P Forward Looking Sonar

Attitude Sensor R Depth Pressure Sensor

Rate Sensor P Propulsion Motor Controller

Rate Sensor Q Propulsion Motor

Rate Sensor R Propulsion Sensor

Global Positioning System Fin Actuator Motor Controller

Doppler Velocity Sonar Fin Actuator1

Temperature Sensor Fin Actuator2

Saltwater Detector Sensor Fin Actuator3

Battery1 Fin Actuator4

Battery Scanner System Fin Sensor1

Power Distribution System Fin Sensor2

Vehicle Controller Fin Sensor3

Autonomous Controller Fin Sensor4


DOE: UUV System Responses

7

Type Response Description Rationale

Perf

orm

an

ce

Position Error (t) Deviation from baseline mission

path over time

Position errors cause data

collection errors

Attitude Error (t)

[φ,θ,ψ]

Deviation from baseline attitude

over time

Attitude errors cause data

collection errors

Speed Error (t) Deviation from baseline speed

over time

Speed influences

execution time, stealth,

energy

Reco

vera

bilit

y Energy Consumption Energy consumption for mission Must operate within

available energy limits

Mission Time Total mission time Establish expectations for

recovery/communication

Surface Position

Error

Deviation from designated end-of-

mission surface point

Large errors on surfacing

impact recovery


Environmental Experiment Design

Available Environmental Factors (3)

Uniform Current Magnitude & Direction

Prior Terrain Knowledge

DOE Design

2 Level, 3 Factor Full Factorial – min/max

levels with median center point

Center point detects curvature

8

Min Median Max

Current

Magnitude 0 Knots 1Knots 2Knots

Current

Direction 0° 90° 180°

Map

Mismatch 0% 50% 100%

0°

180°

90°50%

Mismatch

100%

Mismatch

RunOrder CenterPt

CurrentMagnitude

(knots)

Current Direction

(deg)

Map Mismatch

(%)

1 1 2 0 100

2 1 2 180 100

3 1 0 0 100

4 0 1 90 50

5 1 0 0 0

6 1 2 180 0

7 1 0 180 100

8 1 2 0 0

9 1 0 180 0

Experimental Design with Center Point0%

Mismatch

Actual Terrain A priori Terrain MapCurrent

Direction


Short Mission Evaluation Scenario

Short Mission Goals

Terrain avoidance

Waypoint following

Case Study Scenario

Design

Vary ocean currents,

map quality

~ 300 seconds

Approach & avoid

terrain on way to

waypoint

Basis of all case study

simulations

9

Draper Simulation

Framework (DSF)

Mission Visualization


Execution & Data Analysis

10

Draper Simulation + Matlab ® Minitab ® MS Excel ®

Baseline

Execute Simulations 1..N

Parse Experiment Data Files

Identify Baseline Data Set

E ...E 2

E 1

Delta E ...Delta E 2

Delta E1

Delta(N)=E(N)-Baseline

Box-Cox Transformations

General Linear Model (GLM):

• Regression

• Main Effects & Interactions

• T-Test Values

0 10 20 30 40 50 60 70 80

propulsion Omission

Att Psi Omission

Att theta Omission

pressure Omission

fin7 Omission

fin5 Omission

fin6 Omission

Att r Omission

fin4 Omission

sonar fls Omission


Att P Omission

sonar rsls Omission

sonar lsls Omission

0 nominal

Att q Omission

Att Phi Omission


|T|

T-Test Ranking

(Pivot Tables)

Graphing

Box-Coxx2,√x ...

Max of |T-value| Column Labels

Row Labels MapMismatch

Energy**2

pressure_omission

Mission Time 143.44

attitude_rOmission 143.44

pressure_omission

fins_fin4Omission 116.2

Surface Position Error**-2

pressure_omission

Main Effects &

Interactions


Results – Recoverability Responses

Many Failures Not Significant

Encouraging robust behavior

Vehicle recoverable from ~12

other failures

Candidates for Higher Fidelity

Tests:

Map Mismatch (navigation

error) with attitude sensor &

fin failures

Pressure sensor failures in

strong currents

Caveats

Propulsion failures not

registered as “significant”

Scenario setup/duration and

simulation boundaries mask

effects

11

0 20 40 60 80 100 120 140 160

pressure_omission

attitude_rOmission

pressure_omission

fins_fin4Omission

pressure_omission

Ener

gy*

*2

Mis

sio

n T

ime

Surf

ace

Po

siti

on

Er

ror*

*-2

Recovery Responses: Signficant Factors & Failures

CenterPt

CurrentDirection

CurrentDirection*CurrentMagnitude

CurrentDirection*MapMismatch

CurrentMagnitude

CurrentMagnitude*MapMismatch

MapMismatch

|T|


Results – Mission Performance Responses

Key Failures to Test Further

Certain attitude & rate

Sensors (Phi, p, q)

All Sonars (agrees with

Markov)

Observations

Sensor failures more

important than fin failures

Similar sensor failure

rankings in other responses

Propulsion rank likely

incorrect for longer

scenarios

Position Control

12

0 10 20 30 40 50 60 70 80

propulsion Omission

Att Psi Omission

Att theta Omission

pressure Omission

fin7 Omission

fin5 Omission

fin6 Omission

Att r Omission

fin4 Omission

sonar fls Omission


Att P Omission

sonar rsls Omission

sonar lsls Omission

0 nominal

Att q Omission

Att Phi Omission


|T|

current only rank


Conclusions

Lifecycle Autonomous System Test design process

developed & evaluated

Unmanned Underwater Vehicle (UUV) Case Study

Markov Reliability Analysis – shows utility, needs

better integration

Design of Experiments – valuable insights to

prove relationships and focus subsequent tests

Future Work

Integration of Markov Analysis with Draper

Simulation

Fault injection into software layers of the system

Innovations in Engineering


Supplemental Slides


UUV Reliability Studies

Little Published Reliability Data

Many University projects

Inconsistent record keeping

Multi-Year Reliability Studies on Autosub

UK Oceanographic Survey UUV

Extreme deep diving

Terrain following

Science data collection

The Ocean is a Harsh Mistress

Majority of failures in [sensor] hardware (leaks, shorts)

Humans caused more faults than software

Estimated 63% Survival Probability for long Under-Ice mission

Lessons

Need Test & Evaluation rigor in development (happening now)

Don’t always assume software is the long-pole

15

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

5

10

15

20

25

Sensing

Hardware

Vehicle

Hardware

Human Software Ship

Autosub Fault Pareto4 Years, 240 Missions, 1600 NM

Relia

bility

Autosub Reliability Data & Model

Data from Griffiths, G. et al., “On the Reliability of the Autosub

Autonomous Underwater Vehicle”, Underwater Technology, 2002





Sensors (Phi, p, q)


Markov)

Observations

Equivalent to Position

Control, plus pressure

sensor failures

Repeat of Phi & q as top

ranking failures (not

predicted by Markov)

Attitude Control

16

current only rank

0 10 20 30

Att Psi Omission

Att theta Omission

Att r Omission

fin7 Omission

fin4 Omission

fin5 Omission

fin6 Omission

pressure Omission


sonar fls Omission

sonar rsls Omission

sonar lsls Omission

Att P Omission

0 nominal

Att q Omission

Att Phi Omission

Attitude Error: CurrentMag

|T|

current only rank


0 10 20 30 40

propulsion Omission

Att theta Omission

Att r Omission

fin7 Omission

fin4 Omission

Att Psi Omission

pressure Omission

fin5 Omission

Att q Omission

fin6 Omission

Att Phi Omission

sonar fls Omission


Att P Omission

0 nominal

sonar rsls Omission

sonar lsls Omission

Speed Error: CurrentMag

|T|




Sensors (Phi, p)


Markov)

Observations

Repeat of Phi as high

ranking failure

All performance

responses highly

dependant on current

Consistent high

rankings of similar

failures

Speed Control

17

current only rank


Example Results: Position Response

18

-400-300

-200-100

0100

200300

0

500

1000

1500

2000

2500

0

20

40

60

80

100

120

Crosstrack (ft)

UUV Path During Select Bathymetric Sonar Failures

Downrange (ft)

Dep

th (

ft)

Nominal Failure Case

Bathy Fails, Map Mismatch 100%

Bathy Fails, 50% Map Mismatch, 2 knot Side Current

Bathy Fails, 4 knot Tail Current, Mission Incomplete

System Baseline

Documents

Model Based Test Planning for Autonomous Systems...Example Results: Position Response 18-400-300-200-100 0 100 200 300 0 500 1000 1500 2000 2500 0 20 40 60 80 100 120 Crosstrack (ft)