The LVC Continuum for Test and Evaluation

11100 Johns Hopkins Road Laurel, MD 20723-6099

The LVC Continuum for Autonomous System Test and Evaluation

Robert Lutz Chris Eaton, Ph.D.Chief Engineer Project ManagerIntelligent Combat Platforms Intelligent Combat Platforms240-228-7599 [email protected] [email protected]

DISTRIBUTION STATEMENT A - APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED.

mailto:[email protected]

mailto:[email protected]

What is Autonomy?

• Autonomy will be critical for future “dull, dirty, and dangerous” military operations.- Controls costs/personnel loading and improves mission success rates.

• Artificial intelligence can be used to increase the sophistication of the decision-making capabilities in autonomous systems.

Tele-operation Automatic Autonomous

Operator interface similar to pilot control but with a remote control link

Tele-operator interface but with many automated sequences to

reduce pilot workload

Operator interface gives high level mission-like commands with the

autonomy engine doing the details

The actions of an autonomous system will depend upon its organic sensors and/or external feeds to produce a correct “world view”.

World View

DISTRIBUTION STATEMENT A - APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED."

What Can Go Wrong with Autonomy?

Autonomous systems respond to unpredictable change by devising a new course of action – trust must be achieved through rigorous testing

Testing must assure that autonomous decisions will: Obey constraints and achieve objectives set by human supervisors Not produce unacceptable unintended consequences Outcomes and behaviors “learned in operational use” must be recorded/ assessed

3

• Autonomous systems respond to unpredictable change by devising a new course of action –rigorous testing is an important aspect of building trust in the system.

• Testing must assure that autonomous decisions will:• Obey constraints and achieve objectives set by human supervisors• Not produce unacceptable unintended consequences• Outcomes and behaviors “learned in operational use” must be recorded/ assessed

2015 I/ITSEC - 15348USATODAY – 3/12/2019


Facets of Autonomous Systems Development• Eight Facets of Autonomous Systems

Development• Two-way interaction between all facets• Breaking down the interactions between

all facets allows understanding of gaps & influences

• Framework to begin providing meaningful requirements

• Influence Requirements for Test & Evaluation

Holistic View of Autonomous System Development Necessary for Successful T&E

26 September 2019 4DISTRIBUTION STATEMENT A - APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED."

LVC Environments

Real People Operating Real Equipment

Manned & Unmanned systems in actual environment performing mission elements

LIVEReal People Operating Simulated Equipment

Manned cockpit/operator simulators, red & blue force entity simulation

VIRTUALSimulated People Operating Simulated Equipment

Blue Force simulation providing identical software & performance characteristics as a live entity

CONSTRUCTIVE

Distribution Statement A: Approved for public release; distribution is unlimited.DISTRIBUTION STATEMENT A - APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED."

Why LVC for Autonomous System T&E?

Live-Virtual-Constructive Environments are the key to simulation and stimulation of autonomy

Distribution Statement A: Approved for public release; distribution is unlimited.

• Cost• Richness of many operational environments

cannot be replicated on test ranges in an affordable way

• Synthetic forces can augment live forces as needed to contain test expenditures

• Safety• Untested autonomous vehicles are a safety

hazard for other live test assets• Live emitters can create interference for

tests occurring elsewhere on the range• Live shooters engaging autonomous

systems under test is an obvious risk VBS3 from Bohemia Interactive


Agile Methods for LVC Development

7

• Computer simulations are a type of software system, and thus subject to the same measures of “goodness” (metrics) as other software systems• Extensibility, reliability, reusability, …

• LVC simulations can be developed via predictive or adaptive (agile) methods

• While predictive approaches provide very formal, structured methodologies for control of large project teams, they are relatively rigid and are not flexible to changing requirements

• Customers can rarely articulate all of their requirements upfront in the project, suggesting agile methods usually work best• Scrum, Feature-Driven Development

Agile Process Framework

https://speechfoodie.com/agile-scrum-framework-diagram/


The LVC Continuum

8

• The term “LVC Continuum” was first used to reflect how LVC capabilities are used across the entire acquisition lifecycle

• However, when specifically focused on autonomous systems, there is also a continuum of LVC capabilities that support the end-to-end testing process

• Our “LVC Continuum” provides a more detailed illustration of how LVC enables key activities throughout the testing of an autonomous system

System Testing

https://en.wikipedia.org/wiki/Live,_virtual,_and_constructive


Fast-Time Constructive Simulation

9

• There is significant use of fast-time constructive simulations early in an acquisition program• Requirements development, concept

exploration, system design/development

• From the testing perspective, this same class of simulation can be used for (early) requirements validation and test plan development• Agile “build-test-deliver-build”

development strategies• Potential reuse of capabilities from

earlier acquisition phases• Reduce the potential test cases to the

most difficult, stressing conditions

AFSIM VESPA GUI


Real-Time Virtual/Constructive Simulation

10

• Enhance existing fast-time simulation infrastructure to support real-time operation• Necessary for Human-Machine Interface

(HMI) development and testing

• For autonomous systems, must create a realistic operational picture for a mission commander to set goals/objectives for the autonomous system (or team)

• Can be used to study workload issues, early user feedback on symbology/display placement, and conduct validation of real-time system requirements


Software/Hardware-in-the-Loop

11

• Reusing previous iteration, replace simulated system components with actual software/hardware

• For autonomous systems, the objective is to integrate the autonomy software and processor with host vehicle control systems in a laboratory environment

• LVC stimulus transitions from effects to actual messages/data formats with associated throughput, latencies, etc.

• Provides evidence for software/hardware certification prior to live testing Simulation Monitor

ControllerHardware

Controller InterfaceDevice


Enhanced OPFOR

12

• Operational testing, and some aspects of developmental testing, require a more realistic and reactive OPFOR than is typical in earlier iterations • For machine learning autonomy engines,

simple fixed OPFOR tactics can be recognized and behaviors adjusted to artificially optimize system performance

• Allows operationally-relevant system performance measures to be obtained

• Mission context provides a viable basis for establishing user trust in the autonomous system(s)

• Provides foundation for user training

12

Simulation Monitor

ControllerHardware



Live System Integration

13

• Reusing previous iteration, replace simulated Systems Under Test (SUT) with live test vehicles• May start with live surrogates before

transitioning to the target platform• Should not require interface changes, as

previous iterations employed actual message formats

• OPFOR or other simulated test articles (e.g., targets) can also be replaced with live assets

• Can introduce live test assets gradually over multiple tests

• Validation of simulated system behaviors and performance via comparisons with live SUTs improves user trust in the system Simulation Monitor

ControllerHardware



LVC Exemplar - Safe Testing of Autonomy in Complex, Interactive Environments (TACE)

• An architecture for autonomy testing

• Implements an onboard watchdog function

• Sophisticated LVC test infrastructure

• Records outcomes for “black box” testing


TACE Architecture – Ground Component

Ground Infrastructure

On Board AUVSUT Communication Server

Ground Communication Server

Visualizer

Synthetic Force

GeneratorSensor Models

Communication Models

Black Box Monitor

TENA Data Logger

TACE TENA Execution

2-Way Robust

Real Time C2/Data

Links

LVCLVCLVC

Applications

Test Data BaseTACE Test Executive

TACE Watchdog ServerTACE Safety Constraints

Test Manager Safety Manager


TACE Architecture – Air Component

Vendor Autonomy Engine

(This is the autonomy that is being tested)

Vend

or M

iddl

ewar

e

Telemetry

Waypoints

Sensed World Data

Vendor Ground Station

• Constraint / Violation Status• Telemetry• Black Box Data Acquisition• Who is in Control• Remediation Status

TACE Ground Station

Ground InfrastructureOn Board AUV

Test Platform(unmanned

platform with on-board autonomy

engine

VendorAutopilot

TACE WatchdogTelemetry

Waypoints

TAC

E M

iddl

ewar

e

Safety Monitor

Remediation Engine

Synthetic Data Injector

Bac

kup

GN

C

Native C2 Link

TACE Watchdog

Safety Monitor

Remediation Engine

Synthetic Data Injector

Back

up

GNC

• Constraint / Violation Status• Telemetry• Black Box Data Acquisition• Who is in Control• Remediation Status


TACE LVC Continuum

LoiterPoint A

LoiterPoint B

LoiterPoint A

LoiterPoint BEvolving

Framework of LVC

Capabilities


Summary• LVC simulation will be a critical enabler of testing procedures for autonomous systems in the

future• A continuum of LVC capabilities are needed throughout the different test stages

- Fast-time constructive simulation - Real-time virtual/constructive simulation- Real-time live/virtual/constructive simulation

• Agile software development techniques combined with extensive software reuse can lower both cost and schedule risk as opposed to more traditional predictive approaches


Acknowledgements• Co-Authors

- Dr. Reed Young- Dr. Eddie White

• Contributors- Kristine Ramachandran- Dr. Bill D’Amico- Dr. Dave Scheidt


Documents

The LVC Continuum for Test and Evaluation