Tom Gullion

SRMV Toolsmithtgullion@msisinc.com

(608) 315-2780

Executing UML Models as Verification Oracles

Overview

• Charter– Execute UML models as oracles of behavior exhibited by software

under test• Challenge

– Testing facilities for flight control software and related domains are expensive

• in terms of cost and resources and access– Verification teams often find it difficult to get adequate access to

those testing facilities• Goal

– Reanimating test logs from the test labs against System Reference Models

– That is, verify the SRM against test logs. And vice versa!

The Challenge• Testing facilities for flight control software and related

domains are expensive– in terms of cost and resources and access

• Verification teams often find it difficult to get adequate access to those testing facilities

• Access to logs from test runs is common• Access to software simulations may happen• Access to live test rigs could possibly happen…

• How can we leverage this test data in Verification?

Test Data Reanimation

• Data from a test log is used as a stream of input events

• SRM is implemented/extended as a test oracle

• Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test

Basic Reanimation Approach

test definition

test definition

comand/eventstack

comand/eventstack

ReanimateSRM

ReanimateSRM

data convertedto stream of events

data convertedto stream of events

Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test

Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test

SRM astest oracle

SRM astest oracle

execexec

Test log expectations

• Logs from testing / simulation facilities will have timeframes, command names, parameter data and other metadata

• We won’t be interested in all of it (at least, at first)

• We may even prefer to extract relevant data from multiple sources for our own scenarios (e.g., testing across component boundaries)

Case study: Ejection Seat Behavior

Test scenario

• Example: the testing facility ran a simulation of a flight where the crew ejected

Reanimate Ejection Seat Events

• Focus only on ejection seat commands in test logs

• Configure model runner environment

• Execute commands against SRM to reanimate the events

Reanimation details

COMMENT START SCRIPT Flight with Eject 3330 EJECTION_SEAT arm6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT

FRAME: 3330 EJECTION SEAT PHASE CHANGE TO Armed

FRAME: 6322 EJECTION SEAT PHASE CHANGE TO Ready

FRAME: 6323 EJECTION SEAT PHASE CHANGE TO Ejected

FRAME: 6326 EJECTION SEAT PHASE CHANGE TO Separated

FRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChute

Scenario complete: SUCCESS

Input events(Frame, Command, Parameter(s))

Input events(Frame, Command, Parameter(s))

ResultsResultsplay

back

again

st

SRM

mod

el ele

men

t(s)

play

back

again

st

SRM

mod

el ele

men

t(s)

Ejection seat scenario results

• Use reanimation results to verify logged events match SRM modeled behavior

• But this scenario only verifies a single slice of the SRM. Let’s expand the reanimation to include flight modes…

Reanimating multiple statemachines

3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT

3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT

Input events(Frame, Command, Parameter(s))

Input events(Frame, Command, Parameter(s)) ResultsResults

play

back

again

st

SRM

mod

el ele

men

t(s)

play

back

again

st

SRM

mod

el ele

men

t(s)

FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 EJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 EJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS

FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 EJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 EJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS

Flight with Ejection seat scenario results

• Now we’ve verified logged events match SRM modeled behavior in two statemachines

• This scenario only verifies proper event sequencing within its scope. But we’re not checking preconditions and other rules.

• Extend interfaces defined in SRM to implement rules…

Reanimating the SRM

3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT

3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT

Input events(Frame, Command, Parameter(s))

Input events(Frame, Command, Parameter(s))

ResultsResults

play

back

again

st

SRM

mod

el ele

men

t(s)

play

back

again

st

SRM

mod

el ele

men

t(s)

FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 FlightController: Ejection seat armedEJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 FlightController: Canopy releasedFlightController: Ejection seat positionedFlightController: Ejection seat ejectedFlightController: main chute deployedEJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS

FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 FlightController: Ejection seat armedEJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 FlightController: Canopy releasedFlightController: Ejection seat positionedFlightController: Ejection seat ejectedFlightController: main chute deployedEJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS

Complete scenario results

• Now we’ve verified logged events match SRM modeled behavior in two statemachines as well as a flight controller class

• From this framework, it is now possible to add in telemetry data and other command data.

• This structure is setup for reuse across other reanimations.

live data feedlive data feed

Live Test Rig

Live Test Rig

Extensible Reanimation Approach

comand/eventstack

comand/eventstack

Pluggable input data types;Pluggable exec environments

Pluggable input data types;Pluggable exec environments

SoftwareSimulator

SoftwareSimulator

execexec

ReanimateSRM

ReanimateSRM

adversecondition data

adversecondition data

test definition(s)

test definition(s)

Extensible Reanimation Approach

Pluggable input streams allow for future use of• Other test logs (related tests, adverse conditions, etc.)

• "live" data feeds

• the use of accredited simulations as live or stale data sources

Some or all these data sources can be used without changing the overall approach

Conclusions• This approach is appropriate for complex collections of analysis

cases– Simple analysis cases probably do not require reanimation

• We are building part of a larger construct– Analyze adverse conditions against specific tests logs or accredited

simulations– Enables use/reuse of SRM without extensive modification– Enables traceability from reanimation result back through SRM to

originating requirements– Allow distributed machines to play different roles (data source vs data

analysis)– Flexibility for machines (real and virtual) to interface with existing

infrastructure (IV&V not required to manage all machines)• This approach embraces the existing overall framework and is

designed to scale

Applying the 3 Questions

• What is the system supposed to do?– Verified by success and failure scenarios

• What is the system not supposed to do?– Verified by success and failure scenarios

• What does the system do under adverse conditions?– Verified by adverse conditions test data

– Can predict possible failure scenarios that don’t seem adequately handled, propose test scenarios for dev team

Acknowledgements

• Thanks to NASA IV&V for the opportunity to present these concepts

• Special thanks to Frank Huy, Bill Stanton, Todd Gauer, Karl Frank, Steve Driskell, Dan Sivertson, Tom Hempler and Jeff Micke for technical contributions and assistance

Questions? / Comments?

Thank you!

Tom Gullion

tgullion@msisinc.com

(608) 315-2780