Model-based Testing of a Software Bus - Applied on Core Flight Executive

1© 2014 Fraunhofer USA, Inc.

Center for Experimental Software Engineering

Model-based Testing of a Software Bus

applied on Core Flight Executive

Dharmalingam Ganesan, Mikael Lindvall

Dave McComas (NASA GSFC)

Presented at Flight Software Workshop, Pasadena, CA, 2014



Traditional Test Automation

Only test execution is automated

E.g. Junit, CuTest, etc.

Test cases are manually constructed

Effort intensive

Some issues with traditional testing:

Insufficient coverage of off-nominal behaviors

Tests are too-detailed with low-level details

Not easy to test multi-tasking architecture

More on this later.



Model-based Testing (MBT)

The tester develops a model (a.k.a. testing models)

instead of writing suite of test cases

The model becomes the test oracle

Test cases are auto-generated from the model

based on requirements, API documentations

Key benefits:

Tester works at a high-level of abstraction

Innumerable number of test cases derived from the model

Triggers many, if not all, off-nominal behaviors

Precise specification



System under test (SUT)

• The Core Flight Software System (CFS)• a mission-independent, platform-independent, Flight

Software (FSW) environment integrating a reusable core flight executive (cFE)

• The CFS is a product-line developed by the NASA Goddard Space Flight Center (GSFC)

• CFS is implemented in C

• cFE layer of the CFS is open source



cFE/CFS Context Diagram

Inter-task Message Router (SW Bus)

TranspondersCommands

Real-time Telemetry (UDP)

Comm Cards

File downlink

(CFDP)

Summit Chip

Mass

Storage

System

CFDP File

Transfer

FileManager

Local

Storage

DataStorage

Event

ServicesExecutive

Services

Time

Services1553 Bus

Support

Software

BusCommand

Ingest

Telemetry

Output

Table

Services

EDACMemory

Scrubber

Self

Test

Memory

DwellInstrument

ManagerChecksum

Memory

ManagerGN&C

Applications

(4)

Mission Apps

cFE core App

CFS Applications

Stored

Commanding

Software

Scheduler

Health &Safety

Manager

House-

keeping

Limit

Checker



MBT of Software Bus

cFE has a software bus (SB) Apps communicate indirectly using the SB

Publish-Subscribe architectural style

SB has unit-tests (developed by NASA GSFC) Good coverage but

not taking multi-tasking into consideration

Goals: find defects related to multi-tasking (difficult!)

Generate test cases for SB

Generate the “bubbles” (the apps on previous slide)

Demonstrate the applicability of MBT

Developed an approach of testing SB

Allows testing of multi-tasking architectures



Scope of the current model

Modeled the following behaviors Create Pipes (to hold messages)

Delete Pipes

Subscribe to Messages

Send Messages

Receive Messages

Multiple Apps (dynamically instantiated)

Innumerable test cases (in C) auto-generated

Model based on Microsoft’s Spec Explorer Tool

Will get back to this



Challenges of testing a SB

To test a SB we need apps that publish-subscribe

Apps are runtime tasks that communicate using the

SB

Apps publish/subscribe to messages

Each app cannot decide on its own the correctness

Correctness depends on the global state of the system

E.g: subscribe(msg), RecvMsg() may not work if no other

task is publishing any message

The order of execution of tasks also matters

Need a test architecture!



Test Architecture – Key Ideas

Parent/Child architecture for testing

Each test case is a parent

Each test case runs as an app

At run-time, one or more child tasks are spanned by

the parent

Model controls the behavior of the parent

All test assertions are decided by the parent

All child tasks share the codebase

How the parent and children communicate?

Why not just use the software bus itself?



Test Architecture

CFE

Paren

tChild

1

Child

2

Child

n

Command Pipe

All child tasks share the codebase



Test Architecture – Key Ideas

Each child task subscribes to all commands, such as Create Pipe, Delete Pipe, Subscribe, etc.

Parent broadcast commands to all child tasks Communication uses CFE infrastructure

Task id of the child is also part of the message struct

Only the target child can perform a certain command

Child tasks perform the command and send the return code back to the parent Child tasks send out a result msg

Parent task subscribes to the result msg

The parent verifies test assertion

Asserts are generated from the model



Spec Explorer – Brief Background

Tester develops a model (a.k.a. model program)

Spec Explorer runs as a plug-in to MS Visual Studio

Model programs are written in C# like syntax

The model program is a simplified version of the

SUT

Spec Explorer generates state machines from

models

Also checks whether model satisfies invariants

Helps in validating the model

Test cases are automatically derived from state

machines

SUT’s behavior is automatically compared with

model

Tests failure: Deviation between model and SUT



Abstractions for the model program

Model program is another implementation of the

SUT

But we do not want to create two implementations

No one wants to maintain two implementations

No one wants to develop a system two times

How did we create a simplified version of the SUT?

Key idea: Apply abstractions

Model of GPM, which uses CFE



Sample abstractions for a model

Model is agnostic to multi-tasking complexity of the

SB

Model has a very simple message structure

Message is modeled as an int (not C structures)

Message queues/pipes are also abstracted

Finite depth

Message queues are modeled as simple sequences

instead of using shared memory

No pointers, threads, semaphores at the model level

Very simple data structure using very basic data types

int, boolean, maps



Structure of the model program

E.g.: Number of apps, pipes

Represent the state of the bus

Used for excluding the uninteresting states

Which states are good for terminating tests

Models the actual logic of the software bus

Preconditions for enabling the rule methods to fire

Generates values for rule methods

Utilties for rule methods and guards



State Data



Fragments of the model program

Rules are enabled only if Condition.IsTrue returns true



Fragments of the guards



Slicing the model for specific tests



Generated from the model program

We generate

the model!

In “regular”

MBT you have

to manually

create the

model.



Generated test sequences - sample

Each chain is a test case



SUT Adapter

Adapter wraps the SUT

Converts data/commands from the model into SUT’s

syntax

Adapter simplifies modeling complexity

Methods of the model should map to the adapter

Our adapter is in C#

We “print” test code from our adapter in C

Converts C# tests into C tests

Recall that CFE’s SB interface is in C



Our abstracted interface for Testing

int32 InitApp_w(int32 appName);

int32 CreatePipe_w(int32 appName, int32 pipeName, int32

pipeDepth);

int32 DeletePipe_w(int32 appName, int32 pipeName);

int32 Subscribe_w(int32 appName, int32 msgId, int32 pipeName);

int32 UnSubscribe_w(int32 appName, int32 msgId, int32

pipeName);

int32 RcvMsg_w(int32 appName, int32 pipeName, int32*

actualMsgId);

int32 SendMsg_w(int32 appName, int32 msgId);



Sample generated test casevoid Parent_TestAppMain( void ) {

int32 status;

uint32 RunStatus = CFE_ES_APP_RUN;

Parent_TestAppInit();

status = InitApp_w(APP_0);

assert(status == CFE_SUCCESS);

status = CreatePipe_w(APP_0, PIPE_0, 1);


status = Subscribe_w(APP_0, MSG_0, PIPE_0);


status = Subscribe_w(APP_0, MSG_1, PIPE_0);


status = UnSubscribe_w(APP_0, MSG_1, PIPE_0);


CFE_ES_ExitApp(RunStatus);

}



Advantages of using Model-Based Testing

The model focuses on the domain (easier to

understand)

Instead of being source code oriented (harder to

understand)

We automatically generate an endless number of

executable test cases (high coverage)

Instead of manually writing individual test cases

The information is in one place: in the model, easy to

maintain

Instead of being spread out (hard to maintain)

The test cases can easily be run over and over again



Advantages of using Spec Explorer

Generated tests are pretty readable

This is due to the ability to slice models into smaller

models

Data parameters are well handled

E.g., Model can be configured to test multiple apps

Models are programs

Ideal for programmers (who prefer coding)

Models can be formally verified

Invariants encoded in the model help to validate the

model



Challenges with Spec Explorer

Modeling errors can lead to infinite state machine

Need to be careful even for small models (e.g., int

parameters)

Syntax for slicing the model is powerful but not that

easy

Easy to misuse some of (algebraic) operators for slicing

Completeness of our slices

Did we miss any combination of behaviors during slicing?

Model debugging. For example:

Why a new state was generated?

Where/Why the invariants are violated?



Applicability to other flight software

The same approach is applicable to other types of

sw

Requirements are that

The software has an interface (e.g. API, GUI)

Through which commands (stimuli) can be sent

Through which results (responses) can be received

Need (some) specification

Optional: Sample test cases, API usage examples



Conclusion

MBT works well for testing of multi-tasking

architecture

In this case of a software bus

Parent/Child test architecture facilitates testing

Individual tasks cannot decide correctness of their own

Parent coordinates with children and asserts correctness

Models and generated state machines: a good spec!

Innumerable number of test cases from the model

Test cases are agnostic to cFE syntax but still executable

Need to be careful in managing the model’s

complexity

Abstraction is important

Otherwise the model will be as complex as the system under

test



Acknowledgement

Jan-Philip Quirmbach (Fraunhofer Intern)

cFE is open source – not an issue for foreign interns

Alan Cudmore (NASA GSFC)

OSMA SARP:

Martha Wetherholt (NASA HQ)

Ken Rehm (NASA IV&V)

Ricky A. Forquer (NASA IV&V)

This work was partly funded by SARP



Questions

Dharma Ganesan ([email protected])

mailto:[email protected]