For Official Use Only ©2010 BAE Systems 30 June 2010 FOPEN GXP System Framework Presented at the FOPEN GXP Kickoff Meeting At SRC on September 22, 2010

For Official Use Only

©2010 BAE Systems 30 June 2010

FOPEN GXP System FrameworkPresented at the FOPEN GXP Kickoff Meeting

At SRC on September 22, 2010

By Dr. Michael J. Shea

©2010 BAE Systems 2


Software ApplicationsSoftware Applications

Framework based on RIVET - Rapid Integration, Validation, and Engineering Technologies

• Supports Open Software Architecture with Integration Technology Tools

• Standards-based middleware and formats allow easy incorporation of components, services and systems

• Application developers kits isolate algorithm developers from details of transport and data marshalling

• Speeds Integration of R&D Systems Composed of Complex Algorithmic Applications

• Functional Test Units (FTUs) automatically identify instances of non-compliance with the stated architecture

• Interactive Analysis Packages (IAPs) support rapid - and effective – root cause analysis in huge datasets

• Allows for Rapid Evolution to Deployment and Improved Long Term Sustainability

• Extensive software and functional verification allows for improved testing on evolving functionality

• Code auto-generation allows for efficient evolution of the system architecture to address new capabilities

SensorsSensors

Control & MonitoringControl &

Monitoring

System Status

Comms.

RAM Loading

FTU Errors

Situational Awareness

In App Out

Transport InterfaceData Interface

FTUsFTUs

Data Player

Data Logger

XML FileXML File

IAPs

XML File



Application Developer’s Kits (ADKs) Allow Developers to Focus on Algorithms, Not Integration

FusionData Interface

TransportLayer APIFile HornetQTCP UDP

In Out Operator InterfaceData Interface


In Out

FTU

Data InterfaceTransportLayer API

SAN HornetQTCP UDP

In OutExtern GatewayData Interface


In OutSensor CommsData Interface

TransportLayer APISAN HornetQTCP UDP

In Out

Data StorageData Interface

TransportLayer APISAN HornetQTCP UDP

In Out

• Application Developer’s Kits (ADKs) Provide a Flexible Software Architecture• Simple XML-Beans-like API (Set, Get, Send Receive) for data exchanges (C++, Java, etc.)• Automatic downcasting allows interface extensions without changes to legacy applications

• Auto-Generation Reduces Turn-Around from Weeks to Hours for Architecture Changes• XML Beans for Java; CppGen for C++; Currently developing Matlab and .Net extensions• Automatically generate validators for common system testing of the transport objects



Application Developer Kit Provides• Human Readable Interface Control Document (ICD) which specifies the XML messages exchanged between components, comprising

• An XML schema• Semantic constraints and tests• The ability to rapidly respond to changes in ICD by auto-generating

marshalling/unmarshalling code • (See RIVETS sample ICD)

•Libraries for the middleware calls• Single, simple software API (Get and Send Methods) for all transports, e.g.,

JMS or file based• Can use file-based transport for development (ease of use) and JMS for

production (superior performance)• Just takes a parameter change, zero lines of code changed in application

• Test Harness that allows for components to be tested• Use provided sample data• Execute Functional Test Units



Functional Test Units Provide Automated Identification of Non-Compliance Errors

• Simple Syntax – Length of vector, String vs Float• Obvious Semantics – range checking, probabilities sum to 1• Mathematical Properties (Covariance is Pos-Def)• Multi-Field (within Message) Tests

• Track contained with Stated Bounding region•Multi-Message Constraint Tests

• Sensor Action in Response to Sensor Request• Set value method caches intermediate results. Test method has access to local cache• Note: Multi-Field and Multi-Message use same underlying implementation



Run Monitoring Gives an Overview of Errors in Run

• Quick insight into running system• Provides look back into saved Archives



Time Shift FTU teaches us the value of embedding the likely failure modes into the Test Suite

• FTU Developed As Result of Interactive Error Analysis

• Poor Chi-Squared Analysis Indicated A Problem• User Attempted To Minimize Error By Removing

Biases• Time Shift Bias Detected and FTU Generated To

Hypothesize Time Shift as Cause of Error

• Many Variants Surfaced Throughout the Program

• 84 Second Shift on Video – “Yeah, the Clock on the Plane Were Off that Day”

• 14 Second Shift Resulting from Difference between UTC and GPS

• 28 Second Shift – Firmware Sign Error in Sensors• 4 Seconds Shift from Tracker Error• 1 Hour Shift from Time Zone Error

-40 -20 0 20 400

20

40

60

80

100

120

140

Chi-S

quar

ed A

vera

ge V

alue

FTU Detects 14 Second Offset in Input Data



Testing Harness Configuration and Monitoring

Get System Under Test

Get Test Infrastructure

Configure System

Get Test Data / Sim Configuration

Run Test

Monitor System Running

Analyze Run

Archive Results



How Automated Testing Comes together in the SIL

1. Component tests at developer site using ADK with Acceptance Test Harness

2. Component delivered with VDD3. Component installed and tested at SIL with identical Acceptance Test

Harness ensures valid delivery4. Component tested within live system configuration5. Automated FTUs identify any known issues6. Components with any issues notified with accompanying data sets and

scripts to reproduce errors7. New data set added to Acceptance Test Suite8. RepeatIf components test without known issues:9. IAPs explore statistical semantic and performance10. Potential issues identified and components notified11. If failure condition verified, new FTU developed to track issue over time12. Repeat



Strawman FOPEN GXP Ground Station

RD Maps (STANAG 4607)

Framework (provides networking and message transport)

FORESTER Interface /

FOUO

Key

Grey lines represent ethernet connections

Black arrows indicate message transmission directions

AD

Disc

RD Maps (STANAG 4607)

MTI Reports (STANAG 4607)RD Feature (XML)

Activity Reports (XML)

GSE

Tracks (XML)Group States (XML)Composition Likelihoods (XML)

All data products sent to analyst Display

• The figure shows the simplest scenario in which the Modules act sequentially (and all pass their output to the Display).

• Input and output of each component to be defined during the course of the program

• Opportunities for Modules to provide feedback to each other

• Opportunities for Modules to provide feedback to sensor

Disc

GatewayResourceManager



Integration at SRC• Vendors will develop

components, validate with canned data at their own facilities• Can use the ADK or implement

their own tools

• SRC will integrate system at their Software Integration Lab (SIL)• Will use Framework and ADK

to perform system test with canned data

• SRC will perform final test at their SIL• Live data generated from roof-

mounted sensor



Conclusion• Framework will be used to integrate the FOPEN GXP Ground Station• Framework provides a variety of tools that speed the integration

process• The ADK makes these tools available to all developers



Backups



System Specification Uses Industry Standard Tools to Precisely Define the Syntax and Semantics of Interfaces

• XML Schema Definitions Provide Standard To Rigorously Define Interfaces• XSDs allow for COTS tools to create, review, and

perform simple validations• Computer readable format allow for autogeneration of

additional tools• Non-proprietary interfaces reduce contractor ramp-up

times and increase future extensibility• Semantically Tagged ICDs Enable Tool And Test

Reuse Across Interfaces and Programs• Defines meaning and constraints of messages beyond

format or type• Testing and validation tied to semantically tagged

reusable message subcomponents• Pre-existing subcomponents can be wrapped in non-

proprietary interfaces and inserted into new systems while leveraging semantically based tool set

COTS tools allow quick generation of human and computer readable interfaces

<xs:simpleType name="HandsetId"> <xs:annotation> xs:documentation>Handset Identifier, type: uint32, valid range: 0-99, 0 indicates Base Stim</xs:documentation> </xs:annotation> <xs:restriction base="xs:int"> <xs:minInclusive value="0"/> <xs:maxInclusive value="99"/> </xs:restriction></xs:simpleType> <xs:simpleType name="RadioType"> <xs:annotation> <xs:documentation>Radio Type, type: char[32], valid range: model </xs:documentation> </xs:annotation> <xs:restriction base="xs:string"> <xs:length value="32"/> </xs:restriction></xs:simpleType>



DataFile

Functional Test Units (FTUs) Provide Flexible and Extensible Automated Compliance Testing

• Functional Test Units Evaluate a Component’s Compliance to Architecture Semantics• Semantic tests ensure messages are

consistent with their ICD description• Semantic Validation Core (SVC) is pluggable

as a library or a stand-alone service, either real-time or off-line

• SMEs define architecture compliance through ICDs, anyone can run SVC

• Functional Tests are Re-Used Across and Within Programs• Pre-defined tests are automatically composed

to test in accordance w/ ICD• User-defined tests for any message are linked

in symmetrically to baseline tests• Adding tests to regression suite ensures

resolved issues stay fixed

Middleware

App 1 App 2 SVCResults

SVCResults

Wrapper

Real-time SVC can be used along side an application or off-line to listen to and

validate messages

Wrapper

Middleware



Statistical Analysis FTUs Reveal Subtle and Hard-to-Find Model Issues

• Statistical Analysis Isolates Causes of System Failure

• Can’t model component’s complex transfer functions

• Single sample tests are unreliable due to stochastic nature of components

• Components provide implied distributions (e.g., covariance matrix)

• FTUs Decouple Component Consistency from Performance

• Sample component output normalized error relative to truth data

• Evaluate statistics, e.g., Chi-Square, each stage of processing in the system

• Jumps in Chi-Square value indicate component failure or model mismatch

App 1

App 2

Sensor

Truth

Measurements

State Estimates

State Estimates

FTU 2

FTU 3

FTU 1



Interactive Analysis Packages (IAPs) Mines Massive Data Sets to Speed Up Tedious Analysis of Complex System Issues

• IAPs Use FTUs to Isolate Specific Instances of Potential Failures or Model Mismatches• Here, Chi-Square tests are run across every

detection and track estimate in a scenario• The user quickly sorts data and scans across

multiple metrics (e.g., position-only or position and velocity)

• She prioritizes the most egregious values for deeper investigation

• IAPs Provide Focused Analysis to Rapidly Delve into the Root Causes of Failures• Here, the user selects specific tracks to analyze

further• The IAP pulls data from multiple sources, e.g.,

detections, tracks, and truth• She browses different aspects and views of the

data elements to rapidly deepen her understanding of the underlying conditions

IAPIAP

True target velocity relative to sensor isreported with a sign inversion



In App Out

Transport InterfaceData Interface

Distributed System Testing with ADKs Provides Meaningful Pre-Integration Testing, Acceptance Testing and Bug Reproduction

• Test Infrastructure Supports Geographically Diverse Contractors• Message based system design enables generic

test harness• Data player sends messages in the same order and

frequency as they occur In the system• Transport agnostic communications allow for light

weight harness at contractor sight• FTUs and IAPs Provide Automated Insight Into

Live And Replayed Runs at Each Site• SVC provides semantic validation through SME-

generated tests• IAPs and Statistical FTUs available for in-depth data

analysis• PTUs automate performance validation and alerting

when truth data is available

FTUsFTUsIAPsIAPsAPP

Data Player

Data Logger

Test harness with associated functional tests gives contractors the means to

locally simulate the system conditions

XML FileXML File



Example: Identification of Difference in ICD Semantic Interpretation

0 20 40 60 80 1000

50

100

150

200

• Basic Radar Tracking Use Case • Single Target, Good Sensor Coverage• This Should Work!

• How the Problem was Identified:• Looks Good to Naked Eye, but …• FTU Detected Issues at Particular Times• Track State Estimates Repeating• Tracker Outputs Repeated Estimates when no

Measurement Occurred

• Results:• Early Identification in Key Semantic

Misunderstanding between Applications• Automated System Detected Likely Problem

Points

Approximately 14 Sigma

Chi-S

q St

atist

ics

Position Plot Looks Fine

FTU Identifies Unlikely Statistical Outliers



Example: Time Offsets are Common in Real-Time Integrated System

• Initial Case• Sensor Simulator Bug• Resulted in a 7 Second Time Shift

• Many Variants Surfaced Throughout the Program• 84 Second Shift on Video – “Yeah, the

Clock on the Plane Were Off that Day”• 14 Second Shift Resulting from Difference

between UTC and GPS• 28 Second Shift – Firmware Sign Error in

Some Sensors• 4 Seconds Shift from Tracker Error• 1 Hour Shift from Time Zone Error

-40 -20 0 20 400

20

40

60

80

100

120

140

Chi-S

quar

ed A

vera

ge V

alue

FTU Detects 14 Second Offset in Input Data



Example: Fisheye Effect Corrupts Video Data

• Engineers Noted Poor Video Tracker Performance• FTUs Isolated a Set of Problem Cases• Some Tracks were Consistent• Others were Unreasonably Inconsistent

• Issue Tracked Down to a Problem in the Video Metadata• Focal Length Metadata Inaccurate at Low

Grazing Angles• Resulted in a Distortion Proportional to the

Targets Position in the Image• Metadata Correction Removed the Issue FTU Identifies Statistical Bias Caused by

Errors in Video Metadata

-Track-Truth-Covariance



RIVETs Help Prepare VADER Exploitation Ground Station (VEGS) Radar Exploitation Capabilities for Theater Deployment

• Open Architecture Ensures EGS is Compatible with Current & Future Theater Infrastructure• Development by Northrop Grumman, BAE

Systems, Army, and w/ NCCT Interface Module• Non-proprietary, standards-based ICDs and

transport, e.g., XML, NITF, STANAG, JMS• VEGS ADKs Enable Distributed Development,

Integration, and Testing Prior to Integration in SIL• Developers use same comms libraries as in real-

time system => no code change at SIL• Test harness contains collected and simulated

data => rigorous pre-integration testing• Test Units (FTUs and PTUs) Provide Message

Validation And Performance Assessments• Distributed with ADKs to support developers• FTUs used in SIL or field for validation; PTUs

used when truth is availableCoherent Change

Detection

Coherent Change Detection

Long-TermTracking

Long-TermTracking

DismountExploitation

DismountExploitation

VADER Ground Station

Application

Communications Backbone

OperatorInterface

ExploitationGateway

VADER RadarVADER Radar



RIVET has been Used on a Diverse Set of Research and Development Efforts

• RIVET Supports Research Efforts Allowing the System to Efficiently Evolve as the Understanding of the Problem Evolves• Distributed test harnesses allowed reproduction of system issues at local sites to track down

subtle bugs on the Dynamic Tactical Targeting program • Automatically generated interface and test code allowed rapid system reconfiguration to explore

system dynamics on CleanSweep program• Transport independence of ADKs allowed for a last minute switch to OpenAMQ when JBoss

issues bottlenecked system performance on the CORTEX program• RIVET Supports Development through System Deployment

• Open architecture infrastructure has allowed for the upgrade of the system with new sensors and exploitation components at each new deployment site for the SeeCoast program

• Application isolation allows for the simultaneous development of the Net Track system on both Intel and embedded PowerPC architecture platforms

• ADKs allow third party exploitation and tasking of a sensor with a proprietary interface under the VADER Exploitation Ground Station program

• Auto-persistence of system messages allowed for offline testing and offline forensic analysis for the MASINT Tactical Information Fusion program

Documents

For Official Use Only ©2010 BAE Systems 30 June 2010 FOPEN GXP System Framework Presented at the FOPEN GXP Kickoff Meeting At SRC on September 22, 2010