1

Analytical Support for Rapid Initial Assessment

Charles Twardy, Ed Wright, Kathryn Laskey,

Tod Levitt, Kellen Leister, Andy Loerch

George Mason University C4I Center

2

Topics •  Rapid Initiative Assessment (IA) challenges

•  Overview of Mason’s IA methodology

•  Example

2

3

Challenge: Analysis Support for Initial IA We focus here:

rapid initial assessment.

Models can also be reused here and beyond.

3

4

•  Rigorous, rapid, consistent, re-usable analytic justification for JIEDDO initiative assessments

•  Fulfills critical need as warfighter requirements grow while budgets tighten and scrutiny increases

Solution: Analysis Support for Initial IA

Rapid Initial IA Requirements •  Provide rapid assessments (days to

weeks) •  Model dependence of relevant Measures

of Effectiveness (MOEs) on system & environmental variables

•  Use available knowledge •  Identify information collection priorities •  Be consistent, repeatable, & extensible

5/19/10

1.  _ 2.  _

3.  Implement as Probabilistic Model

4.  Exercise Model & Analyze Results

5.  Determine Sensitive Parameters

6.  Report Results

Initiative Assessment Process

6

1.  Identify MOEs

2.  Generate Explanation

Example MOEs: Casualties per Incident Time to Complete Mission Weapons Intelligence Gathered

Partial Explanation Example: If there is an IED detonation during robot neutralization, Blue soldiers are not exposed. The robot may be damaged or destroyed.

Bayesian network (BN) model for

EOD robot

7

IA Approach Benefits •  Consistent framework for

assessing initiatives.

•  Clearly communicates to decision makers, the assessed impact, potential tradeoffs, and the mechanism by which it works.

•  Makes the explanation structured, explicit, executable, and reusable.

•  Perform what-if, try scenarios, test understanding, perform sensitivity analysis.

•  Enable development of more informative test plans.

•  Identify relevant MOEs.

•  Generate an Explanation of how the initiative is expected to affect MOEs.

•  Implement the explanation as a probabilistic model.

•  Execute & analyze model to assess performance

•  Determine the “sensitive parameters” (SPs) to help prioritize information collection.

ExplanationProbabilistic Model •  Generate explanation of how initiative affects MOE

–  Clutter can interfere with the ability of the sensor to detect IEDs and cause false positives

•  Implement explanation as Bayesian network (BN) –  Structured, explicit, executable, and reusable –  Models how initiative is likely to perform in

operation –  Supports what-if and sensitivity analysis

5/19/108

CPT for Sensor_Result

9

Available Knowledge •  SMEs (at JIEDDO and elsewhere) •  JUONS and other needs statements •  Initiative documentation •  Current suite of equipment & capabilities •  Additional contractor knowledge •  Blue and Red TTPs •  Previous initiatives •  Previous models •  Previous tests

9

New Class

?

New Initiative

Select model From

Repository

No

Model Analysis

Add

Performance Assessment &

Sensitive Parameters

OFFLINE: Enhance models

Yes

Model Sufficient?

Yes

Add

No (n+1)th

Iteration Model

nth Iteration Model

Model Development Spiral

Recursive Spiral Prototyping

10

Time Avail

?

No

Yes

Develop 1st Iteration Model

Modify Model

Model Repository

Analysis

5/19/1011

Try Scenarios in the Model, and examine the effect on the MOEs

For each MOE, find the most influential variables:

Intel. Potential redDetonatesRobot

redDetonation probDisableSuccess robotProbEffective

robotReadiness

Calculate individual link strengths:

Vary some parameters over their range:

12

Specific Technologies RECCE I

Cougar 6x6 Platform

Gyrocam (VOSS)

Remote Wpn Sys

EOD Robot

Comms

Duke v1

RECCE II Adds

LNS

Duke (v2)

EOD Robot In the Remote Deployment System

Remote Wpn Sys

VOSS on Mast

LNS

Photo from the (S) ATEC C&L Report, July 2008

Example 1 EOD Robot

Assess EOD Robot

- New Class? Yes

Select MOEs

Build 1st Iteration Model

13

New Class

?

New Initiative

Select model From

Repository

No

Model Analysis

Add

Performance Assessment and

Sensitive Parameters

OFFLINE: Enhance models

Yes

Model Sufficient?

Yes

Add

No (n+1)th

Iteration Model

nth Iteration Model

Recursive Spiral Prototyping

Time Avail?

No

Yes

Develop 1st Iteration Model

Modify Model

Model Repository

New Initiative

New Class

?

Yes

Develop 1st Iteration Model

MOEs by Tenet *

14

Tenet Potential MOEs Predict (Intell. Gathered)… Mission

Time (Cost)

Prevent Number and relative proportion of each type of IED tactic, Number or percentage of interceptions, raids, captures before emplacement, #IEDs/mission mile

Detect-Air P(detect), False Alarm Rate, Sweep Width, Rate of Advance Detect-Ground

P(detect), False Alarm Rate, Sweep Width, Rate of Advance, P(spot)

Neutralize P(neutralize), Neutralize Time, Intelligence Gathered Mitigate Casualties/Attack, KIA/Attack, WIA/Attack, Damage/Attack

Some MOEs suggested by Perry et al., Minimizing the Threat from Improvised Explosive Devices in Iraq, RAND 2007 Some from the RECCE II Initiative Evaluation Plan (AMSAA, August 2008)

*Tenet: JIEDDO divided initiatives into “tenets” which roughly follow the “left of boom” timeline.

15

Identification of MOEs Assumptions The EOD robot provides a capability to remotely neutralize (disable or detonate) an IED. If the robot is not available or not successful, a soldier will neutralize the IED.

MOE Assumptions and Considerations Time Robot may take longer than an EOD soldier

If the robot is unsuccessful, we still must use a soldier P(neutralize by robot)

Distinguish disable from destroy

Casualties or Damage per Attack

•  Replace with generalized, qualitative P(damage) •  If Red detonates the IED during robot neutralization, soldiers are not

exposed. The robot may be damaged or lost. •  If the robot is unavailable, or fails, then a soldier will be at risk. •  If the IED is not spotted, robot has no effect on damage / casualties.

P(collecting valuable Intelligence)

•  If Blue disables the IED, it can be examined for forensic intelligence. •  If Blue detonates it, there may be some intelligence collected before

the detonation. •  If Red detonates it, there is little intelligence gained.

16

Implement Explanation as BN Model assumes IED is present and successfully detected. •  If robot is available and working

correctly, it can be used to attempt to disable or detonate an IED.

•  If the robot succeeds in disabling the IED, we can gather forensic intel.

•  Little intelligence can be collected if the robot detonates the IED.

•  If there is a Red detonation during neutralization, Blue soldiers are not exposed. The robot may be damaged or destroyed.

•  If the robot is not available or not successful, a soldier will be at risk while disabling the IED.

•  Using the robot may take longer than using an EOD soldier.

•  If unsuccessful, a soldier must still disable the IED.

1

2

2

3

3

1

Example 1 EOD Robot (2)

Assess EOD Robot

- New Class? Yes

Build 1st Iteration Model Add 1st Iteration Model to Repository

Time Available? No

Run the Model, Analysis

17

New Class

?

New Initiative

Select model From

Repository

No

Model Analysis

Add

Performance Assessment and

Sensitive Parameters

OFFLINE: Enhance models

Yes

Model Sufficient?

Yes

Add

No (n+1)th

Iteration Model

nth Iteration Model

Recursive Spiral Prototyping

Time Avail?

No

Yes

Develop 1st Iteration Model

Modify Model

Model Repository

New Initiative

New Class

?

Yes

Develop 1st Iteration Model

Select model From

Repository

Add

nth Iteration Model

Model Repository

Model Sufficient?

No Time Avail?

No

Model Analysis

Performance Assessment and

Sensitive Parameters

Robot Analysis 1: View Effects If the robot is not available … a soldier will be at risk while disabling the IED.

If a robot is available and it is working correctly, it can be used to attempt to remotely disable or detonate an IED.

Lower risk to soldier, more time

If the robot succeeds in disabling the IED, it can be examined for forensic intelligence. Less intelligence can be collected if the robot detonates the IED.

18

19

Robot Analysis 2: Sensitive Parameters by MOE EOD Robot: Top 5 Sensitive Parameters by MOE.

•  Assuming robotAvailable, and excluding deterministic functions

ClearTime Intelligence Damage redDetonatesRobot redDetonatesRobot redDetonation

redDetonation redDetonation redDetonatesRobot robotReadiness probDisableSuccess robotProbEffective

robotProbEffective robotProbEffective robotReadiness probDisableSuccess robotReadiness --

Next Steps (as time allows):

•  Investigate sensitive parameters in more detail

•  Extend / refine the model: additional variables, situations; extend or refine the state space of important variables; refine local probability distributions.

•  Identify knowledge requirements for the sensitive parameters

•  Seek additional information: SMEs, system documents, data collection, …

20

Summary Challenge or Need IA Methodology Solutions

Short timeline Spiral development: start simple, extend later Reuse models; Standard flow starting from MOEs

Little quantiative data / need to assess prior to testing

Probabilistic models can use available expert and prior knowledge as soft constraints; initiative models make use of any existing models onto which they are added

Need analytical support Probabilistic models are explicit representations of how the initiative is thought to work, and can be executed.

Prioritize information collection

Sensitivity analysis in the model can rank variables by influence, and show the effect of parameter changes

Consistent, repeatable, extensible

Standardized methodology based on MOEs leads to consistent assessment across initiatives. Model reuse provides repeatability and extensibility

Integrate with Portfolio Management

MOE statistics from the model feed into PM approach: casualties/damage, time, effectiveness

21

Questions?