A System Analysis Study Comparing Reverse Engineered

Combinatorial Testing to Expert Judgment

Atlee M. Cunningham, Jr., Jon Hagar, Ryan J. HolmanLockheed Martin

Jon.d.hagar@lmco.com

2

Agenda

Introduce the trade study space: the F16 and Combinatorial Test (CT) problem

Define the F16 Failure

Present the steps of the CT study

Cover the results

Conclusions

3

INTRODUCTION: F16 VENTRAL FIN STUDY AND APPLYING COMBINATORIAL TESTING

Evaluate the use of Combinatorial Testing (CT) to a real “problem”

–Used a historic F-16 problem and data–See if CT could be used in place of or to support an

expert• Save time and/or people• Mixed example, between what was done and what

could have been done

Problem space–Interacting factors (good for CT)–Outside of the software testing–System-Hardware failure resolution and design evaluation –Demonstrate CT is viable

4

F16 Failure Case Study

During production and maintenance of the F-16 fighter aircraft a structural problem immerged

– Buffeting of the F-16 ventral fins has provided a classic example of structural fatigue

of such aerodynamic surfaces by an upstream source of severe turbulent wakes

– These fins are very thin surfaces, about 5 ft. chord and 2 ft. span, composed of three

wedge like surfaces that taper down to edge thicknesses of 0.05 inches, all of which

makes the fins susceptible to turbulence buffeting

– Examples of possibly interacting turbulence sources include: various centerline

stores, side slips and inlet lip spillage during rapid decelerations

The historic work done by Atlee M. Cunningham, Jr. and Ryan J. Holman

The Combinatorial analysis and case study was primarily done Jon Hagar with support from Atlee Cunningham as the “expert”

5

ORIGINAL F-16 PROBLEM DETAILS

Added 2 avionics LANTIRN pods on the F-16 just aft of the inlet on the lower fuselage directly upstream of the ventral fins

Avionics pods in general are often not very aerodynamic in shape and hence can produce very turbulent wakes

The damage to the right hand ventral fin on first flight with LANTIRNS

– Originally, the primary source of the fin’s fatigue and loss was high speed throttle chops

that produced severe turbulence from inlet lip spillage during rapid decelerations where

the throttle was suddenly pushed to idle position

A comparison of the ventral fin response to LANTIRN and throttle chop turbulence was done

– The response levels are about the same; however, constant buffeting by the LANTIRNS

produced much higher fatigue damage per flight hour as compared to that due to the

transient throttle chop

– As a result, several major structural re-designs of the fins and other associated structures

followed over the following years that incrementally improved the fatigue life of these

components

6

F16 with LANTIRN Pod and Ventral Fin

7

Failure: Damaged Ventral Fin

But Why (what parameters and interactions)?

8

ORIGINAL ANALYSIS HISTORY

After a number of years more the problem continued. As a result, a detailed analysis of the flight data was performed by

Atlee Cunningham , yielding

–Showed that the most severe buffeting of the ventrals consistently occurred with only LANTIRN pods on the aircraft and with high speed throttle chops at Mach 0.95 on the clean aircraft

–Anomalous trends were also seen in throttle chop data with LANTIRNs where response levels were 3-to-4 times as high as level flight with LANTIRN

–Recognizing that the very thin ventrals (leading edge thickness is 0.05 in.) would probably be subject to leading edge separation at small angles of side slip

–Flow change resulted in a large increase in the slope of side force with side slip angle, which would have a significant impact on dynamic loads due to large amplitude turbulence

9

Do More Testing and Analysis (by experts)

A low speed small scale wind tunnel test was conducted to

– Explore various aspects of ventral aerodynamics and effects of modifications

– Data were obtained with 1/5 scale models of the fin mounted on the wind tunnel wall and

rotated for incidence effects

– Testing was to determine “sensitivities” (variables and values) but had to be designed by

expert

Flight Tests were conducted

– Three of these four fins, plus several early block ventral fins, were tested on an early Block 15

F-16. The fins consisted of:

(1) the baseline fin, “BSLN,” the standard Block 40 ventral fin;

(2) the “MMC” fin, the Block 40 fin with 40% stiffer skins of MMC aluminum material

(3) the “MMCNC” fin, the MMC fin with an added rounded “nose cap” glove with a NACA 0012

airfoil section of 5 inch chord

(4) the “NACA” fin, the Block 40 modified to have a full span, full chord airfoil section that

eliminated the sharp leading edge and sharp tip section of the fin

– An expert had to define test program (combinations) for these too = hundreds of hours

10

DEFINING CONDITIONS FOR CT

What was the situation(s) that brought the failure on? Factors considered include:

– Aircraft (AC)

– Maneuver

– Speed (Mach)

– Altitude

– Aircraft add on structures (tanks, pods, etc.)

Which design solutions (4 fin designs) might solve the problem with different aircraft configurations:

– Block 15

– Block 40

11

IDEALIZED ANALYSIS STEPS USING CT (HYPOTHETICAL RECONSTRUCTION)

NIST ACTS Combinatorial Tool was used to “reverse” engineer a test program

–Other tools were considered

–Open source nature was deciding factor

This can be viewed as a “reverse” or “Re” engineering case study

– We were trying to see if the tool would replicate the historic test program without an system expert

– Test planning using a CT tool (not the expert)

A series of idealized steps were done using the tool

12

CT Step 1

Historic “first” test program - clean baseline configuration, which in the example are F16s in block 15 and 40 in “clean” configuration, and apply “testing” to points associated

Input to tool (equivalence classes):

Tool produced: 90 test cases (similar to actual effort) with 2 way

Aircraft 15 , 40

Altitude (s) 5k, 10k, 15k, 20k, 30k, 40k, 50k  

Maneuvers hi-speed throttle , slow accel/dwell , L/R 5deg side slip , L/R 360 roll , R/L 360 roll, R/L 5deg side slip, Med accel/dwell, R-L-R-L banking, Hi-speed to Low, 360 nose roll

 

Mach(100th) 40, 50, 60, 70, 80, 90, 100, 110, 120  

Parameters: Variable:

13

Step 2: Refinement of the test program

Step 2 a more refined set of analyses would have been done based on information from:

– Step 1

– Historic databases

– More detailed wind tunnel analyses

– Supplemental water tunnel analysis

– Flight data and constraints were available

This effort confirmed design work

– Produced 30 test cases from 2 way coverage

Parameters: Variable:Aircraft 15 , 40Alt 5, 10, 15Mach(100th) 60, 80, 85, 90, 95LANTIRN on, off

14

Step 3: Final Design Flight Test Program

Parameters: Variable:

AC-BLK&Ventral-Fin-Config

Blk15-Blk15 ventral, Blk15-Blk40 ventral, Blk15-MMC ventral, Blk15- MMC ventral + cap, Blk40-Blk40 ventral, Blk40-MMC ventral, Blk40-MMC Ventral+ nonosecap, Blk40-NACA

LANTIRN [on, off]Alt [5, 10, 15]Mach (100ths) [60, 70, 80, 85, 90, 95]Maneuver Block [basic, basic +]

Tests for a flight test program

Number of test (cases) generated with 2 way coverage: 72

15

Conclusions An example where combinatorial test could have aided

– Provided another test design method for teams to use

– Reduce the "shotgun” approach and expert judgment needed for situations dealing with “many” parameters

–Showed CT can support a system failure (fault isolation) analysis Historic data useful in a CT proof on concept (case study example) Lockheed Martin will continue advocating CT as a technique

–Looking for pilots and more data points

–Would be interesting to compare to DOE approaches

–How to get Engineers to start using Other items noticed

–Tool interchange (operability), particularly into a test automation framework

–Constraints were “tricky”

– Interface to/from Model based testing would be useful

16

Summary

Demonstrated Combinatorial Test tool could have supported the F16 problem (or other hardware, software, system test/analysis)

–Expert felt results would have been similar

–Approach could support other programs

Open source tool worked

–Commercial tools worked too

–Supports move from “theory” to real use

Supported an “non” software area

– System Design/Failure Evaluation