““Flight Test Is Flight Test Is As Flight Test Does”As Flight Test Does”

Guide to Safe and Effective Flight Test

George CusimanoFlight Test Safety Workshop5 May 2010 1

Mamma always said: Mamma always said: ““Stupid is as stupid does” Stupid is as stupid does”

Forrest Gump circa 1994

• Meaning – You are what you do

• Applies to flight test as well as life in general

2

Safe & Effective Flight Test Is . . .

• Desired perceptions– Culture of safety– Manage risks– Credible and unbiased– Efficient and effective– Legitimate results– Requirements validation

3

. . . As Flight Test Does

• Real actions do not always reflect good intentions– Program pressure – test is an obstacle to progress• Cost• Schedule

– “We can do more with less” attitude– Poor risk assessments• Inadequate• Incomplete

– Poor systems knowledge

4

Guide to Positive ACTIONSACTIONS

• Truths for Safe and Effective Flight Test– Common sense– Historical basis– Mindful of safety

• “Stupidity” prevention

5

Guide to Positive ACTIONSACTIONS

• All models are wrong• Believe the data until it is proven wrong• Don’t be like a box of chocolates• Always challenge• When in doubt – Don’t• Testing the wrong thing right is as costly as testing

the right thing wrong• If at first you don’t succeed – check your plan

6

All Models are Wrong!

• Model: Representation of reality– Bounded by assumptions and simplifications– Valuable to the development process

• Flight Test: Reality – Validates models

– If predictions are incorrect – fix the models!

7

Believe the Data . . . Until . .

• Flight Test is truth . . . IF– Good instrumentation– Test configuration– On the correct condition– Applying the correct technique, correctly

• If the results are different than expectation . . .– Slow down– Stop

. . . until the differences are resolved

8

Believe the Data(Example)

• A tracking algorithm was supposed to run at 100 Hz.• During test the control room observed 10 Hz– Assumed to be a TM anomaly

• Post flight analysis revealed that 10 Hz was real.

9

Box of Chocolates

Mamma always said: Mamma always said: ““Life is like a box of chocolates, Life is like a box of chocolates,

you never know what you are going to get” you never know what you are going to get” Forrest Gump circa 1994

• Flight test should NEVER be like a box of chocolates; we should always know what to expect!

10

Set Expectations

• Predict flight test results– For every test point– Based on models and/or analyses

• If you do not know what to expect, then you do not understand the system

• If you do not understand the system, then you may not be testing the right thing right

• If you are not testing the right thing right, then you should not be conducting the test.

11

Set Expectations(Example)

• Objective: Expand forward CG limit– 12” forward of existing limit

• Incremental approach– First flight planned @ 1” in front of baseline limit

• Difficulty rotating with full aft stick; nearly departed• Did not understand why – decided to land . . .

12

Always Challenge

• What if something unexpected happens?– What will we do?– Have a plan

• Why did the system behave as it did?– Systems knowledge– Significance of error

• What next?– Do nothing – proceed– Suspend test point, but continue the mission– Stop the mission

13

Always Challenge(Example)

• Designed to test fighter thrust vectoring technology• High AOA test point• Pilot: “Something is not right. I show 280 KIAS and

20 alpha.”• Pilot did not persist in knowing why• Control room did not suspend test while they

investigated• Result: Lost aircraft

14

When in Doubt – Don’t!

• If test results do not meet expectation– Ponder– Wonder– Debate– Decide

• Listen to your instincts• The obligation to stop a test resides with every

member of the test team

Proceeding without knowledge = disaster15

When in Doubt(Example)

• Objective: Expand forward CG limit– 12” forward of existing limit

• Incremental approach– First flight planned @ 1” in front of baseline limit

• Difficulty rotating with full aft stick; nearly departed• Did not understand why – decided to land . . . . & try again• Repeated problem on 2nd takeoff• Post flight – CG 13” in front of existing forward limit

16

Testing the Wrong Thing Right;Testing the Right Thing Wrong

• Both are costly and inefficient• How we get trapped:– Not having well defined test objectives – Not understanding the system under test

17

Testing Right(Example)

• RADAR test at maximum slant range• Weather forced the test to higher altitude and

closer horizontal range • Four hours of data collected• Result: Invalid test– Test had to be repeated

18

If at First You Don’t Succeed --Check Your Plan

• Based on well defined requirements– Relevant test objectives – Data required to meet the objectives – Methods needed to collect that data– How the collected data will be analyzed to show

compliance – Must include resources

19

Check Your Plan (cont.)

• Without a good test plan you do not know: – Why you are testing– How you will test– When you are finished testing– What to do with the collected data

• Plans change– Configuration control is required

20

Check Your Plan(Example)

• F-117• Night• Navigation triangles with autopilot• Pilot decided to hand fly one turn– Disengaged autopilot– Black hole phenomenon

• Almost departed the aircraft.

21

Configuration . . .

• Test article– Testing the right thing

• Test plans– Testing requirements

• Instrumentation– Valid data

• Test methods and conditions– Testing right

Know What You’re Testing22

Common Element

Test Article

Test Plan Instrument Methods &

Conditions

Models are Wrong Not real Limited by

simulationNot part of simulation

Limited by simulation

Believe the Data

SetExpectations

AlwaysChallenge

When in Doubt

23

Common Element

Test Article

Test Plan Instrument Methods &

Conditions

Test the Right Thing

Planning is Everything

24

Summary

• “Stupid is as stupid does”– Testing without knowledge is stupid

• Do NOT be like a box of chocolates– Always know what to expect

25

Know What You are TestingKnow What You are Testing

““That’s all I have to say about that !”That’s all I have to say about that !”Forrest Gump circa 1994

26