1

Flight Testing Of Fused Reality

Visual Simulation System

Justin Gray, Systems Technology, Inc.

13th Annual AIAA Southern California Aerospace Systems and Technology (ASAT) Conference April 30th 2016, Santa Ana, CA

2

• General Objective

• Fused Reality

• Inflight System Description

• Test Methods and Conditions

• Test Results

• Outcomes and Observations

• Lessons Learned

• Conclusions

Overview

3

General Objective• To assess handling qualities, there is no substitute for flight test.

~ Simulators have improved fidelity, but the human physical response to the dynamic response of the test aircraft can still not be adequately replicated in a simulator.

~ Flight testing comes at a much higher cost than simulation, especially when additional flight assets, such as aerial refueling tankers, are required for the testing.

• What if important handling qualities evaluations tasks that include aerial refueling, formation flying, etc. could be accomplished without the support aircraft?

~ This will not only save valuable flight test dollars, but it will also provide enhanced safety, since additional flight test assets will not be placed in close proximity to the test aircraft.

~ A flight test evaluation system based on a novel head mounted display visual system that combines virtual and real world elements was developed by Systems Technology, Inc.

• This presentation describes a flight test evaluation of such a system known as Fused Reality.

4

Fused Reality: Helicopter Cabin Gun Trainer

• Fused Reality combines real time video via a camera on a head mounted display and virtual elements to provide an overall immersive environment.

• The original concept used a real-time blue screen technique as the keying technology that distinguishes real from virtual:

~ Wherever the keying color is seen in the video, the color is replaced by the virtual scene..

~ IR markers are used to track the position of real world objects that allows for a seamless interaction between the real and the virtual world.

5

Fused Reality: Initial Inflight System• Initially, a cockpit window border keying technology using two color bands

was employed:

~ Under consistent lighting conditions, the approach worked well and led to an effective demonstration.

~ As the program advanced, limitations of the border technology were exposed as integration with an actual aircraft was attempted.

~ Sensed electromagnetic properties in the out-the-window scene were used to successfully key out the cockpit window area.

• This allowed for an inflight technology demonstration and evaluation using the Calspan Learjet Inflight Simulator.

6

Fused Reality Inflight System Description: System Architecture

7

Fused Reality Inflight System Description: System Components

IR tracking cameras

8

Test Methods and Conditions: Evaluation Aircraft – GA8 Airvan

• The GippsAero GA-8 Airvan (N215AV) is a strut braced, high wing, eight seat, Federal Aviation Administration (FAA) type certified aircraft.

• The GA-8 has fully reversible flight controls driven by conventional mechanical pulleys and pushrods.

• Because of modifications, the aircraft has a special airworthiness certificate in the experimental category for research and development.

9

Test Methods and Conditions: Safety & Evaluation Pilots

• Approximately ten hours of flight time in the NTPS Airvan were used in support of this program:

~ Roughly half of those hours dedicated to system integration and checkout

~ The remaining hours to formal evaluation sorties

• Four NASA Armstrong test pilots participated in the formal evaluation flights over several days in mid-January 2015.

10

Test Methods and Conditions: Evaluation Tasks

Drogue Tracking Formation Flying – Stencil Mode

Formation Flying – Virtual ModePrecision Landing

11

Test Methods and Conditions: Evaluation Procedures

• Precision Landing

~ 500’ AGL Virtual Altitude

~ Centerline approach and landing

~ Offset approach and landing (200’ offset, correction at 200’ AGL)

~ Desired/Adequate Landing Boxes: ±25’/±50’ lat, ±250’/±500’ lon

~ Desired/Adequate Airspeed: ±5/±10 KIAS

• Drogue Tracking

~ 5 minutes of practice followed by 30 second evaluation

~ Desired/Adequate (Lateral/Vertical): ±6’/±12’

• Formation Flying – Virtual & Stencil Mode

~ 5 minutes of practice followed by 30 second evaluation

~ Desired/Adequate (Station Keeping Guides): ±5’/±10’

• NVG Mode Demonstration, time permitting

12

Test Results: Fused Reality Video

13

Test Results: Precision Offset Landing

• Example Pilot 4 offset landing run shown here received HQR 3

• Pilot 4: “Lag is not a factor at all… …what I am seeing is the airplane through the system and the system is not in the way.”

• Successful “landings” achieved by all four evaluation pilots

• Indicates that the FR system can be used to conduct repeatable precision approach and landing evaluations at altitude.

0 10 20 30 40 50 60 70 80-100

0

100

200

300

400

500

Time (sec)

La

tera

l an

d V

ert

ica

l P

ath

to

Ru

nw

ay

(ft

)

Lateral Offset

Virtual Altitude

0 10 20 30 40 50 60 70 800

10

20

30

40

50

60

70

80

90

100

Time (sec)

Air

sp

ee

d (

KIA

S)

14

Test Results: Drogue Tracking

• Example Pilot 4 drogue tracking run shown here received HQR 8

• Pilot 4: “Very loose laterally.” “This is really hard.” “A little more field-of-view would help the fore/aft cues.”

• The Airvan was never intended to perform such a demanding task and control limitations, particularly in the lateral axis, were challenging for the pilots.

140 145 150 155 160 165 170-10

-8

-6

-4

-2

0

2

4

6

8

10

Time (sec)

Tra

ck

ing

Err

or

(ft)

Lateral Tracking Error

Vertical Tracking Error

140 145 150 155 160 165 170-60

-50

-40

-30

-20

-10

0

Time (sec)

Fo

re/A

ft L

on

git

ud

inal

Err

or

(ft)

15

Test Results: Formation Flying

• Example virtual mode formation flying task shown here received HQR 4

• Pilot 4: “The fore/aft was the hardest thing for me to learn, but once I got a better sense for that, it made it a little easier.”

• The evaluation pilots were able to successfully formation fly with a virtual target aircraft using the FR system in full virtual and stencil modes.

150 155 160 165 170 175 180-5

-4

-3

-2

-1

0

1

2

3

4

5

Time (sec)

Tra

ck

ing

Err

or

(ft)

Lateral Tracking Error

Vertical Tracking Error

150 155 160 165 170 175 180-10

-8

-6

-4

-2

0

2

4

6

8

10

Time (sec)

Fo

re/A

ft L

on

git

ud

inal

Err

or

(ft)

16

Lessons Learned

• Checkout flights were vital to insure that the Fused Reality system was properly integrated with the test aircraft.

• For flight test projects with limited flight hours, inflight trouble shooting should only be done when absolutely necessary.

~ Better to land and fix the problem on the ground than burn flight hours trying to solve the issue in the air.

• Different flight test organizations have different rules, processes, procedures and philosophies, but the common goal is safety first.

~ Communication, respect, an open mind, and concentration on that common goal enable successful collaborations.

• When at a standstill, the FR navigation can drift because it keys off of ground track, but as soon as the aircraft is moving, the system locks on to the correct position.

17

Lessons Learned (continued)

• FR system needs to initialize from a stabilized flight condition at the start of each evaluation run.

• Sun angle was at times a factor for the IR cameras used for the evaluation pilot head tracking and for the video camera attached to the HMD.

~ Best results were achieved when the cameras were not pointed directly at the sun.

• Regarding the sun angle, when the sun angle was to the evaluation pilot’s right, the right eye display could wash out.

~ NVG-like blinders attached to the HMD could alleviate this issue.

• Regarding the evaluation tasks:

~ Markings that provided next to the virtual runway that provided an aim point for the lateral offset would aid the precision offset landing.

~ Field-of-view of the HMD can limit longitudinal closure cues used in the drogue tracking and formation flying tasks.

18

Conclusions

• Using a NTPS Gippsland GA-8 Airvan research aircraft, a new stand-alone system for closed-loop handling qualities assessments was successfully demonstrated that allows virtual out-of-the-window elements to be combined with the real-world cockpit visual environment.

• As a result of these flight tests, all of the issues uncovered in the initial flight test campaign on the Calspan Learjet Inflight Simulator and during the initial GA-8 checkout flights have been corrected and tested in flight.

• Many enhancements for Fused Reality as well as other potential training uses were identified by the team which will aid in improving and commercializing this technology.

19

Questions?