The Safety Benefits of LOSA: A Manufacturer’s View

The Safety Benefits of LOSA: A Manufacturer’s View

The Safety Benefits of LOSA: A Manufacturer’s View

R. Curtis Graeber, Ph.D. Senior Technical Fellow

Chief Engineer, Human FactorsBoeing Commercial Airplanes

International LOSA & TEM Workshop, Toulouse, France November 16, 2006

Int’l. LOSA & TEM Workshop

Today’s TopicsToday’s Topics

A Proactive Approach to Safety

TLC-Boeing Collaboration

LOSA’s Unique Contribution to Safety

Learning from LOSA data

Future Opportunities

A Proactive Approach to Safety

TLC-Boeing Collaboration

LOSA’s Unique Contribution to Safety

Learning from LOSA data

Future Opportunities


Boeing’s Approach to Safety Boeing’s Approach to Safety

Be proactive in research and promoting safety

Learn from incidents as well as accidents

Learn from the experience of our customers

Continuously improve the Airplane Life Cycle process

Seek Global Involvement – Airlines– Other Manufacturers/Sub-contractors– Regulatory agencies– Safety and flight crew organizations

Be proactive in research and promoting safety

Learn from incidents as well as accidents

Learn from the experience of our customers

Continuously improve the Airplane Life Cycle process

Seek Global Involvement – Airlines– Other Manufacturers/Sub-contractors– Regulatory agencies– Safety and flight crew organizations

Global Aviation Safety Roadmap

Industry commits to unify plans through Roadmap

5/06 Gabon-5

ICAO Global Aviation Safety Roadmap

Goals and Objectives:

• Provide a common frame of reference for all stakeholders

• Coordinate and guide safety policies and initiatives worldwide to reduce the accident risk for commercial aviation

• Avoid duplication of effort and uncoordinated strategies

• Encourage close industry and government cooperation on common safety objectives

Timescale: Near (2006-10) and Medium Term (2010-14)


Boeing’s Safety Approach: Focus on Human Factors

Boeing’s Safety Approach: Focus on Human Factors

Collaborate globally on external research– NASA, FAA, NLR

– Madrid BRTE

– Customer airlines

– Universities (Illinois, Michigan, Cranfield, Delft, Massey, etc.)

– University of Texas HF Project/The LOSA Collaborative

• Create safety tools to support SMS at airlines– Flight Operations Support Program (FOSP)

– Procedural Event Analysis Tool (PEAT)

– Maintenance & Ramp Error Decision Aids (MEDA, REDA)

– Cabin Procedure Investigation Tool (CPIT)

– Line Operations Safety Audits (LOSA)

Collaborate globally on external research– NASA, FAA, NLR

– Madrid BRTE

– Customer airlines

– Universities (Illinois, Michigan, Cranfield, Delft, Massey, etc.)

– University of Texas HF Project/The LOSA Collaborative

• Create safety tools to support SMS at airlines– Flight Operations Support Program (FOSP)

– Procedural Event Analysis Tool (PEAT)

– Maintenance & Ramp Error Decision Aids (MEDA, REDA)

– Cabin Procedure Investigation Tool (CPIT)

– Line Operations Safety Audits (LOSA)


The LOSA Collaborative (TLC)Boeing MOU Agreement

The LOSA Collaborative (TLC)Boeing MOU Agreement

• Signed May 2002

• Assures airline and crew member confidentiality

• Enables TLC to provide customers a two-way channel of communication with Boeing

• Establishes a mutually valuable research and safety partnership


The Value of CollaborationThe Value of Collaboration

For The LOSA Collaborative:

• Manufacturer’s perspective and expertise in global aviation safety

• Scientific and expert technical support

• Detailed design knowledge

• Comparisons with findings from other sources including FOSPs

• Expanded opportunity to benefit all airlines

• Enhanced resources (data collection software)

For The LOSA Collaborative:

• Manufacturer’s perspective and expertise in global aviation safety

• Scientific and expert technical support

• Detailed design knowledge

• Comparisons with findings from other sources including FOSPs

• Expanded opportunity to benefit all airlines

• Enhanced resources (data collection software)


The Value of CollaborationThe Value of Collaboration

For Boeing:

–Synergistic research partnership

–Invaluable, objective insights about how our products are operated in “normal” situations

–Access to information about crew errors during normal operations (ARCHIE)

–Insights about how crews recover from errors

–Objective crew performance data to guide design of future airplanes and procedures


Improving the Crew InterfaceImproving the Crew Interface

• Accidents Investigations

• Incident Investigations

• Crew Reports: ASAP, STEADES, etc

• Recorded airplane data: FDM/FOQA

• Normal Operations

– Airline Feedback

– LOSA

– Boeing FOSPs, in-flight observations

• Accidents Investigations

• Incident Investigations

• Crew Reports: ASAP, STEADES, etc

• Recorded airplane data: FDM/FOQA

• Normal Operations

– Airline Feedback

– LOSA

– Boeing FOSPs, in-flight observations

Potential Data Sources:


Accident DataAccident Data

• Example: NTSB, AIB, BASI etc.

• Extraordinary detail about rare single events

• Can be a high value data source when applicable

• Very dependent upon expertise and processes of investigating authority

• Poor comparability and usually too few worldwide to understand HF issues

• Always too late!

• Example: NTSB, AIB, BASI etc.

• Extraordinary detail about rare single events

• Can be a high value data source when applicable

• Very dependent upon expertise and processes of investigating authority

• Poor comparability and usually too few worldwide to understand HF issues

• Always too late!


Incident and Crew Report DataIncident and Crew Report Data

ECCAIRS, ASRS, CHIRP, ASAP, STEADES

• Key factors frequently identified

• Participant recollection can be suspect

• Level of detail varies, especially regarding equipment

• Extremely high value data source to corroborate other data

• Can support precursor search

ECCAIRS, ASRS, CHIRP, ASAP, STEADES

• Key factors frequently identified

• Participant recollection can be suspect

• Level of detail varies, especially regarding equipment

• Extremely high value data source to corroborate other data

• Can support precursor search


FDM/FOQA DataFDM/FOQA Data

• Proprietary and closely protected by operators

• Very detailed objective airplane outcome data from normal operations

• Uniquely suited for multi-flight statistical analysis

• Limited in scope

• Requires information about context and crew actions to understand “Why”

• Proprietary and closely protected by operators

• Very detailed objective airplane outcome data from normal operations

• Uniquely suited for multi-flight statistical analysis

• Limited in scope

• Requires information about context and crew actions to understand “Why”


LOSA DataLOSA Data Structured format that accurately captures

– Facts and events in normal operations

– Background that puts facts & events in context

Provides more than single flight context

Can provide better detail than incident reports

Greater accuracy than anecdotal analysis

Can support rate of occurrence analysis

Reveals how crews successfully manage errors

Enables focused observations on issues of interest

Structured format that accurately captures – Facts and events in normal operations

– Background that puts facts & events in context

Provides more than single flight context

Can provide better detail than incident reports

Greater accuracy than anecdotal analysis

Can support rate of occurrence analysis

Reveals how crews successfully manage errors

Enables focused observations on issues of interest


Boeing’s Use of LOSA DataBoeing’s Use of LOSA Data

• Search for threats and errors related to safety of our fleet (e.g., automation related errors)

• Close the “loop” on design

• Correlate results with other data to gain convergent validity

• Capitalize on observations in ARCHIE to discover trends and patterns

• Search for threats and errors related to safety of our fleet (e.g., automation related errors)

• Close the “loop” on design

• Correlate results with other data to gain convergent validity

• Capitalize on observations in ARCHIE to discover trends and patterns


Other LOSA BenefitsOther LOSA Benefits

• Proactively discover new safety issues/ through manufacturer/airline inquiry

• Assist in prioritizing risks and identifying the real safety issues

• Help define “Best Practices!”

• Validate or better formulate other safety initiatives

• Provide design and procedural insights about different Boeing types and models

Experienced-Based Airplane Lifetime Safety Cycle

Experienced-Based Airplane Lifetime Safety Cycle

Continuous feedback of information

Boeing design and regulatory

requirements

Validateand

certify

Delivery

ProduceDesign

ProblemResolution

Reports

•Previous DR&O

•FAR’s, •JAR’s•Customer requirements

NewTechnology

•SR&O’s•Design reviews

•Analyses•Testing

Analyses

Flight Testing

LaboratoryTesting

DATA• 300,000

telexes per year

• 85,000 accidents, incidents to date

Lessons Learned

ModificationReviewBoard

Testrequirements

In-ServiceOperation

Servicebulletins, etc.

BCA In-Service Safety Process

SRB’s*

Data Corrective Action Management

PotentialIssue

Analysis

AccidentInvestigation

PASC

Functional Safety

Councils

Issue Classification

7-23-02 SPS-032

Safety Review Board Members

CSD

SRP

AccidentData

In-Service Events

Lead Airline Process Involvement

Flight Test

Factory

COSP filterFAA EIB = Engineering Investigation Board

SRB = Safety Review Board*

IncidentData

COSP Bulletin Board EIBs*

Potential Supplier

Safety Issues

Issue Database

BCA In-Service Safety Process

LOSA


LOSA Data - Influence on Flt Deck DesignLOSA Data - Influence on Flt Deck Design

• LOSA provides valid performance data on flight crew successes and errors

• Flight deck designers can learn from both categories• Leverage successes into even better crew-centered

designs

• Change / enhance control operations to design away from errors

• LOSA data provides a framework for future flight deck designs and enhancing existing designs


Visits to The LOSA CollaborativeVisits to The LOSA Collaborative

Annual 2-Day Visits, Safety Pilot + Human Factors Specialist

August 2005 – Reviewed approximately 400 sets of ARCHIE

data, primarily 747-400 and 777

– Airfield and Dates De-identified

– Airline De-identified

October 2006– Reviewed entire archive of 5609 sets of data


2005: Where Are The Threats?2005: Where Are The Threats?

• Threats in the pre-departure phase of flight are highest.

• Where English is not the native language

• Traffic compression is a growing threat

• Pre-flight after major maintenance often do not adequately prepare the airplane for the aircrew

• Threats in the pre-departure phase of flight are highest.

• Where English is not the native language

• Traffic compression is a growing threat

• Pre-flight after major maintenance often do not adequately prepare the airplane for the aircrew


An Example: Checklist ErrorsAn Example: Checklist Errors

LOSA data confirm our previous reports (Boorman 2001; Holder ‘03, ‘04)

Value added by LOSA data– Provided rates of occurrence– Separated occurrences by phases of flight– Added context to better understand each error

Most frequent reasons for checklist errors are:– Late checklist– Checklist not performed to completion – Items on checklist left open– Wrong checklist– Not knowing which checklist to use (training &

proficiency)


Percent of Flights with Checklist Errors

Percent of Flights with Checklist Errors

0%

5%

10%

15%

20%

25%

737 757 767 777


777 Checklist777 Checklist

• We knew electronic checklist could provide a great advancement in safety

• Operator experience confirms we succeeded

• We designed a lot of the traditional checklists problems out

• Anecdotal and FOSP information has identified a few potential enhancements

• Seeking more information, such as LOSA, to focus our enhancement efforts


Hitting The Target On ErrorsHitting The Target On Errors

• There seems to be a correlation between error and delays.

• Most errors are inconsequential!

• Callouts for un-stabilized approaches and procedural callouts are not solving handling errors.

– When deviations occur, crew members identify it with a callout only ½ the time!


Phase 1 – Discovering ErrorsPhase 1 – Discovering Errors

• 165 B777 LOSA flights

• 11 of 165 flights had checklist errors (7%)

• 13 checklist errors in 11 flights

• 3 errors reported as intentional

• 1 error led to additional error

• No undesired aircraft states


Phase 2 – Classifying errorsPhase 2 – Classifying errors

Omitted checklist

Forgot checklist

Did not call for checklist

Self-initiated checklist

Checklist performed late or at wrong time

Checklist not performed to completion (e.g., open item that gets forgotten)

Checklist performed from memory

Omitted checklist

Forgot checklist

Did not call for checklist

Self-initiated checklist

Checklist performed late or at wrong time

Checklist not performed to completion (e.g., open item that gets forgotten)

Checklist performed from memory


Phase 3 – Learning the ContextPhase 3 – Learning the Context

Pilot forgot to ask for checklist

PNF forgot he left an item open on CKL

PF did not call for “After-Landing” CKL

Landing CKL initiated at 2000 ft (late)

Late completion of landing CKL

FO ran “Engine Start”, “Pushback” CKL from memory

Most errors occur on descent/approach (7/13) and pre-departure (4/13) phases

Pilot forgot to ask for checklist

PNF forgot he left an item open on CKL

PF did not call for “After-Landing” CKL

Landing CKL initiated at 2000 ft (late)

Late completion of landing CKL

FO ran “Engine Start”, “Pushback” CKL from memory

Most errors occur on descent/approach (7/13) and pre-departure (4/13) phases


Phase 4 – Design SolutionPhase 4 – Design Solution

• New checklist alerting will be integrated into new airplane designs to prevent:

− omission

− forgetting, and

− late execution of critical normal checklists

• Plus other data-driven enhancements to 777 and 787 baseline checklist design


Altitude Deviations Are CommonAltitude Deviations Are Common

• Many occur due to distractions and high workload situations

– Deviation around weather

– Fuel in center tank operational requirements

– ATC

– Deviation around traffic

– Missed Approaches

– Traffic compression


Altitude Deviation SolutionAltitude Deviation Solution

• Possibly Design

• More emphasis in the Flight Crew Training Manuals to bring about awareness

• More emphasis during LOFT and CRM training to bring about awareness

• More structured/defined cross check process


2006 ARCHIE Review:Unstabilized Approaches

(5609 total observations as of September 2006)(5609 total observations as of September 2006)

245 Unstable Approaches - 4 %245 Unstable Approaches - 4 %

59 Landed Outside Touchdown Zone

(Landed long, short, firm or off-center – 24%)

59 Landed Outside Touchdown Zone

(Landed long, short, firm or off-center – 24%)

182 Exceeded Company Established Parameters

(Landed within touchdown zone – 74%)

182 Exceeded Company Established Parameters

(Landed within touchdown zone – 74%)

4 Executed Missed Approach

(no further error - 5%)

4 Executed Missed Approach

(no further error - 5%)


2006 ARCHIE– Unstabilized Approaches

179 LOSA OBSERVATIONS – BOEING AIRPLANES179 LOSA OBSERVATIONS – BOEING AIRPLANES

78% UNSTABLIZED APPROACHES

– VISUAL APPOACH


– VISUAL APPOACH

56% CAPTAIN FLOWN (BASE RATE) 56% CAPTAIN FLOWN (BASE RATE)

4 Executed Missed Approach (179 Boeing – 2%) 4 Executed Missed Approach (179 Boeing – 2%)


2006 ARCHIE – Unstabilized Approaches


– VISUAL APPOACH


– VISUAL APPOACH

NOTE: CALLOUTS ABSENT

PM – NO COMMUNICATION OR CALLOUTS

PF – NO ALERTING TO DEVIATIONS

NOTE: CALLOUTS ABSENT

PM – NO COMMUNICATION OR CALLOUTS

PF – NO ALERTING TO DEVIATIONS


LOSA Methodology as Research ToolLOSA Methodology as Research Tool

University of South Australia Qantas-CASA-AIPA Long-Haul Fatigue Study

Full-Mission Simulation– Fresh crews

– Recently returned crews from trip

Structured LOSA observations at key decision points in scenario

University of South Australia Qantas-CASA-AIPA Long-Haul Fatigue Study

Full-Mission Simulation– Fresh crews

– Recently returned crews from trip

Structured LOSA observations at key decision points in scenario

Note: See Thomas et al., FSF-IATA-IFA International Air Safety Seminar Oct 2006 for reports

35

International Trip (US or Europe)

5 Days Rest (Home)

Simulator Session (Sydney)

4 Days Recovery (Home)

or

sleep/wake/fatigue ratings

Research Overview

36

Three Hour Protocol

Simulator Scenario

120

SIMULATOR

PerformanceTasks

10

Interview

30

Subjective Ratings

10

POST-SIM

PerformanceTasks

10

Subjective Ratings

10

PRE-SIM

Dispatch

30

Decision-making Event

Simulator Session

37

38

Scenario Structure

workload

low

high

T1 Dispatch with Thrust Reverser “Locked Out”

T1

T2 Change in Loadsheet

T2

T3 Change in Duty RWY

T3

T4 Clearance to Lower Altitude on Departure

T4

T5 Higher Cruise Level & Expedite Climb

T5

T6 High Speed Descent & Track Shortening

T6

T7 ATC QNH Error

T7

T8 Failure to Provide Clearance to Land

T8

CDE

CDE Critical Decision Event

Pre-Flight Take-Off

Climb

CruiseDescent - Approach - Landing

39

1. Obtain Melbourne WX/TTF

2. Time to obtain Melbourne WX/TTF

3. Below alternate criteria

4. Cross-check figures

5. Number of options considered

6. Runway 16 available

7. Diversion to another airport

8. Time to decision

9. Time to execute decision

10.Decision review

Situation Awareness

COA Development

Decision

Evaluation

Decision Variables


Future Questions For LOSAFuture Questions For LOSA

• How is TCAS being used in an operational context?

− Are crews proactive or reactive with regards to potential TA/RA’s?

• Is there any correlation between threat rates versus performance errors?

• What are “The Best Crew Practices”?− Attributes of crews who make few errors?

− Attributes of crews who do an excellent job in managing or trapping errors?

− Why do some crews commit more errors than others?


ConclusionsConclusions

• LOSA is a unique tool that provides a “window” into crew performance during normal ops

• By cooperating, we can greatly multiply LOSA’s potential for improving global aviation safety– Consistent implementation is the key– Will learn together what we don’t know– Will help us to anticipate future system risks– More questions are being raised than answered

• Success is critically dependent on the protection of safety data such as LOSA

• Boeing is committed to strongly supporting LOSA as a key safety tool


Thank You For YourThank You For YourAttention!Attention!

Documents

The Safety Benefits of LOSA: A Manufacturer’s View