Identifying Risk

Enhanced Data Analysis & Reporting

K imber ly Py leD i rector , Po l icy & Per fo rmance

FAA A i r T ra f f ic Organ i zat ionK i m b e r l y . P y l e @ f a a . g o v

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20192

Organization

REGULATOR

7,300+U.S. commercial airlines and air operators

3,300+ Federally-funded airports

867,450+ Active Pilots

• FAA: UNIQUE STRUCTURE

AIR TRAFFICSERVICES PROVIDER

7,000+Takeoff and landings per hour

660,000,000+Passengers

37,000,000,000+Cargo revenue ton miles of freight/year

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20193

Air Traffic Organization

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20194

Safety & Technical Training

Kimberly PyleDirectorPolicy &

Performance

Abigail SmithDirector

Technical Training

Tony SchneiderDirector (A)

Safety

SMS, Audits, Safety Metrics,

Global Safety Harmonization

Quality Assurance, Investigations, VSRP,

Runway Safety, Human Performance

Management

Technical Training Development,

Simulators, Recurrent Training

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20195

• Driven by Data

Safety Management System

Gathering Performance Data from Employee Reports and Automated Reporting

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20196

Evolution of Safety Data Collection

“Operational Errors” or “Pilot Deviations” reported by Employees; Metrics included a limit on the number of events

“Losses of Separation”reported by Employees and collected automatically, plus VSRP; New Metrics

2012 July 2019

Events reported by Employees andcollected automatically using ARIA to score risk

2004 ----------------------------------------------

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20197

2012 Policy Changes

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20198

Voluntary Safety Reporting Introduced

Tech-Ops Safety Action Program for Technicians

Air Traffic Safety Action Program for Air Traffic Controllers

Confidential Information Sharing Program with Airlines

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 20199

Better Data Through Reporting

BEFORE• Short• Vague

AFTER• Comprehensive• Detailed

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201910

• Augmenting “Mandatory Occurrence Reports” by Employees• Collected at the National Level• Gathering significant data on compliance anomalies

Automated Detection & Reporting

Traffic Analysis and Review Program (TARP) deployed in 2012

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201911

How does ATO leverage this technology?

Identify Potential Loss of

Separation Events

Review events where

< 66% separation maintained

/

Quality Assurance Staff

validate all “Risk Analysis” events

RAP

Conduct Risk Analysis Process

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201912

• Insight into system performance drove procedural changes

Benefits of the Current System

Opposite Direction Operations

Converging Runway Operations

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201913

• Compliance does not equal risk

• Only detects one type of event

• Significant human review required

• Difficult to maintain

What are the Limitations of the Current System?

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201914

SFO Wrong Surface Approach Replay

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201915

How does Risk-based Safety Assurance Differ from Compliance-based?

Compliance-based Safety Assurance

Risk-based Safety Assurance

“Does this violate a ruleor regulation?”

“How much risk is associated with the

observed behavior?”

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201916

Example: Compliant, High Risk

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201917

Principles of a Risk-based Approach to Event Identification

Find the Highest Risk

Events

Reduce Time Validating

Events

Increase Risk Analysis Events

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201918

Aviation Risk Identification and Assessment

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201919

How does ARIA Calculate Risk?

Risk

Time

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201920

How does ARIA Calculate Risk?

Risk

Time

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201921

How does ARIA Calculate Risk?

Risk

Time

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201922

How does ARIA Calculate Risk?

Risk

Time

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201923

How does ARIA Calculate Risk?

Risk

Time

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201924

How does ARIA Calculate Risk?

Risk

Time

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201925

How does ARIA Calculate Risk?

Risk

Time

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201926

Principles of a Risk-based Approach to Event Identification

Increase Risk Analysis Events

IFR/ VFR Missed Approach Terrain Proximity Surface Risk

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201927

• ARIA is currently integrated into FAA Quality Assurance processes

• ATO plans to use ARIA to identify airborne events beginning July 2019

• Surface version of ARIA is currently going through V&V

• Use ARIA data to assist FAA with barrier performance

• Neutral taxonomy in RAP process will allow us to gather data

• Options for each barrier are: Effective, Degraded, Failed, N/A, Not Scored

Next Steps

F e d e r a l A v i a t i o n A d m i n i s t r a t i o n

Safety & Procedures Safety Forum, Brussels June 5, 201928

Data Fuels Corrections that Enhance Safety

• PROVIDING DRAMATIC RESULTS

Source: NTSB. Accident Rates are per 100,000 Flight Hours. Flight hours compiled by FAA.

“Collaboration andthe use of more data fuels the corrections that drive down the

accident rate.” - former NTSB Chairman Chris Hart

Identifying Risk

Enhanced Data Analysis & Reporting

K imber ly Py leD i rector , Po l icy & Per fo rmance

FAA A i r T ra f f ic Organ i zat ionK i m b e r l y . P y l e @ f a a . g o v

Slide Number 1Slide Number 2Slide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23Slide Number 24Slide Number 25Slide Number 26Slide Number 27Slide Number 28Slide Number 29