51
Federal Aviation Administration Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015 International Civil Aviation Organization (ICAO) Operational Data Link Familiarization Seminar Session 3 – Preparation for data link implementation – Post-implementation monitoring (compliance) (ANSP and Operator): Nairobi, Kenya 2-6 November 2015

International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

  • Upload
    lamcong

  • View
    264

  • Download
    4

Embed Size (px)

Citation preview

Page 1: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

International Civil Aviation Organization (ICAO) Operational Data Link Familiarization Seminar

Session 3 – Preparation for data link implementation – Post-implementation monitoring (compliance)

(ANSP and Operator): Nairobi, Kenya 2-6 November 2015

Page 2: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Presentation Overview Introduction → guidance material ANSP data collection

▫ CPDLC transaction time/continuity ▫ ADS-C report delivery time/continuity ▫ SATVOICE transaction time/continuity ▫ SATVOICE position report delivery time/continuity ▫ Availability

ANSP performance monitoring and analysis ▫ Data filtering ▫ Time/continuity ▫ Availability ▫ Monitoring reports for regional and global use

Regional performance monitoring and analysis Regional problem reporting and resolution Benefits of PBCS

2

Page 3: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Introduction

• Guidance material for post-implementation monitoring and compliance contained in ICAO Doc 9869 – Performance-Based Communication and Surveillance (PBCS) Manual – Appendix D: CPDLC and ADS-C – Appendix E: SATVOICE

• Formerly contained in the Global Operational Data Link Document (GOLD) and the Satellite Voice Guidance Material (SVGM)

3

Page 4: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

DATA COLLECTION

4

Page 5: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

CPDLC Transaction Time/Continuity (1 of 3)

• CPDLC data set: controller-initiated transactions that receive a single DM 0 WILCO response A DM 0 WILCO response following a DM 2 STANDBY is not measured

• Table D-1 in PBCS Manual contains CPDLC data collection points

– Most of required 19 data points can be extracted from either the ACARS or ATN B1 header or the CPDLC application message, or calculated based on the other data points

– Aircraft type and operator will need to be matched from a separate database using the aircraft registration as the common point

5

Page 6: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

CPDLC Transaction Time/Continuity (2 of 3)

• CPDLC analysis is based on measurement of: – Actual Communication Performance (ACP)

→Required Communication Monitored Performance (RCMP) – Actual Communication Technical Performance (ACTP)

→Required Communication Technical Performance (RCTP) – Pilot Operational Response Time (PORT)

→RCP PORT

• Suggested that ANSP also conduct regular analysis of the message use statistics for the current CPDLC message set for the purpose of assessing usage trends and future development of CPDLC applications

6

Page 7: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

CPDLC Transaction Time/Continuity (3 of 3) ACP

Up Down

Sent) (UplinkReceived) (WILCOACP −= RCMP

( )

+

≅ ΔDown

2ΔUpACTP RCTP

ACTPACPPORT −≅ RCP PORT

1 Uplink Sent Date/time ATSU sent CPDLC clearance to the aircraft

2 MAS Received Date/time ATSU receives the MAS for the CPDLC clearance

3 WILCO Sent Date/time aircraft sends WILCO response for the CPDLC clearance

4 WILCO Received Date/time ATSU receives WILCO response for the CPDLC clearance

1 42 3

The measurements (in seconds) are calculated as follows:

Page 8: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

ADS-C Report Time/Continuity • Actual Surveillance Performance (ASP)

→Required Surveillance Performance (RSP)

• Table D-3 in PBCS Manual contains ADS-C data collection points

– Most of required 12 data points can be extracted from either the ACARS or ATN B1 header or the ADS-C application message, or calculated based on the other data points

– Aircraft type and operator will need to be matched from a separate database using the aircraft registration as the common point

8

ASP = {time the ADS-C report is received at the ANSP} – {time at position extracted from the decoded ADS-C basic group}

Page 9: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

SATVOICE Transaction Time/Continuity • SATVOICE communication performance analysis is based on the

calculation of – Actual Communication Performance (ACP) →RCP time allocations for communication transaction (RCMP)

• The analysis uses the measurement of transit and response times related to clearances sent via SATVOICE that receive a single readback response

• Table E-1 in PBCS Manual contains SATVOICE transaction data collection points – Most of required 9 data points can be extracted from either the

ACARS or the ACARS application message, or calculated based on the other data points

– Aircraft type and operator will need to be matched from a separate database using the aircraft registration as the common point

9

Page 10: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

SATVOICE Position Report Delivery Time/Continuity

• Actual Surveillance Performance (ASP) →Required Surveillance Performance (RSP)

• Note: Because the accuracy of the time-over-position within the ACARS position report message is only to the minute (e.g. 15:11) while the accuracy of the timestamp of receipt at the ANSP is to the second (e.g. 15:11:11) the accuracy of the measurement of the surveillance performance will be limited to the minute

• Table E-2 in PBCS Manual contains SATVOICE position report data collection points

– Most of required 12 data points can be extracted from either the ACARS header or the ACARS application message, or calculated based on the other data points

– Aircraft type and operator will need to be matched from a separate database using the aircraft registration as the common point

10

ASP = {time the report is received at the ANSP} – {time-over-position extracted from the decoded ACARS message }

Page 11: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Availability (1 of 2) • To calculate the actual availability of CPDLC and ADS-C ANSP and

of SATVOICE service provision data should be collected for outages greater than 10 minutes CSP notified system outages Detected outages that are not observed by or notified by the CSP

• For each outage the following information should be collected: a) Time of CSP outage notification: In YYYYMMDDHHMM format or “Not

Notified” if no CSP notification received b) CSP Name: Name of CSP providing outage notification if applicable c) Type of outage: Report media affected SATCOM, VHF, HF, ALL d) Outage start time: In YYYYMMDDHHMM format e) Outage end time: In YYYYMMDDHHMM format f) Duration of Outage: In minutes

11

Page 12: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Availability (2 of 2)

• Example of unreported outage – large ADS C downlink delays observed from 3 aircraft during the

period from 11:20 to 12:13

12

Aircraft registration Aircraft time ANSP system time Downlink time (Seconds)

ZKSUI 11:55:38 12:12:52 1,034 ZKSUI 11:44:42 12:12:19 1,657 ZKSUI 11:23:21 12:08:32 2,711 ZKSUJ 11:41:54 12:12:01 1,807 ZKSUJ 11:26:18 12:09:42 2,604 ZKSUJ 11:20:34 12:07:39 2,825 ZKOKG 11:53:52 12:12:51 1,139

Page 13: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Data Format

Data may be stored in database or text format

When sharing raw data (e.g. with the regional monitoring entity) it is suggested to be sent as a .csv file

13

Page 14: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

ANSP PERFORMANCE MONITORING AND ANALYSIS

14

Page 15: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Overview

• Collected data are used to monitor the time/continuity of CPDLC and SATVOICE transactions, and ADS-C and SATVOICE report delivery

• At a minimum, monitoring should be conducted for: ►Aggregate system performance (all data combined) ►All media types ►All message type(s) ►All operators ►All aircraft types ►All airframes

15

Page 16: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Data Filtering - CPDLC • Aim is to include only those CPDLC transactions for which there is a

critical communications requirement when applying reduced separation standards – i.e. intervention messages

• The following transactions are filtered out: – Uplink messages with any response other than DM 0 WILCO, including

messages with DM 2 STANDBY responses followed by DM 0 WILCO – Non-intervention route messages (UM 79, UM 80, UM 81, UM 82, UM 83,

UM 84, UM 91, and UM 92) – Contact instructions (UM 117 – UM 123) – RESUME NORMAL SPEED (UM 116)

• Note: the removal of all contact instructions (UM 117 – UM 123) may drastically reduce the monthly data set for some smaller ANSPs and make it difficult to assess ACTP. For this reason some ANSPs may retain these (UM 117 – UM 123) transactions when assessing ACTP only

16

Page 17: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Data Filtering – ADS-C

• Duplicate ADS-C reports should be removed from the data set prior to analysis – Occurs when ADS-C report is sent and the acknowledgement

(ACK) from the GES is not received within a defined period of time causing the aircraft system to resend the report (typically during media transitions)

Only the ADS C report with the earliest receipt time should be kept in the data set

• ADS-C reports with delivery times of zero or less than zero should be filtered out – These times represent cases where the ADS-C basic group

timestamp extracted as seconds since the most recent hour was incorrectly decoded into the HH:MM:SS format by the ATS unit’s system

17

Page 18: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Data Filtering - SATVOICE • When SATVOICE is used after failed attempts on HF, the

observed performance may indicate excessive delays in the SATVOICE performance

• Analysis should include these data to reflect actual operational performance from the controller perspective and then determine whether procedures could potentially mitigate the effects of these delays ►e.g. the radio operator may consider using the SATVOICE

directly when it can be determined to provide a more reliable communication than HF

18

Page 19: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Data Filtering - Outage Periods

• The outage data collected to measure availability should also be used for filtering the ADS-C, CPDLC and SATVOICE data sets

• All ADS-C reports, CPDLC transactions, SATVOICE transactions, and SATVOICE position reports occurring during applicable outage periods reported by the CSP should be removed

• All ADS-C reports, CPDLC transactions, SATVOICE transactions, and SATVOICE position reports occurring during applicable unreported outages detected by the ANSP should also be removed

19

Page 20: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Cumulative Distributions to Measure Performance • Filtering the data limits the size of the sample that will be used to create the

cumulative distributions of data • When providing cumulative distributions of the data, a sufficient sample size

should be determined taking into account a number of factors, such as: – Type of data that will be considered in the sample – Cost, time and difficulty in collecting the data – Existing knowledge about the underlying technologies and implementation – Variability of the data collected – The specific criterion that the data sample will be measures against – Level of confidence desired in the estimated result

• Once a sufficient sample of filtered data has been collected, the next step is to calculate a cumulative distribution for each of the performance parameters to be measured:

– ACP, ACTP, PORT for the CPDLC application – ASP for the ADS-C application – ACP only for SATVOICE intervention capability – ASP for the SATVOICE position reports

20

Page 21: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Graphical Analyses

• It is recommended to begin with graphical analysis of the data as this method is useful for clearly depicting the performance and facilitating the identification of performance problems

• The cumulative performance should be shown in comparison to the relevant parameter values for the transaction times and corresponding continuity requirements – For example - when measuring the cumulative ACP against

RCP 240, the RCP 240 safety and efficiency requirements should be included:

• 240 seconds at 99.9% • 210 seconds at 95.0%

21

Page 22: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

PBCS Time/Continuity Performance Criteria

Performance Measure

Percentage of Messages Required to

Meet Criteria

RSP180 Criteria

(sec)

RSP400 Criteria

(sec)

RCP240 Criteria

(sec)

RCP400 Criteria

(sec)

ASP 95% 90 300

99.9% 180 400

ACTP 95% 120 260

99.9% 150 310

ACP 95% 180 320

99.9% 210 370

PORT 95% 60 60

22

Page 23: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Sample Graphical Analysis Format

23

Page 24: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example – Assessing Annual ACP

24

Page 25: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example – Assessing ASP by Station ID

25

Page 26: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

GES LOCATION(S) SATELLITE/ REGION SITA Identifier

ARINC Identifier

Burum, Netherlands

Inmarsat I-3 AOR-E AOE2 XXN

Inmarsat I-3 AOR-W AOW2 XXW

Perth, Australia

Inmarsat I-3 IOR IOR2 XXI

Inmarsat I-3 POR POR1 XXP

Fucino, Italy

Inmarsat I-4 EMEA EUA1 XXF

Inmarsat I-4 EMEA SBB EME9 XXB

Paumalu, Hawaii, US

Inmarsat I-4 Americas AME1 XXH

Inmarsat I-4 Asia-Pac APK1 XXA

Inmarsat I-4 Americas SBB AMR9 XXU

Inmarsat I-4 Asia-Pacific SBB PAC9 XXS

Kobe and Hitachiota, Japan

MTSAT Japan MTS1 --

Phoenix, Arizona, US Iridium Global IGW1 IG1

26

Page 27: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Tabular Analysis

• It is sometimes helpful to view and report the results in tabular format o For reporting performance at a high level

• Aggregate analysis and analysis by media type

o When there is an impractical amount of series associated with a particular subset to be clearly displayed on a chart

• Analysis by operator

27

Page 28: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example - Performance by Media Type

28

Media Type

ADS-C CPDLC

Count of ADS-C Downlink Messages

ADS-C 95%

ADS-C 99.9%

Count of CPDLC

Transactions

ACTP 95%

ACTP 99.9%

ACP 95%

ACP 99.9%

PORT 95%

Performance Criteria RSP 180 RCP 240

Aggregate 1,286,267 98.2% 99.3% 45,754 99.7% 99.8% 99.0% 99.3% 96.5%

SAT 1,019,933 98.1% 99.3% 41,822 99.7% 99.8% 99.1% 99.3% 96.5%

VHF 261,232 98.8% 99.5% 3,529 99.9% 99.9% 99.2% 99.4% 96.9%

HF 5,096 73.4% 86.1% 5 -- -- -- -- --

New York July – December 2014

53,570 flights

Page 29: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015 29

Example - Observed Performance by Operator New York FIR July – December 2014

Page 30: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example - Summary of Performance by Operator New York FIR

• There were 96 operators with at least 100 ADS-C messages during this 6-month period

• Summary of how many operators meet criteria for each performance measure:

30

Page 31: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Identifying Poor Performers

• There are many potential causes of degraded performance

• Experience has shown that poor performance may be attributed to a specific aircraft in a fleet – Can be identified by graphing the monthly data for a fleet by

aircraft registration

• Techniques such as graphing the positions of all delayed messages on a geographical display have identified areas for further investigation

31

Page 32: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example – Poor Performer

• De-sensitized operator code: BH • Fleet of 12 – B763 aircraft

– 9 using Iridium → 3/9 meet 95% RSP180 – 3 using I-3 → all meet 95% and 99.9% RSP180

• DSP: SITA • Issue identified with 1 airframe (BH4) performing well

below 95% and rest of fleet

32

Page 33: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015 33

Page 34: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example – Graphing Positions • Each ADS-C position annotated with station ID and associated

delay, e.g. “XXU-136” = XXU station ID and 136 sec delay • Clear pattern of higher delays at transition area between VHF and

SAT

34

Page 35: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example - Monitoring Availability of CPDLC and ADS-C Jun 2014 to May 2015

35

Meets safety and reliability criteria

Meets safety criteria only

Does not meet safety or reliability criteria

Page 36: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Monitoring Reports for Regional and Global Use

• Each ANSP within a region should compile monitoring reports at the interval agreed by the regional forum

• A tabular format can be used to report on the observed system performance in terms of the availability and time/continuity parameters specified in the applicable RCP and RSP specifications

• Appendix D of PBCS Manual contains sample reports for: – Service availability

– RCP and RSP

– Operator with different aircraft types/systems in its fleet

36

Page 37: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

REGIONAL PERFORMANCE MONITORING AND ANALYSIS

37

Page 38: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Overview • Consistent data provided by each of the ANSPs within a region can

be aggregated to create a regional PBCS monitoring report in graphical or tabular form

• Options for data sharing from individual ANSPs ► Raw .csv files with data in formats described in Appendices D and E of

the PBCS Manual ► Data containing the cumulative distributions calculated by the ANSP

‒ The regional PBCS monitoring program would specify the time period of interest, the subset(s) of interest, the required filtering and the required format to ensure consistency between the data sets)

• These regional performance metrics should be made available to all interested stakeholders – Regional website should be considered to enhance the distribution of

metrics • It is recommended that regions implement monthly performance

reporting to obtain system performance metrics

38

Page 39: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

REGIONAL PROBLEM REPORTING AND RESOLUTION

39

Page 40: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Overview

• All stakeholders should be actively involved in the problem reporting and resolution process – All aircraft operators in a region must have the opportunity to

become involved in the process – CRAs should be pro-active in getting all aircraft operators and

other stakeholders to register and participate in the process

• The problem identification and resolution process for each individual problem consists of: o Data collection phase o Problem analysis and coordination with affected parties to

secure a resolution o Recommendation of interim procedures to mitigate the problem

(as necessary)

40

Page 41: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015 41

Page 42: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

ISPACG-CRA / NAT DLMA

– PRs filed via ISPACG-CRA, NAT DLMA Problem Reporting website: http://www.ispacg-cra.com/

Website hosted by Airways Corporation of New Zealand Limited

– Now used for:

• CRA for South Pacific (ISPACG FIT)

• CRA for North, Central, East Pacific (IPACG FIT)

• DLMA for North Atlantic (NAT CNSG)

• FIT-Asia for South China Sea, Bay of Bengal, Indian Ocean

– Continue to get new entities registered with website

42

Page 43: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

PR Status Definitions

• RAISED - the PR has been filed by the originator but has not yet been processed by the CRA

• ACTIVE - CRA has processed the PR and allocated a CRA # and someone to investigate it. During this phase the PR is under investigation

• OPEN - The investigation is complete however some form of correction is required before it can be closed

• CLOSED AS DUPLICATE - Closed because problem is already covered under another PR

• CLOSED - Corrective action has been implemented or PR is a non-problem

43

Page 44: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

PR Type Definitions

• Website choices: AIR, GROUND, NETWORK, TBA • CRA tracking breaks out as:

– AIR – procedural – Problem due to flight crew action – AIR – technical – Problem due to avionics fault – GROUND – Problem due to issue at ATSU – NETWORK – Problem at GES or in network – Mult - Problems occurred in more than one area – None - Problem was a non-problem – TBA – To Be Assigned – problem type not yet determined

44

Page 45: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Problem Report Metrics Growth in Number of PRs per Year

45

162 178

112

258

319

49

413

0

50

100

150

200

250

300

350

400

450

2010 2011 2012 2013 2014 2015 (to date) 2015 (predicted)

49 PRs as of 26 Feb 2015

Page 46: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Problem Report Metrics

46

133

36 79

55

11 9

NATNOPACSOPACASIACANADAEUROPE

All Problem Reports by Region Feb 12, 2014 - Feb 12, 2015

141

139

41

2 ANSPs

Operators

CRA (incl.CRASA-J)DSPs

All Problem Reports by Agency Type

Feb 12, 2014 - Feb 12, 2015

Page 47: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example: PR 1508-MM • PBCS analysis, ASP by station ID, in New York FIR highlighted several

paths with notably lower performance, including XXH, data link service provided by ARINC using the Inmarsat I-4 Americas satellite via the ground station in Paumalu, Hawaii, US

• Upon further investigation, notable variation between operator/aircraft type combinations, with the top user of XXH barely meeting the 95% criteria for RSP180

• PR submitted 5 Feb 2014 • The issue has been worked between the operator, Honeywell, and

Boeing since then • Initial solution was to change ORT settings – however this did not

resolve the issue • Honeywell has indicated that a specific incompatibility between certain

Honeywell SATCOM avionics and the Inmarsat Classic Aero network is the likely cause of this problem and are continuing to work on a solution

47

Page 48: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example: PR 1508-MM ASP by Station ID for New York FIR

48

Page 49: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Example: PR 1508-MM ASP by Operator/Aircraft Type for New York FIR – Jul to Dec 2013

49

Page 50: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015 50

Example: PR 1508-MM ASP by Operator/Aircraft Type for New York FIR – May 2015

Page 51: International Civil Aviation Organization (ICAO ... Link... · International Civil Aviation Organization (ICAO) Operational Data ... Performance-Based Communication ... ICAO AFI Data

Federal Aviation Administration

Post Implementation Monitoring (Compliance) ICAO AFI Data Link Seminar, 2-6 November 2015

Benefits of PBCS

PBCS Monitoring • Ensures actual system performance is maintained in

accordance with RCP-RSP specifications – Actual communication performance is measured against

appropriate RCP specifications – Actual surveillance performance is measured against RSP

appropriate specifications

• Provides effective way to improve system performance – Analysis tools can be and are shared (e.g. G-PAT) – Local results can be exchanged regionally and globally