Continuous Monitoring Approach - Updatecoscap-na.com/sites/default/files/13SCM_CMA rev1_0.pdf · •Introduce ICAO CMA Online Framework •Filing of Differences (EFoD) •USOAP Reports

Continuous Monitoring Approach - Update

13th Meeting of the Steering Committee Jeju Island, ROK

16 – 18 April 2013

1/30/2014 13th Steering Committee 1

• Review ICAO USOAP Continuous Monitoring Approach

• Introduce revised / updated PQs (May 2013) and impact on States’ LEI

• Annex 19 • Introduce ICAO CMA Online Framework • Filing of Differences (EFoD) • USOAP Reports • Actions Required of States • CMA Training

1/30/2014 2 13th Steering Committee

Objective

USOAP CMA

COSCAP 2013 3


USOAP CMA


USOAP CMA Components

• CSA audits

• Safety audits

• ICAO Coordinated Validation Missions (ICVMs)

• Online monitoring activities

• Mandatory Information Requests (MIRs)

• Protocol findings

• Significant Safety Concerns (SSCs)

• Corrective Action Plans (CAPs)

• Analysis of safety risk factors

• Evaluation of State’s safety management capabilities

• States

• Internal stakeholders

• External stakeholders

Collection of safety information

Determination

of State safety

risk profile

Update of LEI and status of SSCs

Prioritization and conduct of USOAP CMA activities

Collection of Safety Information

COSCAP 2013 6

States provide:

• The State Aviation Activity Questionnaire (SAAQ)

• Compliance Checklists (CCs) or Electronic Filing of Differences (EFOD)

• The self-assessment checklist

• Updated CAPs



Internal stakeholders include:

• ICAO Secretariat Bureaus/Sections

• Regional Offices (ROs)



External stakeholders include:

• EASA;

• EC;

• EUROCONTROL;

• IATA;

• CANSO;

• Interstate Aviation Committee (IAC);

• Airports Council International (ACI); and

• other national, regional, supranational and international organizations recognized by ICAO. Note: These organizations conduct activities that generate safety information.



Determination of

State safety risk profile

COSCAP 2013 10

State Safety Risk Profile

Safety risk factors

Safety risk indicators

Safety risk profile

Safety risk factors

• Previous USOAP activity results

• Level of traffic in the State

• Progress made by the State in resolving USOAP deficiencies

Safety risk indicators

• LEI vs. traffic (exposure)

• Existence of SSC(s)

• Level of aviation activities for each audit area

• Projected growth of aviation activities

• Level of acceptability of State’s CAPs

• Progress in implementation of State’s CAPs

• Ongoing or planned assistance projects

• Major changes in the organizational structure

Safety risk profile

• CMO determines the safety risk profile which is generated by determining safety risk factors and indicators.

COSCAP 2013 11

Prioritization and conduct of

USOAP CMA activities

COSCAP 2013 12

• States are prioritized by CMO, based on their safety risk profile and information including:

• Input from the States/ROs on States’ progress in implementing their CAPs

• Input from the States/ROs on States’ progress in resolving identified SSCs

• Specific requests from States/ROs for conduct of a USOAP CMA activity



• Scope of an ICVM based on: – Level of aviation activity in the State

– State’s self-assessment

– Level of progress reported by State in implementing CAPs

– Level of progress reported by States in addressing not satisfactory PQs

– Request by a State (cost-recovery ICVM)

• Duration of ICVM determined by scope.



• Scope of an Audit (safety/CSA) based on:

– All relevant PQs

– Level of aviation activity in the State

– State’s self-assessment

– Request by the State (cost-recovery audit)

• Duration of Audit is determined by scope




USOAP CMA Components

• CSA audits

• Safety audits

• ICAO Coordinated Validation Missions (ICVMs)

• Online monitoring activities

• Mandatory Information Requests (MIRs)

• Protocol findings

• Significant Safety Concerns (SSCs)

• Corrective Action Plans (CAPs)

• Analysis of safety risk factors

• Evaluation of State’s safety management capabilities

• States

• Internal stakeholders

• External stakeholders

Collection of safety information

Determination

of State safety

risk profile

Update of LEI and status of SSCs


Update of LEI

and

Status of SSCs

COSCAP 2013 17

• ICAO Validates collected safety information

• Validation enables ICAO to continuously update State LEI status

• State LEI for the previous audit cycle is reported on iSTARS


Update of Lack of Effective Implementation (LEI)

PQs have been revised and updated

• Applicable in May 2013.

• New / amended PQs will impact State’s LEI – deletion of some PQs

– new PQs

– merging of existing PQs

• Detail later in Presentation



Mandatory Information Request (MIR) can be issues by CMO when: • SAAQ, CCs and/or PQs are not submitted,

are out-dated or are contradictory to other available information

• CAPs are not submitted or are not kept up-to-date by State

• Available information is insufficient • Concerns are raised by internal / external

stakeholders.



Mandatory Information Request (MIR)

• States required to provide status of compliance with PQs

• “Manage State self-assessment” tool on the OLF

• CMO will communicate with States through MIRs to seek additional information



Status of PQs may be changed through the CMO validation process based on:

• CAPs or other information received from States, supported by appropriate evidence

• Information received from ICAO ROs, recognized organizations and other stakeholders

• information received from States in response to MIRs





• With the new online monitoring activities, CMO may

review and validate some PQs related to CE-1 to CE-5

off-site and from ICAO HQ.

• However, validation of PQs related to CE-6 , CE-7,

and CE-8 will typically require an on-site activity.


Status of Significant Safety Concerns (SSCs)

“SSC occurs when the audited State allows the holder of an authorization or approval to exercise the privileges attached to it, although the minimum requirements established by the State and by the Standards set forth in the Annexes to the Chicago Convention are not met, resulting in an immediate safety risk to international civil aviation.”

Reference: EB 2010/7 dated 19 February 2010


Status of SSCs

SSCs unresolved in 12 States

SSCs identified by ICAO

SSCs resolved through corrective actions taken by the States

SSCs resolved by immediate actions taken by the States prior

to being posted on the ICAO website

34

17

12

5


Status of SSCs Continuous monitoring process

Ongoing monitoring of evidence and information collected from the State and other sources

USOAP CMA on-site activity

Evidence collected points to a SSC

• Team leader brings it to the attention of the State as soon as it is discovered

• State may initiate corrective actions immediately

• Team leader provides all relevant information to C/CMO

SSC Committee

convened

to validate

Preliminary

SSC

identified

Critical Elements of the

State Safety Oversight System

COSCAP 2013 27

ICAO Audits to determine States safety oversight capabilities include:

• Assessing the effective implementation of the 8 CEs of a safety oversight system

• Verifying State implementation of: – all safety-related ICAO SARPs

– associated procedures

– guidance materials

– best safety practices


Critical Elements of State Safety Oversight System


8

Resolution

of Safety

Concerns

1

Legislation 2

Operating

Regulations

3

Organization,

Safety Oversight

Functions

5

Guidance,

Procedures

& Info.

6

Licensing &

Certification

Obligations

7

Surveillance

& Inspection

Obligations

ESTABLISH

IMPLEMENT

• The State shall promulgate a comprehensive and effective aviation law, consistent with the size and complexity of the State’s aviation activity and with the requirements contained in the Convention on International Civil Aviation, that enables the State to regulate civil aviation and enforce regulations through the relevant authorities or agencies established for that purpose.

• The aviation law shall provide personnel performing safety oversight functions access to the aircraft, operations, facilities, personnel and associated records, as applicable, of service providers.


CE-1: Primary aviation legislation

• The State shall promulgate regulations to address, at a minimum, national requirements emanating from the primary aviation legislation, for standardized operational procedures, products, services, equipment and infrastructures in conformity with the Annexes to the Convention on International Civil Aviation.

Note The term “regulations” is used in a generic sense and includes but is not limited to instructions, rules, edicts, directives, sets of laws, requirements, policies, and orders.


CE-2: Specific operating regulations

• The State shall establish relevant authorities or agencies, as appropriate, supported by sufficient and qualified personnel and provided with adequate financial resources. Each State authority or agency shall have stated safety functions and objectives to fulfill its safety management responsibilities.

• The State shall ensure that inspectors are provided with guidance that addresses ethics, personal conduct and the avoidance of actual or perceived conflicts of interest in the performance of official duties.

Note In addition, Appendix 5 to Annex 6, Part I, and Appendix 1 to Annex 6, Part III, require the State of the Operator to use such a methodology to determine its inspector staffing requirements. Inspectors are a subset of personnel performing safety oversight functions.


CE-3: State system and functions

• The State shall establish minimum qualification requirements for the technical personnel performing safety oversight functions and provide for appropriate initial and recurrent training to maintain and enhance their competence at the desired level.

• The State shall implement a system for the maintenance of training records.


CE-4: Qualified technical personnel

• The State shall provide appropriate facilities, comprehensive and up-to-date technical guidance material and procedures, safety critical information, tools and equipment, and transportation means, as applicable, to the technical personnel to enable them to perform their safety oversight functions effectively and in accordance with established procedures in a standardized manner.

• The State shall provide technical guidance to the aviation industry on the implementation of relevant regulations.


CE-5: Technical guidance, tools and provision of safety-critical information

• The State shall implement documented processes and procedures to ensure that personnel and organizations performing an aviation activity meet the established requirements before they are allowed to exercise the privileges of a license, certificate, authorization and/or approval to conduct the relevant aviation activity.


CE-6: Licensing, certification, authorization and/or approval obligations

• The State shall implement documented surveillance processes, by defining and planning inspections, audits, and monitoring activities on a continuous basis, to proactively assure that aviation license, certificate, authorization and/or approval holders continue to meet the established requirements. This includes the surveillance of personnel designated by the Authority to perform safety oversight functions on its behalf.


CE-7: Surveillance obligations

• The State shall use a documented process to take appropriate corrective actions, up to and including enforcement measures, to resolve identified safety issues.

• The State shall ensure that identified safety issues are resolved in a timely manner through a system which monitors and records progress, including actions taken by service providers in resolving such issues.


CE-8: Resolution of safety issues

• Definitions of the 8 CEs of a safety oversight system are now an Appendix of Annex 19

• Guidance on the eight CEs is provided in ICAO Doc 9734, Part A


Critical Elements of State Safety Oversight System

Annex 19 – Safety Management

COSCAP 2013 39



The Air Navigation Commission, at the fourth and fifth meetings of its

190th Session on 8 May 2012, considered proposals developed by the

Safety Management Panel (SMP) to transfer the provisions on safety

management responsibilities and processes from existing Annexes for

consolidation in new Annex 19 — Safety Management, and related

consequential amendment proposals to existing Annexes developed by

the Secretariat.

The proposed new Annex 19 and consequential amendments to

Annexes 1, 6, 8, 11, 13 and 14, Volume I are envisaged for applicability

on 14 November 2013.



Given the increasing complexity of the global air transportation system and

the interrelated nature of aviation activities required to assure the safe

operation of aircraft, this Annex provides the means to support the

continued evolution of a proactive strategy to improve safety performance.

The Standards and Recommended Practices (SARPs) in this Annex are

intended to assist States in managing aviation safety risks, with the

objective of continuously reducing the number of aviation accidents and

incidents.

The foundation of a proactive safety strategy is based on the

implementation of the State safety programme (SSP) of a State and will

continue to rely on the successful implementation of existing international

provisions and accident/incident investigation activities.

The 4 components of the SSP framework are:

• State safety policy and objectives;

• State safety risk management;

• State safety assurance; and

• State safety promotion.




SSP Roll-out

The roll-out will include details on how USOAP CMA will monitor the

implementation of the SSP.

The ICAO Secretariat will be proposing to Council a detailed roll-out plan

of the SSP to guide States in its implementation.

USOAP CMA Audit Areas

COSCAP 2013 44


Audit Areas Civil aviation organization

(ORG)

SAAQ

Primary aviation legislation & civil aviation regulations

(LEG)

Chicago Convention

Personnel licensing & training (PEL)

Annex 1

Aircraft operations (OPS)

Annexes 6, 9, 18

PANS-OPS

Aircraft accident and incident investigation (AIG)

Annex 13

Airworthiness of aircraft (AIR)

Annexes 6,7,8 and 16

Air navigation services (ANS)

Ann 2,3,4,5,10,11,12,15 PANS-ATM

Aerodromes and ground aids (AGA)

Annex 14

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013+

PUBLIC

STATES

1997: Voluntary Assessment Programme, Fully Confidential (Annexes 1-6-8)

1999: USOAP Audit Summary Reports to all States (Annexes 1-6-8)

2005: USOAP CSA Audit results full transparency to all States

2006: SSC introduced, fast track notification to all States (restricted web site)

2001: Generic, non-State specific LEI results globally and by region

2005: Public access to LEI, Critical Element results by State. All States provided consent

2006: Mechanism to make full USOAP results available to the public with State consent. 1st cycle audits 45% of States

SSCs published on the CMA on line framework Proposed layout of the SSCs for the public to receive State feed back

2014 Unresolved SSCs to be made available to the public in the format and conditions approved by Council

ICAO has identified a significant safety concern with respect to the ability of [State] to properly oversee the [insert airlines (air operators); airports; aircraft; or air navigation services, as applicable] under its jurisdiction. This does not necessarily indicate a particular safety deficiency in the [insert airlines (air operators); airports; aircraft; or air navigation services, as applicable] but, rather, indicates that the State is not providing sufficient safety oversight to ensure the effective implementation of applicable ICAO Standards. Full technical details of the ICAO findings have been made available to [State] to guide rectification, as well as to all ICAO Member States to facilitate any actions that they may consider necessary to ensure safety. [State] has undertaken to regularly report progress on this matter to ICAO.

As of January 2013, safety oversight information is available

on the ICAO public website.

COSCAP 2013 46

Revision of Protocol Questions

(PQs)

COSCAP 2013 47

• PQ revision

• Change in PQ methodology

• Applicability (May 2013)

• PQ mapping and addressing revised PQs

• Impact LEI Score

• Actions required by States


Revision of Protocol Questions (PQs)

• In preparation for USOAP CMA, all PQs from the CSA cycle were revised and updated to reflect the latest changes in ICAO provisions.

• The review also took into consideration comments from States, ICAO (ROs, ANB, etc.), USOAP audit team members and external stakeholders.

• Under the USOAP CMA, PQs will be reviewed and updated on a periodic basis to reflect emerging issues and changes in Annex provisions and guidance materials.


Overview of the PQ revision


Overview of the PQ revision • The CSA cycle had a total of 975 PQs addressing

the 8 audit areas of LEG, ORG, PEL, OPS, AIR, AIG, ANS, AGA.

• The new USOAP CMA PQs have a total of 1016 PQs, including several revised PQs

– Deleted PQs: 15

– PQs merged with other PQs: 53

– New PQs: 105

5 PQs without a CE assignment have been deleted.

FROM A FINDING & RECOMMENDATION BASED TO A PROTOCOL BASED

Change in Methodology

COSCAP 2013 51


2.011

3.101

3.103

4.027

5.021

5.023

5.045

7.031

7.053

7.203

7.271

7.327

…

One

CAP/CAP

update

including:

Item 1

.

.

.

.

.

.

.

.

.

.

.

Item 20

Finding

text The current

organizational

structure of the

CAD is not

determined in

sufficient detail

as to provide a

clear definition of

the regulatory

and safety

oversight

functions and

responsibilities

related to PEL,

IOS, AIR< ANS

and AGA. In

addition, the

available

organizational

charts have not

been approved

yet.

ORG/01

One CAP

assessment

ORG/01 ORG

PEL

ANS

.

.

.

Finding

text

One

CAP/CAP

update

2.011 One CAP

assessment

Finding

text

One

CAP/CAP

update

3.101 One CAP

assessment

Finding

text

One

CAP/CAP

update

7.327 One CAP

assessment

From To

PQ Mapping & Addressing Revised PQs

COSCAP 2013 53

• Revisions to PQs have resulted in minor changes in LEIs of all States.

• LEIs on the CMA OLF should be considered live, since their status can change on a continuous basis.

• With the introduction of the revised PQs, States will regularly update their status of compliance and provide evidence on the CMA OLF (through the “Manage State self-assessment” and “Manage CAP” functionalities).



• PQs that are currently “satisfactory” will remain satisfactory, unless new evidence results in a change of status.

• Status of “not satisfactory” PQs may be changed by CMO on the basis of existing evidence provided for some PQs related to CE-1 to CE-5.

• PQs related to CE-6, CE-7 and CE-8 will typically require an evaluation through an on-site USOAP CMA activity to verify effective implementation.



Highlights of Changes

Per

PQ Area

COSCAP 2013 56

• Addition of new PQs on new ICAO Standards and Recommended Practices (SARPs) and guidance material

• Moving of certain LEG and ORG PQs to other audit areas (i.e. OPS and ANS)

• Harmonization of text for all PQs


General

• Area-specific PQs previously under LEG are now moved to the relevant audit areas, such as:

– 2 PQs on interception of civil aircraft OPS and ANS;

– PQ on units of measurement ANS; and

– PQ on journey log book and documents to be carried on board an aircraft OPS.


LEG

• New PQs have been added on mechanisms to avoid perceived or potential conflict of interest of inspectorate staff.

• 5 PQs have been deleted that were not associated with any CEs (3 ORG, 1 PEL and 1 OPS; related information now obtained from State Aviation Activity Questionnaire or SAAQ).


ORG

• ICAO Doc 9379 — Manual of Procedures for Establishment and Management of a State’s Personnel Licensing System (Second Edition) has been published in 2012.

• All references (in “ICAO ref.” column) have been revised to include ICAO Doc 9379.


PEL

• 2 PQs on flight data recorders have been moved to AIR (AIR PQs 5.204 and 5.205)

• These PQs have been moved from LEG to OPS: – 1 PQ on interception of civil aircraft

– 1 PQ on journey log book

– 1 PQ on documents to be carried on board an aircraft


OPS

• New PQs have been added

• Examples: – Aircraft registration (identification plate and use of

registration marks – AIR PQs 5.120 and 5.130);

– Safety Management System (SMS – AIR PQs 5.372 and 5.373); and

– AIR-related approval of CAT II and III instrument approaches (AIR PQs 5.330 and 5.331).


AIR

• New PQs have been added on procedures for inclusion of safety recommendations in draft final report forwarded to States and organizations concerned.


AIG

• Revisions include PQs on:

– New RVSM monitoring requirements;

– Performance-based navigation (PBN) implementation;

– Implementation of SMS/State Safety Programme (SSP); and

– Runway safety programme.


ANS

• PQs associated with multiple CEs (CE-2, CE-5, CE-6 and CE-7) are split into PQs associated with single CEs

• New CE-2 PQs have been added to address all chapters of Annex 14, Volume I

• New PQs have been added on minimum qualifications for aerodrome inspectors


AGA

Actions Required

by

States

COSCAP 2013 66


Collection & Processing of CMA Info

On-line Framework

COSCAP 2013 68

• A suite of web-integrated applications and centralized database systems to support the implementation of CMA activities

• Includes:

– Collection of safety-related information and documents from different sources

– Monitoring and reporting of safety oversight activities by Member States


CMA On-line Framework


State Dashboard

NCMCs to view at a glance: Status of CMA information completion Status of overall LEI/EI Status of CAP progress


Access Control • NCMCs

– Have a full control of user accounts for the State • Issuing additional user accounts • Deactivating invalid user accounts • Setting individual user rights/permissions

• Individual Users – Manage user profile information – Personalize password

Note: If you are not an NCMC and want to have access to the online framework, please contact the NCMC(s) of your State

• User account

– composed of “ID or Email address” and password

– duplicate IDs or email addresses are not accepted

• Access rights

– 3 types (no access, read-only or read/write)

– CC/EFOD : access rights can be granted per Annex

– PQ, CAP and F&R : access rights are permitted for all technical areas


Access Control

Update on

Electronic Filing of Differences (EFoD)

COSCAP 2013 73


States

Difficulty in comprehending

and interpreting

Annex provisions

Lack of knowledge

and expertise

Lack of effective national

process and procedures

Lack of States’ commitment

ICAO

Not at all or not properly file differences

Not publish significant

differences in AIPs

Misleading, inconsistent, insufficient guidance

Limited value of

Supplements

Frequent Annex Am’t

and unrealistic

applicability dates

• March 2011, Council agreed to use EFOD as an alternative means of filing of differences

• Jan 2011, CC data from 153 States were migrated into the EFOD system


Filing Differences – Transition

Keep up-to-date compliance

checklist pursuant to MOU for

USOAP CMA

File differences pursuant to

Article 38 of Chicago

Convention

• In April 2011, States were requested to (SL AN 1/1-11/28):

• Recognize EFOD as an alternative means of filing of differences

• Verify and confirm the data in EFOD which had previously been submitted through CC under USOAP. – Over 100 States have been updating data in EFOD – 45 States sent replies (34 States completed

verification, 11 States submitted plan for completion)


Filing Differences – Transition


Procedures on the Use of EFOD

• Considered and approved by Council at its 195th Session

• To be Published by Secretary General in 2nd Quarter 2013

• Table of Contents 1. Introduction/Scope

2. Access to the System

3. Languages

4. Annex Information

5. Compliance/Differences Information

6. System performance

7. References

• Static Report

• Final reports based on onsite activities conducted since the first cycle of USOAP

• Published according to predetermined timelines after the closing meeting of an onsite activity


Previous USOAP Report

Dynamic Report

• Based on the most recent online and/or onsite activity for each audit area/CE


CMA USOAP Report

• Understanding of the USOAP CMA methodologies

• Essential knowledge required to participate in USOAP CMA activities

• Enhance the competencies in the areas addressed by USOAP CMA

• NCMC and others


USOAP CMA Computer Based Training (CBT)

• Manage user accounts of the CMA online framework

• Keep up-to-date SAAQ and CC/EFOD

• Provide a CAP update and/or progress per corresponding protocol finding

• Provide self-assessment result on each PQ with a priority on the PQs that have a “Not Satisfactory” status

• Respond to MIR in a timely manner


Actions required by States

Wayne Chapin

Chief Technical Advisor

COSCAP North Asia

[email protected]


Questions & Comments

mailto:[email protected]



Documents

Continuous Monitoring Approach - Updatecoscap-na.com/sites/default/files/13SCM_CMA rev1_0.pdf · •Introduce ICAO CMA Online Framework •Filing of Differences (EFoD) •USOAP Reports