(ASMM) and Assessment ATM Safety Maturity Model 1 APAC Your Service Provider for Success ATM Safety Maturity Model (ASMM) and Assessment EUROCONTROL Safety R&D Seminar Barcelona 2006

APA-PHO-AP0A7G-1.0b-0 1

APAC Your Service Provider for Success

ATM Safety Maturity Model (ASMM) and Assessment

EUROCONTROL Safety R&D SeminarBarcelona 2006

ATM Safety Maturity Model (ASMM) and Assessment

EUROCONTROL Safety R&D SeminarBarcelona 2006

[email protected]

[email protected]

www.apac.at



ExperienceExperience• EAD • SES activities

– SESIS– SESFARR– Recognised Organisation

• Safety assessment activities– Assessment of SAM as an AMC with ESARR 4– EAD– OATMS

• Definition of safety management systems– Austro Control– Slovenia Control

• Studies/Research– ASMM– Knowledge management database



SES ActivitiesSES Activities• SESIS

– Supporting the development of guidance material for• ANSPs for the implementation • NSAs – certification

• SESFARR– Supporting the development of a questionnaire to

assess the status of the implementation of the SES regulations (EU member states and associated states)

– Supporting the development of the report• Recognised Organisation

– APAC was recognised according to Art. 3 Regulation (EEC) No.550/2004 of March 2004 by the Austrian National Supervisory Authority



End of 2004End of 2004

• Common Requirements ante portas

• ESARRs

• A large number of safety-related standards

• Dozens of GUIs, partially in draft status

• SWAL seems to be something esoteric for suppliers

• ...



SAFETY (ATM) Maturity Model BasisSAFETY (ATM) Maturity Model Basis

R

E

Q

U

I

R

E

M

E

N

T

S

European CommissionRegulation

No. 550/2004

Common Requirements

EurocontrolESARR 5

Safety Regulatory Requirement for ATM Services' Personnel

ISO 9001

EurocontrolESARR 3

Use of Safety Management

Systems by ATM Service Providers

EurocontrolEATMP

Safety Policy

Eurocontrol & European

CommissionESARR 1

Safety Oversight in ATM

National Supervisory Authority

Air Navigation Service Provider

Supplier and/or ANSP

EurocontrolESARR 2

Reporting and Assessment of

Safety Occurrences in ATM

EurocontrolESARR 7

ATM Procedures(RVSM, GNSS, Data Link, etc.)

EurocontrolESARR 6

Software in ATM Systems

EurocontrolESARR 4

Risk Assessment and Mitigation in ATM

EurocontrolESARR 6

Software in ATM Systems

Guidance Material

EurocontrolSafety

Assessment Methodology

(SAM)and Guidance

Material

EN 61508Functional Safety of electrical/electronic/

programmable electronic safety-related systems

ApplicableICAO SARPs

(Standards and Recommended

Practices)

FullCompliance Determine

Compliance

Scope of the Safety

Capability Maturity Model

Legend:ATM

Extension Capability Maturity Model

System SCMM

Questionnaire

ATM Extension

Questionnaire

ESARR Technical System

Development Questionnaire

EN 61508Questionnaire

Technical System/Service Development

Level

Responsibility

Air Traffic Management Service Provision Level

Supervision Level

EUROCAE ED78A,

(Guidelines for approval of the

provision and use of Air

Traffic Services supported by data communications)

Approved Means of

Compliance

Determine Compliance

Means of Compliance to be taken into Account



The GoalThe Goal

• Establish clear pass/fail criteria to reach safety objectives

• Repeat ISO9001/Bootstrap (BICO) success by added value to a pass/fail assessment

• Benchmarking of SMS

• Target environments– ANSPs (results re-usable for NSA certification)

– ATM system supplier



AT

M C

MM

Mod

el

Dev

elop

men

t

Res

earc

h on

ex

istin

g m

ate

rial

Dev

elop

men

t of C

ore

Saf

ety

Cap

abili

ty M

odel This phase includes the

definition of the best practices

1

3

5

7

9

11

13

Months

I

II

III

IV

V

Phase

Dev

elop

men

t of t

he

Sys

tem

SC

MM

Q

ues

tionn

aire

Dev

. of t

he

ES

AR

R

Sys

tem

Dev

elop

men

t Q

ues

tionn

aire

Dev

elop

men

t of t

he

EN

615

08

Qu

estio

nnai

re

AT

M E

xten

sion

Q

uest

ion

naire

Test / Validation

Update

Final Delivery

Update Update Update

Model Development PlanModel Development Plan

Full Common Requirements Scope



Cooperation with UniversityCooperation with University• Evaluation of available maturity models –

ISO15504 (SPICE) selected as basis

• Combination of ATM/safety requirements with ISO15504

• Draft questionnaires for all areas

• Validation in ANSP environment only partially achieved

• Validation in industrial environment of the ISO15504/EN61508 model



2005 Target Environment Response2005 Target Environment Response

• ANSPs – ANSP - do not seem to be too interested in numbers and seem to consider benchmarking a threat rather than a help.

• ATM suppliers and automotive suppliers are interested, but– We have to speak a language they can

understand = EN61508

– Focus on projects (system aspects: ISO9001)

– Separation of development and safety maturity is not acceptable (development and safety aspects have to be covered in one assessment)



The Supplier Safety Maturity Model

The Supplier Safety Maturity Model

• EN 61508 is not a development standard and project oriented but very successfully applied and widely accepted in industry

• Combination of ISO15504 (SPICE) with EN 61508 leads to a sound and robust model capable to become an AMC for SWAL requirements

• Both are well established standards• ISO15504 provides a reference system for

benchmarking and a well defined algorithm for scoring



The Assessment BasisThe Assessment Basis

• ISO15504 (SPICE)

• EN 61508

• Allow uncertainties to keep the assessment effort acceptable –mitigation of resulting risks by application of common sense



Capability LevelCapability Level

Planned-and-Tracked Level

Informally-performed Level

Well-defined Level

Quantitatively-controlledLevel

• Performing the process

• Planning performance • Disciplined performance• Verifying performance• Tracking performance

• Defining a standard process

• Performing the defined process

• Establishing measurable quality goals

• Objectively managing performance

Continuously-ImprovingLevel

• Improving organisational capability

• Improving process effectiveness



Architecture of the QuestionnaireArchitecture of the Questionnaire

12 G

EN

ER

IC

PR

AC

TIC

ES

5 GE

NE

RIC

P

RA

CT

ICE

S

3 GE

NE

RIC

P

RA

CT

ICE

S

5 GE

NE

RIC

P

RA

CT

ICE

S

............



Example Engineering Level 1Example Engineering Level 1

Engineering Processes

QUESTION REMARKS FOR SCORING SPICE SIL 2 Methods&Techniques 61508

Level 11.1: Perform Processes

2.1 Develop software requirements

Develop software requirements:

Determine software requirements; analyse software requirements;

R: Computer-aided specification tools; Tools without preference for one particular design method;

Establish, analyse and refine the software requirements.

determine operating environment impact; evaluate requirements with customer; update requirements for next iteration

R: Describe some critical parts with semi-formal methods e.g.: Logic-Function Block Diagrams, Sequence Diagrams Dataflow Diagrams, Finite State Machine/State Transition Diagrams, Time Petri Nets, Decision Truth TableR: Formal Methods including for example, CCS (Calculus of Communicating Systems), CSP(Communicating Sequential Processing), HOL, LOTOS, OBJ, temporal logic VDM and Z



Example Engineering Level 2Example Engineering Level 2Level 2

2.1: Planning Performance2.10 Allocate

resourcesAllocate adequate resources (including people) for performing the process category "engineering".

Evidence of resource allocation exists; records/plan indicate resources are allocated to perform job tasks

2.11 Assign responsibilities

Assign responsibilities for developing the work products and/or providing the services of the process category "engineering".

Assigned responsibilities are recorded; representative understands the process and tasks he is responsible for

2.12 Document the process

Document the approach to performing the process category "engineering" in standards and/or procedures.

Tasks to be performed; inputs and outputs; entry/exit criteria; control points; internal and external interfaces; process measurements

2.13 Provide tools Provide appropriate tools to support performance of the process category "engineering".

Adequate training in the operation of the tool; documentation and/or instructions are available for the tool; support for the tool is available

2.14 Ensure training Ensure that the individuals performing the process category "engineering" are appropriately trained in how to perform the processes.

Training is available for tools;training curriculum covers all tasks;resources are allocated for training

2.15 Plan the process Plan the performance of the process category "engineering".

WBS; project standards; special needs; reuse strategy; resource estimation; risks; schedule



Rating ProcessRating Process



Rating SchemeRating Scheme

• The rating is represented as follows:N...Not Adequate 0 <= x < 16

P...Partially Adequate 16 >= x < 51

L...Largely Adequate 51 >= x < 86

F...Fully Adequate 86 >= x <= 100

• If a question is not applicable, it is not taken into account for scoring.



Ratings for the LevelsRatings for the Levels

• 1st Step: all best practices for Level 1 will be evaluated with N, L, P and F.

• 2nd Step: calculation of the evaluation average

• 3rd Step: result will be mapped to the interval of N,L,P and F. You received the values for Level 1.

• The same process will be used for the Generic Practices of Level 2, 3, 4 and 5.



Example of a Derived RatingExample of a Derived Rating

Leve

l 1

Best Practices Scoring3.1 0.85993.2 0.85993.3 0.85993.5 0.85993.9 1

Results per LevelLevel 1 0.8879Level 2 0.8599Level 3 0.7899Level 4Level 5

Sum Levels 2.75



Example ResultsExample ResultsAssessments Result Example, October 2005 and October 2006

0,00

0,50

1,00

1,50

2,00

2,50

3,00

3,50

4,00

4,50

5,00

SIL 2

Oct. 05

Oct. 06



Description Audit Ref ActionProject

Recording reasons of decisions 1) Record reasons of decisions, requirements/(sepcification and design process

Independ safety manager 2) Define an independ safety manager :- Update Quality manual.- List tasks

V&V responsible 3)4.2) - 1.85

company or project level- Update Quality manual- List tasks- Update default PMP/PQP

Indenpendence between tester and developer 4) - Update Quality manuel- Update default PMP/PQP

Configuration management 12) - Update CVS procedure, and a $Name tag- Update source code tempalte

Working environment 4.3) - 1.5 Define of a reuse strategy and processSafety

Quality manual to adress safetyInitiation of a safety-life cycle: policy, persons, activities, documentation, phases

5)1.2) - 0.153.1) - 1.854.1) - 1.854.2 ) - 1.85

Safety management procedures to be integrated in PQP and QS : Defined strategy, vision and culture (management goals) and communication.Identify person, department and organisation if charge of independent safety activities- List tasks- Update Quality manual

Implement safety management during development

1.4) - 0.151.5) - 1.85

- Update SDP (define safety specific analysis of hazardous incidents and operations and maintenance performance, safety validation procedures, periodical functional safety auditsSDP : safety requirements, identification of safety-related functions, clear interface to non-safety-related functions

Safety management after delivery 1.6) - 0.15 Definition of procedures for initiatin and approving modifications to the system incl. Responsibilities and documents

Update reporting system 1.7) - 1.75 Define procedure for maintaining accurate information on potential hazards and safety related system

Results as Basis for Process Improvement

Results as Basis for Process Improvement

• Action Plan

• Charts allow easy visualisation of improvement (“management compatible“)



Questions

Discussion

Questions

Discussion

Thank you for your attention

Documents

(ASMM) and Assessment ATM Safety Maturity Model 1 APAC Your Service Provider for Success ATM Safety Maturity Model (ASMM) and Assessment EUROCONTROL Safety R&D Seminar Barcelona 2006