The SACAA Professional

Pilot Examination System Mr Blake Vorster

Senior Manager Aviation Personnel Standards

SA Civil Aviation Authority

1

The SACAA Professional

Pilot Examination System

• The Historical Safety Risk

• Examination Considerations

• Risk Mitigation

• Process

• Pass Rate and Examination System

Architecture

2

The Historical Safety Risk

• Prior to January 2014

Reliant on Windows 98 – could collapse.

Fixed examination papers per subject.

Some questions based on unreferenced

documents.

Papers compromised.

3

The Historical Safety Risk

• Prior to January 2014

Students studying to pass

the known examination

questions.

Examination fixing.

Syllabus outdated.

4

Examination Considerations

ICAO definition of competency:

• Competency. A combination of skills,

knowledge and attitudes required to perform

a task to the prescribed standard.

(Annex 1 — Personnel Licensing Chapter 1)

5

Examination Considerations

• “To operate an aircraft safely, a pilot must have an

understanding of technical subjects such as aircraft

engines, instruments, meteorology and navigation.

• To make the appropriate decisions for the safe operation

of an aircraft, the pilot must exercise judgement. This is

based on training, knowledge and experience…”

Human Performance and Limitations in Aviation P.3

(R.D. Campbell & M. Bagshaw)

6

Examination Considerations

• “Traditionally, pilot training utilises knowledge, skill

and experience.

• Knowledge has to be gained before a skill can be

developed, and then improved by experience.”

Human Performance and Limitations in Aviation P.111

(R.D. Campbell & M. Bagshaw)

7

Examination Considerations

• “Having the knowledge to be successful and

applying that knowledge successfully are two

different scenarios. We must possess the

knowledge in the first place, organise the

knowledge in a manner that makes it usable, and

then know when to apply it.”

Briefings on Patient Safety P.5 (Medical Article)

(2008 HCPro, Inc)

8

Examination Considerations

• “Knowledge-based errors occur when situations

arise and we have not been prepared to address

them (i.e., no rules exist). In these cases, we must

rely on our basic knowledge and apply it to the new

situation.”

Briefings on Patient Safety P.6 (Medical Article)

(2008 HCPro, Inc)

9

Risk Mitigation

• Revise and revamp the theoretical

knowledge examination syllabi.

• Use the existing PPL examination system

software on a secure island server.

• Security improved.

10

Process

• Use reliable reference material.

• CPL and ATPL syllabi benchmarked

against EASA.

• Extensively work-shopped with industry.

• Regulatory progress followed for at least

12 months with approval in August 2012.

11

Process

• Syllabi published in January 2013 with an

implementation date of 1 August 2013.

• Revised on 1July 2013 to read 1 January

2014.

12

Process

• Industry informed in various manners e.g.

notices to Approved Training Organisations and

notices published on the SACAA website.

• Information was shared on at least 18

conferences for Designated Flight Examiners

and Flight Instructors during 2013.

13

Process

• There was regular communication with industry

for 16 months prior to implementation of the

new syllabi.

14

Process

Recommended reference

material from which the

examinations would be

compiled was published.

15

Pass Rate and Examination

System Architecture

• No comparison between the old (i.e. pre-

2014) exams towards the CPL and ATPL

and the new database.

• Many of the new questions have been

obtained directly from EASA – high quality.

16

Pass Rate and Examination

System Architecture

• Off-the-shelf system – customised for SACAA.

• Entire syllabus captured.

• Random generation of questions.

• Detractors order changed.

• Excess of 10 000 questions in data base.

• Each question verified.

17

Pass Rate and Examination

System Architexture

• Drop in the pass rate – there is a strong

correlation between the level of preparation

and the exam scores.

• Use the syllabus and the recommended

reference material.

• Students from some schools doing well.

18

Pass Rate and Examination

System Architecture

More emphasis on the

evaluation of practical

knowledge (i.e. understanding).

19

Pass Rate and Examination

System Architecture

• System of Continuous Improvement.

20

Thank you!!!

21