Embed Size (px)
Citation preview
National Fighter Procurement Secretariat
Secrétariat national d’approvisionnement en chasseurs
Industry Engagement RequestQuestionnaire 1
Assessment Methodology
Meeting with Companies
May 03, 2013
2
• To provide companies with details on the assessment methodology that will be used in assessing your responses to Questionnaire 1
• To provide clarifications and provide answers to questions
• To provide companies with information on next steps
• We are committed to an open, fair and transparent process
Purpose
Industry Engagement RequestQuestionnaire 1
• Canadian fighter mission and threat information across two time horizons (2020-2030 and 2030+).
• Responses to include defined mission configurations:– Air-to-Air;– Air-to-Ground; and– Air-to-Surface.
• Responses to include capability developments (planned upgrades).
• Responses to include descriptions of capability, production and supportability.
3
PRINCIPLES
The following Principles will guide all participants involved in theassessment of the responses:• Consistency
– Team leads will ensure that the assessment is well documented, including all qualitative and professional military judgement aspects.
– Individual aircraft are being assessed against each Canada First Defense Strategy (CFDS) mission.
– No aircraft will be screened out. All options will remain on the table. Final product will be a high level risk assessment.
• Objective– The National Fighter Procurement Secretariat will provide oversight through all
steps of the assessment process
• Use of Third Parties– Responses will be shared with the Independent Review Panel (i.e. Panel)
members and any significant differences of opinion between assessment teams will be brought to the attention of the Panel.
4
Overview of Q1 Assessment• The “Capabilities” questions of the Industry Engagement Request will be
assessed using a 3-step assessment process:– Step 1 – Assessment of Measures of Performance (MOP);– Step 2 – Assessment of Measures of Effectiveness (MOE); and– Step 3 – Assessment of Aerospace Capabilities weighted against Canada
First Defense Strategy (CFDS) Mission and criticality to result in “Operational Risk” assessment of each aircraft for each of the missions.
• The “Production and Supportability” questions will be assessed using a 2-step assessment process:
– Step 1 – Assessment of responses by technical Subject Matter Experts teams on Military Strategic Assessment Factors – Impact Statements; and
– Step 2 – Assessment of responses by the Royal Canadian Air Force Senior leadership level from a missions perspective to result in “Strategic Risk” assessment of each aircraft for each of the missions.
• Both the “Operational Risk” and “Strategic Risk” will then be consolidated into an overall risk profile for each aircraft against each mission.
5
Step 1 of Capabilities Assessment• Assessments being performed using responses to the 17 areas of
« Capabilities » contained in the Industry Engagement Request.• National Defence teams assessing specific areas, with oversight from the
National Fighter Procurement Secretariat. • Qualitative assessment - independent of missions.• Results in Measures of Performance assessment for each aircraft.
6
Sensors Radio Frequency Sensors
Electro-optical or Infrared Sensors
Air-to-Air Weapons
Weapons Air-to-Ground Weapons
Air-to-Surface Weapons
Non-Kinetic Weapons (Electronic Attack)
Self Protection
Radio Frequency Self Protection
Infrared Self Protection
Countermeasures
Avionics Data Link
Pilot Workload
Communications
Sensor Integration
Performance Radio Frequency Signature
Infrared Signature
Engine/Airframe
Combat Radius/Endurance
Evaluation Grid for Step 1
• Performance rating of individual systems
• Rating scale from 0-10 to allow sufficient fidelity
• Classified systems information represented by a number (supported by classified references)
Rating Definition
10 Excellent Without appreciable deficiencies9
8 Very Good Limited only by minor deficiencies7
6 Good Limited by moderate deficiencies5
4 Poor Limited by major deficiencies3
2 Very Poor Significantly limited by major deficiencies1
0 Non-Existant
7
Step 2 of Capabilities Assessment• Three Royal Canadian Air Force teams (four members each)
performing this assessment, with oversight from the National Fighter Procurement Secretariat:
• Including fighter pilots and at least one Air Force Air Weapon Controller on each team.
• Significant differences of opinion between assessment teams will be brought to the attention of the Panel.
• The following inputs will be used in this assessment:• Scores and input from the results of Step 1;• Companies responses to the Industry Engagement Request,
including the mission configuration systems; • Vignettes detailed in Appendix “A” of the Industry Engagement
Request;• Detailed threat capabilities from Canadian Forces Threat
Assessment;• Detailed fighter task lists; and• Other information available to Government.
• Fighter Measures of Effectiveness areas of consideration detailed in the Industry Engagement Request.
8
Information on Missions as per the Canada First Defence Strategy (CFDS)
1 – Conduct daily domestic and continental operations, including in the Arctic and through NORAD (Vignette 1)
2 – Support a major international event in Canada, such as the 2010 Olympics (Vignette 2)
3 – Respond to a major terrorist attack (9/11-type scenario under CFDS Mission 1 through NORAD) (Vignette 5)
4 – Support civilian authorities during a crisis in Canada such as a natural disaster (No Vignette)
5 – Lead and/or conduct a major international operation for an extended period
[including complex peace enforcement operation, and coalition state-on-state war-fighting] (Vignettes 3 & 4)
6 – Deploy forces in response to crises elsewhere in the world for shorter periods (Vignette 6)
Of the 6 Canada First Defense Strategy missions above, only 5 relate to the 6 vignettes. Mission 4 (Support civilian authorities during a crisis in Canada such as a natural disaster) will be analyzed together with Mission 2, understanding the fighter’srole in this mission is generally minor.
9
Information on Aerospace Capabilities
1. Defensive Counter Air (DCA)2. Offensive Counter Air (OCA)3. Strategic Attack4. Close Air Support (CAS)5. Land Strike6. Tactical Support to Maritime Operations (TASMO)7. Intelligence, Surveillance, and Reconnaissance (ISR)
10
Information on Fighter Measures of Effectiveness
1. Awareness2. Survivability3. Reach and Persistence4. Responsiveness5. Lethality6. Interoperability
11
Step 2 of Capabilities Assessment Overview
• Step 2 will first generate “raw” scores for Measures of Effectiveness (MOEs).
• Raw results will then be weighted against the applicable mission to come up with an Operational Level Aerospace Capability Score for each aircraft against each mission over the 2 timeframes.
• Mission weightings are sensitive and not included as part of this presentation.
12
Evaluation Grid for Step 2 Raw Scores
• Effectiveness rating of contributing system performances.
• Multiple systems will contribute to each measure of effectiveness.
• Will vary across all missions.
• Rating scale from 1-10 to allow sufficient fidelity.
• Qualitative, and thus requires Subject Matter Expert input.
• Will result in “Raw” scores for Step 2.
13
Rating Definition
10 Excellent No appreciable limitations in delivering the desired effect9
8Very Good Only minor limitations in delivering the desired effect
7
6Good Moderate limitations in delivering the desired effect
5
4Poor Major limitations in delivering the desired effect
3
2 Very Poor Severe limitations in delivering the desired effect1
14
Step 2 Example CFDS MissionAerospace Capability
1. Defence of Canada – Baseline and Arctic
Defensive Counter Air Vignette 1
ThreatThe Canadian Fighter is conducting normal daily and contingency North American Aerospace Defence (NORAD) missions at normal alert levels, and is prepared to react to elevated alert levels. Operations are conducted from main operating bases as well as Forward Operating Locations and missions may be over land and over water. Operations are being conducted in a combined and/or joint environment with NORAD forces. The threats for these missions can be air and maritime-based.
2020-2030: Non-state civilian aircraft (including low/slow)State long-range aviation Classified Threat List (NORAD)
Tasks
- Rapidly establish and maintain air presence/policing/deterrence for a tactically significant period of time in an area of operations.- Detect, validate, characterize, assess, and warn of unwanted and unauthorized air activity approaching or operating within North American domestic airspace including potential attacks against North America. (NCI 10-175, 1.2.1)
- Detect, Identify, Determine Intent, Monitor, Intercept, Report, Negate, or Destroy potential air threats.
2030+: Non-state civilian aircraft (including low/slow)State long-range aviation and escortsNon-state use of stand-off missiletechnologiesClassified Threat List (NORAD)
15
Defensive Counter Air
(MOEs) using Vignette 1 in 2020 to 2030
timeframe
Aircraft X
Aircraft Y
Lethality 2 9
Survivability 4 9
Responsiveness 9 6
Reach and Persistence 9 6
Interoperability 5 8
Awareness 6 8
Tech Area Contributing System Aircraft X Aircraft Y
Sensors Radio Frequency Sensors 3 8
Electro-optical or Infrared Sensors 7 7
Weapons Air-to-Air Weapons 2 8
Air-to-Ground Weapons
Air-to-Surface Weapons
Non-Kinetic Weapons 0 5
Self Protection Radio Frequency Self Protection 2 9
Infrared Self Protection 5 7
Countermeasures 5 8
Avionics Data Link 5 10
Communications 7 7
Sensor Integration 2 8
Pilot Workload 8 9
Performance Radio Frequency Signature 4 9
Infrared Signature 3 6
Engine/Airframe 9 6
Combat Radius/Endurance 9 7
Step 2 Example – Assessment of Raw Scores
16
Step 2 Example – Capability Raw Scores
MissionAerospace Capability
Measures of Effectiveness (Aircraft X)
Lethality SurvivabilityReach andPersistence
Responsiveness Interoperability Awareness
1. Defence
of Canada(Baseline and Arctic)
Defensive Counter Air 2 4 9 9 5 6
Offensive Counter Air
Strategic Attack
Close Air Support
Land Strike
TASMO
ISR
5.8
Example Only
17
Step 2 Example – Capability Raw Scores
MissionAerospace Capability
Measures of Effectiveness (Aircraft Y)
Lethality SurvivabilityReach andPersistence
Responsiveness Interoperability Awareness
1. Defence
of Canada(Baseline and Arctic)
Defensive Counter Air 9 9 6 6 8 8
Offensive Counter Air
Strategic Attack
Close Air Support
Land Strike
TASMO
ISR
7.66
Example Only
18
Step 2 - Capability (Weighted)
MissionAerospace Capability
(Weighted) Measures of Effectiveness (Aircraft X) (2020-2030)
Lethality SurvivabilityReach and
Persistence Responsiveness Interoperability Awareness
1. Defence
of Canada
(Baseline and
Arctic)
Defensive Counter Air
Offensive Counter Air
Strategic Attack
Close Air Support
Land Strike
TASMO
ISR
Applying weighting function to each measure of effect allows capability to be properly quantified within that Mission
6.7Example Only
19
Step 2 - Capability (Weighted)
MissionAerospace Capability
(Weighted) Measures of Effectiveness (Aircraft Y) (2020-2030)
Lethality SurvivabilityReach andPersistence
Responsiveness Interoperability Awareness
1. Defence
of Canada
(Baseline and
Arctic)
Defensive Counter Air
Offensive Counter Air
Strategic Attack
Close Air Support
Land Strike
TASMO
ISR
Applying weighting function to each measure of effect allows
capability to be properly quantified within that Mission
7.1Example Only
20
Step 3 of Capabilities Assessment
• This last step of the Capabilities Assessment takes into account: a) the weighting of each Fighter Aerospace Capability within a particular mission and b) criticality as follows:
• DCA – XX%, OCA – XX%, TASMO – XX%, ISR – XX%;• Mission Critical (MC), Mission Essential (ME), or Mission Routine (MR);
• The Operational Level Score obtained at the end of Step 2 is aggregately weighted and assessed by Royal Canadian Air Force against the mission criticality matrix.
• Aerospace capability limitations will carry risk forward dependant on their mission criticality (translation matrix on slide 22 expands).
CFDS Mission 1 (example)
6.1
TASMO ISRLand StrikesCASStrategic
AttackOCADCA
7.37.1Aircraft
Y
Aircraft D
Aircraft C
Aircraft B
Aircraft A
ISRTASMOLd Strk.CASStrat Atk.
OCADCA
CFDS Mission 1 (example)
ECCMission
criticality (C/E/R)
X%X%X%Weightingroles VS
CFDS
Weighting of Fighter Roles / CFDS missions+ Mission Criticality
Step 3a – Operational Mission Weighting
21
Operational level scoring (before Mission criticality) 2020-2030
CFDS 1CFDS
2/4CFDS 3 CFDS 5 CFDS 5 CFDS6
V1 V2 V5 V3 V4 V6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Aircraft Y 7.0
Operational Risk 2020-2030
CFDS 1CFDS
2/4CFDS 3 CFDS 5 CFDS 5 CFDS 6
V1 V2 V5 V3 V4 V6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Aircraft Y
Aircraft E
Step 3b - Mission Criticality Translation Matrix
22
DefinitionsMission Critical: A capability that delivers a direct effect as part of its primary function. This effect Is evaluated as critical to mission success and will pose risk to mission success if it is not employed.
Mission Essential: A capability that is an essential enabler to Mission Critical Capabilities; the lack of one or more Mission Essential Capabilities will pose risk to mission success.
Mission Routine: A capability that is required for the mission but either as a routine supporting function or a very low likelihood of employment. Only in cases where multiple Mission Routine Capabilities constitute a systems failure will any significant risk be posed to the mission.
Insignificant limitationsMinor limitationsModerate limitationsMajor limitationsSevere limitations
Low operational riskMedium operational risksignificant operational riskHigh operational riskVery high operational risk
Weighted Capability Assessment of each
Aerospace Capability Criticality of
Aerospace Capability
Maximum Operational Assessment
+ =
+ =
+ =
Mission Critical
Mission Essential
Mission Routine
Operational level scoring (before Mission criticality) 2020-2030
CFDS 1 CFDS 2/4 CFDS 3 CFDS 5 CFDS 5 CFDS 6
Vignette1
Vignette 2 Vignette 5 Vignette 3 Vignette 4 Vignette 6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Example
Step 3 – “Operational Risk” Assessment
Mission Criticality Translation Matrix
Operational Risk 2020-2030
CFDS 1 CFDS 2/4 CFDS 3 CFDS 5 CFDS 5 CFDS 6
Vignette 1
Vignette 2Vignette
5Vignette
3Vignette
4Vignette
6
Aircraft A
Aiircraft B
Aircraft C
Aircraft D
Aircraft E
Example
Step 3 - End State “Operational Risk”
Rating Definition
10 Insignificant limitations in accomplishing the mission.9
8 Minor limitations in accomplishing the mission7
6 Moderate limitations in accomplishing the mission.5
4 Major limitations in accomplishing the mission3
2 Severe limitations in accomplishing the mission.1
Definition
Low operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Medium operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Significant operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
High operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Very high operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
+ =
+ =
+ =
MC
ME
MR23
24
Assessment of the “Production and Supportability” Questions• Step 1 – Review of responses by technical Subject Matter
Expert teams – no scoring.
• Step 2 – Assessment of responses by the Royal Canadian Air Force Senior leadership level from a Canada First Defense Strategy missions perspective to result in “Military Strategic Risk” assessment of each aircraft against the missions.
• Both Steps will be overseen by the National Fighter Procurement Secretariat.
• Significant differences of opinion between assessment teams will be brought to the attention of the Panel.
Factor Analysis through Deductive ReasoningWhere does the Industry response fit inside the framework?
25
Strategic Assessment Definition:There is a possible risk that…
Factors
The mission need is…
The threat is assessed as…
CFDS
Likelihood
Possible Impacts
Required mitigations
Considerations
So What?
Industry responses to Industry Engagement Request
Open source study
Government-to-Government Information
Other products from the Evaluation of Options
Source Documentation
So What?Strategic RiskAssessment
Deductions(step 1) (step 2)
(used during step 2 only)
Acquisition• This Strategic Assessment Factor (SAF) is an assessment of
factors, other than cost, that would affect acquisition of aircraft types. This includes:
• planned production periods of aircraft type;
• the manufacturer’s ability to continue or re-establish production periods;
• the ability of the manufacturer to offer a complete “cradle to grave” program, including; training, in-service / life cycle support and disposal;
• the manufacturer’s successful completion of any required developmental work before the aircraft are ready for acquisition; and
• the ability of aircraft type to be certified for airworthiness under Canadian Forces regulations.
26
Supportability and Force Management• An assessment of the overall / long-term supportability and
aspects related to management of the fighter force. This includes factors such as:
the required quantity of aircraft; managed readiness postures and sustainment rations; the training system and production of appropriately trained pilots and
maintenance / support personnel (including use of simulators); training considerations for maintenance / aircrew personnel; aircraft mission availability rates and aircrew / technician workload; long-term availability of components needed for aircraft maintenance; implementation and sustainment of supporting infrastructure; the ability to operate from required locations; weapons compatibility and support; and the suitability of a reprogramming capability including its flexibility and
responsiveness for new threats and/or theatres of operation or new capabilities.
27
Integration
• This Strategic Assessment Factor assesses broad interoperability within the Canadian Forces and with allied forces. This includes interoperability with air-to-air refuelling services (other than Canadian Forces), common ground/spares support with allies and their supply lines, training systems, data sharing, communications, STANAGs, and the ability to feed data into Canadian Forces and Government of Canada networks taking into consideration national security requirements.
28
Growth Potential
• This Strategic Assessment Factor assesses the growth potential and technological flexibility to respond to unforeseen future advances in threat capabilities, to implement required enhancements to fighter technology, and to evolve as needed to meet the Canadian Forces’ needs. Factors include analysis of the architecture of aircraft types, power and cooling capability for new systems.
29
30
Response toIER Strategic Assessment Factors
Acquisition
Supportability &Force Management
Integration
Growth Potential
Future Security Env’t
Other reports resulting from evaluation ofoptions
Step 1Aircraft - SAF Assessment
- Technical SME level working groups
- Problem space defined and scoped by the use of a “possible risk” based questionnaire
- SME comment on likelihood; then- Comment on possible impacts and
required mitigations to the overall fighter program
- Information to be collated and considered in Step 2.
Step 2SAF - CFDS assessment
- Senior leadership level Working group (fighter heavy)
- Problem space defined and scoped by the Task 2 report
- “Considering the likelihood of possible impacts to the overall Fighter Program, and based on your Professional Military Judgment, how would you rate the strategic risk to the RCAF ability to successfully fulfill the fighter contribution to each CFDS mission?”
Open Source/Other Info and Gov-to-Gov Info
End State - Strategic Risk
Strategic Risk 2020-2030
CFDS 1CFDS
2/4CFDS 3 CFDS 5 CFDS 5 CDFDS 6
V1 V2 V5 V3 V4 V6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Example
Definition
Low strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Medium strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Significant strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
High strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Very high strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Final Stage – Overall Mission Risk Assessment
End State: Mission Risk
Strategic Risk Operational Risk
31
End StateMission Risk
Operational Risk 2020-2030
CFDS 1CFDS
2/4CFDS 3 CFDS 5 CFDS 5 CFDS 6
V1 V2 V5 V3 V4 V6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Aircraft Y
Mission Risk(fusion of Operational and Strategic risk)
RCAF to apply professional military judgement
Strategic Risk 2020-2030
CFDS 1CFDS
2/4CFDS 3 CFDS 5 CFDS 5 CFDS 6
V1 V2 V5 V3 V4 V6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Aircraft Y
C Air Force endorsement of Final Report
Definition
Low risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Medium risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Significant risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
High risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
Very high risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
32
Mission Risk 2020-2030
CFDS 1 CFDS 2/4 CFDS 3 CFDS 5 CFDS 5 CFDS 6
V1 V2 V5 V3 V4 V6
Aircraft A
Aircraft B
Aircraft C
Aircraft D
Aircraft E
Aircraft Y
33
Mission Risk
• Mission Risk will be an outcome of the fusion of Strategic and Operational risk by the same RCAF senior leaders assessment teams who took part in step 2 of the Strategic Assessments.
• Decision support tools, such as but not limited to a mission risk translation matrix and the DND/CF Integrated Risk Management Guidelines will be used to determine a rating.
• Final assessment will be informed by, but not limited to the decision support tools provided. Assessors will be allowed to deviate from the mission risk translation matrix used by + or – one risk level.
• Rationale behind a final assessment that deviates from the mission risk translation matrix used will be provided to the Independent Review Panel for review.
Summing Up• We are committed to an open, fair and transparent process
– IER and methodology posted publicly on Government website
– All questions and answers during IER shared simultaneously with all companies
– Details of assessment methodology shared with companies in advance of IER response deadline
– NFPS overseeing entire assessment process
– Significant differences of opinion between assessment teams will be brought to the attention of the Panel
– The process will result in an assessment of benefits and risks associated with each aircraft against each mission. Canada will release a summary report that respects commercial sensitivities and classified information restrictions.
34
