NATO PROJECT ASSESSMENT MODEL · MMT Model Management Team PL Project Liaison PM Project Manager PMT Project Manager Team PR Peer Reviewer VIII ORIGINAL. AQAP 2050 (Edition 1) Chapter

AQAP 2050 (Edition 1)

NATO PROJECT ASSESSMENT MODEL


(September 2003)

I ORIGINAL


Page blank

II ORIGINAL


NORTH ATLANTIC TREATY ORGANIZATION

NATO STANDARDISATION AGENCY (NSA) NATO LETTER OF PROMULGATION September 2003 1. AQAP-2050(Edition 1) – NATO PROJECT ASSESSMENT MODEL is a NATO/PFP UNCLASSIFIED publication. The agreement of interested nations to use this publication is recorded in STANAG 4107. 2. AQAP-2050(Edition 1) is effective on receipt. 3. It is permissible to distribute copies of this publication to Contractors and Suppliers and such distribution is encouraged. Jan H ERIKSEN Rear Admiral, NONA Chairman NSA

III ORIGINAL


Page blank

IV ORIGINAL


Record of Changes

Change Date Date Entered Effective Date By Whom Entered

V ORIGINAL


Table of Contents Page number List of Acronyms 1 Chapter 1 - Overview 1

1.1 References 1 1.2 Objectives 1 1.3 Application 2

Chapter 2 - Assessment Model 3 Chapter 3 - Roles and Responsibilities 4

3.1 Assessment Sponsor (AS) 4 3.2 Project Manager (PM) 4 3.3 Assessment Team (AT) 4 3.4 Assessment Team Leader (ATL) 4 3.5 Project Liaison (PL) 4 3.6 Independent Peer Reviewer-Optional (PR) 4 3.7 Enterprise Customer (EC) 5 3.8 Assessment Management Board (AMB) 5 3.9 Model Management Team (MMT) 5

Chapter 4 - Assessment Process 6

4.1 Process 6 4.1.1 Activities 6 4.1.2 Tasks 7

4.2 Establish Assessment Capability 8 4.2.1 Purpose 8 4.2.2 Tasks 8 4.2.3 Responsibility 8 4.2.4 Products 8

4.3 Adopt Assessment Capability 8 4.3.1 Purpose 8 4.3.2 Tasks 9 4.3.3 Responsibility 9 4.3.4 Products 9

4.4 Initiate Assessment 9 4.4.1 Purpose 9 4.4.2 Tasks 9 4.4.3 Responsibility 9 4.4.4 Products 10

4.5 Plan Assessment 10 4.5.1 Purpose 10 4.5.2 Tasks 10 4.5.3 Responsibility 10 4.5.4 Products 10

4.6 Conduct Assessment 11 4.6.1 Purpose 11

VI ORIGINAL


4.6.2 Tasks 11 4.6.3 Responsibility 11 4.6.4 Products 11

4.7 Analyze and Report Assessment Results 11 4.7.1 Purpose 11 4.7.2 Tasks 11 4.7.3 Responsibility 12 4.7.4 Products 12

4.8 Follow-up 12 4.8.1 Purpose 12 4.8.2 Tasks 12 4.8.3 Responsibility 13 4.8.4 Products 13

4.9 Analyze and Improve 13 4.9.1 Purpose 13 4.9.2 Tasks 13 4.9.3 Responsibility 14 4.9.4 Products 14

Annex A - Assessment Issue Description A-1

VII ORIGINAL


LIST OF ACRONYMS

AMB Assessment Management Board AP Assessment Process AS Assessment Sponsor AT Assessment Team ATL Assessment Team Leader EC Enterprise Customer MMT Model Management Team PL Project Liaison PM Project Manager PMT Project Manager Team PR Peer Reviewer

VIII ORIGINAL


Chapter 1 - Overview The purpose of this publication is to describe and provide guidance on NATO Project Assessment Model (NPAM). The NPAM is a model for assessing projects to, identify specific project issues, provide recommendations, and extract cross-project systemic issues. This document describes the model, and provides guidance at high-level on its use, and feedback of results into common information repository(s). 1.1 References The material included in this standard is based on the Program Assessment Architecture developed by the U.S. Department of Defense under the Tri-Service Assessment Initiative. a. Allied Quality Assurance Publication (AQAP) 100 Ed 3, The NATO Policy on an

Integrated Systems Approach to Quality Through the Life Cycle. b. ISO10006:1997, Quality Management -- Guidelines to quality in project

management. c. Phased Armaments Programming Systems (PAPS). d. AQAP 2000 NATO Policy on an Integrated Systems Approach to Quality

Through the Life Cycle. e. ISO 19011:2002, Guidelines for quality and/or environmental management

systems auditing.

1.2 Objectives

The objectives of the NPAM are as follows:

a. Provide assistance to Project Management Teams (PMTs) in order to:

(1) Identify issues within and outside of their control; (2) Recommend areas for improvement; (3) Provide follow up assistance as needed; (4) Improve organization-wide processes.

b. Focus on project management issues throughout the system lifecycle. Noted

that NPAM is a tool to assess the project itself (not to assess any stakeholders -PM, PMT or contractor/subcontractor-).

c. Enable identification and use of common/systemic issues across projects

through:

(1) Monitoring statistical outcomes;

1

(2) Establishing benchmarks;


(3) Maintaining and disseminating lessons learned. d. Be consistent with NATO quality policy and guidance; based upon approved

International standards and best practices.

1.3 Application

The NPAM is intended for: a. NATO and national implementation for acquisition, development and/or

maintenance projects. b. Collaborative multi-national projects. c. Bi-lateral agreements with Partnership for Peace Nations.

2


Chapter 2 - Assessment Model The NPAM is comprised of an Assessment Process (AP) and a documented set of issue categories that the assessment should cover. The issue categories provide a consistent baseline by which assessment results are reported. The AP is comprised of eight activities, which are described in this document. The activities include: a. Establish Assessment and Maintain Capability. b. Adopt Assessment Capability. c. Initiate Assessment. d. Plan Assessment. e. Conduct Assessment. f. Analyze and Report Assessment Results. g. Follow up. h. Analyze and Improve. The Issue Categories provide a top-level structure for identifying project issues. All of the issue categories should, at least, be considered by the Assessment Team (AT). Those issues which are deemed to be high priority will be subject to more detailed analysis. Additional issue categories may be added to the assessment. The Issue Categories are: a. External Constraints. b. Mission Requirements. c. Financial. d. Resources. e. Management. f. Processes. g. Product. h. Schedule. i. User / Customer. j. Project Specific. It can be chosen only a few issue categories for conducting the assessment. Also, in many cases it would be convenient to conduct more than one assessments during the project life cycle.

3


Chapter 3 - Roles and Responsibilities 3.1 Assessment Sponsor (AS)

The requestor of the assessment. For a specific project, the AS could be the Project Manager (PM) or sometimes the Enterprise Customer (EC) when is related to mentioned specific project or some other authority involved in this specific project and interested in project outcomes. This person determines the scope and use of the assessment. This person also ensures that assessment results and recommendations are reviewed for quality and consistency at both the project and enterprise levels and provides feedback to the Model Management Team (MMT). This feedback includes the assessment of process performance and non-attributable (sanitized) assessment data.

3.2 Project Manager (PM)

The assigned manager for a specific project. Responsible for all aspects of the project and related decisions. The primary user of the project assessment results.

3.3 Assessment Team (AT)

This team is comprised of a Team Leader and selected participants from National and/or NATO resources. The Assessment Team is responsible for executing the core activities of the assessment process after assessment initiation. Team makeup is based on the projected assessment requirements. Team members are independent from the assessed project.

3.4 Assessment Team Leader (ATL)

Responsible for the planning and conduct of a specific assessment in accordance with the Assessment Method and related guidance. The ATL is appointed by and interfaces directly with the AS. The ATL organizes and manages the AT.

3.5 Project Liaison (PL)

Representative from the project/organization who will interface with the AT. The PL is responsible for providing background information to the team, and assistance with site visit logistics and planning.

3.6 Independent Peer Reviewer-Optional (PR)

The person or persons designated to independently review the assessment findings, results, conclusions, and recommendations for a given assessment. The PR evaluates the assessment results for compliance with the assessment process and to

4


verify that assessment conclusions and recommendations are consistent with the findings. The PR is not a member of the AT.

3.7 Enterprise Customer (EC)

A customer for enterprise level analysis of results and products related to a specific project. Examples of EC are national/international acquisition organizations. These organizations could be NATO agencies or commands, NATO/PfP Nations or MoD/Defence organizations.

3.8 Assessment Management Board (AMB)

Responsible for overall model policy, maintaining the assessment capability, and monitoring its use, effectiveness, and efficiency. The AMB is supported by the MMT. Note: It is envisioned that this function will be performed by a NATO organization.

3.9 Model Management Team (MMT)

This team is responsible for the maintenance and improvement of the NPAM as well as for developing training concept and material. Responsible for non-attributable (sanitized) data collection, systemic analysis, and reporting, and assessment process performance analysis. Note: It is envisioned that this function will be performed by a NATO organization.

5

AQAP-2050 (Edition 1)

Chapter 4 - Assessment Process 4.1 Process 4.1.1 Activities There are eight activities which comprise the assessment process: a. Establish and Maintain Assessment Capability. b. Adopt Assessment Capability. c. Initiate Assessment. d. Plan Assessment. e. Conduct Assessment. f. Analyze and Report Assessment Results. g. Follow up. h. Analyze and Improve.

Figure 1 depicts the overall NATO Assessment Process. The characteristics of the process are described below.

1. ESTABLISHand MAINTAINASSESSMENTCAPABILITY

by AssessmentManagement Board

1. ESTABLISHand MAINTAINASSESSMENTCAPABILITY

by AssessmentManagement Board

NATOASSESSMENT

METHODOLOGYAQAP-2050

NATOASSESSMENT

MODELAQAP-2050

2. ADOPTMETHODOLOGYby Enterprise Customer

2. ADOPTASSESSMENTCAPABILITY

by Enterprise Customer

Core Activities

4. PLANASSESSMENT

by Team Leader

4. PLANASSESSMENT

4. PLANASSESSMENT

by Team Leader

4. PLANASSESSMENT

5. CONDUCTASSESSMENT

by Team


by AT


by Team


by AT

6. ANALYSE& REPORTby Team Leader

6. ANALYSE& REPORT

by AT/ ATL

6. ANALYSE& REPORTby Team Leader

6. ANALYSE& REPORT

by AT/ ATL

3. INITIATEASSESSMENT

by Sponsor

3. INITIATEASSESSMENT

by Sponsor

REPOSITORY:- SANITIZED DATA- PROCESS FEEDBACKby Methodology Team

REPOSITORY:- SANITIZED DATA (controlled by PM)

- PROCESS FEEDBACKMaintain and Improve by MMT

7. FOLLOW UPACTION

by Project Manager

7. FOLLOW UPACTION

by PM

8. ANALYSE& IMPROVE

by Methodology Team

8. ANALYSE& IMPROVE

by MTT

REPORTFINAL

ASSESSMENTREPORT

UpdateThroughAssessmentManagementBoard

SYSTEMICISSUES

SYSTEMICISSUES

STAKEHOLDERSSTAKEHOLDERSSTAKEHOLDERSSTAKEHOLDERSSTAKEHOLDERSSTAKEHOLDERS

SPONSORSPONSOR

FEEDBACKFEEDBACK

SPONSOR andSPONSOR, PM and AT

PROJECT

REPORT

INITIAL ASSESSMENTREPORT

Presentationto PM - AT

by ATL

NATO PROJECT ASSESSMENT MODEL - PROCESS MODEL

6


From a wide range of analysed assessments, systemic issues contents a description of common issues arose during the assessment performance, typical problems found, key issues to deal with in a specific project, etc. Four of the assessment process activities are designated as “core” assessment activities. The “core” activities are:

a. Initiate Assessment. b. Plan Assessment. c. Conduct Assessment. d. Analyze and Report Assessment Results.

These core activities encompass the assessment tasks and interactions necessary for specific project assessments. These activities are the primary responsibility of the designated project AT.

4.1.2 Tasks

Each activity in the AP is comprised of tasks, responsibility and products. The tasks delineate the specific actions required to implement the activity. Individual tasks may be tailored to best address the characteristics of the project being assessed, the assessment time frame, and the assessment scope. The AT should document the manner in which the tasks are tailored in the assessment plan. Assessments are conducted on individual projects in accordance with the AS requirements. There are different types of assessments, generally defined in terms of the purpose and scope of the information needs of the AS and the PM. They include:

a. Assessments of project general “situation”. b. Assessments focused on specific project issues. c. Assessments focused on specific project organizations. d. Software and/or system assessments.

Assessments are focused on the success of the project. No individual assessment results are provided by the AT to AS and/or EC without the concurrence of the PM. The PM controls the release and dissemination to MMT of all project specific assessment results. The MMT is responsible for analyzing and reporting systemic issues to stakeholders (whoever involved in the AP) in order to support enterprise level management and decision-making.

Each activity that comprises the AP is individually described in the following sections (Figure 1 refers). The activities are described in terms of:

a. The purpose of the activity. b. The tasks which comprise the activity. c. Responsibilities for the implementing the activity.

7


d. Products of the activity.

4.2 Establish Assessment Capability 4.2.1 Purpose

The Establish Assessment Capability activity is to implement and maintain the NPAM. This includes providing the necessary infrastructure to maintain and improve the capability and to perform systemic analyses.

4.2.2Tasks 4.2.2.1 Implement repository and method of use; 4.2.2.2 Implement training concept and materials; 4.2.2.3 Establish a MMT; 4.2.2.4 Develop and implement model performance measurement process and an

associated database. 4.2.2.5 Maintain the AQAP-2050 document. 4.2.3 Responsibility AMB.

4.2.4 Products

List of capable assessment offices; MMT; Training materials; Systemic analysis results and associated information technology infrastructure.

4.3 Adopt Assessment Capability 4.3.1 Purpose

The Adopt Assessment Capability activity is the process by which the NPAM is adapted/translated for application. Note: This involves ensuring the technical and management infrastructure, and qualified technical resources, are available to effectively conduct assessments. Implementation of the tasks associated with this activity results in an organization capable of performing assessments within that Enterprise and improving the manner in which the assessments are conducted.

8

AQAP-2050 (Edition 1) 4.3.2 Tasks 4.3.2.1 Implement the standard assessment model. 4.3.2.2 Establish assessment team member qualifications and associated team

selection criteria. 4.3.2.3 Develop and update the participant’s qualifications, training, and experience

database. 4.3.3 Responsibility EC.

4.3.4 Products

Adopted Model documentation; Assessment tool, technique, and reference library and associated components; Model performance measurement database; Assessment education and training courses and materials; AT members capability database.

4.4 Initiate Assessment 4.4.1 Purpose

The Initiate Assessment activity is the first of the four “core” assessment activities and must be initiated only by AS. This activity includes those management and technical tasks required to initiate the AP. Follow-up assessments may be initiated based upon recommendations from a previous assessment report, or may be a result of feedback by a PM when previous assessment recommendations did not provide necessary corrective action.

4.4.2 Tasks 4.4.2.1 Establish the need for, and the agreement to conduct an assessment. This task

includes determination of the scope of the assessment to be conducted. 4.4.2.2 Appoint the ATL. 4.4.2.3 Appoint the PL (in accordance with PM's proposal). 4.4.3 Responsibility AS.

9

AQAP-2050 (Edition 1) 4.4.4 Products

Agreement to conduct an assessment; Assessment scope; Identification of the PL. 4.5 Plan Assessment 4.5.1 Purpose

The Plan Assessment activity includes those management and technical tasks required to conduct the assessment. Implementation of these tasks results in the coordination of a project AT, a consensus plan/approach for conducting a project assessment, and an initial profile of project characteristics and issues in accordance with the assessment scope.

4.5.2 Tasks 4.5.2.1 Obtain, disseminate, and review background information. This includes

characteristics and associated project documentation. 4.5.2.2 Communicate with the PM to identify concerns. From this communication,

determine the initial project issues. 4.5.2.3 Identify team participants based upon the initial project issues, project

domain/functional area, and assessment scope. Team members should be a mix of experienced assessors and subject matter experts related to project initial list of issues.

4.5.2.4 Ensure team members are trained in the NPAM. 4.5.2.5 Obtain appropriate confidentiality agreements for participants, in order to avoid

that data collected during project assessment activities can be spread out. 4.5.2.6 In cooperation with the team members, plan the project assessment and

document the approach. Planning information may include: Assessment objectives and strategy; Assigned assessment resources; Selected assessment issue categories and prioritization; Assessment schedule; Information needs/sources; Assessment products.

4.5.3 Responsibility ATL.

4.5.4 Products

Initial issues; AT; Confidentiality agreements; Assessment plan.

10

AQAP-2050 (Edition 1) 4.6 Conduct Assessment 4.6.1 Purpose

The Conduct Assessment activity includes identification of strengths, concerns, and risks of the assessed project based upon consideration of the NPAM Issue Categories (Annex A), and in-depth investigation into critical project issues. This investigation may include customer and supplier site visits (only to prime and first tier sub-suppliers), interviews, and review of objective evidence. The Conduct Assessment activity includes validating and updating the initial assessment issues.

4.6.2 Tasks 4.6.2.1 Identify and document issues in accordance with the assessment scope and the Issue Categories. Confirm and/or corroborate findings that support the identified issues. The ATL will maintain an interface with PL to discuss any changes to the assessment plan. 4.6.3 Responsibility AT. 4.6.4 Products

Documented findings; Initial observation briefings as necessary. 4.7 Analyze and Report Assessment Results 4.7.1 Purpose

The Analyze and Report Assessment Results activity includes tasks required to relate the issues that have been individually evaluated, analyze and integrate the results in terms of cause and effect, and report the results with appropriate recommendations for the project. As a result of this activity, the assessment customers (AS and PM) understands key project issues and receives actionable recommendations. In addition, sanitized project assessment results (findings arose during the assessment of a specific project and related exclusively to the NPAM application itself and the AP) should be made available for systemic analysis.

4.7.2 Tasks 4.7.2.1 Conduct in-depth analysis of key project issues. 4.7.2.2 Document findings and results to create the final assessment issue profile.

11

AQAP-2050 (Edition 1) 4.7.2.3 Evaluate and describe issues in terms of associated project objectives,

assumptions, and constraints. Address strengths, problems, risks, and lack of information related to issue findings.

4.7.2.4 Prioritize the issues with respect to project impact. 4.7.25 Analyze issues to determine cause and effect. 4.7.2.6 Generate recommendations related to the issues and causes, and determine

whether a future assessment should be recommended. 4.7.2.7 Develop a draft assessment report. It must reflect the consensus of the AT. The

report may be reviewed by a PR. The draft report is reviewed with the PM. The PM is given the opportunity to comment and/or add additional information.

4.7.2.8 Finalize the report and disseminate to the PM and the AS. 4.7.2.9 Provide feedback to the MMT about issues concerning the application of model

including required performance measures, assessment model related lessons learned, and sanitized issues from the assessment.

4.7.2.10 Solicit feedback on the assessment from the AT, the PM and AS to the MMT. 4.7.3 Responsibility ATL. 4.7.4 Products

Final Assessment Report to the PM and AS; Assessment feedback and sanitized assessment results to the MMT.

4.8 Follow-up 4.8.1 Purpose

The Follow-up activity includes those tasks required to produce and implement the action plan.

4.8.2 Tasks 4.8.2.1 The PM should use the Final Assessment Report to determine and prioritize the

actions that will be taken to address the issues. 4.8.2.2 Determine and prioritize actions that will be taken by the project and others. 4.8.2.3 Designate resources, schedules and responsibility for the actions.

12

AQAP-2050 (Edition 1) 4.8.2.4 PM must report to AS about the Project Manager's action plan. Also he will

provide to AS reports about the Action Plan fulfillment (when it would be required by AS or systematically).

4.8.2.5 Determine need for a future assessment. 4.8.2.6 Track actions to closure. 4.8.3 Responsibility PM.

4.8.4 Products Project Manager’s action plan.

4.9 Analyze and Improve 4.9.1 Purpose

The Analyze and Improve activity includes tasks required to determine systemic issues that are affecting project performance. Further, this activity includes tasks required to maintain the NPAM and evaluate its performance. This includes analysis of sanitized assessment results and process feedback across a wide range of assessments.

4.9.2 Tasks 4.9.2.1 Perform statistical analysis from sanitized assessment data to determine

systemic issues, including state of practice in compliance with current allied publications.

4.9.2.2 Determine recommended changes to this allied publication and other

appropriate actions based upon the above. 4.9.2.3 Determine the benefits of applying the NPAM based upon feedback received

from assessed PMs and Sponsor. 4.9.2.4 Make systemic issues available for use by all stakeholders. 4.9.2.5 Suggest maintenance and improvement actions for the NPAM and associated

products based upon feedback received to the Assessment Management Board.

13

AQAP-2050 (Edition 1) 4.9.2.6 Maintain and improve the supporting information technology infrastructure,

including the repository. The following is a list of sanitized assessment data that could be included in the repository: Assessments Results (project profile, assessment plan, assessment schedule, ...) and Lessons Learned (both customer and AT feedback).

4.9.3 Responsibility MMT.

4.9.4 Products

Systemic issue analysis information; Proposed updates to the NPAM and supporting materials; Proposed updates to allied publications and other actions; Updates to the information technology infrastructure, including the repository.

14

ANNEX A to AQAP-2050 (Edition 1)

ANNEX A – ASSESSMENT ISSUE DESCRIPTION

A-1


Issue Category Issue Sub - Issue 1. External Constraints 1.1 Regulatory Environment 1.1.1 Legal

1.1.2 Policy 1.1.3 Audits & Assessments

1.2 Workplace Environment 1.2.1 Cooperation 1.2.2 Morale 1.2.3 Culture

1.3 Political Environment 1.3.1 Legislative Agendas 1.3.2 Customer’s Agenda 1.3.3 Supplier’s Agenda

1.4 Other Constraints 1.4.1 Corporate 1.4.2 Industrial Scheme

2. Mission Requirements 2.1 Operational Requirements 2.1.1 Reasonableness 2.1.2 Stability 2.1.3 Dependencies 2.1.4 Change Tolerance

3. Financial 3.1 Funding 3.1.1 Sufficiency 3.1.2 Timeliness 3.1.3 Continuity/Stability 3.1.4 Flexibility

3.2 Budget 3.2.1 Allocation 3.2.2 Variance 3.2.3 Control

4. Resources 4.1 Personnel 4.1.1 Qualifications 4.1.2 Staffing 4.1.3 Availability

4.2 Facilities 4.2.1 Capital Equipment 4.2.2 Infrastructure

4.3 Tools 4.3.1 Support Tools 4.3.2 Information Systems

4.4 Government Furnished 4.4.1 Equipment 4.4.2 Information

4.5 Supplier Furnished 4.5.1 Commercial Off-The-Shelf (COTS) 4.5.2 Non-Developed Items (NDI) 4.5.3 Developed Items (DI)

4.6 Selection of Prime Contractors/Supplier

4.6.1 Integrity 4.6.2 Longevity 4.6.3 Sub-suppliers

5. Management 5.1 Acquisition Strategy 5.1.1 Acceptability 5.1.2 Feasibility 5.1.3 Suitability

5.2 Project Planning 5.2.1 Acceptability 5.2.2 Feasibility 5.2.3 Suitability

5.3 Project Management 5.3.1 Organisation 5.3.2 Suitability 5.3.3 Monitoring and Control 5.3.4 Change Management

A-2


Issue Category Issue Sub - Issue

A-3

5.4 Contracting and Subcontracting

5.4.1 Conditions/Constraints 5.4.2 Cost Accounting 5.4.3 Progress Tracking 5.4.4 Arrangements 5.4.5 Timeliness 5.4.6 Change Management

5.5 Communication 5.5.1 Interfaces 5.5.2 Openness 5.5.3 Teamwork

5.6 Quality Management 5.6.1 Quality Management System

5.7 Documentation Management

5.7.1 Availability 5.7.2 Adequacy

6. Processes 6.1 Conformance 6.1.1 Compliance 6.1.2 Process Employment 6.1.3 Process Compatibility

6.2 Capability 6.2.1 Fitness for Purpose 6.2.2 Efficiency 6.2.3 Improvement

7. Product 7.1 Requirements

7.1.1 Completeness 7.1.2 Correctness 7.1.3 Feasibility 7.1.4 Stability 7.1.5 Traceability

7.2 Design

7.2.1 Architecture 7.2.2 Scale 7.2.3 Complexity 7.2.4 Technology Effectiveness 7.2.5 Interoperability 7.2.6 Reuse

7.3 Attributes

7.3.1 Usability 7.3.2 Performance 7.3.3 Dependability 7.3.4 Reliability/Availability 7.3.5 Supportability/ Maintainability 7.3.6 Reusability 7.3.7 Portability 7.3.8 Efficiency 7.3.9 Integrity 7.3.10 Survivability 7.3.11 Safety

7.4 Technical Constraints

7.4.1 Human Factors 7.4.2 Health & Safety 7.4.3 Security 7.4.4 Natural Environment Protection

8. Schedule 8.1 Constraints

8.1.1 Feasibility 8.1.2 Dependencies 8.1.3 Contingency Plans

8.2 Progress

8.2.1 Estimation Accuracy 8.2.2 Visibility 8.2.3 Progress Performance


Issue Category Issue Sub - Issue 8.2.4 Rework

9. User/Customer 9.1 Satisfaction

9.1.1 Involvement 9.1.2 User Acceptance

9.2 Transition

9.2.1 Transition Support 9.2.2 Training

9.3 Support

9.3.1 Maintenance 9.3.2 Assistance 9.3.3 Warranty

10. Project Specific

Table 1. Issue Structure

A-4


Chapter 0 - Introduction 0.1 Issue Categories

The issue categories are used to capture the high-level concerns (risks, problems and uncertainties/lack of information) generated during the assessment. They are applicable to the whole effort being assessed, i.e., the organizational, systems and software aspects of a project or enterprise. The issues are dynamic and multi-dimensioned in nature. This means that an issue will change dependent on time (e.g., life cycle phase), stakeholder perspective (e.g., customer or supplier), priority of importance, triggering event (i.e., the event(s) that initiated the issue), and the objectives, assumptions and constraints that exist when the issue is being examined.

0.2 Issue Structure Application

The list of issues is not meant to be exhaustive, but rather a minimum set of concerns that the ATL and team members should survey to help initiate and sustain the assessment. Further, the issues are only described in general terms and are not intended to cover the specifics of every possible situation the team may find itself assessing. It is required that each issue category will be reviewed by the AT as part of the issue identification process. However, it is not expected that every issue or sub-issue will necessarily be addressed during an assessment. It is not expected that every team member will be an expert in every issue category. However, the team as a whole should be able to relate the general issue descriptions to the specific circumstances of the effort under assessment. Additionally, some issues necessarily overlap or are repeated because of certain items that require emphasis based upon a perspective or triggering event. Again, the issue categories should be used as a starting point only, and be supplemented with other domain specific questionnaires, surveys, taxonomies, personal experience, etc., as the situation requires. A “Project Specific” issue category exists to allow an AT to add new issues to the structure where appropriate. The sub-issues have a customer and supplier perspective from which they may be assessed. Consequently, the AT shall be aware that there is both a customer and supplier aspect. Many times the assessment guidance provided in the perspectives are nearly identical, but sometimes they are not. For example, the political agendas of the customer and supplier may be very different from one another, while their ability to hire personnel may be very similar. Be aware that even when a sub-issue’s

A-5


perspective appears the same, the actions or recommendations to address the issue may be very different because of the unique perspective involved. The guidance provided follows a convention where the word “evaluate” is used to examine things that are factual in nature, i.e.; they can be objectively measured. The word “contrast” is used to try to show differences, while “compare” is meant to convey similarities. The word “assess” is used to render some sort of judgment. Finally, the ATL should endeavor to look for issues beyond this list to reduce the possibility of “issue blindness”, i.e., seeing only what is on the list of issues.

A-6


Chapter 1 - External Constraints Scope: This issue category concerns the context in which the project or enterprise operates, the boundaries that constrain it, and any externalities that may influence it. 1.1 Regulatory Environment

Scope: This issue concerns the regulatory constraints under which the project or enterprise operates.

1.1.1 Legal

Scope: The effort’s compliance with international, national and local laws. Customer Perspective: Examine general legal constraints, impacts of acquisition regulations, and potential changes in legislation. Evaluate the effort’s compliance with appropriate legal requirements. Assess whether the level of compliance is acceptable. Supplier Perspective: Examine legal requirements imposed on the effort. Evaluate copyrights, patents, trade secrets or intellectual property rights. Assess the impact of non-compliance with or changes to the legal requirements.

1.1.2 Policy

Scope: The current governmental, NATO policies, and industry standards that influence the effort’s acquisition, development, operational or sustaining engineering. Customer Perspective: Examine policy constraints, policy that may be changing, and/or policies that mandate the use of specific technology solutions. Evaluate these policies for consistency. Assess whether these policies have any implications on the effort. Supplier Perspective: Examine internal corporate policies (e.g., purchasing, finance, personnel administration, labor, at contract “at risk” policies) as well as governmental policies. Evaluate the constraints these policies place on the effort. Assess the impact of changes in policy.

1.1.3 Audits & Assessments

Scope: The compliance with previous external and internal reviews/audits/ assessments performed on the effort.

A-7

Customer Perspective: Examine previous reviews/audits/assessments performed on the assessed project and those performed on projects of a similar nature. Evaluate the recommendations and their implementations, which recommendations were not


followed, and the rationale for not following the recommendations. Assess the adequacy of the previous reviews/audit/assessments and their recommendations.

Supplier Perspective: Examine previous reviews/audits/assessments. Evaluate the recommendations and their implementations, which recommendations were not followed, and the rational for not following recommendations. Assess the validity and/or impacts of not following previous recommendations (e.g., the likelihood of passing the next major decision milestone).

1.2 Workplace Environment

Scope: This issue concerns the general situation existing within the stakeholders’ workplace.

1.2.1 Cooperation Scope: The level of teamwork existing among the effort’s stakeholders.

Customer Perspective: Examine the work agreements in place, the teamwork that exists, including cooperation within internal groups. Evaluate the willingness to compromise, the number and types of confrontations that occur (e.g., between customer and sponsor, customer and supplier), the conflict resolution process, and cooperation received from supplier as well as projects/programs with which the effort must interface. Assess whether the level of cooperation is acceptable. Supplier Perspective: Examine the work agreements in place and the level of teamwork that exists. Evaluate the acceptance of reasonable direction and instructions from customer or other parties, the degree of commitment to effort by the stakeholders, the general openness towards resolving problems and risks. Evaluate the cooperation that exists within the organization (e.g., with corporate headquarters) as well as among subcontractors/vendors. Assess the impact of changes in the level of cooperation on the effort.

1.2.2 Morale

Scope: The degree of enthusiasm and commitment that exists among the effort’s stakeholders and their personnel. Customer Perspective: Examine the morale of the effort’s stakeholders. Evaluate the personnel turnover rate, personnel satisfaction, leadership visibility, the number of crisis management meetings, the number of changes in schedule. Assess whether the morale is acceptable.

A-8


Supplier Perspective: Examine the morale of the effort’s stakeholders. Evaluate the personnel turnover rate personnel satisfaction, the current reputations of the stakeholders, the productivity levels, the promptness of remedial actions. Assess the impacts of a change in morale on the effort.

1.2.3 Culture

Scope: The unwritten rules of organizational behavior that exist within the effort’s stakeholders that may influence the success of the effort. Customer Perspective: Examine the organizational culture (customer and supplier). Evaluate their organizational assumptions, how decisions are made, unwritten codes of behavior, and where power resides. Assess whether the varied organizational cultures support the effort or whether culture incompatibilities will ensue. Supplier Perspective: Examine the organizational cultures (customer, supplier, vendors, and subcontractors). Evaluate their codes of conduct, vision statements, what is socially valued, internal paths leading to success. Assess the impacts of changes or incompatibilities in culture.

1.3 Political Environment

Scope: This issue concerns the general situation existing within the overall political environment.

1.3.1 Legislative Agendas

Scope: Current and pending legislation, a sense of Governmental or other legislative bodies or institutions that may influence the effort’s success. Customer Perspective: Examine current and pending legislation. Evaluate the potential impacts of pending/proposed legislation, related support (e.g., lobbying groups, newspapers, think tanks) on the effort. Assess whether the legislative agendas support the effort. Supplier Perspective: Examine the potential impacts of pending/proposed legislation; and supporting organizations. Assess the impacts of changes in legislative agendas on the effort.

1.3.2 Customer’s Agenda

Scope: The political objectives of effort’s customer and/or sponsor. Customer Perspective: Examine political objectives of the customer, the customer’s sponsor, support contractors and consultants. Compare and contrast their respective political objectives. Evaluate the actions of the sponsors to safeguard the effort and

A-9


their willingness to champion the effort. Assess whether the agendas support the customer’s agenda for the effort. Supplier Perspective: Examine the customer’s agenda. Evaluate impact of the customer’s (or supporting contractors) political goals conflicting with the technical goals of the effort. Assess the impact of changes in the customer’s agendas.

1.3.3 Supplier’s Agenda

Scope: The political (market and economic) objectives of the effort’s supplier and other team members. Customer Perspective: Examine the supplier’s (and its subcontractors) agenda. Evaluate the influence of the supplier on international or political environment, the economic strength of supplier, the current competitive environment, the supplier’s long and short-term market/financial goals. Assess whether the supplier’s agenda support the customer’s agenda. Supplier Perspective: Examine all the stakeholders’ agendas. Compare and contrast the political and economic goals of supplier and team members against technical/customer’s goals for project. Evaluate the actions of the supplier to safeguard the effort and champion the effort (e.g., risk sharing). Assess the impacts of agenda conflict or changes in anyone’s agenda.

1.4 Other Constraints

Scope: This issue concerns other external constraints that impact the project. 1.4.1 Corporate

Scope: The current company-wide policy procedures and processes regulated by the own company for its function that influence the effort’s acquisition. Customer Perspective: Examine company policy constraints, procedures and best practices to be applied to the effort, policy that may be changing, and/or policies that establishes or propose the use of specific technology solutions. Evaluate these policies, procedures and best practices for consistency. Assess the implications of such policies, procedures and best practices on the effort. Supplier Perspective: Examine company policy constraints, procedures and best practices to be applied to the effort, policy that may be changing, and/or policies that establishes or propose the use of specific technology solutions. Evaluate these policies, procedures and best practices for consistency. Assess the implications of such policies, procedures and best practices on the effort.

A-10

ANNEX A to AQAP-2050 (Edition 1) 1.4.2 Industrial Scheme

Scope: This issue concerns the industrial scheme (e.g. joint venture, international Competitive Biding, Partnership, …) under which the project will be executed and its impact on the assessed project. Customer Perspective: Examine the scheme to see how management is shared among the participant companies, and evaluate associated rules and procedures. Evaluate the rules established to regulate their participation, responsibilities and distribution of tasks and charges to meet contract requirements. Assess the responsibilities determined for each participant and their contribution to the effort. Supplier Perspective: Examine the scheme to see how management is shared among the participant companies, and evaluate associated rules and procedures. Consider the marketplace, legal and data sharing environments. Assess the responsibilities, budget and financial distribution of each participant with respect to their contribution to the effort.

A-11


Chapter 2 - Mission Requirements Scope: This issue category concerns how well a mission requirement can be supported or the validity of a mission-related capability against a threat. 2.1 Operational Requirements

Scope: This issue concerns the forces that can influence the mission requirement.

2.1.1 Reasonableness

Scope: The operational requirements and the effort’s technical approach. Customer Perspective: Examine the operational requirements. Assess the reasonableness of the requirements. Assess whether the implementation is reasonable given the proposed technology and acquisition, development and/or support approach. Supplier Perspective: Examine the operational requirements. Assess the reasonableness of the requirements. Assess whether the implementation is reasonable given the proposed technology and acquisition, development and/or support approach. Assess the impact of operational requirement changes to the reasonableness of the effort.

2.1.2 Stability

Scope: The stability of the operational requirements. Customer Perspective: Examine the threat environment, operational doctrine, legislation, and/or operational & test feedback. Evaluate each for changes (e.g., threat projections, pending doctrinal changes, interoperability requirements, legislative proposals) that might modify the mission requirements. Assess the impact of operational requirements change/volatility on the effort. Supplier Perspective: Examine the threat environment, operational doctrine, legislation, and/or operational & test feedback. Evaluate each for change (e.g., pending doctrinal changes, interoperability requirements, legislative proposals) that might modify the mission requirements. Assess the impact of operational requirements change/volatility on the effort’s functionality, cost and schedule.

2.1.3 Dependencies Scope: The impact of external dependencies on the operational requirements.

A-12

Customer Perspective: Examine operational requirements. Evaluate their dependency on the existence of other mission requirements (e.g., those provided by


other platforms). Assess the impact of changes in mission requirements on the effort and/or by the effort on other missions.

Supplier Perspective: Examine operational requirements. Evaluate their dependency on the existence of other mission requirements (e.g., those provided by other platforms). Assess the criticality of this dependency on the capability to support the effort’s mission.

2.1.4 Change Tolerance

Scope: The capability of the effort to manage the impacts of changes to the mission requirements. Customer Perspective: Examine the risk management, change management, requirements management, and rework procedures. Assess their robustness. Assess whether they can smoothly handle a change in mission requirements. Supplier Perspective: Examine the risk management, change management, requirements management, and rework procedures. Assess their robustness. Assess whether they can smoothly handle a change in mission requirements.

A-13


Chapter 3 – Financial Scope: This issue category concerns the process of monetary control being exercised over the effort. 3.1 Funding Scope: This issue concerns the process involved in funding the effort. 3.1.1 Sufficiency

Scope: The amount of funding available for the effort and the support for continued funding of the effort. Customer Perspective: Examine the funding profile and expenditure rates. Evaluate the adequacy of the funding amounts against the effort’s objectives. Assess the continued funding support of effort by sponsors. Assess whether the funding and support are acceptable. Supplier Perspective: Examine the funding received. Evaluate its adequacy against contractual requirements and obligations. Assess any impacts of funding slow-down and/or reductions. Assess the impacts of changes in the support of effort by customer and internal sponsors.

3.1.2 Timeliness

Scope: The funding profile and timeline. Customer Perspective: Examine the funding profile and timeline. Compare and contrast the funding timeline against contractual requirements and expenditures. Evaluate whether the funding profile maps to when the resources are required, factoring in lead times. Assess the adequacy of the timeline’s fit against the requirements and expenditures. Supplier Perspective: Examine the funding profile and timeline. Compare and contrast the funding timeline against contractual requirements and expenditures. Assess the impacts of changes in the funding timeline on current contractual requirements and liabilities (e.g., cash flow requirements).

3.1.3 Continuity/Stability

Scope: The funding profile and timeline.

A-14

Customer Perspective: Examine the funding profile and timeline. Evaluate the continuity and stability of the funding profile. Assess whether the continuity/stability of the funding is acceptable.


Supplier Perspective: Examine the funding profile and timeline. Evaluate the continuity/steadiness of the funding profile. Assess the impacts of changes in continuity/stability (e.g., a sudden reprogramming of the funding) against effort’s cost, schedule, and technical objectives.

3.1.4 Flexibility

Scope: The funding profile, timeline, funding process and the type (i.e., “color”) of funding available. Customer Perspective: Examine the funding profile, timeline, funding process and funding type. Evaluate the capability to reallocate or transfer funding. Assess whether there is sufficient funding flexibility. Supplier Perspective: Examine the funding profile, timeline and process. Evaluate the capability to absorb changes to funding. Assess the impacts of changes in funding profile on the effort.

3.2 Budget

Scope: This issue concerns the process of budgetary control exercised over the effort.

3.2.1 Allocation

Scope: The budgetary process and disbursements. Customer Perspective: Examine the budgetary process and disbursements. Evaluate the appropriateness of the funding allocation (e.g., the allocation of funding to subcontractors) as well as the value for money spent. Contrast the allocation against current and proposed work. Assess whether the allocation is acceptable. Supplier Perspective: Examine the budgetary process and disbursements. Compare and contrast the current and proposed work against budget allocations and expenditures. Assess mix and balance of the allocations against the work performed. Assess the impact of changes to the allocation mix (e.g., a funding shortfall).

3.2.2 Variance

Scope: The budgeting process and expenditures. Customer Perspective: Examine the budgeting process and expenditures. Evaluate the accuracy and precision of the budgetary expenditures. Contrast the funding against expenditures. Assess the impact of any variance in estimated versus actual budget. Assess whether the accuracy of the budget is acceptable.

A-15


Supplier Perspective: Examine the budgeting process and expenditures. Contrast the funding against expenditures. Assess the impact of any variance in estimated versus actual budget. Assess the impact of changes in the budgetary information.

3.2.3 Control

Scope: The budgetary process and expenditures. Customer Perspective: Examine the budgetary control process. Evaluate how funds are spent, monitored and reported (e.g., comparing budgeting allocated funds and expenditures profile). Assess whether the financial controls are acceptable. Supplier Perspective: Examine the budgetary control process. Evaluate how funds are spent, monitored and reported. Assess the impacts of changes to the financial control process.

A-16


Chapter 4 – Resources Scope: This issue category concerns the assets needed and available to meet the effort’s objectives, including but not limited to personnel, facilities, training, etc. 4.1 Personnel

Scope: This issue concerns the quality of the expertise available to the effort.

4.1.1 Qualifications

Scope: The relevant knowledge, experience and training of the personnel assigned to the effort. Customer Perspective: Examine the expertise of the personnel (customer and supplier) assigned to the effort. Contrast the relevance of the expertise, domain knowledge, and training against effort’s needs. Assess the adequacy of the expertise. Supplier Perspective: Examine the expertise of the supplier personnel assigned to the effort. Contrast the relevance of the expertise, domain knowledge, and training against effort’s needs. Assess the impacts of any change in the level of expertise.

4.1.2 Staffing

Scope: The staffing profile. Customer Perspective: Examine staffing profile. Evaluate the customer and supplier staffing. Compare and contrast the staffing adequacy, mix, turnover, apportionment balance, and timeliness of staffing against current and proposed work. Assess the adequacy of the staffing profile. Supplier Perspective: Examine staffing profile. Compare and contrast the staffing adequacy, mix, turnover, apportionment balance, and timeliness of staffing against current and proposed work. Assess whether the personnel bid for are assigned to the effort. Assess any changes in the staffing profile.

4.1.3 Availability

Scope: The current economic/organizational situation in regard to personnel supply and demand. Customer Perspective: Examine the competitive marketplace as well as constraints on personnel availability (e.g., personnel ceilings, promotion ceilings, internal competition for qualified personnel, etc.). Evaluate the availability of qualified

A-17


personnel to support the effort (customer and supplier). Assess whether there exists an ability to hire and retain qualified personnel. Supplier Perspective: Examine the competitive marketplace. Evaluate the availability of qualified personnel to support the effort (e.g., experienced Ada programmers). Assess the ability to hire and retain qualified personnel. Assess the circumstances that support or negatively impact the ability to put together a qualified team of people. Assess the impacts of changes in the marketplace.

4.2 Facilities

Scope: This issue concerns the organizational technology, e.g., plant and machinery, required for supporting the effort.

4.2.1 Capital Equipment

Scope: The profile of the capital equipment required in supporting the effort. Customer Perspective: Examine the capital equipment provided. Contrast the equipment against the effort’s requirements. Assess whether the capital equipment is sufficient. Supplier Perspective: Examine the capital equipment provided. Contrast the equipment against the effort’s requirements. Assess the impacts of any changes in capital requirements.

4.2.2 Infrastructure

Scope: The infrastructure required for supporting the effort. Customer Perspective: Examine the infrastructure provided (e.g., office space, facilities, ergonomic, and safety aspects, etc.). Contrast the infrastructure against the effort’s requirements. Assess whether the infrastructure is adequate. Supplier Perspective: Examine the infrastructure provided. Contrast the infrastructure against the effort’s requirements. Assess the impacts of changes in infrastructure.

4.3 Tools

Scope: This issue concerns the range of supporting tools, techniques, etc., required for supporting the effort.

A-18

ANNEX A to AQAP-2050 (Edition 1) 4.3.1 Support Tools

Scope: The range of tools required for acquiring, developing or maintaining the effort. Customer Perspective: Examine the support tools provided. Contrast the tools functional match against the effort’s requirements. Assess whether the tools provided are sufficient in number and quality. Supplier Perspective: Examine the support tools provided. Contrast the tools against the effort’s requirements. Assess the level of trained personnel on the tools. Evaluate if the tools provided are sufficient in number and quality. Assess the impacts of any changes in development tools.

4.3.2 Information Systems

Scope: The information systems required in supporting the effort. Customer Perspective: Examine the information systems provided that support the effort (e.g., MIS, financial accounting, communications/ information capture including web-based capabilities). Assess their adequacy against both the customer and supplier’s requirements to manage the effort. Supplier Perspective: Examine the information systems provided that support the effort (e.g., MIS, financial accounting, communications including web-based capabilities). Evaluate their adequacy against the effort’s needs. Assess the impacts of information system changes.

4.4 Government Furnished

Scope: This issue concerns the items available from the government that are being used (or planned for use) in the effort.

4.4.1 Equipment

Scope: The Government Furnished Equipment (GFE) required to support the effort. Customer Perspective: Examine the GFE provided. Compare and contrast GFE against the effort’s requirements. Evaluate its availability, reliability, maintainability, supportability, etc. Assess whether the GFE used is adequate. Supplier Perspective: Examine the GFE provided. Compare and contrast the GFE against the effort’s requirements. Evaluate its availability, reliability, maintainability, supportability, etc. Assess the impacts of any changes in GFE.

A-19

ANNEX A to AQAP-2050 (Edition 1) 4.4.2 Information

Scope: The Government Furnished Information (GFI) required to support the effort. Customer Perspective: Examine the GFI provided. Compare and contrast the GFI against the effort’s requirements. Evaluate its availability, accuracy, reliability, etc. Assess whether the GFI is adequate. Supplier Perspective: Examine the GFI provided. Compare and contrast the GFI against the effort’s requirements. Evaluate its availability, accuracy, reliability, etc. Assess the impact of changes in GFI.

4.5 Supplier Furnished

Scope: This issue concerns the items available from the supplier that are being used (or planned for use) in the effort.

4.5.1 Commercial Off-The-Shelf (COTS)

Scope: The commercial off-the-shelf equipment/software required in support of the effort. Customer Perspective: Examine the COTS being used. Compare and contrast the COTS against the effort’s requirements. Evaluate its availability, reliability, supportability, etc. and any data rights issues. Assess its position in the marketplace and evaluate its longevity. Assess the adequacy of the COTS used. Supplier Perspective: Examine the COTS being used. Contrast the COTS against the effort’s requirements. Evaluate its availability, reliability, supportability, etc., and any data rights issues. Assess its position in the marketplace and evaluate its longevity. Assess the impacts of any changes in COTS use.

4.5.2 Non-Developed Items (NDI)

Scope: The NDI (excluding COTS) required in support of the effort. Customer Perspective: Examine NDI the being used (e.g., a unique, once developed software/hardware item that is not available on the open market). Compare and contrast the NDI against the effort’s requirements. Evaluate its availability, reliability, supportability, etc., as well as any data rights issues. Assess the adequacy of the NDI used. Supplier Perspective: Examine the NDI being used. Contrast the NDI against the effort’s requirements. Evaluate its availability, reliability, supportability, etc., as well as data rights issues. Assess the impacts of any changes in NDI use.

A-20

ANNEX A to AQAP-2050 (Edition 1) 4.5.3 Developed Items (DI)

Scope: The DI required in support of the effort. Customer Perspective: Examine the DI being used. Compare and contrast the DI against the effort’s requirements. Evaluate its availability, reliability, supportability, etc. Assess the adequacy of the DI used. Supplier Perspective: Examine the DI being used. Contrast the DI against the effort’s requirements. Evaluate its availability, reliability, supportability, etc. Assess the impacts of any changes in DI use.

4.6 Selection of Prime Contractor/Supplier

Scope: This issue concerns how the prime contractor/supplier influences the effort. 4.6.1 Integrity

Scope: The prime contractor/supplier being used on the effort. Customer Perspective: Examine the prime contractor/supplier. Evaluate its history, economic viability, reputation and past performance. Assess the likelihood of meeting its commitments. Assess whether its integrity is acceptable. Supplier Perspective: Examine the prime contractor/supplier. Evaluate its history, economic viability and reputation and past performance. Assess the likelihood of meeting its commitments and expectations. Assess the impact of changes in the prime contractor/supplier meeting its commitments/ expectations.

4.6.2 Longevity

Scope: The prime contractor/supplier being used on the effort. Customer Perspective: Examine the prime contractor/supplier. Evaluate its history, economic viability and past performance. Assess its likelihood of staying in business. Assess whether its longevity potential is acceptable. Supplier Perspective: Examine the prime contractor/supplier. Evaluate its history, economic viability and performance. Assess its likelihood of staying in business. Assess the impact of changes in the prime contractor/supplier.

A-21

ANNEX A to AQAP-2050 (Edition 1) 4.6.3 Sub-suppliers

Scope: This issue concerns how sub-suppliers influence the effort.

Customer Perspective: Examine how the sub-suppliers are selected, controlled, and whether they are satisfactory. This includes their history, economic viability, reputation and past performance, likelihood of meeting their commitments, their integrity, and longevity. Examine the ability of the customer to influence the sub-suppliers through the prime contractor. Assess the effort’s dependency on these sub-suppliers. Supplier Perspective: Examine how the supplier ensures appropriate selection, control and performance of the sub-suppliers. This includes their history, economic viability, reputation and past performance, likelihood of meeting their commitments, their integrity, and longevity. Evaluate the supplier’s dependency on these sub-suppliers.

A-22


Chapter 5 – Management Scope: This issue category concerns the capability to plan, resource, control and monitor the effort. 5.1 Acquisition Strategy

Scope: This issue concerns the effectiveness of the effort’s acquisition strategy and the process for implementing and sustaining the strategy.

5.1.1 Acceptability

Scope: The acquisition strategy. Customer Perspective: Examine the acquisition strategy. Evaluate the strategy against governmental policies and laws and Industrial scheme. Assess the strategy against sponsor policy. Assess the strategy’s strengths and weaknesses against the effort’s objectives. Assess whether the acquisition strategy is acceptable. Supplier Perspective: Examine the acquisition strategy. Contrast the strategy against successfully completing the effort. Assess the impacts of changes to the acquisition strategy.

5.1.2 Feasibility

Scope: The acquisition strategy, policy, process and resources; Contractual management; Project performance management; Contract tracking; Management of risk. Customer Examine: The acquisition strategy, policy, and process. Evaluate the strategy for its ease of implementation and likelihood of success. Contrast the strengths and weaknesses of their implementation. Assess whether the acquisition strategy and process are acceptable. Supplier Perspective: Examine the acquisition strategy, policy, and process. Assess the impact of the strategy on the day-to-day operations of the effort. Evaluate the strengths and weaknesses of their implementation. Assess the impact of changes to the acquisition strategy and process.

5.1.3 Suitability

Scope: The effort’s management team; The acquisition strategy, policy, process and resources; Contractual management; Project performance management; Contract tracking; Management of risk.

A-23


Customer Perspective: Examine the acquisition strategy, policy, and process. Evaluate the capability of the management team (i.e., the customer) to implement them. Assess whether the management team is adequate. Supplier Perspective: Examine the acquisition strategy, policy, and process. Evaluate the capability of the management team (i.e., the supplier and subcontractors) to implement them. Assess the implications of this capability on the success of the effort. Assess the impacts of change to the management team.

5.2 Project Planning

Scope: This issue concerns the effectiveness of the effort’s planning approach.

5.2.1 Acceptability

Scope: The political and organizational aspects of the project plan and planning process. Customer Perspective: Examine the project plan and planning process. Evaluate the plan against mission requirements and the customer’s agendas. Contrast the plan’s strengths and weaknesses. Assess the thoroughness of the planning process. Assess whether the project plan meets technical and political objectives. Supplier Perspective: Examine the project plan and planning process. Evaluate the plan against mission requirements. Contrast the strategy’s strengths and weaknesses. Assess the thoroughness of the planning process. Assess the impacts of changes made to the project plan.

5.2.2 Feasibility

Scope: The technical aspects of the project plan. Customer Perspective: Examine the project plan. Evaluate the plan for its technical feasibility. Assess whether the project plan is technically possible. Supplier Perspective: Examine the project plan. Evaluate the plan for its technical feasibility. Assess the impact of changes to the technical aspects of the plan.

5.2.3 Suitability

Scope: The effort’s management team and the management of the project plan and planning process. Customer Perspective: Examine the project plan and planning process. Evaluate the capability of the management team (i.e., the customer’s) to implement and control them Assess whether the management team is capable.

A-24


Supplier Perspective: Examine the project plan and planning process. Evaluate the capability of the management team (i.e., the supplier’s and subcontractors’) to implement and control them. Assess the implications of this capability on the success of the effort. Assess the impact of changes to the management team.

5.3 Project Management

Scope: This issue concerns the instantiation and performance of the project management.

5.3.1 Organization

Scope: The effort’s management organizational structure. Customer Perspective: Examine the project management teams (i.e., customer’s and supplier’s) organizational structure. Contrast the strengths and weaknesses of the structure. Assess whether the organizational structure is acceptable. Supplier Perspective: Examine the project management teams (i.e., supplier’s and subcontractors’) organizational structure. Contrast the strengths and weaknesses of the structure. Assess the impact of changes to the management structure.

5.3.2 Suitability

Scope: The project management teams. Customer Perspective: Examine the project management teams (i.e., the customer’s and supplier’s). Evaluate the capability of the management teams to successfully implement the effort. Assess whether the management teams are capable. Supplier Perspective: Examine the project management teams (i.e., supplier and. subcontractors). Evaluate the capability of the management team to successfully implement the effort. Assess the impact of changing the management team.

5.2.4 Monitoring and Control

Scope: The process and practices used to oversee, track and evaluate progress and activities of the project. Customer Perspective: Evaluate the capability of the project management teams (customer and supplier) to track progress, oversee the effort, and make decisions based upon information collected. Examine the measurement processes, risk management programs, and their impact on the decision-making process. Assess their performance and accuracy.

A-25


Supplier Perspective: Evaluate the capability of the project management teams (supplier and sub-supplier) to track progress, oversee the effort, make decisions and report information. Examine the measurement processes, risk management programs, and their impact on the decision-making process. Assess their performance and accuracy.

5.3.4 Change Management

Scope: The ability of the effort to manage changes in the day-to-day working environment. Customer Perspective: Examine the risk, problem and change management procedures in place. Evaluate their comprehensiveness. Assess the procedures’ capability to handle both routine and unplanned changes. Assess whether the effort’s capability to manage change is acceptable. Supplier Perspective: Examine the risk, problem, change, requirements, rework and configuration management procedures. Evaluate their comprehensiveness. Assess procedures’ capability to handle both routine and unplanned changes.

5.4 Contracting and Subcontracting

Scope: This issue concerns how the contractual process influences the effort.

5.4.1 Conditions/Constraints

Scope: The contractual constraints placed on the effort determined by the general terms and conditions. Items such as contract type (e.g., payment terms and contractual requirements), restrictions (e.g., conditions or directives imposed) and dependencies (e.g., use of products outside of direct control) should be considered. Customer Perspective: Examine the effort’s contracts. Evaluate the responsibilities of the customer, divisibility of the contract, any substantial performance requirements, contract discharge requirements, and modification procedures. Assess whether the contractual requirements are adequate and necessary. Supplier Perspective: Examine the effort’s contracts. Evaluate the incentive structure, reporting of progress, rights in technical data and software, key personnel requirements, warranties, and any unusual restrictions. Assess the impact of the contract or changes to the contract on meeting the mission requirements.

A-26

ANNEX A to AQAP-2050 (Edition 1) 5.4.2 Cost Accounting

Scope: The effort’s cost monitoring and control system, e.g., the WBS.

Customer Perspective: Examine the effort’s cost monitoring and control system. Evaluate the completeness, timeliness and accuracy of the system. Evaluate the interoperability of cost data among stakeholders. Assess whether the cost accounting system is acceptable. Supplier Perspective: Examine the effort’s cost monitoring and control system. Evaluate the completeness, timeliness and accuracy of the system. Evaluate the interoperability of cost data among supplier and subcontractors. Assess the impacts of change to the cost accounting system.

5.4.3 Progress Tracking

Scope: The contracting process. Customer Perspective: Examine the contracting process. Evaluate the process and mechanisms for tracking the progress of the contract. Assess whether the tracking process is acceptable. Supplier Perspective: Examine the contracting process. Evaluate the process and mechanisms for tracking the progress of the contract. Assess the impacts of changing the contract tracking process.

5.4.4 Arrangements

Scope: The contract team. Customer Perspective: Examine the effort’s supplier team. Evaluate the contractual arrangements (e.g. Joint Venture, Consortium, etc.) between the customer and the supplier, and the suppliers themselves. Assess whether the contract arrangements are acceptable. Supplier Perspective: Examine the effort’s supplier team. Evaluate the contractual arrangements. Contrast the contracted mix and balance of the team’s expertise against the effort’s requirements (e.g., who is responsible for specific technical expertise). Assess the impacts of changing the team’s mix.

5.4.5 Timeliness

Scope: The contracting process. Customer Perspective: Examine the contractual process. Evaluate how timely the contract conditions are negotiated. Assess whether the time taken is acceptable.

A-27


Supplier Perspective: Examine the contractual process. Evaluate how timely the contract conditions are negotiated. Assess how much “at risk” work can be sustained. Assess the impact of changes in negotiation time.

5.4.6Change Management

Scope: The contracting process. Customer Perspective: Examine the contracting process. Evaluate how easily contract conditions contracts can be modified and under what conditions. Assess whether the capability to modify the contract is acceptable. Supplier Perspective: Examine the contracting process. Evaluate how easily contract conditions contracts can be modified and under what conditions. Assess the flexibility of the contract. Assess the impacts of modifying the contract.

5.5 Communication

Scope: This issue concerns the communication environment that exists; the level of openness; the trust among the effort’s stakeholders; how communication among the stakeholders influences the effort.

5.5.1 Interfaces

Scope: The internal and external communication interfaces. Customer Perspective: Examine the communication interfaces among the stakeholders and other relevant programs/projects (e.g., projects that the effort must interface with). Examine the channels established for communicating with supplier and possible sources of noise. Evaluate the complexity, efficiency and timeliness of the communication process. Evaluate whether the communication process allows passing of correct and clear information without any misunderstanding. Assess whether the communication interfaces are acceptable. Supplier Perspective: Examine the communication interfaces among the stakeholders. Examine the channels established for communicating with customer and possible sources of noise. Evaluate the complexity, efficiency and timeliness of the communication process. Evaluate whether the communication process allows passing of correct and clear information without any misunderstanding. Assess whether the communication interfaces are acceptable. Assess the impact of changing the communication interfaces.

A-28

ANNEX A to AQAP-2050 (Edition 1) 5.5.2 Openness

Scope: The communication process among the effort’s stakeholders.

Customer Perspective: Examine the communication process that exists within the customer’s organization, to and within the supplier, and any other programs/projects the effort must communicate with. Evaluate the degree to which information is or can be shared openly. Assess the impediments and disincentives to openly sharing information. Assess whether the openness of communications is sufficient.

Supplier Perspective: Examine the communication process among the stakeholders, within internal teams (e.g., IPTs), and within individual supplier organizations. Evaluate the degree to which information is or can be shared openly. Assess the impediments and disincentives to openly sharing information. Assess the impact of changing the openness of communications.

5.5.3 Teamwork

Scope: The ability of the personnel assigned to the effort to work together as a team. Customer Perspective: Examine the teaming arrangements and work groups (e.g., IPTs). Evaluate how well the teams trustfully cooperate and coordinate activities. Assess the impact of any conflicts. Assess whether the level of teamwork is acceptable. Supplier Perspective: Examine the teaming arrangements and work groups (e.g., IPTs). Evaluate how well the teams cooperate and coordinate activities. Assess the impacts of changes in teamwork.

5.6 Quality Management

Scope: This issue concerns the quality system in place, its effectiveness and its improvement.

5.6.1 Quality Management System

Scope: The implementation of the documented quality system at organizational and project level. Customer Perspective: Examine the existence and implementation of a quality system. Evaluate any measures taken to monitor its effectiveness and inspect any internal or external assessment results. Also evaluate the implemented measures for continuous improvement and the compliance of the quality system with any national, international, NATO or “de facto” standards. Assess whether the quality management system supports achievement of project objectives.

A-29


Supplier Perspective: Examine whether a quality system is documented, in use, and methodically followed up at organizational and project level. Evaluate that measures are taken to ensure its effectiveness and continuous improvement. Evaluate pursuit of compliance of the quality system with any national, international, NATO or “de facto” standards. Assess whether the quality management system supports achievement of project objectives.

5.7 Documentation Management

Scope: This issue concerns the documentation used or produced to support the project’s activities and to describe the product.

5.7.1 Availability

Scope: The documentation used or produced to support project’s activities and to describe the product. Customer Perspective: Examine the organizational, project and product documentation. Assess whether the relevant documentation is available to the customer at the required time and place. Supplier Perspective: Examine the organizational, project and product documentation. Assess whether the relevant documentation is available to the project team at the required time and place.

5.7.2 Adequacy

Scope: The documentation used or produced to support project’s activities and to describe the product. Customer Perspective: Examine the organizational, project and product documentation. Evaluate whether the process and project documentation are controlled and supports the customer related activities. Evaluate whether the product documentation is controlled and describes the customer’s view of the product. Assess whether the documentation is adequate. Supplier Perspective: Examine the organizational, project and product documentation. Evaluate whether the process and project documentation are controlled and supports the project team. Evaluate whether the product documentation is controlled and describes the product. Assess whether the documentation is adequate.

A-30


Chapter 6 – Processes Scope: This issue category is concerned with the measures taken to ensure that adequate processes are engineered, their capability assessed, and improvement measures taken by the organization. 6.1 Conformance

Scope: This issue concerns the conformance of the processes employed in the effort with any recognized standards.

6.1.1 Compliance

Scope: The organizational, supporting and product/service realization processes employed in the effort. Customer Perspective: Examine the compliance of the processes used on the effort against chosen standards or models. Evaluate measures taken to assess performance and to maintain and improve the compliance ratings. Assess whether level of compliance is acceptable. Supplier Perspective: Examine the compliance of the processes used on the effort against chosen standards or models. Evaluate measures taken to assess performance and to maintain and improve the compliance ratings. Assess the impact of changes in compliance. Assess whether level of compliance is acceptable.

6.1.2 Process Employment

Scope: The processes employed on the effort. Customer Perspective: Examine the usage of the processes on the effort. Evaluate whether the processes are followed in practice. Assess whether level of employment is adequate. Supplier Perspective: Examine the usage of the processes on the effort. Evaluate whether the processes are followed in practice. Assess whether level of employment is adequate. Assess the impact of deviations from the process.

6.1.3 Process Compatibility

Scope: The processes employed across the effort. Customer Perspective: Examine the processes of all the stakeholders on the effort. Evaluate their consistency and any impact on the project. (e.g., are defects or lines of code tracked in the same manner). Assess whether any incompatibility is acceptable.

A-31


Supplier Perspective: Examine the processes of all the stakeholders on the effort. Evaluate their consistency and any impact on the project. (e.g., are defects or lines of code tracked in the same manner). Assess whether any incompatibility is acceptable.

6.2 Capability

Scope: This issue concerns the overall effectiveness of employed processes.

6.2.1 Fitness for Purpose

Scope: The processes, steps, methods and supporting tools employed on the effort. Customer Perspective: Examine the utility of the processes used on the effort. Evaluate how well suited and/or capable the processes are for supporting the effort. Assess whether the processes used are acceptable and adequate. Supplier Perspective: Examine the utility of the processes used on the effort. Evaluate how well suited and/or capable the processes are for supporting the effort. Assess whether the processes used are adequate. Assess the impact of changes to the processes.

6.2.2 Efficiency

Scope: The processes, steps, methods and supporting tools employed on the effort. Customer Perspective: Examine the productivity of the processes used on the effort. Contrast productivity against similar efforts. Assess the degree of process concurrency/overlap. Assess whether the productivity is acceptable. Supplier Perspective: Examine the productivity of the processes used on the effort. Contrast productivity against similar efforts. Assess the degree of process concurrency/overlap. Assess the impact of changes in productivity.

6.2.3 Improvement

Scope: The processes, steps, methods and supporting tools employed on the effort. Customer Perspective: Examine the processes used on the effort. Evaluate how they are (to be) improved over time. Assess whether the improvement process is acceptable. Supplier Perspective: Examine the processes used on the effort. Evaluate how they are (to be) improved over time. Assess the impact of changes in the improvement process.

A-32


Chapter 5 – Product Scope: This issue category concerns the characteristics of the product(s)/services being developed or maintained by the effort. 7.1 Requirements

Scope: This issue concerns the characteristics of the requirements imposed on the effort’s product.

7.1.1 Completeness

Scope: The product and support requirements. Customer Perspective: Examine the product and support requirements. Evaluate their completeness in depth and breadth. Assess the traceability of the requirements to mission requirements. Assess whether the degree of completeness is acceptable. Supplier Perspective: Examine the product and support requirements. Evaluate their completeness in depth and breadth. Assess the impact of changes in requirement completeness.

7.1.2 Correctness

Scope: The product and its support requirements. Customer Perspective: Examine the product and support requirements. Evaluate their correctness. Assess whether the level of correctness is acceptable. Supplier Perspective: Examine the product and support requirements. Evaluate their correctness. Assess the impact of changes in requirements’ correctness.

7.1.3 Feasibility

Scope: The product and support requirements. Customer Perspective: Examine the product and support requirements. Evaluate their feasibility. Assess whether their level of feasibility is acceptable. Supplier Perspective: Examine the product and support requirements. Evaluate their feasibility. Assess the impact of changes on requirement feasible.

A-33

ANNEX A to AQAP-2050 (Edition 1) 7.1.4 Stability

Scope: The product and support requirements. Customer Perspective: Examine the product and support requirements. Evaluate their likelihood of changing. Assess whether the stability of the requirements is acceptable. Supplier Perspective: Examine the product and support requirements. Evaluate their likelihood of changing and the capability of the requirements change-management process to highlight unplanned requirements changes (e.g., requirements creep). Assess the impact of changes to the stability of the requirements.

7.1.5 Traceability

Scope: The product and support requirements. Customer Perspective: Examine the product and support requirements. Evaluate the traceability of the requirements across the lifecycle. Assess whether the requirements can be traced back to the agreed requirement specifications. Supplier Perspective: Examine the product and support requirements. Evaluate the traceability of the requirements across the lifecycle. Assess whether the requirements are referenced from the initial development to final user acceptance.

7.2 Design

Scope: This issue concerns core attributes of the product being produced by the effort.

7.2.1 Architecture

Scope: The effort’s products and supporting infrastructure. Customer Perspective: Examine architectural design and implementation of the product and supporting infrastructure. Evaluate their robustness and tolerance to change (e.g., can new functionality be added easily). Assess whether the architecture of the product and infrastructure are acceptable. Supplier Perspective: Examine architectural design and implementation of the product and supporting infrastructure. Evaluate their robustness and tolerance to change. Assess the impacts of changes to the architecture.

A-34

ANNEX A to AQAP-2050 (Edition 1) 7.2.2 Scale

Scope: The effort’s products and supporting infrastructure. Customer Perspective: Examine the effort’s products and infrastructure. Evaluate the capability to scale these up or down over time (e.g., can data throughput be increased easily). Assess whether the capability for scaling is acceptable. Supplier Perspective: Examine the effort’s products and infrastructure. Evaluate the capability to scale these up or down over time. Assess the impacts of changes in the capability to scale the product or infrastructure.

7.2.3 Complexity

Scope: The effort’s products and supporting infrastructure. Customer Perspective: Examine the effort’s products and infrastructure. Evaluate their complexity (e.g., coupling, cohesion, number of interfaces, communication protocols, etc.) both individually and together. Assess the impact of this complexity on the effort. Assess whether the level of complexity is acceptable. Supplier Perspective: Examine the effort’s products and infrastructure. Evaluate their complexity (e.g., coupling, cohesion, number of interfaces, communication protocols, etc.) both individually and together. Assess the impact of changes to the level of complexity.

7.2.4 Technology Effectiveness

Scope: The effort’s products and supporting infrastructure. Customer Perspective: Examine the effort’s products and infrastructure technologies. Assess the ability to recognize the need for technology infusion. Evaluate the ability to use new technology. Assess whether the degree of technology effectiveness is acceptable. Supplier Perspective: Examine the effort’s products and infrastructure technologies. Compare and contrast their the timeliness and effectiveness of the technology with respect to the effort’s objectives and overall mission requirements. Assess the ability to predict, plan for and utilize technology to its potential. Assess the impact of changes in technology effectiveness.

A-35

ANNEX A to AQAP-2050 (Edition 1) 7.2.5 Interoperability

Scope: The effort’s products and supporting infrastructure. Customer Perspective: Examine the product and supporting infrastructure. Evaluate their interoperability requirements with other systems such as working environments, compatibility of technologies, capability and capacity of the interfaces, adherence to standards. Assess their interoperability maturity/readiness. Assess whether the interoperability of the product is acceptable. Supplier Perspective: Examine the product and supporting infrastructure. Evaluate their interoperability requirements with other systems such as working environments, compatibility of technologies, capability and capacity of the interfaces, adherence to standards. Assess their interoperability maturity/readiness Assess the impact of changes to product capability.

7.2.6 Reuse

Scope: Reuse of components already designed. Customer Perspective: Examine whether the concept of reuse is an integral element of the system/software engineering processes in place. Evaluate whether mechanisms are established to verify appropriateness of the reusable elements for their intended purpose. Assess the adequacy and effectiveness of reused components. Supplier Perspective: Evaluate implementation of the concept of reuse as an integral element of the system / software engineering processes in place. Evaluate the mechanisms to verify the appropriateness of the reusable elements for their intended purpose. Assess the adequacy and effectiveness of reused components.

7.3 Attributes

Scope: This issue concerns the fitness for use of the effort’s product.

7.3.1 Usability

Scope: The product and its (proposed) operation. Customer Perspective: Examine the product and its (proposed) operation. Evaluate their likelihood of meeting user needs and expectations. Evaluate the level of complexity required in its operation. Evaluate the training requirements and the training supplied. Assess whether the usability level is acceptable. Supplier Perspective: Examine the product and its (proposed) operation. Evaluate their likelihood of meeting user needs and expectations. Evaluate the level of

A-36


complexity required in the product’s operation. Evaluate the training requirements and the training supplied. Assess the impacts of changes to product usability.

7.3.2 Performance

Scope: The technical performance/functional correctness required of the software/system products, e.g., speed, size, bandwidth, and integration with related systems or components. Customer Perspective: Examine the effort’s technical requirements and degree of functional correctness at this point in the effort. Compare and contrast with current capability. Assess whether targeted performance and actual performance are acceptable. Supplier Perspective: Examine the effort’s technical requirements and degree of functional correctness at this point in the effort. Compare and contrast with current capability. Assess impacts of any shortfall in targeted performance. Assess impacts of any changes in targeted or actual performance.

7.3.3 Dependability

Scope: The product and its (proposed) operation. Customer Perspective: Examine the product and its supporting infrastructure. Contrast the dependability of the technology used against respective targets. Assess whether the dependability is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Contrast the dependability of the technology used against respective targets. Assess the impacts of changes to the dependability targets.

7.3.4 Reliability/Availability

Scope: The product and its (proposed) operation. Customer Perspective: Examine the product and its supporting infrastructure. Contrast the reliability/availability of the technology used against their respective targets. Assess whether the reliability/availability is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Contrast the reliability/availability of the technology used against respective targets. Assess the impacts of changes to the reliability/availability targets.

A-37

ANNEX A to AQAP-2050 (Edition 1) 7.3.5 Supportability/Maintainability

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and its supporting infrastructure. Contrast the level of supportability/maintainability in operation against technical/mission requirements. Assess whether the level of supportability/maintainability is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Contrast the level of supportability/maintainability in operation against technical/mission requirements. Assess the impact of changes in maintainability.

7.3.6 Reusability

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and its supporting infrastructure. Evaluate their technology, design, architecture, etc., for reusability, and contrast the level of reusability against reusability targets. Assess the reusability policy for impacts on schedule, cost, reliability, etc. Assess whether the amount of reusability is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Evaluate their technology, design, architecture, etc., for reusability, and contrast the level of reusability against reusability targets. Assess the reusability policy for impacts on schedule, cost, reliability, etc. Assess the impact of changes in the level of reusability.

7.3.7 Portability

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and supporting infrastructure. Assess the length of time the product will likely remain on the current technology. Evaluate the ease of their execution using different technology. Assess whether the degree of portability is acceptable. Supplier Perspective: Examine the product and supporting infrastructure. Evaluate the ease of their execution on different technology. Assess the impact of having to change the porting of the product or infrastructure.

7.3.8 Efficiency

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and its supporting infrastructure. Evaluate their technology, design, architecture, etc., for efficiency of use (e.g.,

A-38


impacts on throughput, utilization, and timing constraints) and contrast the level of efficiency achieved against targets. Assess whether the amount of level of efficiency is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Evaluate their technology, design, architecture, etc., for efficiency of use and contrast the level of efficiency achieved against targets. Assess the impact of changes in the level of efficiency.

7.3.9 Integrity

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and its supporting infrastructure. Evaluate whether engineering and verification/validation processes are in place to ensure product integrity. Evaluate system completeness, correct cohabitation/interoperability with sub-systems as well as successful and correct operation. Assess whether the level of integrity is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Evaluate whether engineering and verification/validation processes are in place to certify system completeness, correct cohabitation/interoperability with sub-systems as well as successful and correct operation. Assess the impact of changes in the level of integrity.

7.3.10 Survivability

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and its supporting infrastructure. Evaluate whether appropriate scenarios are established and executed to verify continuity of specified functions after abnormal working conditions with a schema of graceful degradation. This includes verification of the ability to fulfill specified functions even during an intentional or accidental attack. Assess whether the level of survivability is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Evaluate established and executed scenarios to verify continuity of specified functions after abnormal working conditions with a schema of graceful degradation. This includes verification of the ability to fulfill specified functions even during an intentional or accidental attack. Assess the impact of changes in the level of survivability.

A-39

ANNEX A to AQAP-2050 (Edition 1) 7.3.11 Safety

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and its infrastructure. Evaluate whether the project considers the implementation of safety-related requirements through the definition of safety scenarios and cases. Evaluate safety-related verification and validation activities. Assess whether the level of safety of the product is acceptable. Supplier Perspective: Examine the product and its infrastructure. Evaluate whether the project considers the implementation of safety-related requirements through the definition of safety scenarios and cases. Evaluate safety-related verification and validation activities. Assess the impact of change on the level of safety of the product.

7.4 Technical Constraints

Scope: This issue concerns the risks inherent within the development or operation of the effort’s product or the product itself.

7.4.1 Human Factors

Scope: The product and its supporting infrastructure. Customer Perspective: Examine the product and its supporting infrastructure. Evaluate both for their consideration of human factors. Evaluate the human factor requirements and their impacts on ease of use, ease of learning, ease of remembering, etc. Assess whether the consideration of human factors of the effort is acceptable. Supplier Perspective: Examine the product and its supporting infrastructure. Evaluate both for their consideration of human factors. Evaluate the human factor requirements and their impacts on ease of use, ease of learning, ease of remembering, etc. Assess the impact of changes to the human factors aspects of the effort.

7.4.2 Health & Safety

Scope: The product use, its supporting infrastructure, and the processes used to manage, develop or support the product. Customer Perspective: Examine the product use, its supporting infrastructure, and the processes used to manage, develop or support the product. Evaluate each against health and safety requirements. Assess whether the consideration of health and safety requirements is acceptable.

A-40


Supplier Perspective: Examine the product use, its supporting infrastructure, and the processes used to manage, develop or support the product. Evaluate each for health and safety requirements. Assess whether the health and safety requirements are feasible. Assess the impact of changes to the health and safety requirements on the effort.

7.4.3 Security

Scope: The product, its supporting infrastructure, and the processes used to manage, develop or support the effort. Customer Perspective: Examine the product, its supporting infrastructure, and the processes used to manage, develop or support the product. Evaluate each for security requirements. Assess whether the security requirements are acceptable. Assess whether the consideration of security requirements is acceptable. Supplier Perspective: Examine the product, its supporting infrastructure, and the processes used to manage, develop or support the product. Evaluate each for security requirements. Assess whether the security requirements are acceptable. Assess the impact of changes to the security requirements on the effort.

7.4.4 Natural Environment Protection

Scope: The product across its life cycle, and its supporting infrastructure. Customer Perspective: Examine the product, and its supporting infrastructure. Evaluate the requirements for natural environment protection. Assess whether the consideration of natural environment protection provisions is acceptable. Supplier Perspective: Examine the product, and its supporting infrastructure. Evaluate the requirements for natural environment protection, Assess whether provision of natural environment protection requirements is acceptable. Assess the impact of changes to the natural environment protection requirements of the effort.

A-41


Chapter 8 – Schedule Scope: This issue category concerns the ability to develop and execute a consolidated realistic schedule. 8.1 Constraints

Scope: This issue concerns factors that constrain the project’s ability to meet schedule.

8.1.1 Feasibility

Scope: The past and current schedules. Customer Perspective: Examine past and current schedules of the customer and proposed by the supplier. Assess schedules against customer needs and imposed target dates. Review assumptions made, life cycle model selected, resources allocated. Determine inconsistencies between customer and supplier schedules and assumptions. Assess ability to meet milestones based upon past performance and all factors reviewed. Supplier Perspective: Examine past and current schedules imposed by the customer and proposed by the sub-supplier. Assess the schedules against customer needs and imposed target dates. Review assumptions made, life-cycle model selected, and resources allocated. Determine inconsistencies between customer, supplier and sub-supplier schedules and assumptions. Assess ability to meet milestones based upon past performance and all factors reviewed.

8.1.2 Dependencies

Scope: This issue concerns the effects of dependencies (technical, schedule, personnel, etc.) upon the effort. Customer Perspective: Examine the system and sub-systems schedules and their relationships, considering contractual arrangements, technical and resource constraints. Evaluate the number and types of dependencies and couplings and their rate of change. Assess the level of dependency, its manageability and impacts on the effort. Assess whether the level of dependency is acceptable. Supplier Perspective: Examine the schedule, proposed by the supplier with respect to its ability to meet contractual arrangements, technical and resource constraints. Evaluate the number and types of dependencies and couplings. Assess the level of dependency, its manageability and its impacts on the effort. Assess the impacts of changes in dependency.

A-42

ANNEX A to AQAP-2050 (Edition 1) 8.1.3 Contingency Plans

Scope: This issue concerns the plans established for acting in case an undesirable event impacts the schedule. Customer Perspective: Examine the plans for alternative actions to be taken in the case of an undesirable event (i.e. test failure) or other impact to the project’s schedule. Assess whether the plans are realistic and relevant, and based upon identified project risks. Supplier Perspective: Examine the plans for alternative actions to be taken in the case of an undesirable event (i.e. test failure) or other impact to the project’s schedule. Assess whether the plans are realistic and relevant, and based upon identified project risks.

8.2 Progress Scope: This issue concerns the capability of the effort to assess progress.

8.2.1 Estimation Accuracy

Scope: The past and current schedules and estimation process. Customer Perspective: Examine past and current schedules and estimation process. Evaluate the accuracy of the schedules, including how often the schedule requires re-estimation. Assess the degree to which the current schedule is inaccurate and how the estimation process contributes to the inaccuracy. Determine whether estimation accuracy is acceptable. Supplier Perspective: Examine past and current schedules and estimation process. Evaluate the accuracy of the schedules, including how often the schedule requires re-estimation. Assess the degree to which the schedule is inaccurate. Assess how estimation inaccuracy is used to improve the estimation process. Assess the impact of schedule or process changes.

8.2.2 Visibility

Scope: The current schedule and estimation process. Customer Perspective: Examine the current schedule and estimation process. Evaluate whether sufficient schedule detail is visible among the stakeholders. Assess how timely the schedule information is reported, especially deviations from schedule. Assess whether the schedule visibility is acceptable. Supplier Perspective: Examine the current schedule. Evaluate whether sufficient schedule detail is visible among the stakeholders. Assess how timely the schedule

A-43


information is reported, especially deviations from schedule. Assess the impacts of changing the level of visibility into the schedule.

8.2.3 Progress Performance

Scope: The past and current schedules and estimation process of the stakeholders. Customer Perspective: Examine the past and current schedules. Evaluate the actual schedule against planned progress targets (e.g., milestones, reviews, earned value). Assess whether the actual schedule progress meets expected progress. Determine the effort remaining to assess whether the progress made is acceptable. Supplier Perspective: Examine the past and current schedules. Evaluate the actual schedule against planned progress targets. Assess whether the actual schedule progress meets expected progress. Determine the effort remaining to assess whether the progress made is acceptable.

8.2.4 Rework

Scope: The past and current schedules and estimation processes of the stakeholders. Customer Perspective: Examine the past and current schedules. Evaluate the number of schedule deviations due to unplanned rework (e.g., technology failure, changes in mission requirements, vendor/supplier delays, staffing issues, etc.). Assess the likelihood of the amount of planned rework being inaccurate. Assess how the estimation process accounts for rework. Assess whether the amount of rework is acceptable within the current schedule. Supplier Perspective: Examine the past and current schedules. Evaluate the number of schedule deviations due to unplanned rework. Assess the likelihood of the amount of planned rework being inaccurate. Assess how the estimation process accounts for rework. Assess the impacts of a change in rework.

A-44


Chapter 9 - User/Customer Scope: This issue category concerns those aspects of the project that support the user or customer.

9.1 Satisfaction

Scope: This issue concerns the level of customer and/or user satisfaction that exists with the products developed or services provided.

9.1.1 Involvement

Scope: The user/customer interface to the effort. Customer Perspective: Examine user/customer interface to the effort. Evaluate how well the user and customer are consulted on the product’s specification, design, testing and overall progress. Assess how well the user and customer are involved. Assess whether the level of consultation is acceptable. Supplier Perspective: Examine user/customer interface to the effort. Evaluate how well the user and customer are consulted on the product’s specification, design, testing and overall progress. Assess how well the user and customer are involved. Assess the impact of changes to the level of consultation.

9.1.2 User Acceptance

Scope: The product and its relationship to its users. Customer Perspective: Examine the product. Evaluate whether the product has been validated against the mission requirements. Assess how well it meets the users’ needs and expectations. Assess whether the level of product utility/usability is acceptable. Supplier Perspective: Examine the product. Evaluate whether it has been validated against the contractual requirements. Assess whether it meets the mission requirements. Assess how well the product meets the users’ needs and expectations. Assess the impact to a change in utility/usability.

9.2 Transition

Scope: The transition support provided to the users of the products or services acquired, developed or supported by the effort.

A-45

ANNEX A to AQAP-2050 (Edition 1) 9.2.1 Transition Support

Scope: The transition support plan and (proposed) implementation. Customer Perspective: Examine the transition support plan and (proposed) implementation provided to the users. Evaluate whether the transition plan is complete and is adequately supported. Assess whether the level of transition support is acceptable. Supplier Perspective: Examine the transition support plan and (proposed) implementation provided to the users and customer. Evaluate whether the transition plan is complete and is adequately supported. Assess the impacts of changes to the level of transition support.

9.2.2 Training

Scope: The customer training program. Customer Perspective: Examine the training program. Evaluate the training curriculum, supporting material and equipment. Assess both the depth and breadth of the training program and instructor expertise. Assess the training program for timeliness. Assess whether the training program is acceptable. Supplier Perspective: Examine the training program. Evaluate the training curriculum, supporting material and equipment. Assess the comprehensiveness of the training program and instructor expertise. Assess the training program for timeliness. Assess the impact of changes to the training program.

9.3 Support

Scope: This issue concerns the operations and maintenance support provided for the customer by the supplier.

9.3.1 Maintenance

Scope: Product related maintenance support provided for the customer by the supplier. Customer Perspective: Examine the contractual maintenance requirements. Evaluate whether they address the type of maintenance, level of effort, duration, maintenance categories, verification mechanisms as well as terms and conditions. Assess whether the delivered support complies with the contractual terms and conditions. Supplier Perspective: Examine the contractual maintenance requirements. Evaluate feasibility and whether they are duly followed. Assess the impact of changes on the

A-46


A-47

level of maintenance support. Assess whether the delivered support complies with the contractual terms and conditions.

9.3.2 Assistance

Scope: The level of support provided to the customer to operate and maintain the product. Customer Perspective: Examine the assistance requirements. Evaluate whether they address the type of assistance, level of effort, duration, verification mechanisms as well as terms and conditions. Assess whether the delivered assistance is adequate and complies with the contractual terms and conditions. Supplier Perspective: Examine the contractual assistance requirements. Evaluate feasibility and whether they meet project needs and are duly followed. Assess the impact of changes on the level of contractual assistance. Assess whether the delivered contractual assistance complies with terms and conditions.

9.3.3 Warranty

Scope: The provider’s responsibility to remove defects after product acceptance. Customer Perspective: Examine the warranty requirements. Evaluate whether they address the type of warranty, required level of effort, duration, verification mechanisms as well as terms and conditions. Consider commonality of warranty requirements of the system and related sub-systems. Assess whether the actually delivered warranty support complies with the contractual terms and conditions. Supplier Perspective: Examine the contractual warranty requirements. Evaluate feasibility, whether they meet product needs and are duly followed. Assess the impact of changes on the level of warranty. Assess whether the delivered warranty complies with terms and conditions.

Documents

NATO PROJECT ASSESSMENT MODEL · MMT Model Management Team PL Project Liaison PM Project Manager PMT Project Manager Team PR Peer Reviewer VIII ORIGINAL. AQAP 2050 (Edition 1) Chapter