Presented to: NDIA PMSC By: Keith Kratzert Date: January 29, 2009 Federal Aviation Administration Improving Program Performance at FAA

Presented to: NDIA PMSC

By: Keith Kratzert

Date: January 29, 2009

Federal AviationAdministrationImproving Program

Performance at FAA

2 2Federal AviationAdministration

Improving Program Performance at FAAJanuary 29, 2009

NEWS FLASH!

• On January 22, 2009 in the update of their High-Risk List, the GAO removed the FAA ATC Modernization Program from their High-Risk List

“GAO is removing this program from its 2009 High-Risk List because of FAA’s progress in addressing most of the root causes of its past problems and the agency’s commitment to sustaining progress.”



EVM at FAA

• FAA goal for Major Programs is to manage Major DME programs using EVMS in accordance with OMB Circular A-11 and ANSI/EIA 748

– Manage/measure all resources

– Program level, not just contractors

– All contractual effort measured, including FFP

– Requires changes in behavior by all parties



Recent Accomplishments

• Established Portfolio/Program Performance Metrics (PPPM)– Now Implementing Agency Wide

• Closing out first contractor acceptance review

• First Program Level IBR conducted, second in planning

• First program surveillance results documented



FAA P3M

Financial Schedule Technical Resources External Interest

Program Manager

Y Y Y G Y Y

Financial Measure of cost performance of work performed

EVM Metrics

Cost Performance Index (CPI) G

Program Cost Reserve Y

Remaining Cost Risk Factor (TCPI / CPI) G

Cost Variance -At-completion (CVAC) % G

Obligation Rate Y

Schedule

Measure of schedule

performance of work performed

EVM Metrics

Schedule Performance Index (SPI) G

Program Schedule Reserve R

Schedule Variance-At-Completion (SVAC) %

Y

Level 1 AMS Milestones

Negative Deviations R

Acquisition Schedule Milestones

Negative Deviations G

Corporate Workplan Goals

Negative Deviations n/a

Technical

Measure of technical metrics and milestones

of work performed

Requirements Stability G

System Defects Y

Test Results Y

Deployment n/a

Value of remaining High Risks n/a

Technical Variance At-Completion (TVAC)

n/a

Resources Measure of

current funding and staffing

Prime Contractor G

Support Contractors G

FAA G

Funding G

Staffing G

External Interest Assessment of

External Reviews

IG G

GAO Y

OMB G

Other G

Program Summary Metrics

Supporting

Metrics



Contractor Acceptance Review

• Progress Assistance Visit conducted July 9 – 10, 2008

• Validation Review conducted September 22 – 26, 2008

• Most recent review of CAR closeout activity January 22, 2009

• Anticipate formal acceptance in February 2009



Program Level IBR

• Conducted IBR on the Surveillance and Broadcast System (SBS) October 10, 2008– One day review– Six CAM interviews– Two outside technical experts

• Next is the System Wide Information Management (SWIM) Program, February 2009



Surveillance Results

• The purpose of the Primary Program Performance Surveillance was to verify the reliability and consistency of program performance data

• Results:

Data is Reliable:

ASDE-X ASR-11 ATCBI-6 NEXCOM TFM WAAS6 27%

Some data is questionable:

ADS-B ASR-9 Ph II ATOP ASWON ERAM IFPA STARS SWIM TAMR II

9 40%

Questionable Data: ITWS SASO 2 10%



Current Evaluations of FAA EVM

• GAO report published GAO-08-756• DOT IG, review of EVM and Security

– Draft report out



GAO-08-756, Recommendations• Modify acquisition policies governing EVM to:

– require the use of a product-oriented standard work breakdown structure– enforce existing EVM training requirements and expand these requirements to

include senior executives responsible for investment oversight and program staff responsible for program oversight, and

– define acceptable reasons for rebaselining and require programs seeking to rebaseline to (1) perform a root cause analysis to determine why significant cost and schedule variances occurred and (2) establish mitigation plans to address the root cause.

• Direct the ERAM program office to work with FAA’s Value Management Office to:

– determine the root causes for the anomalies found in the contractor’s EVM reports and

– develop a corrective action plan to resolve these problems. • Direct the Value Management Office to improve its oversight processes

by:– including an evaluation of contractors’ performance data as part of its program

assessment criteria, when FAA has the authority to do so, and – distinguishing between programs that collect earned value data on fully integrated

programs and those that do not in its agency wide progress reports to provide transparency to decision makers.



FAA EVM Summary

• Continuing to evolve both policy and practice• Initiated Program Level Surveillance• Initiated FAA Major Contractor Acceptance• Addressing GAO recommendations



Backup information



GAO Report GAO-08-756“FAA has taken important steps to oversee program compliance with

EVM policies, but its oversight process lacks sufficient rigor.”

Seven Key Components of an Effective EVM Policy

Policy component Assessment of FAA policy

Establish clear criteria for which programs are to use EVM Fully met

Require programs to comply with national standards Fully met

Require programs to use a standard structure for defining the work products that enables managers to track cost and schedule by defined deliverables (e.g., hardware or software component)

Partially met

Require programs to conduct detailed reviews of expected costs, schedules, and deliverables (called an integrated baseline review)

Fully met

Require and enforce EVM training Partially met

Define when programs may revise cost and schedule baselines (called rebaselining) Partially met

Require system surveillance—routine validation checks to ensure that major acquisitions continue to comply with agency policies and standards

Fully met



FAST.FAA.GOV



FAA EVM



Contractor Pre-Acceptance



Contractor Acceptance Process



FAA Contractor Acceptance• An EVM Systems Implementation Plan is prepared by the

contractor and reviewed by the FAA. • The contractor EVMS is designed and implemented. • The Validation Review Plan is prepared by the FAA and

provided to the contractor. • An EVMS Progress Assistance Visit may be conducted

(Optional). • The Validation Review is conducted; the Validation Review

Report is prepared. • Corrective actions are identified, addressed and resolved. • The contractor EVMS is accepted and the Letter of Systems

Acceptance or Advance Agreement is issued to the contractor. Usually, Agencies require an acceptable surveillance plan and system change management plan from the supplier to obtain an Advance Agreement.

• EVMS surveillance commences.

Documents

Presented to: NDIA PMSC By: Keith Kratzert Date: January 29, 2009 Federal Aviation Administration Improving Program Performance at FAA