RFP Improvement Study

Proposals & EvaluationsProposals & Evaluations– A Government/Industry Perspective – A Government/Industry Perspective

Bill StockmanDayton Aerospace, Inc.

2 © 2006 Dayton Aerospace, Inc.

Cost Issues Today

FAR 12 to FAR 15 Conversions Berry Amendment AFMC Sponsored RFP Improvement Study Source Selection Approaches


FAR 12 to FAR 15 Changes

12.214 -- Cost Accounting StandardsCost Accounting Standards (CAS) do not apply to contracts and subcontracts for the acquisition of commercial items when these contracts and subcontracts are firm-fixed-price or fixed-price with economic price adjustment (provided that the price adjustment is not based on actual costs incurred).

15.406(b) When cost or pricing data are required, the contracting officer shall require the contractor or prospective contractor to submit to the contracting officer (and to have any subcontractor or prospective subcontractor submit to the prime contractor or appropriate subcontractor tier) the following in support of any proposal:(1) The cost or pricing data.(2) A certificate of current cost or pricing data, in the format specified in 15.406-2, certifying that to the best of its knowledge and belief, the cost or pricing data were accurate, complete, and current as of the date of agreement on price or, if applicable, an earlier date agreed upon between the parties that is as close as practicable to the date of agreement on price.

• What is the impact on the cost community when existing programsSwitch mid-stream from FAR 12 to FAR 15 cost requirements?

• “Commercial-like” programs from the mid-1990s that were competed but no competition left today


Buy America & Berry Amendment Provisions FY 2004 Defense Department Authorization Bill

The Berry Amendment requires the U.S. Defense Department (DOD) to buy certain products -- judged essential to our military readiness - with 100% U.S. content and labor. These products include clothing and other textile items, specialty steel, and food. In addition to the Berry Amendment, certain other "Buy American" provisions apply to DOD.

ISSUE:Rep. Duncan Hunter, Chairman of the House Armed Services Committee, has strengthened and expanded both the "Berry" and "Buy American" provisions in House version of the FY 2004 Department of Defense (DOD) Authorization bill by:

•Tightening the waiver process so that the Berry Amendment is not waived for inappropriate and arbitrary reasons. (Berry)•Clarifying that the Berry Amendment requirements must be met throughout all levels of the supply chain. (Berry)•Directing DOD and prime defense contractors to purchase U.S. made machine tools & dies and specialty metals. (Buy American)•Increasing the minimum U.S. content for defense procurement from 50% to 65%. (Buy American)•Requiring the Secretary of Defense to create a break-out list of critical components and technologies fundamental to our national defense effort, with a phased-in requirement that these items be 100 % U.S. made. (Buy American)


Berry Amendment & the Cost Community

Cost of compliance on new programs Cost of compliance on existing programs

Determining current content for primes and subs Determining cost to change design to comply Impact on current production programs Impacts on reliability Cost benefit analysis Waivers

Improving Proposals through better RFPs Improving Proposals through better RFPs – A Government/Industry Perspective – A Government/Industry Perspective

Bill Buzzell & Bob KayuhaDayton Aerospace, Inc.


Purpose of StudyPurpose of Study

Improving Proposals through better RFPs – A Government/Industry Perspective

Basic premise is that if we can improve RFPs- It will result in better proposals,

- Which will result in better executed programs

This study was accomplished under contract with the US Air Force HQ AFMC.This study was accomplished under contract with the US Air Force HQ AFMC.


DAI Study ApproachDAI Study Approach

DAI Group SessionDAI Group Session

Industry On-lineSurvey

Industry On-lineSurvey

One-on-OneInterviews

One-on-OneInterviews

Industry GroupSession

Industry GroupSession

Analysis &Documentation

Analysis &Documentation

Final ReportFinal Report PresentationsPresentations

• Generated initial subject areas• Guided Survey & Interview questions

• Blind Survey• Non-Attribution (company or individual)• Results guided Industry Session

• Brainstormed problems• Categorized problems into groups• Prioritized problems in each group• ID’d owners & provided recommendations

• Consolidated data• Summarized trends and major findings

• HQ AFMC• APMP Conference Presentation


Statement of Objectives – Focus of PresentationStatement of Objectives – Focus of Presentation

1. Identify specific industry capture strategies

2. Organization of industry capture teams

3. Methods used to develop a proposal

4. Identifying known problems with Air Force RFPs

5. List industry best practices and lessons learned

6. Examples from actual programs

7. Providing specific recommendations for process improvement


RFP Study ResultsRFP Study Results

Survey Results

Interview Results

Group Session Results


Survey ResultsSurvey Results

What are the major government drivers inhibiting a good RFP?

0% 10% 20% 30% 40% 50% 60% 70% 80%

Inexperienced evaluators

RFPs not reviewed by decision makers with experience

Functional “pet rock” requirements disrupt the RFP

DRFP released before acquisition strategy fully thought out

Selection criteria used not specified in Section M

RFP developed by inexperienced people – too much “cut and paste”

Other



Which of the following RFP timetable issues normally cause you the most problems?

0% 10% 20% 30% 40% 50% 60% 70% 80%

Locked in dates for submission

Not enough time to respond

Delay in release of final RFP

Major changes between DRFP and final RFP (with no submittal extension)

Government doesn’t realize cost impact to proposal schedule slips

Other



Which of the following RFP program schedule issues (as dictated in an RFP) normally cause you the most problems?

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Government mandated program schedule or milestones

Last minute re-phasing of funding in final RFP release

Funding profile unrealistic (can’t match a realistic program schedule)

Government mandating a one-for-one match between the WBS and the Integrated Master

Schedule tasks

Other



Which of the following parts of the RFP usually cause the most problems in terms of inconsistent program guidance?

0% 10% 20% 30% 40% 50% 60% 70% 80%

Overall Instructions

Section L Instructions

Section M Instructions

Cost Instructions

Past Performance Instructions

Other


Survey ResultsSurvey Results Which of the following parts of the RFP usually cause the most

problems in terms of inconsistent format guidance?

0% 10% 20% 30% 40% 50% 60% 70% 80%

Overall Instructions

Section L Instructions

Section M Instructions

Cost Instructions

Past Performance Instructions

Other



Which of the following government prioritization issues normally cause you the most problems?

0% 10% 20% 30% 40% 50% 60% 70% 80%

Poor program technical definition

Too prescriptive/restrictive program requirements

Bad or non-existent pre-RFP requirements review process

Funding Profile – incompatible funding profile to program requirements

Conflicting government priorities (e.g., conflicting functional requirements, conflicting

priorities within different sections of the RFP)

Other



Which of the following government RFP review process issues do you feel result in a poorly prepared RFP?

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

No industry input to cost ( & schedule) of program (funding required) prior to RFP release

Draft RFP process not handled well (e.g., badly coordinated, too short a timeframe, etc.)

RFP poorly coordinated within government

Poorly run Q&A process

Other



Survey Results

Interview Results



Interview ResultsInterview ResultsAcquisition Strategy

• Make sure the RFP reflects the acquisition• Past RFPs are applied incorrectly to new acquisitions• Use a gate process – start with past performance and screen bidders, then move to SOW, IMP/IMS etc• Need execution team on both sides - not just evaluation team and proposal team – “avoid bait & switch”

Acquisition Strategy• Make sure the RFP reflects the acquisition• Past RFPs are applied incorrectly to new acquisitions• Use a gate process – start with past performance and screen bidders, then move to SOW, IMP/IMS etc• Need execution team on both sides - not just evaluation team and proposal team – “avoid bait & switch”

Pre-Proposal Discussion• Don’t use only one industry day – Have a series of industry days & provide early feedback to contractor• Negotiate proposal page limitations during Industry Days• Be absolutely explicit in answering contractor Q&As Risk Reduction review – look at entire program – facilities, people etc, rather than just engineering status More discussion is better – need to develop relationship between all Govt/Industry functional players

Pre-Proposal Discussion• Don’t use only one industry day – Have a series of industry days & provide early feedback to contractor• Negotiate proposal page limitations during Industry Days• Be absolutely explicit in answering contractor Q&As Risk Reduction review – look at entire program – facilities, people etc, rather than just engineering status More discussion is better – need to develop relationship between all Govt/Industry functional players

Requirements• Make sure Section L and Section M agree• Past Performance evaluation criteria that clearly reflects the most important requirements • Past Performance criteria should reflect risk items.• Section L & M conflicts generate the biggest issues in services type proposals – RFP Traceability matrix • Govt shouldn’t hide assumptions or concerns (risks) – if it is considered a risk, then tell the contractors

Requirements• Make sure Section L and Section M agree• Past Performance evaluation criteria that clearly reflects the most important requirements • Past Performance criteria should reflect risk items.• Section L & M conflicts generate the biggest issues in services type proposals – RFP Traceability matrix • Govt shouldn’t hide assumptions or concerns (risks) – if it is considered a risk, then tell the contractors


Interview Results - continuedInterview Results - continuedExperience/OJT

• Main problem – experience level in program office• Senior leadership needs to be more proactive• Use more senior review – but they need to be experienced and have the right information• Need to get back to basics – overall process is sound, mainly a lack of discipline• Lack of experience, especially in the contracts area – training is a big issue• Smaller contracting shops at centers not disciplined – award size “driven under the radar”• Variability extensive between centers – standardization needed - simplify task order procedure• Need independent risk assessment of proposals

Experience/OJT• Main problem – experience level in program office• Senior leadership needs to be more proactive• Use more senior review – but they need to be experienced and have the right information• Need to get back to basics – overall process is sound, mainly a lack of discipline• Lack of experience, especially in the contracts area – training is a big issue• Smaller contracting shops at centers not disciplined – award size “driven under the radar”• Variability extensive between centers – standardization needed - simplify task order procedure• Need independent risk assessment of proposals

Formal Training• Risk Day – good concept – no one understands risk!! Force training for new people

Formal Training• Risk Day – good concept – no one understands risk!! Force training for new people

Proposal Response Govt needs to ask for a risk confidence level in proposal assumptions Oral proposals over written

Proposal Response Govt needs to ask for a risk confidence level in proposal assumptions Oral proposals over written

Scheduling IMP/IMS – Govt doesn’t understand what they are asking for – can’t evaluate it when they get it IMP/IMS growth – IMS too large – ask for in rolling wave

Scheduling IMP/IMS – Govt doesn’t understand what they are asking for – can’t evaluate it when they get it IMP/IMS growth – IMS too large – ask for in rolling wave



Survey Results

Interview Results



Industry Group SessionIndustry Group Session

Acquisition Strategy Experience/OJT Formal Training Funding Pre-proposal Discussion Proposal Response Requirements Scheduling

Major Categories of Findings


Acquisition Strategy FindingsAcquisition Strategy FindingsFinding Score Description OPR Recommendation

Lack of a well thought out total Program Acquisition Strategy

62

No clear roadmap

Lack of disciplined planning

Lack of experience and proper Senior level review

PM PEO

ASP training to the SS team

Keep it simple – consider incremental baseline reviews

Coordinate with Contractor and Warfighter early

Acquisition Strategy Changes

49

Requirements changes after RFP release

Programmatic changes drive schedule slippages which drive increased B&P costs

SAF/ ACE

PM

User

Involve Industry upfront in reqmts generation/refinement

Changes/slips approved at higher management level

Evaluation Criteria 43

Unclear or mismatched criteria to the requirements

CMMI not effectively used as evaluation criteria

PM

ACE

Use ASP to review evaluation criteria

Govt Red Team the criteria

Lack of Understanding the Program Cost Risk

29

On down selects competitions there are multiple technical TIMS, but cost TIMS are rarely conducted

SAF/ ACE

PM

SAF/ACE needs to establish minimum cost risk criteria

Actively address contractor’s cost estimating assumptions

Government desire to be integrator

20Misunderstanding about the Govt’s integration role

PMFormally define categories of Govt support – use ASP process for vetting


Experience/OJT FindingsExperience/OJT FindingsFinding Score Description OPR Recommendation

Inexperienced Program Managers

45

Lack of management overlap due to re-assignment

Formal training but little actual acquisition experience

HQ AFMC

SAF/ AQ

Need active mentoring process

Need effective career management plans for future senior PMs

Inexperienced technical evaluators

38

Reductions in workforce have stretched resources

Failure of civil service to hire primarily at entry level

SAF/ ACE

HQ AFMC

Develop pool of technical evaluators at each center

Formal SS training coupled with OJT

Contractor Questions 34Questions not clearly answered prior to formal RFP release

PM

PCO

Need effective review process for release of Govt answers

Build time into acq schedule

Use additional Industry Days to address questions

Government and Industry Interface

27Not working as partners when the DRFP is written

PMPM should set the tone and ensure a level playing field

Award Fee Contracts 21

Failure to identify Award Fee Evaluation criteria prior to contract award

Contractors need to know criteria prior to submission

SAF/ ACE

Need formal policy addressing Award Fee Criteria in RFPs

Award Fee Plans should be negotiated prior to contract award


Formal Training FindingsFormal Training FindingsFinding Score Description OPR Recommendation

Lack of a clear, integrated RFP

51

Leads to inconsistencies, omissions & contradictions

A symptom of inadequate training and experience

PM

PCO

Process issue under the control of the PM and PCO

Build reqmts matrix to align Section L, M, SOO, etc

T&Cs not always appropriate for the type of contract being solicited.

31

Contractor wastes valuable time sorting incorrect clauses

A symptom of inadequate contracts experience

PCOTraining and templates will help

Early preparation and review of the model contract during DRFP

Inconsistency between what Contracts and Engineering expects from the awarded contract

29

Reflects and inadequate Acquisition Plan and ASP

Lack of experience in creating an executable contract

PM

Ensure contract execution is foremost in the development & structure of the contract

Feedback from Industry/User

Little or no risk identification by government

28

Govt doesn’t understand - Not presented in the written draft RFP or at the industry day

Government does not solicit inputs from Industry

PM

Need formal proposal and program risk training

Don’t hide risks – Need open discussion of risk during Industry Days

Government training on RFP writing is deficient or non existent.

26

Systemic issue - tendency in Program Office to press forward without proper preparation/training

SAF/

ACE

Need ACE guidance and well-defined training program

Make mandatory before DRFP process is initiated


Funding FindingsFunding FindingsFinding Score Description OPR Recommendation

Unrealistic funding profiles and total program estimate from the beginning

67

Funding profile does not reflect work scheduling

Inefficient profile – shortage of funding early

Failure of leadership to recognize/acknowledge program uncertainties

PM

SSA

ASP must address funding profile vs. system requirements and expected delivery times

PM must facilitate trade-off and cost impacts discussions between contractors and user

SSA must be final decision maker if shortfall identified after the fact

Too many changes to RFP requirements which are cost prohibitive

40Lack of control of the RFP by the PM, allowing user changes late in the process

PM

Requirements need to be well defined prior to RFP release

Use industry feedback on DRFP/SOW to finalize requirements

Spiral development is requested, but Section L asks for waterfall

23

Lack of effective review prior to RFP release

Can result in improperly proposed program costs

PM

ASP

Govt Red Team review of the RFP

Should be addressed as part of the ASP


Pre-Proposal Discussion FindingsPre-Proposal Discussion FindingsFinding Score Description OPR Recommendation

Discussions with industry

57

Inadequate opportunity for technical discussions prior to formal RFP release

RFP authors work in isolation

PM

PM sets the tone and “requirement” for industry participation

Mandatory Industry Days

Draft RFP 48

Inadequate use of Draft RFPs

No follow-up face-to-face meeting with industry after Draft RFP comments

PM

SAF/

ACE

Should be a firm reqmt and part of the ASP and acquisition schedule

Need standard DRFP procedure

Internal Government Discussions

37

Lack of coordination between contracts, engineering, user

Too much “Govt only” discussion on key issues

PMPM responsibility to assure balance and flow of information within Govt and out to industry

Proposal Schedule Slippages

28

Formal RFP released too soon resulting in both proposal and program schedule adjustments

Post RFP release schedule slips can drive significant additional B & P costs

PM

PCO

Responsibility of ASP process to hold program to a realistic and executable RFP schedule

PM and SSA must agree on the RFP schedule and make necessary allowances for holding to it through award


Proposal Response FindingsProposal Response FindingsFinding Score Description OPR Recommendation

Not enough lead time to respond to RFP

46

Especially when no DRFP

Lack of effective planning

No understanding of industry process

PMEarly planning with industry involvement

Industry Day & DRFP

RFP Q&A response time 38

Government takes too long to answer questions from industry on program (Industry Day, DRFP, etc.)

PCO

Establish reasonable disciplined schedule to receive industry comment and government response

Schedule slippages ... drive B & P dollars...after company budget has been approved for the year

31Too often government changes solicitation schedule in the middle of the process

PM

Establish schedule & stick to it

Change only as last resort

Inform Industry early if slip required

In many competitions there are multiple technical TIMS, but cost TIMS rarely conducted

25Government spends a lot of time on technical issues and not enough on cost issues

FM

Conduct Cost TIMS parallel with technical TIMSs to help develop a common understanding of the cost risk

Lack of appreciation of contractor effort to submit a fully compliant proposal

25Government personnel have very little understanding of industry proposal process

DAU

AQ

Provide material on generic industry proposal process in acquisition and source selection training


Requirements FindingsRequirements FindingsFinding Score Description OPR Recommendation

Not enough discussion on requirements - interface with user is inadequate

34

Inadequate opportunity to discuss requirements with tech team and users

Results in unreasonable expectations

PM & User

More focus on Industry Days - include Users

Initiate an early requirements risk management process

Section L and M issues 33Still – inconsistencies between Sections L & M

MDA / PEO

Need an independent last scrub (red team/murder board) prior to RFP release

Unrealistic funding profiles and total program estimate from the beginning

33Inconsistency between requirements and available funding

PM / FM / User

Early industry involvement to establish reasonable level of requirements for available funding

RFP Changes 27

After release of the RFP

Not enough upfront time taken to fully define requirements

PM / User

Allow sufficient time before RFP to nail down requirements.

Do not allow any but imperative Senior Group directed changes

SOW requirements/ schedule conflicts

23SOW requirements inconsistent with other parts of RFP

PM/ PEO

Industry/Government communication open and early.

Disciplined independent review process of RFP

Take time to do it right


Scheduling FindingsScheduling Findings

Finding Score Description OPR Recommendation

Inadequate funding for the requirements

43The scheduling aspect of this finding is a mismatch of expecting too much too soon

PMEnsure a balance between cost, schedule and technical requirements

Large changes to the RFP requiring major proposal re-submittals

41Changes in RFP timeline causes significant impact to industry participants

PM

Government needs to understand the actual industry proposal process.

Establish a disciplined RFP schedule and don’t change it

SOW requirements

/schedule conflicts40

Over restriction of schedule for the funding profile and requirements

PM

Early interface with industry to allow them to assist in definition of a realistic schedule for the funding and requirements

RFPs released with no or inaccurate information

33Cut and paste results in RFPs that do not match program or missing data

PMIndependent review of RFP.

Interface with industry

IMP/IMS 31

Ineffective use of this tool.

Limitations that make it impossible to demonstrate the entire program

PMTraining on purpose and use of IMP/IMS

Do not put limitations on IMS

Three Analysis Examples


Resource Analysis PlanAA BB CC DD EE FF

Narrative Review Yes Min Yes Yes Yes Yes

Detailed Labor Dist Yes Yes Yes Yes Yes YesBy FTEFTE Summary/Rate Yes Yes Yes Yes Yes Yes

Detailed Labor Rate Yes Yes Yes Yes Yes Yes

Subcontr Labor Anl Yes Yes Yes Min Diff Imbedded

O/A Rates Yes Yes Yes Yes Yes Yes

HW Refresh Details Yes Yes Yes No Yes No

HW Costs Yes Yes Yes Min Yes No

CLIN Summary Yes Yes Yes Yes Yes

Summary Cost B/O Yes Yes Yes Yes Yes Yes

WBS summary Yes Yes Yes Min Diff Yes

Transition WBS Yes Yes Yes No Diff No

Transition B/O Yes Yes Yes Yes No Yes

Trans Labor rates Yes Yes Yes Yes Yes Yes

Subs WBS B/O Yes Yes Yes

Trans Sched Yes yes Yes Yes yes yes

FTE Trans Ramp

SS FTEs

O/A Rate Impacts

Mgmt Metrics

Small Business

Dollarization TBD TBD TBD TBD TBD TBDS/W/RContract ClauseImpacts


Combined PRAG/Resource Allocation

VRRSRNRNO PP

WBS 1.2 WBS 1.3 WBS 1.4 WBS 1.5 Total TotalFTEs $$ FTEs $$ FTEs $$ FTEs $$ FTEs $$

AA 24.9% 18.9% 62.6% 50.6% 83.7% 76.9% 27.6% 21.6% 36.5% 30.4%A 6.8% 9.8% 20.7% 27.8% 11.2% 15.0% 10.4% 14.2%B 19.3% 15.5% 9.9% 7.5%CTI 18.1% 16.1% 9.3% 7.9%D 10.7% 15.9% 3.0% 5.6% 16.3% 23.1% 7.0% 10.6%E 4.4% 4.5% 4.9% 3.9% 10.1% 9.3% 5.6% 5.2%F 23.0% 23.6% 5.2% 5.4%G 9.8% 10.7% 5.0% 5.2%H 20.3% 22.8% 4.5% 5.2%I 2.4% 3.0% 7.8% 10.4% 2.9% 3.7%J 6.0% 5.8% 1.3% 1.3%K 2.4% 3.0% 1.2% 1.5%L 1.2% 2.5% 0.6% 1.2%M 1.8% 1.9% 0.4% 0.4%N 1.0% 1.8% 0.2% 0.4%


Total Evaluated Costs

• In theory, if one dollarizes all risks and color ratings, then all bids are “normalized” to green/low• This would allow for a simple selection of lowest total evaluated cost• Reality normally is that some items can’t be dollarized, so best value is the rule

AA BB CC DD EE FF

Total Proposal Costs 2,744,441,539$ 2,887,974,133$ 2,658,372,238$ 2,959,151,881$ 2,316,283,803$ 2,371,052,616$

Direct Costs/Savings (105,274,578)$ (86,537,850)$ (81,927,900)$ (87,735,050)$ (98,232,915)$ (90,768,200)$

Direct Benefits/Risks 72,837,396$ 160,554,692$ 341,698,715$ 85,188,829$ 334,654,804$ 393,705,250$

Eval Factors--SWR (30,163,824)$ (37,783,380)$ (11,436,000)$ (5,704,000)$ (8,457,480)$ (13,855,732)$

Total Evaluated Cost 2,681,840,534$ 2,924,207,595$ 2,906,707,053$ 2,950,901,660$ 2,544,248,211$ 2,660,133,934$