AQUANTITATIVEACQUISITIONPROCESSMODELINGAPPROACHTOWARDEXPEDITING

SYSTEMSENGINEERING

YvetteRodriguez06April2017

USCCenterforSystemsandSoftwareEngineering2017AnnualResearchReview

ResearchMotivation:BetterBuyingPower(BBP)3.0

• BBP3.0InitiativetoEliminateUnproductiveProcessesandBureaucracybymeansofstreamliningprocessesandreducingcycletime.

• BBP3.0Initiative:EliminateUnproductiveProcessesandBureaucracy

• Unnecessaryandlow-valueaddedprocessesanddocumentrequirementsareasignificantdragonacquisitionproductivityandmustbeaggressivelyidentifiedandeliminated.

“BetterBuyingPower(BBP)istheimplementationofbestpracticestostrengthentheDefenseDepartment'sbuyingpower,improveindustryproductivity,andprovideanaffordable,value-

addedmilitarycapabilitytotheWarfighter.”

ResearchFocus:Pre-MilestoneBData

• DepartmentofDefense(DoD)decision-makingduringearly(pre-MilestoneB)systemsengineeringprocesseshavelastingimpacts,bothpositiveandnegative,throughoutthelifecycle.

• SourcesofdataonDODsystemsengineeringtimelinesanddecisionprocessesprovidedquantitativeimprovementinsights.

DoD ReviewCycle

Letter

Figure 5: DOD Levels Reviewing Information Requirements

Page 15 GAO-15-192 Acquisition Reform

Many different functional organizations within each level review the information before the document is approved. The number of organizations conducting reviews varies depending on the information included in each document. A few documents that include a wide breadth of information can be reviewed by many offices at each level. For example, Air Force acquisition strategies, that on average took over 12 months to complete for the programs we surveyed, can be reviewed by

USAirForceReviewProcessExample

• 30LevelsofOfficeoftheSecretaryofDefense(OSD)Reviews

• 22ServiceLevelsofAcquisitionExecutive(SAE)Reviews

• 7LevelsofProgramExecutiveOffice(PEO)Reviews

ProgramExecutiveOffice(PEO)LevelReviews

1. ProgramExecutiveOffice2. FinanceFunctionalStaff3. DeputyProgramExecutiveOfficer4. EngineeringFunctionalStaff5. ContractingFunctionalStaff6. ProgramExecutiveOfficerExecutionGroup7. LogisticsFunctionalStaff

ServiceAcquisitionExecutive(SAE)LevelReviews

1. AssistantSecretaryoftheAirForceforAcquisition(ServiceAcquisitionExecutive

2. AssistantSecretaryoftheAirForceInstallations&Environment

3. AirForceLogistics,Installations,&MissionSupport

4. AirForceOperations,Plans,&Requirements

5. AirForceIntelligence,Surveillance,&Reconnaissance

6. AirForceFinancialManagement&Comptroller

7. AirForceTest&Evaluation8. AssistantSecretaryoftheAirForceSmall

BusinessPrograms9. AssistantSecretaryoftheAirForceChief

InformationOfficer10. AssistantSecretaryoftheAirForceTest&

Evaluation(PolicyandPrograms)11. AirForceOperations,Plans&

Requirements(OperationalCapabilityRequirements)

12. AirForceLogistics,Installations&MissionSupport(Logistics)

13. AssistantSecretaryoftheAirForceInstallations&Environment(Logistics)

14. AirForceIntelligence,Surveillance,&Reconnaissance(Strategy,Plans,Doctrine&ForceDevelopment)

15. AssistantSecretaryoftheAirForceChiefInformationOfficer(Policy&Resources)

16. AssistantSecretaryoftheAirForceDeputyGeneralCounselforAcquisition

17. AirForceFinancialManagementandComptrollerDeputyAssistantSecretary(CostandEconomics)

18. AirForceFinancialManagementandComptrollerDeputyAssistantSecretary(Budget)

19. AssistantSecretaryoftheAirForceDirectorateofScience,Technology&Engineering

20. AssistantSecretaryoftheAirForceDirectorateManagementPolicy&ProgramIntegration

21. AssistantSecretaryoftheAirForceDirectorateofContracting

22. AirForceAcquisitionCapabilityDirectorate

OfficeoftheSecretaryofDefense(OSD)LevelReviews

16. Director,SystemsEngineering17. Director,CostAssessmentandProgram

Evaluation18. Director,DevelopmentalTest&Evaluation19. Director,AcquisitionResources&Analysis20. DeputyAssistantSecretaryofDefense,

Manufacturing&IndustrialBasePolicy21. PrincipalDeputyUnderSecretaryofDefense

(Acquisition,Technology,&Logistics)22. Director,InternationalCooperation23. AssistantSecretaryofDefense(Acquisition)24. Director,PerformanceAssessmentandRoot

CauseAnalysis25. AssistantSecretaryofDefense(Logistics&

MaterialReadiness)26. AssistantSecretaryofDefense(Legislative

Affairs)27. DeputyUnderSecretaryofDefense(Installations

andEnvironment)28. Director,DefenseProcurementandAcquisition

Policy29. DeputyGeneralCounsel(Acquisition&Logistics)30. AssistantSecretaryofDefense(Operational

EnergyPlansandPrograms)

1. DefenseAcquisitionExecutive2. AssistantSecretaryofDefense(Research&

Engineering)3. ViceChairmanoftheJointChiefsofStaff4. DeputyAssistantSecretaryofDefense,

Strategic&TacticalSystems5. UnderSecretaryofDefense(Policy)6. DeputyAssistantSecretaryofDefense,

Space&Intelligence7. UnderSecretaryofDefense(Comptroller)8. DeputyAssistantSecretaryofDefense,

Communication,Command,andControlCyber

9. UnderSecretaryofDefense(Personnel&Readiness)

10. Director,NationalGeospatial-IntelligenceAgency

11. UnderSecretaryofDefense(Intelligence)12. DeputyDirector,CostAssessment13. ChiefInformationOfficer14. Director,DefensePricing15. Director,OperationalTest&Evaluation

Hypothesis&ResearchQuestions• Hypothesis:Thereexistsabaselinesetofcriticalsuccess

factordatavariablesthatidentifyearlyDODAcquisitionprogramslikelytoexperiencedelays.

• ResearchQuestions:– RQ1:Whatearlyacquisitionpredictivedatavariablesactas

criticalsuccessfactorsindistinguishingpreviousExpeditedversusDelayedearly-SEacquisitionprocesses?

– RQ2:Howcanthepeople,product,andprocessorganizationalpracticesidentifiedintheESEFframeworkbetterpromoteexpeditedsystemsengineeringthroughouttheearly-SEDoDacquisitionprocess?

– RQ3:WhatvalueisaddedtotheExpeditedSystemsEngineeringFrameworkthroughtheanalysisofquantitativereporteddata?

ExpeditedSystemsEngineeringFramework

TheoreticalFramework

Pre-MilestoneBProcessofInterest:GenericDoD ContractAwardProcess

MilitaryContractAwardProcess

Findings

BoxPlotOutliers

EstimatedDelay

PhaseA(ESISTOASP)#ofDays

PhaseB(ASPtoASD)#ofDays

PhaseC(ASDtoRFP)#ofDays

PhaseD(RFPtoCA)#ofDays

TotalDays(StarttoCA)

0

200

400

600

800

1000

1200

1400

1600

Boxplots

Mean Minimum/Maximum Outliers(1)

0

5

10

15

20

25

DollarV

alue

Level

Boxplot(DollarValueLevel)

ScatterPlotOutliers

EstimatedDelay

PhaseA(ESISTOASP)#ofDays

PhaseB(ASPtoASD)#ofDays

PhaseC(ASDtoRFP)#ofDays

PhaseD(RFPtoCA)#ofDays

TotalDays(StarttoCA)

0

200

400

600

800

1000

1200

1400

1600

Scattergrams

Mean Median

0

5

10

15

20

25

DollarV

alue

Level

Scattergram(DollarValueLevel)

RegressionResults

ΣDContract Pr ocess = 55.4+ 0.8*DEstimatedDelay−8.1DVContract + 0.6DPhaseA+1.4DPhaseB+ 0.4DPhaseC +1.2DPhaseD

D = Number of DaysDV = Dollar Value Level

-500

0

500

1000

1500

2000

-500 0 500 1000 1500 2000

Days(C

ontractP

rocess)

Pred(Days(ContractProcess) )

Pred(Days(ContractProcess))/Days(ContractProcess)

Given the R2, 91% of the variability of the dependent variable DContractProcess is explained by the 6 explanatory variables.

Conclusions• QuantitativeResults:Theresultssupportthehypothesisidentifying

thatthereexistsabaselinesetofcriticalsuccessfactorsdatavariabletoprovideevidence-baseddecision-makinginexpeditingsystemsengineering.

• StudyResults:TheresultsaddressesfindingsspecifictotheESEFprocessbyidentifyingprocessphasetrends.

• DoD AcquisitionDataCollectionPractices:Anextensivesearchwasconductedtofindanappropriatesetofdatatoconductasignificantquantitativestudyacrossmultipleprogramsanddatalimitationscontinuetomakequantitativeanalysisparticularlychallenging.

• EarlySystemsEngineeringPractices:Thepracticesexploredspecificallyfocusedonthecontractawardprocesswithobservationsbasedonexpertopinion.

LevelsofKnowledgeDistribution

Letter

Figure 5: DOD Levels Reviewing Information Requirements

Page 15 GAO-15-192 Acquisition Reform

Many different functional organizations within each level review the information before the document is approved. The number of organizations conducting reviews varies depending on the information included in each document. A few documents that include a wide breadth of information can be reviewed by many offices at each level. For example, Air Force acquisition strategies, that on average took over 12 months to complete for the programs we surveyed, can be reviewed by

FinalRecommendations• Anearlierunderstandingofthewidersystemicviewofthe

missionobjectivecanprovideawiderandmoreeffectiverangeoftruealternatives(trade-space)inearlysystemsengineeringprocesses.

• Proposedsolutionsapprovedatlowerlevelscanhavebelatedrejectionsanddelayslaterintheprocess,thereforeearlycommunicationbetweenhigherauthoritativelevelsandtheprogramofficeisrecommended.

• Identificationofthespecificdangersinexpeditingpredecessorphasesandconcurrentlyaccomplishingtasksinmultiplephasestowillprovideanimprovedunderstandingofrisksandopportunities.