Analyzing Past Trends Predict Future - DAU Full...employee’s official duties or was done on government time. If the research effort is performed as part of official duties or on

Analyzing Past Trendsto Predict Future

ACQUISITION OUTCOMES

July 2017 Vol. 24 No. 3 | ISSUE 82

Article ListARJ Extra

Analyzing Cost Growth at Program Stagesfor DoD Aircraft Capt Scott J. Kozlak, USAF, Edward D. White,Jonathan D. Ritschel, Lt Col Brandon Lucas, USAF,and Michael J. Seibel

Estimating Firm-Anticipated Defense Acquisition Costs with a Value-Maximizing Framework LTJG Sean Lavelle, USN

Informing Policy through Quantification of the Intellectual Property Lock-in Associated with DoD AcquisitionMaj Christopher Berardi, USAF, Bruce Cameron,and Ed Crawley

The Impact of a Big Data Decision Support Tool on Military Logistics: Medical Analytics Meets the Mission Felix K. Chang, Christopher J. Dente,and CAPT Eric A. Elster, USN

Online-only Article Beyond Integration Readiness Level (IRL): A Multidimensional Framework to Facilitate the Integration of System of SystemsMaj Clarence Eder, USAF (Ret.), Thomas A. Mazzuchi, and Shahram Sarkani

Online-only Article Effectiveness Test and Evaluation of Non-lethal Weapons in Crowd Scenarios: Metrics, Measures, and Design of Experiments Elizabeth Mezzacappa, Gordon Cooke, Robert M. DeMarco, Gladstone V. Reid, Kevin Tevis, Charles Sheridan, Kenneth R. Short, Nasir Jaffery, and John B. Riedener

The Defense Acquisition Professional Reading List Destructive Creation: American Business andthe Winning of World War II Written by Mark R. Wilson andreviewed by Benjamin Franklin Cooling

Mr. James A. MacStravicPerforming the duties of Under Secretary of Defense for Acquisition, Technology, and Logistics

Mr. James P. WoolseyPresident, Defense Acquisition University

Editorial BoardDr. Larrie D. Ferreiro

Chairman and Executive Editor

Mr. Richard AltieriDwight D. Eisenhower School for NationalSecurity and Resource Strategy

Dr. Michelle BaileyDefense Acquisition University

Dr. Don BirchlerCenter for Naval Analyses Corporation

Mr. Kevin BuckThe MITRE Corporation

Mr. John CannadayDefense Acquisition University

Dr. John M. ColombiAir Force Institute of Technology

Dr. Richard DonnellyThe George Washington University

Dr. William T. EliasonDwight D. Eisenhower School for National Security and Resource Strategy

Dr. J. Ronald FoxHarvard Business School

Mr. David GallopDefense Acquisition University

Dr. Jacques GanslerUniversity of Maryland

RADM James Greene, USN (Ret.)

Naval Postgraduate School

Dr. Mike KotzianDefense Acquisition University

Dr. Craig LushDefense Acquisition University

Dr. Troy J. MuellerThe MITRE Corporation

Dr. Christopher G. PerninRAND Corporation

Dr. Mary C. RedshawDwight D. Eisenhower School for National Security and Resource Strategy

Dr. Yvette RodriguezDefense Acquisition University

Dr. Richard ShipeDwight D. Eisenhower School for National Security and Resource Strategy

Dr. Keith SniderNaval Postgraduate School

Dr. John SnoderlyDefense Acquisition University

Ms. Dana StewartDefense Acquisition University

Dr. David M. TateInstitute for Defense Analyses

Dr. Trevor TaylorCranfield University (UK)

Mr. Jerry VandewieleDefense Acquisition University

ISSN 2156-8391 (print) ISSN 2156-8405 (online)DOI: https://dx.doi.org/10.22594/dau.072017-82.24.03

The Defense Acquisition Research Journal, formerly the Defense Acquisition Review Journal, is published quarterly by the Defense Acquisition University (DAU) Press and is an official publication of the Department of Defense. Postage is paid at the U.S. Postal facility, Fort Belvoir, VA, and at additional U.S. Postal facilities. Postmaster, send address changes to: Editor, Defense Acquisition Research Journal, DAU Press, 9820 Belvoir Road, Suite 3, Fort Belvoir, VA 22060-5565. The journal-level DOI is: https://dx.doi.org/10.22594/dauARJ.issn.2156-8391. Some photos appearing in this publication may be digitally enhanced.

Articles represent the views of the authors and do not necessarily reflect the opinion of DAU or the Department of Defense.

Director, Visual Arts & Press Randy Weekes

Managing Editor Deputy Director,

Visual Arts & PressNorene L. Taylor

Assistant Editor Emily Beliles

Production Manager,Visual Arts & Press Frances Battle

Lead Graphic Designer Michael Krukowski

Graphic Designer, Digital Publications Nina Austin

Technical Editor Collie J. Johnson

Associate Editor Michael Shoemaker

Copy Editor/Circulation Manager Debbie Gonzalez

Multimedia Assistant Noelia Gamboa

Editing, Design, and Layout The C3 Group &Schatz Publishing Group

CONTENTS | Featured Research

A Publication of the Defense Acquisition University July 2017 Vol. 24 No. 3 ISSUE 82

p. 386Analyzing Cost Growth at Program Stages for DoD Aircraft Capt Scott J. Kozlak, USAF, Edward D. White, Jonathan D. Ritschel, Lt Col Brandon Lucas, USAF, and Michael J. Seibel

This research examines cost growth factors from Milestone B to various program stages for 30 Department of Defense aircraft programs.

p. 408Estimating Firm-Anticipated Defense Acquisition Costs with a Value-Maximizing FrameworkLTJG Sean Lavelle, USN

The author uses a value-maximizing framework to predict how firms will bid under varying levels of risk sharing, allowing the government to estimate future costs more accurately.

Featured Research

p. 432Informing Policy through Quantification of the Intellectual Property Lock-in Associated with DoD Acquisition Maj Christopher Berardi, USAF, Bruce Cameron, and Ed Crawley

This article introduces a quantitative analysis of intellectual property lock-in trends in DoD acqui-sition. The analysis ultimately quantifies the magnitude of the problem, illustrates trends across 8 fiscal years, and correlates lock-in to internal research and development funding.

p. 468The Impact of a Big Data Decision Support Tool on Military Logistics: Medical Analytics Meets the MissionFelix K. Chang, Christopher J. Dente, and CAPT Eric A. Elster, USN

This study uses a combat simulation to demon-strate how military organizations not directly involved in logistics can use decision support tools to streamline their logistics operations. It further quantifies the benefits that one such tool designed for the military medica l community might generate.


A Publication of the Defense Acquisition University July 2017 Vol. 24 No. 3 ISSUE 82

Beyond Integration Readiness Level (IRL): A Multidimensional Framework to Facilitate the Integration of System of SystemsMaj Clarence Eder, USAF (Ret.), Thomas A. Mazzuchi, and Shahram Sarkani

Data analyses of research aimed at understanding major integration issues of DoD Space Systems has been validated by experts, resulting in development of an integration assessment framework to help assess Integration Readiness Level of System of Systems.

http://www.dau.mil/library/arj

Effectiveness Test and Evaluation of Non-lethal Weapons in Crowd Scenarios: Metrics, Measures, and Design of ExperimentsElizabeth Mezzacappa, Gordon Cooke, Robert M. DeMarco, Gladstone V. Reid, Kevin Tevis, Charles Sheridan, Kenneth R. Short, Nasir Jaffery, and John B. Riedener

This article describes methods for quantitative metrics and analyses for test and evaluation of non-lethal weapons. Results from human testing are also presented.

http://www.dau.mil/library/arj

p. viii2018 Edward Hirsch Aqcuisition and Writing Award Competition

p. xFrom the Chairman and Executive Editor

p. xiiResearch Agenda 2017–2018

p. xviiDAU Alumni Association

p. 574Professional Reading List

Destructive Creation: American Business and the Winning of World War II Written by Mark R. Wilson and reviewed by Benjamin Franklin Cooling

p. 578New Research in Defense Acquisition

A selection of new research curated by the DAU Research Center and the Knowledge Repository.

p. 586Defense ARJ Guidelines for Contributors

The Defense Acquisition Research Journal (ARJ) is a scholarly peer-reviewed journal published by the Defense Acquisition University. All submissions receive a blind review to ensure impartial evaluation.

p. 594Call for Authors

We are currently soliciting articles and subject matter experts for the 2017–2018 Defense ARJ print years.

p. 596Defense Acquisition University Website

Your online access to acquisition research, consulting, information, and course offerings.


Featured Research

DAU ALUMNI ASSOCIATION

2018 EDWARD HIRSCHACQUISITION AND WRITING AWARD COMPETITION

CALL FOR PAPERSResearch topics may include:• Improving Professionalism of the

Total Acquisition Workforce• Career Path and Incentives• Agile Program Management • Agile Software Development• Incorporating Foreign Military

Sales and Direct Contractor Sales Strategies into ProgramsCon-

trolling Costs Throughout the Product Life Cycle

• Acquisition of Defense Business Systems

• Emerging changes to EVM• System Cyber Hardness• Cyber Training and Concepts• Services Management• Should Cost Management

GROUND RULES

• The competition is open to anyone interested in the DoD acquisition sys-tem and is not limited to government or contractor personnel.

• Employees of the federal government (including military personnel) are en-couraged to compete and are eligible for cash awards unless the paper was researched or written as part of the employee’s official duties or was done on government time. If the research effort is performed as part of official duties or on government time, the employee is eligible for a non-cash prize, i.e., certificate and donation of cash prize to a Combined Federal Campaign-registered charity of win-ner’s choice.

• First prize is $1,000. Second and third prizes, if awarded, are each $500.

• The format of the paper must be in accordance with guidelines for articles submitted for publication in the Defense Acquisition Research Journal.

• Papers are to be submitted to the DAU Director of Research: [email protected].

• Papers will be evaluated by a pan-el selected by the DAUAA Board of Directors and the DAU Director of Research.

• Award winners will present their papers at the DAU Acquisition Community Training Symposium, Tuesday, April 3, 2018, at the DAU Fort Belvoir campus.

• Papers must be submitted by De-cember 15, 2017, and awards will be announced in January 2018.

FROM THE CHAIRMAN AND

EXECUTIVE EDITOR

Dr. Larrie D. Ferreiro

The theme for this edition of Defense Acquisition Research Journal is “Using Past Trends to Predict Future Acquisition Outcomes.” The first article is “Analyzing Cost Growth at Program Stages for DoD Aircraft” by Scott J. Kozlak et al. They analyzed 30 military aircraft programs to determine when cost growth occurred during the acquisition and development cycle, and developed some unique and use-ful insights that future programs can use

for prediction. The next article by Sean Lavelle, “Estimating Firm-Anticipated Defense Acquisition Costs with a Value-Maximizing Framework,” uses a value-maximizing framework to predict how firms will bid under varying levels of risk sharing, allowing the gov-ernment to estimate future costs more accurately.

Following this is “Informing Policy through Quantification of the Intellectual Property Lock-in Associated with DoD Acquisition,” by Christopher Berardi, Bruce Cameron, and Ed Crawley, which quantitatively analyzes intellectual property lock-in trends in DoD acquisition and their correlation to internal research and devel-opment funding. Then, Felix K. Chang, Christopher J. Dente, and Eric A. Elster, in “The Impact of a Big Data Decision Support Tool on Military Logistics: Medical Analytics Meets the Mission,” describe a combat simulation tool that showed how to reduce the logistical footprint for blood resupply in a military theatre of operations.

A Publication of the Defense Acquisition University http://www.dau.mil

xii

This issue has two online-only papers. First, “Beyond Integration Readiness Level (IR L): A Multidimensiona l Framework to Facilitate the Integration of System of Systems” by Clarence Eder, Thomas A. Mazzuchi, and Shahram Sarkani, looks at expand-ing the current acquisition practice of characterizing systems by their Technology Readiness Level (TRLs) by using the concept of Integration Readiness Level (IRLs) to address growing inte-gration challenges of System-of-Systems acquisition programs. The second, by Elizabeth Mezzacappa and her co-authors, is titled “Effectiveness Testing and Evaluation of Non-lethal Weapons in Crowd Scenarios: Metrics, Measures, and Design of Experiments.” As the name implies, it discusses test and evaluation methods for benchmarking and comparison of non-lethal weapons intended for use in crowd management situations, the results of which can be used for Analysis of Alternatives and trade-space studies.

The featured book in this issue’s Defense Acquisition Professional Reading List is Destructive Creation: American Business and the Winning of World War II by Mark R. Wilson, reviewed by Dr. Benjamin Franklin Cooling of the National Defense University.

Finally, there are several changes to the Defense ARJ masthead. Sharp-eyed readers will have noticed over the past year that the Research Advisory Board, which had been established to review and provide direction for the research agenda and publications, had been steadily diminishing in size as many of the Board mem-bers departed their senior-level positions during the last year of the previous administration. At the same time, the responsibilities for providing direction to defense acquisition research has been increasingly borne by the Editorial Board, whose makeup is now two-thirds non-Defense Acquisition University members (including international representation). This Editorial Board arrangement now brings the same type of outside experience and perspectives as did the Research Advisory Board. This has led us to disestablish the Research Advisory Board, with its former functions now subsumed by the Editorial Board.

Dr. Mary Redshaw, the last “surviving” member of the Research Advisory Board, has now joined the Editorial Board. Dr. Yvette Rodriguez has also joined, while Dr. Andre Murphy has departed. We thank the former members for their service, and welcome the new ones to continue the strong tradition of advancing the state of knowledge in the defense acquisition community.

July 2017

xiii

This Research Agenda is intended to make researchers aware of the topics that are, or should be, of particular concern to the broader defense acquisition community within the federal government, academia, and defense industrial sectors. The center compiles the agenda annually, using inputs from subject matter experts across those sectors. Topics are periodically vetted and updated by the DAU Center’s Research Advisory Board to ensure they address current areas of strategic interest.

The purpose of conducting research in these areas is to provide solid, empirically based findings to create a broad body of knowl-edge that can inform the development of policies, procedures, and processes in defense acquisition, and to help shape the thought lead-ership for the acquisition community. Most of these research topics were selected to support the DoD’s Better Buying Power Initiative (see http://bbp.dau.mil). Some questions may cross topics and thus appear in multiple research areas.

Potential researchers are encouraged to contact the DAU Director of Research ([email protected]) to suggest additional research questions and topics. They are also encouraged to contact the listed Points of Contact (POC), who may be able to provide general guidance as to current areas of interest, potential sources of infor-mation, etc.

DAU CENTER FOR DEFENSE ACQUISITION

RESEARCH AGENDA 2017–2018


xiv

Competition POCs • John Cannaday, DAU: [email protected]

• Salvatore Cianci, DAU: [email protected]

• Frank Kenlon (global market outreach), DAU: [email protected]

Measuring the Effects of Competition • What means are there (or can be developed) to measure

the effect on defense acquisition costs of maintaining the defense industrial base in various sectors?

• What means are there (or can be developed) of mea-suring the effect of utilizing defense industria l infrastructure for commercial manufacture, and in particular, in growth industries? In other words, can we measure the effect of using defense manufacturing to expand the buyer base?

• What means are there (or can be developed) to deter-mine the degree of openness that exists in competitive awards?

• What are the different effects of the two best value source selection processes (trade-off vs. lowest price technically acceptable) on program cost, schedule, and performance?

Strategic Competition• Is there evidence that competition between system

portfolios is an effective means of controlling price and costs?

• Does lack of competition automatically mean higher prices? For example, is there evidence that sole source can result in lower overall administrative costs at both the government and industry levels, to the effect of lowering total costs?

• What are the long-term historical trends for compe-tition guidance and practice in defense acquisition policies and practices?

July 2017

xv

• To what extent are contracts being awarded non-competitively by congressional mandate for policy interest reasons? What is the effect on contract price and performance?

• What means are there (or can be developed) to deter-mine the degree to which competitive program costs are negatively affected by laws and regulations such as the Berry Amendment, Buy American Act, etc.?

• The DoD should have enormous buying power and the ability to influence supplier prices. Is this the case? Examine the potential change in cost performance due to greater centralization of buying organizations or strategies.

Effects of Industrial Base • What are the effects on program cost, schedule, and

performance of having more or fewer competitors? What measures are there to determine these effects?

• What means are there (or can be developed) to measure the breadth and depth of the industrial base in various sectors that go beyond simple head-count of providers?

• Has change in the defense industrial base resulted in actual change in output? How is that measured?

Competitive Contracting • Commercial industry often cultivates long-term, exclu-

sive (noncompetitive) supply chain relationships. Does this model have any application to defense acquisition? Under what conditions/circumstances?

• What is the effect on program cost, schedule, and performance of awards based on varying levels of competition: (a) “Effective” competition (two or more offers); (b) “Ineffective” competition (only one offer received in response to competitive solicitation); (c) split awards vs. winner take all; and (d) sole source.


xvi

Improve DoD Outreach for Technology and Products from Global Markets

• How have militaries in the past benefited from global technology development?

• How/why have militaries missed the largest techno-logical advances?

• What are the key areas that require the DoD’s focus and attention in the coming years to maintain or enhance the technological advantage of its weapon systems and equipment?

• What types of efforts should the DoD consider pursu-ing to increase the breadth and depth of technology push efforts in DoD acquisition programs?

• How effectively are the DoD’s global science and tech-nology investments transitioned into DoD acquisition programs?

• Are the DoD’s applied research and development (i.e., acquisition program) investments effectively pursuing and using sources of global technology to affordably meet current and future DoD acquisition program requirements? If not, what steps could the DoD take to improve its performance in these two areas?

• What are the strengths and weaknesses of the DoD’s global defense technology investment approach as compared to the approaches used by other nations?

• What are the strengths and weaknesses of the DoD’s global defense technology investment approach as compared to the approaches used by the private sector—both domestic and foreign entities (compa-nies, universities, private-public partnerships, think tanks, etc.)?

• How does the DoD currently assess the relative benefits and risks associated with global versus U.S. sourcing of key technologies used in DoD acquisition programs? How could the DoD improve its policies and procedures in this area to enhance the benefits of global technology sourcing while minimizing potential risks?

July 2017

xvii

• How could current DoD/U.S. Technology Security and Foreign Disclosure (TSFD) decision-making policies and processes be improved to help the DoD better bal-ance the benefits and risks associated with potential global sourcing of key technologies used in current and future DoD acquisition programs?

• How do DoD primes and key subcontractors currently assess the relative benefits and risks associated with global versus U.S. sourcing of key technologies used in DoD acquisition programs? How could they improve their contractor policies and procedures in this area to enhance the benefits of global technology sourcing while minimizing potential risks?

• How could current U.S. Export Control System deci-sion-making policies and processes be improved to help the DoD better balance the benefits and risks associated with potential global sourcing of key tech-nologies used in current and future DoD acquisition programs?

Comparative Studies • Compare the industrial policies of military acquisition

in different nations and the policy impacts on acquisi-tion outcomes.

• Compare the cost and contract performance of highly regulated public utilities with nonregulated “natu-ral monopolies,” e.g., military satellites, warship building, etc.

• Compare contracting/competition practices between the DoD and complex, custom-built commercial prod-ucts (e.g., offshore oil platforms).

• Compare program cost performance in various market sectors: highly competitive (multiple offerors), limited (two or three offerors), monopoly?

• Compare the cost and contract performance of mil-itary acquisition programs in nations having single “purple” acquisition organizations with those having Service-level acquisition agencies.


xviii

DAU ALUMNI ASSOCIATIONJoin the Success Network!

The DAU Alumni Association opens the door to a worldwide network of Defense Acquisition University graduates, faculty, staff members, and defense industry representatives—all ready to share their expertise with you and benefit from yours.Be part of a two-way exchange of information with other acquisition professionals.

• Stay connected to DAU and link to other professional organizations. • Keep up to date on evolving defense acquisition policies and developments

through DAUAA newsletters and the DAUAA LinkedIn Group.• Attend the DAU Annual Acquisition Training Symposium and bi-monthly hot

topic training forums—both supported by the DAUAA—and earn Continuous Learning Points toward DoD continuing education requirements.

Membership is open to all DAU graduates, faculty, staff, and defense industry members. It’s easy to join right from the DAUAA Website at www.dauaa.org, or scan the following QR code:

For more information call 703-960-6802 or 800-755-8805, or e-mail [email protected].

ISSUE 82 JULY 2017 VOL. 24 NO. 3

385We’re on the Web at:

http://www.dau.mil/library/arj 385

Analyzing Cost Growth atPROGRAM STAGES FOR DOD AIRCRAFT

Capt Scott J. Kozlak, USAF, Edward D. White, Jonathan D.

Ritschel, Lt Col Brandon Lucas, USAF, and Michael J. Seibel

This research examines Cost Growth Factors (CGF) at various program stages for 30 Department of Defense aircraft programs. From Milestone (MS) B, the authors determine CGFs at the Critical Design Review (CDR), First Flight (FF), Development Test and Evaluation End, Initial Operational Capability (IOC), and Full Operational Capability. They find development

CGFs are significantly larger than procurement CGFs. Additionally, cost growth primarily occurs early in the program. At CDR, which occurs on average at the 12 percent completion point of a program, aircraft programs had already experienced on average 15 percent of their total program cost growth. The first spike in percent of total cost growth occurs at FF, 35 months or ~3 years after MS B. Lastly, the analysis shows that by IOC (approximately 6.5 years after MS B or 48 percent of program completion), an aircraft program realizes 91 percent of its total cost growth.

DOI: https://doi.org/10.22594/dau.16-763.24.03 Keywords: Cost Growth, Development, Procurement, Analysis, Program Costs

Image designed by Michael Krukowski

388 Defense ARJ, July 2017, Vol. 24 No. 3 : 386-407

Analyzing Cost Growth at Program Stages http://www.dau.mil

Cost growth in weapon systems creates challenges for the Department of Defense (DoD) and often forces difficult decisions regarding acquisition funding. [As noted by Cancian (2010), cost professionals sometimes make a distinction between cost growth and cost overrun; cost growth is more gen-eral and is the term used here.] Such choices could include removing funding from smaller programs, postponing program development, or eliminating programs altogether. To better understand cost growth, the cost community needs to know when cost growth is most likely to occur. This knowledge might allow for better planning of advanced contingencies when a program will need to draw additional funds from another source, more robust baseline estimates to reflect a truly more realistic depiction of the weapon system’s future, or proactive measures in place to mitigate the anticipated spike in cost growth for a program. Supporting this need for knowledge, previous research finds that government decisions account for more than two-thirds of the cost growth experienced in major defense programs (Bolten, Leonard, Arena, Younossi, & Sollinger, 2008). With a total portfolio consisting of 78 programs totaling $1.4 trillion (in Fiscal Year 2015 dollars), even small cost growth percentages have large dollar impacts (Government Accountability Office, 2015). Thus, insight into when cost growth occurs provides valuable data for better informed government decisions that hopefully result in reduced future program costs.

All Major Defense Acquisition Programs (MDAP) are required to pass specific reviews and milestones. Five common stages all major aircraft pro-grams proceed through are: Critical Design Review (CDR), First Flight (FF), Development Test & Evaluation End (DTE), Initial Operational Capability (IOC), and Full Operational Capability (FOC).

389Defense ARJ, July 2017, Vol. 24 No. 3 : 386-407

July 2017

Additionally, the Defense Acquisition System has formal milestones (MS A, MS B, and MS C) that all MDAPs must complete. Although previ-ous research has extensively studied cost growth in MDAPs, no one to our knowledge has analyzed cost growth utilizing program stages as bench-marks. Thus, we assess program cost growth from MS B (official program initiation) to CDR, FF, DTE, IOC, and FOC for the MDAP’s development, procurement, and total acquisition phases. The analysis is limited to DoD aircraft systems as reported in Selected Acquisition Reports (SAR, 2015).

BackgroundSince 1969, MDAPs are required to annually submit SARs to Congress

(Drezner, Jarvaise, Hess, Hough, & Norton, 1993; Porter et al., 2009). SARs outline a weapon system’s status and report current funding estimates as well as actual expenses incurred. Our research utilizes SAR data to evalu-ate program cost estimates and actual costs incurred. Three cost estimates exist within SARs: Planning Estimate (PE), Development Estimate (DE), and Current Estimate (CE) (Calcutt, 1993). PEs are the DoD estimate made during the Concept Exploration and Definition stage (current DoD Instruction 5000.02 now refers to this as Materiel Solution Analysis and Technology Maturation and Risk Reduction phases) of the program life cycle. DEs occur at MS B or the start of Engineering and Manufacturing Development phase of the program life cycle. If a program is complete, the final CE is the actual cost of the program (Calcutt, 1993).

Analysts calculate cost growth from a baseline estimate: the PE, DE, or CE. Typically, the DE at MS B is the baseline estimate for cost growth because an MS B decision results in formal program initiation. As formal cost reports materialize, cost growth becomes easier to track, and it is for this reason the estimator measures cost growth from the DE when possible. Our research investigates cost growth as defined as the increase in cost from the DE to the CE or final estimate of the DoD program

Researchers primarily use two common methods for calculating cost growth. The first method (Equation 1) calculates cost growth as a per-centage of the original cost estimate. In the first method, the estimated cost is subtracted from the actual cost and then divided by the estimated cost (McNichols & McKinney, 1981). The second method (Equation 2) calculates cost growth as a cost growth factor (CGF). The CGF method divides the cost variance (actual) by the estimate (Arena, Leonard, Murray, & Younossi, 2006). A CGF of 1.0 indicates the program did not go over or under the cost estimate, and the actual cost matched the estimated cost.



If the CGF is greater than 1.0, the program sustained growth, calculated as the CGF – 1.0 to determine the percent cost growth. Conversely, if the CGF is less than 1.0, the program did not sustain cost growth; rather, the program cost less than the estimate. For this article, we use CGF (2) to assess cost growth for DoD aircraft programs.

(1)

(2)

(Actual—Estimated)Estimated

ActualEstimated

Previous researchers on DoD MDAP cost growth include Drezner et al. (1993), Christensen (1994), and Arena et al. (2006). Drezner et al. (1993) studied 128 weapon systems utilizing SAR DEs as a baseline. Their research studied CGFs of weapon systems during development, procurement, and total program duration. After accounting for inflation and quantity, they determined individual weapon system cost growth increases on average 2.2 percent per year or about 20 percent through the life of a program. Drezner et al. (1993) also discovered development CGFs were 7 percent greater than procurement CGFs.

Christensen (1994) used Earned Value Management (EVM) data to deter-mine the difference between the original budgeted amount and the estimate at completion. Using EVM in his analysis of cost overrun in DoD weapon systems, Christensen states that cost overruns begin appearing at the 10 percent program completion point. Examining aircraft specific pro-grams, he also discovered that approximately 75 percent of cost overrun occurs by the 50 percent program completion point.

Lastly, the research of Arena et al. (2006) provides information on CGFs for 68 completed programs with similar complexities to programs acquired by the U.S. Air Force. They defined completed weapon systems as systems that have greater than 90 percent production complete. Using SAR reports, Arena et al. divided the data into funding categories, milestones, and commodity type to account for possible changes in correlation with CGFs. The funding categories focused on development and procurement, while the MS category primarily focused on MS B and MS C. The major findings from them include significant cost growth at MS B and MS C, and that completed

391

July 2017

programs reported 46 percent and 16 percent growth, respectively. The two reported CGFs illustrate cost growth bias decreases as a program moves toward completion.

MethodologyOur research uses SAR data to analyze cost growth at five program

stages: CDR, FF, DTE, IOC, and FOC. Specifically, we focus on DoD aircraft programs, which is defined as a fixed-wing, manned aircraft developed for one or more of the U.S. DoD Service branches. Furthermore, our analysis includes only Acquisition Category I (ACAT I) aircraft programs. All ACAT I programs are MDAPs. An MDAP is a program that is not a highly sensitive classified program and that is designated by the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD AT&L) as an MDAP; or that is estimated to require eventual expenditure for research, development, test and evaluation, including all planned increments, of more than $480 million (Fiscal Year 2014 constant dollars) or procurement, including all planned increments, of more than $2.79 billion (Fiscal Year 2014 constant dollars) (Acquisition Category, 2015).

Our research utilizes a database originally built by the RAND Corporation for the Air Force Cost Analysis Agency (AFCAA). This database is populated with SAR data on approximately 330 defense acquisition programs dating back to the 1950s and provides annual funding reports by appropriation as well as calculated cost growth measures (Arena et al., 2006). In addition to the SARs, the AFCAA database also includes significant program dates for DoD aircraft programs: MS B, CDR, FF, DTE, IOC, and FOC. In conjunc-tion with the SAR and AFCAA information, we also use Deagel. Deagel is a nongovernment database that tracks civilian and military aircraft data. If the SAR and AFCAA database lacked a particular stage date for a pro-gram, we reference Deagel. For the B-1B, F-15E, and T-45, we use Deagel’s listed dates for IOC, while for the C-17 we use the listed CDR date. Our final dataset includes 30 DoD aircraft platforms.

We normalize all cost data in order to ensure valid comparisons. The two variables with the biggest effect on cost growth are inflation and order quan-tity (Drezner et al., 1993). The standard approach to account for inflation is to convert all dollars to a single base year value. For each aircraft program, we use the MS B year as the base year for the inflation rates to standard-ize CGFs. We utilize the Office of the Secretary of Defense Comptroller Appropriation (APN) inflation rates to perform these conversions.



Next we normalize for order quantity. SARs list the quantities estimated and produced for each aircraft program. The quantities each aircraft pro-gram produces typically shift throughout a program’s life cycle. In order to standardize the units produced for each aircraft program, the units are standardized to the final production amount. The method used in this article is the same method RAND adopted (Arena et al., 2006). The standardiza-tion process uses learning curves and first unit cost, which are derived from annual funding data provided in each program SAR. If the quantity reported in the baseline estimate is less than the final quantity, we calculate the cost of units not produced and add that value to the baseline estimate. Likewise, if the final quantity produced is less than the baseline estimate, we calculated the estimated cost of additional baseline units and subtract that value from the baseline estimate (Arena et al., 2006)

Lastly, we compute percentage completion of each program. In order to calculate the percentage completion of a program, it is necessary to first identify program completion dates. We use the final reported SAR to sig-nal program completion. The final SAR identifies when all production is complete. However, the Office of the USD (AT&L) can consider terminating the requirement to report SARs when 90 percent of production units are complete or when a program is no longer considered an ACAT I program (SAR, 2015). Because it is uncertain if termination of SAR reports occurs at 90 percent completion or at final production completion, we use the anticipated date of the last production unit completion (in the last SAR report submitted for the program) as the FOC and calculate the percent of completion based off that date.

AnalysisTable 1 displays the complete CGFs for the 30 aircraft weapon systems

in the analysis. The table outlines the acquisition phase: development (DEV), procurement (PROC), or total (TOT), and the program stages: CDR, FF, DTE, IOC, and FOC for each CGF. The blank fields in Table 1 are attributable to either a program not completing a specific stage at the time of this analysis, the program fell below a SAR reporting threshold and no longer required annual reports, or we were unable to find a recorded date for that stage. For example, the F-35 has yet to complete Development Test & Evaluation and the B-1A fell below a reporting threshold in 1978 and was no longer required to make annual SAR reports; therefore, these fields are blank in Table 1.


July 2017



In addition to the CGFs, we calculate the percent of program completion at each stage for all 30 aircraft programs in our database. We plot these CGFs and associated percent program completion in Figures 1–3. Despite the large variability, the points are generally clustered in order of CDR, FF, and finally FOC.

FIGURE 1. DEVELOPMENT COST GROWTH FACTORS BY PERCENT COMPLETE FOR 30 AIRCRAFT PROGRAMS

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

0 20 40 60 80 100

Cost

Grow

th Fa

ctor

Percent Complete— Category = CDR — Category = FF — Category = DTE — Category = IOC — Category = FOC

FIGURE 2. PROCUREMENT COST GROWTH FACTORS BY PERCENT COMPLETE FOR 30 AIRCRAFT PROGRAMS

1

1.2

1.4

1.6

1.8

2

2.2

0 20 40 60 80 100

Cost

Grow

th Fa

ctor


FIGURE 3. TOTAL COST GROWTH FACTORS BY PERCENT COMPLETE FOR 30 AIRCRAFT PROGRAMS

1

1.2

1.4

1.6

1.8

2

2.2

2.4

0 20 40 60 80 100

Cost

Grow

th Fa

ctor


To draw more macro trends, we derive summary descriptive statistics using a statistical discovery software called JMP® (Version 11.2). We examine how many programs and the percentage of these that sustain cost growth; the mean, median, standard deviation, and interquartile range (the differ-ence between the 75 percentile and the 25 percentile); and minimum and maximum CGFs at each stage. Table 2 displays these statistics.

As shown in Table 2, nearly 50 per-cent of the aircraft programs in our

database indicate cost growth at CDR. This percent increases to

approximately 75 percent or more experiencing cost growth at FF, DTE, IOC, or FOC. These trends are similar regardless of acquisition phase. Given this preponderance of cost growth, we focus the remain-ing analysis on those CGFs

greater than 1.0, where cost growth does occur, to further identify macro trends.




July 2017

TABLE 1. COST GROWTH FACTORS (CGF) FOR 30 AIRCRAFT WEAPON SYSTEMS IN THE ANALYSIS

TOT FOC

1.33

0.98

1.20

1.02

1.07

0.87

0.98

1.75

1.43

1.46

0.98

2.15

1.79

1.29

1.30

1.34

1.27

1.43

1.01

1.63

1.07

1.41

1.68

Note. Table 1 outlines the Acquisition Phase: Development (DEV), Procurement (PROC), or Total (TOT), and the Program Stages: Critical Design Review (CDR), First Flight (FF), Development Test and Evaluation End (DTE), Initial Operational Capability (IOC), and Full Operational Capability (FOC) for each CGF. AWACS = Air Warning and Control System; CMUP = Conventional Mission Upgrade Program; Comp = Computer Upgrade; CTOL = Conventional Take-off and Landing; CV = Carrier Variant; Inc = Increment; JDAM = Joint Direct Attack Munition; JSTARS = Joint Surveillance Target Attack Radar System; RERP = Reliability Enhancement and Re-Engining Program; RMP = Radar Modernization Program; RSIP = Radar System Improvement Program.

TOT IOC

1.22

0.91

0.98

1.08

0.88

0.99

1.21

1.47

1.33

1.28

1.45

0.91

2.02

1.05

1.79

1.02

1.19

1.04

1.15

1.31

0.96

1.47

1.69

1.62

1.00

1.02

1.36

1.48

TOT DTE

1.28

1.02

0.99

1.16

0.88

1.07

1.00

1.47

1.30

1.41

1.27

0.90

2.01

1.05

1.60

1.24

1.19

1.04

1.12

1.29

0.96

1.58

1.00

1.04

1.13

1.48

TOT FF

1.22

1.03

1.12

0.99

1.07

0.90

0.89

0.99

1.33

1.08

1.38

1.21

0.87

1.55

1.05

1.48

1.08

1.03

1.03

1.13

1.11

1.01

1.13

1.30

1.63

1.01

1.02

0.99

1.20

TOT CDR

1.03

1.00

0.99

1.00

0.99

0.94

0.93

1.01

1.05

1.00

1.01

1.02

1.06

1.05

0.93

1.08

1.03

1.02

1.00

1.03

1.00

1.05

0.99

1.26

1.34

0.99

0.99

1.10

PROC FOC

1.34

0.92

1.21

0.98

0.95

1.02

1.04

1.72

1.28

2.06

0.90

1.86

1.62

1.18

1.28

1.01

1.08

1.45

1.01

1.62

1.06

1.47

1.70

PROC IOC

1.22

0.86

0.96

1.02

1.05

1.06

1.22

1.45

1.27

1.32

2.05

0.82

1.90

1.02

1.62

0.92

1.23

1.01

1.13

1.35

0.95

1.46

1.82

1.70

0.95

1.00

1.44

1.48

PROC DTE

1.28

0.98

0.96

1.16

1.05

1.14

1.00

1.29

1.31

1.33

1.57

0.81

1.87

1.04

1.62

1.19

1.32

1.01

1.10

1.33

0.96

1.61

0.95

1.02

1.42

1.50

PROC FF

1.22

1.03

1.11

0.98

0.84

1.09

0.99

1.03

1.31

1.09

1.31

1.41

0.78

1.92

1.04

1.53

1.03

1.04

1.00

1.12

1.11

1.02

1.10

1.36

1.66

1.01

1.00

1.00

1.21

PROC CDR

1.03

1.00

1.00

0.99

1.00

1.05

1.00

1.04

1.08

1.00

1.00

0.97

1.92

1.05

0.92

1.03

1.05

1.00

1.00

1.02

1.00

1.03

0.99

1.36

1.36

1.00

1.00

1.10

DEV FOC

1.27

1.30

1.10

1.31

0.95

0.77

0.93

1.81

1.71

1.07

1.12

2.41

2.10

1.83

1.37

1.48

2.51

1.36

0.98

1.64

1.09

0.90

1.53

DEV IOC

1.19

1.20

1.16

1.00

0.77

0.93

1.02

1.54

1.50

1.49

1.07

1.12

2.12

1.04

2.10

1.48

1.09

1.09

1.31

1.15

0.98

1.47

1.53

1.53

1.12

1.08

0.86

1.31

DEV DTE

1.18

1.21

1.17

1.02

0.77

1.02

1.00

1.41

1.26

1.55

1.07

1.11

2.13

1.04

1.48

1.47

1.09

1.09

1.28

1.15

0.98

1.50

1.11

1.10

0.84

1.31

DEV FF

1.19

1.01

1.15

1.05

0.97

0.80

0.81

0.97

1.36

1.06

1.52

1.07

1.12

1.22

1.08

1.38

1.32

0.98

1.07

1.25

1.11

0.95

1.19

1.24

1.50

0.99

1.08

1.02

1.09

DEV CDR

1.09

1.00

0.96

1.05

0.98

0.85

0.88

0.87

1.22

1.00

1.02

1.11

0.98

1.05

0.97

1.32

0.98

1.07

1.00

1.08

0.99

1.12

0.99

1.25

1.24

0.96

1.02

1.07

Aircraft Program

A-10AV-8BB-1AB-1BB-1B CMUP CompB-1B CMUP JDAMB-2 RMPC-5 RERPC-17E-2DE-3A AWACSE-3 AWACS RSIPE-6AE-8 JSTARSEA-18GEF-111AF-14AF-15F-15EF-16A/BF-18A/BF-18E/FF-22F-22 Inc 3.2BF-35 (CTOL)F-35 (CV)P-8AS-3AT-6T-45

395



July 2017

TAB

LE 2

. DE

SCR

IPTI

VE

STA

TIST

ICS

BY

AC

QU

ISIT

ION

PH

AS

E

Max

1.3

2

1.5

2

2.1

3

2.1

2

2.5

1

1.9

2

1.9

2

1.8

7

2.0

5

2.0

6

1.3

4

1.6

3

2.0

1

2.0

2

2.1

5

No

te. T

he

Inte

rqu

arti

le R

ang

e (I

QR

) re

pre

sen

ts t

he

diff

eren

ce b

etw

een

the

75 p

erce

nti

le a

nd

th

e 2

5 p

erce

nti

le.

Min

0.8

5

0.8

0

0.7

7

0.7

7

0.7

7

0.9

2

0.7

8

0.8

1

0.8

2

0.9

0

0.9

3

0.8

7

0.8

8

0.8

8

0.8

7

IQR

0.1

1

0.2

3

0.3

0

0.4

4

0.6

4

0.0

5

0.2

6

0.3

5

0.4

6

0.6

1

0.0

6

0.2

1

0.3

1

0.4

6

0.4

4

Stan

dar

d D

evia

tion

0.1

1

0.1

8

0.2

7

0.3

2

0.4

7

0.1

9

0.2

4

0.2

6

0.3

3

0.3

3

0.0

9

0.1

9

0.2

6

0.2

9

0.3

2

Med

ian

1.0

1

1.0

8

1.13

1.16

1.3

1

1.0

0

1.0

9

1.18

1.2

2

1.2

1

1.0

1

1.0

8

1.15

1.20

1.3

0

Mea

n

1.0

4

1.12

1.2

1

1.26

1.4

1

1.0

7

1.15

1.2

2

1.26

1.2

9

1.0

3

1.13

1.2

1

1.2

5

1.3

2

% P

rog

ram

s w

/ C

ost

Gro

wth

50

%

76%

85

%

82

%

78%

46

%

76%

77

%

75%

83

%

54

%

79%

77

%

75%

83

%

Pro

gra

ms

w/

Cos

t G

row

th

14 22

22

23

18 13 22

20 21

19 15 23

20 21

19

Sam

ple

Si

ze

28

29

26 28

23

28

29

26 28

23

28

29

26 28

23

Cat

egor

y

CD

R

FF

DT

E

IOC

FO

C

CD

R

FF

DT

E

IOC

FO

C

CD

R

FF

DT

E

IOC

FO

C

Pha

se

DE

V

DE

V

DE

V

DE

V

DE

V

PR

OC

PR

OC

PR

OC

PR

OC

PR

OC

TO

TAL

TO

TAL

TO

TAL

TO

TAL

TO

TAL



Table 3 shows the mean and median CGFs for the five program stages (CDR through FOC) and their associated acquisition phases. Table 4 lists the mean and median percent complete and associated percent of total cost growth for each stage. (Note: 100 percent at FOC does not mean that the program has realized 100 percent cost growth; instead, it indicates that whatever cost growth the program will achieve, 100 percent of that cost growth is realized by FOC.) Equation (3) displays the formula necessary to calculate percent of total cost growth, where Stage serves as either the CDR, FF, DTE, IOC, or FOC stage. Table 4 also lists the average months from MS B and associated percent of total cost growth at each stage, again for both the means and medians.

(3)(Stage—1)(FOC—1)

TABLE 3. MEAN AND MEDIAN COST GROWTH FACTOR (CGF) AT TOTAL ACQUISITION PROGRAM PHASES

Program Stage Mean CGF Median CGF

DEV CDR 1.12 1.09

DEV FF 1.19 1.14

DEV DTE 1.26 1.18

DEV IOC 1.34 1.20

DEV FOC 1.56 1.43

PROC CDR 1.16 1.05

PROC FF 1.22 1.11

PROC DTE 1.31 1.30

PROC IOC 1.37 1.32

PROC FOC 1.37 1.28

TOTAL CDR 1.08 1.05

TOTAL FF 1.18 1.12

TOTAL DTE 1.29 1.26

TOTAL IOC 1.35 1.31

TOTAL FOC 1.40 1.34

Note. CDR = Critical Design Review; DEV = Development; FF = First Flight; DTE = Development Test and Evaluation End; FOC = Full Operational Capability; IOC = Initial Operational Capability; PROC = Procurement.

397

July 2017

TABLE 4. AVERAGE TIME COMPLETE (PERCENT AND MONTHS) AND AVERAGE PERCENT COST GROWTH AT TOTAL ACQUISITION

PROGRAM PHASES

Program Stage

Mean % Complete

Mean Months Complete

Median % Complete

Median Months Complete

Mean % Cost Growth

Median % Cost Growth

DEV CDR 13 24.1 12 17.2 22 20

DEV FF 26 43.6 25 34.5 33 32

DEV DTE 49 81.3 44 74.1 47 41

DEV IOC 51 88.9 48 78.1 60 47

DEV FOC 100 185.8 100 176.0 100 100

PROC CDR 13 24.1 12 17.2 44 18

PROC FF 26 43.6 25 34.5 59 39

PROC DTE 49 81.3 44 74.1 83 107

PROC IOC 51 88.9 48 78.1 101 114

PROC FOC 100 185.8 100 176.0 100 100

TOTAL CDR 13 24.1 12 17.2 19 15

TOTAL FF 26 43.6 25 34.5 45 35

TOTAL DTE 49 81.3 44 74.1 72 75

TOTAL IOC 51 88.9 48 78.1 86 91

TOTAL FOC 100 185.8 100 176.0 100 100

Note. CDR = Critical Design Review; DEV = Development; FF = First Flight; DTE = Development Test and Evaluation End; FOC = Full Operational Capability; IOC = Initial Operational Capability; PROC = Procurement.



FIGURE 4. MEDIAN PERCENT TOTAL COST GROWTH VERSUS MEDIAN PERCENT PROGRAM COMPLETION

Perc

ent T

otal

Cost

Grow

th

DevelopmentProcurement Total

0

0.2

0.4

0.6

0.8

1

1.2

0 20 40 60 80 100

Percent Program Completion

Note. Applies to development, procurement, and total acquisition phases for 30 aircraft programs included in the database.

FIGURE 5. MEDIAN PERCENT TOTAL COST GROWTH VERSUSMEDIAN MONTHS FROM MS B

0

0.2

0.4

0.6

0.8

1

1.2

0 50 100 150 200Medi

an P

erce

nt Co

st Gr

owth

Median MonthsDevelopmentProcurement Total

Note. Applies to the development, procurement, and total acquisition phases for the 30 aircraft programs in the database.

399

July 2017

Due to some large CGFs affecting the means as seen in Table 2, we primar-ily address the median values from Tables 3–4 and macro trends of those medians illustrated in Figures 4–5 in the following analysis. Investigating just the FOC stage in Table 3, the median CGFs for the development, pro-curement, and total acquisition phases are 1.43, 1.28, and 1.34, respectively. Therefore, we find that the median CGF for the development phase is sig-nificantly greater than the CGF for either the procurement phase or total acquisition phase. With respect to program cost growth, these CGFs cor-respond to 43 percent, 28 percent, and 34 percent total cost growth for the development, procurement, and total acquisition phases.

According to Table 4, the four program stages of CDR, FF, DTE, and IOC all occur before 50 percent schedule completion. Since IOC is typically the last stage (sometimes DTE has a later date) with a median percent comple-tion of 48 percent, we further analyze these results. For the procurement phase, we find the median CGF represents 114 percent of total realized cost growth at IOC. Thus, the procurement phase realizes all of its cost growth by IOC, despite IOC representing only the 48 percent program completion point. This is not all that surprising given development is mainly complete, and most actual production costs are collected and understood by the time full rate production begins. Additionally, we find that the median CGF at IOC (114 percent) is greater than the median CGF at FOC (100 percent). Visually, this peak is seen in Figure 4. A similar trend holds for the total acquisition phase, whereby 91 percent of total realized cost growth occurs at IOC. This trend for total is not as pronounced as procurement since it also accounts for development costs, which has a much smaller percent of total realized cost growth of 47 percent at IOC. With respect to actual time at IOC, Table 4 shows the median time from MS B to IOC is 78.1 months or 6.5 years. Overall, the descriptive analysis suggests that by IOC, which typically occurs 6.5 years after MS B, a program realizes 91 percent of its total cost growth.

Besides investigating trends seen at IOC, we also assess descriptive trends at CDR, FF, and DTE. For both CDR and FF, the median percent of total realized cost growth for development, procurement, and total acquisition are relatively similar, unlike those at IOC. The percentages for CDR are 20 percent, 18 percent, and 15 percent, while for FF the percentages are 32 percent, 39 percent, and 35 percent, respectively. For DTE, these per-centages start to diverge with median development, procurement, and total acquisition program cost growth at 41 percent, 107 percent, and 75 percent, respectively. Thus, DTE is found to be a pivot point for cost growth.



Investigating the association of median cost growth percentage to median schedule completion percentage at CDR, we find they are relatively com-parable (Table 4). For CDR, the median schedule completion percentage is 12 percent, while the median percentages of total realized cost growth for development, procurement, and total acquisition are 20 percent, 18 percent, and 15 percent, respectively. For FF, this association is again somewhat comparable, but starts to weaken. At FF, the median schedule completion percentage is 25 percent, while the median percentages of total realized cost growth for development, procurement, and total acquisition are 32 percent, 39 percent, and 35 percent, respectively. At DTE, this association further weakens. The median percent of total cost growth is 75 percent, while the total percent of program completion is 44 percent. This is primarily because of the rapid increase of procurement cost. Why this occurs, we do not know; indeed, such rapid increase of procurement cost might suggest an area for future studies and investigation. As discussed previously regarding IOC, 91 percent of total cost growth occurs at 48 percent schedule completion. Overall, we see a steep rise in percent of total cost growth between FF and IOC, primarily attributable to procurement cost.

Investigating procurement costs further, we find the following trends. Median percent of total cost growth at

CDR is 18 percent, while median percent of program com-pletion is 12 percent. At FF, the percent of total cost growth

is 39 percent and median percent of program completion is 25 percent. It is here at FF that the percent of total cost

growth begins to increase more rap-idly than percent of program

completion. At DTE, percent of total cost growth is 107 percent and

percent of program completion is 44 per-cent. At IOC, percent of total cost growth

is 114 percent at 48 percent program com-pletion. As seen in either Figure 4 or 5,

procurement experiences a large increase in percent of total cost growth around

DTE and IOC.

Development percent of total cost grow th does not behave the

same way as procurement cost growth (Table 4). At CDR,

median percent of total cost

401

July 2017

growth is 20 percent at 12 percent program completion. FF percent total cost growth is 32 percent at a program completion percentage of 25 percent. For both CDR and FF, the percent of total cost growth compared to percent of program completion is not too different, ~7–8 percent. However, at DTE the percent of total cost growth is 41 percent and at IOC the percent of total cost growth is 47 percent. Both of these percentages of total cost growth are less than the percent of program completion and far less than the percent of total cost growth experienced with procurement at the same reviews.

Overall, the descriptive analysis highlights some macro trends. Procurement and total program cost growth display similar trends. Both experience the majority of their cost growth prior to the program being 50 percent com-plete. At IOC, median percent of total realized cost growth is 91 percent at 48 percent program completion with respect to overall total acquisition cost. For procurement acquisition only, 114 percent of total realized cost growth occurs at 48 percent program completion. The development phase is different. For development acquisition, only 47 percent of total program cost growth occurs at 48 percent program completion. Additionally, for procurement and total acquisition costs, a large spike in median percent of total program cost growth occurs around FF whereas development cost growth follows a steadier, more linear path.

ConclusionsBuilding a database of 30 aircraft programs comprised of informa-

tion gathered from the SARs, we investigated how CGFs change from CDR to FOC for development, procurement, and total acquisition phases. Despite there being much variability from program to program, as evi-dent in Figures 1–3, noticeable trends soon emerge as we aggregate the data to means and medians. As seen in Tables 2 and 3, over half to three-fourths of the aircraft programs experience median cost growth ranging from 28 percent for procurement to 43 percent for development. Thus, we find that development CGFs are significantly larger than procurement CGFs. These results are comparable to Drezner et al. (1993), who discov-ered development CGFs were 7 percent greater than procurement CGFs. For our database, the average difference was approximately 15 percent.

As previously discussed, the median percent of program completion at IOC is 48 percent and the median percent of total cost growth for total acquisition is 91 percent. Therefore, we identify the CGF of an aircraft program at IOC to be very close to the CGF at program completion.



When comparing development to procurement, procurement is the primary contributor to overall cost growth. With respect to when this cost growth occurs, the major spike in cost growth occurs between FF and DTE. At FF, the median percent of total cost growth is 35 percent at 25 percent program completion. When looking at DTE, median total cost growth is 75 percent at 44 percent program completion. Thus, at DTE, there is a major spike in percent of total cost growth, which could be attributed to a program actu-ally needing to display some capability for the aircraft. From Figure 3, we see DTE does not necessarily occur before IOC for every program. Because of this, DTE can occur after IOC depending on where IOC is identified in a program’s Capability Development Document (CDD). Due to shifts in IOC, the point of greatest CGF could occur at DTE or IOC.

Investigating further into development and procurement, we find signifi-cantly different results for percent of total cost growth versus percent of program completion. For development, median percent of total cost growth at IOC is 47 percent whereas median percent total cost growth for procure-ment is 114 percent. With this information, we are likely to see development cost growth after IOC, but do not expect to see any procurement cost growth after IOC. Although not always, as first noted in Figure 1, IOC typically occurs after DTE. Therefore, this would indicate minimal development costs accrued after IOC. That is not what the data indicated. This is there-fore another area for future research to delve into this counterintuitive finding. The median CGFs for procurement costs spike at DTE and IOC, then slowly decrease at FOC. A similar trend holds for total acquisition costs, but the FOC CGFs are slightly higher due to development costs being more linear during a program’s schedule.

403

July 2017

With respect to possible limitations, we noticed some discrepancies when identifying IOC dates. This is because programs are not required to report IOC at a certain point in the program’s schedule. The Defense Acquisition University’s ACQuipedia (Initial Operational Capability, 2015) defines IOC:

In general, attained when some units and/or organizations in the force structure scheduled to receive a system 1) have received it, and 2) have the ability to employ and maintain it. The specifics for any particular system IOC are defined in that system’s CDD and Capability Production Document.

IOC dates reported by aircraft programs in this research are consistent with the DAU definition, where some aircraft programs report IOC earlier in the schedule than other aircraft programs. This inconsistency of report-ing could affect our findings. However, given the magnitude of median cost growth at IOC, we doubt our findings would change drastically.

Overall, our research quantified the CGFs for 30 aircraft programs at CDR, FF, DTE, IOC, and FOC. We determined the median CGFs at FOC for development, procurement, and total acquisition to be 1.43, 1.28, and 1.34, respectively. These results are comparable to previous findings. Arena et al. (2006) found total CGFs for development, procurement, and total to be 1.58, 1.44, and 1.46 respectively, while Drezner et al. (1993) found total CGFs for development, procurement, and total to be 1.25, 1.18, and 1.20, respectively. Additionally, Christensen (1994) uses EVM data to identify cost overrun beginning as early as 10 percent of program completion. Consistent with Christensen’s findings, we identify a median cost growth of 15 percent for total acquisition costs at CDR, which occurs at a median program completion of 12 percent. Additionally, the first spike in percent of total cost growth occurs at FF—35 months or approximately 3 years after MS B. Lastly, our analysis identifies the amount of time from MS B to each program stage. The median time from MS B to IOC is 78 months or 6.5 years. Therefore, approximately 6.5 years after MS B, a program sustains about 91 percent of its total program cost growth. We submit that understanding a typical cost growth pattern of an aircraft program allows for better timing of program initiation to mitigate funding poaching from other systems when or if cost growth spikes.



ReferencesAcquisition Category. (2015). In Defense Acquisition University ACQuipedia. Retrieved

from https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx?aid=a896cb8a-92ad-41f1-b85a-dd1cb4abdc82

Arena, M. V., Leonard, R. S., Murray, S. E., & Younossi, O. (2006). Historical cost growth of completed weapon system programs (Report No. TR-343-AF). Santa Monica, CA: RAND.

Bolten, J. G., Leonard, R. S., Arena, M. V., Younossi, O., & Sollinger, J. M. (2008). Sources of weapon system cost growth (Report No. MG-670-AF). Santa Monica, CA: RAND.

Calcutt, H. M., Jr. (1993). Cost growth in DoD major programs: A historical perspective. (Executive Research Project, Report No. NDU-ICAF-93-F31). Washington, DC: Industrial College of the Armed Forces, National Defense University.

Cancian, M. F. (2010). Cost growth: Perception and reality. Defense Acquisition Review Journal, 17(3), 389–404.

Christensen, D. S. (1994, Winter). Cost overrun optimism: Fact or fiction? Acquisition Review Quarterly, 1, 25–38.

Drezner, J. A., Jarvaise, J. M., Hess, R. W., Hough, P. G., & Norton, D. (1993). An analysis of weapon system cost growth (Report No. MR-291-AF). Santa Monica, CA: RAND.

Government Accountability Office (2015). Defense acquisitions: Assessment of selected weapon programs (Report No. GAO-15-342SP). Washington, DC: U.S. Government Printing Office.

Initial Operational Capability. (2015). In Defense Acquisition University ACQuipedia. Retrieved from https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx?aid= 87a753b2-99cf-4e63-94dc-9ceab06fc96c

McNichols, G. R., & McKinney, B. J. (1981, October). Analysis of DoD weapon system cost growth using selected acquisition reports. Paper presented at the Sixteenth Annual Department of Defense Cost Analysis Symposium, Arlington, VA.

Porter, G., Gladstone, B., Gordon, C. V., Karvonides, N., Kneece Jr., R. R., Mandelbaum, J., & O’Neil, W. D. (2009). The major causes of cost growth in defense acquisitions (IDA Report No. P-4531). Alexandria, VA: Institute for Defense Analyses.

Selected Acquisition Report. (2015). In AcqNotes [Online encyclopedia]. Retrieved from http://acqnotes.com/acqnote/acquisitions/selected-acquisition-report-sar

405

July 2017

Author Biographies

Capt Scott J. Kozlak, USAF, is a cost analyst at the Air Force Cost Analysis Agency in the Washington, DC metropolitan area. Capt Kozlak holds a BS in Management from the United States Air Force Academy and an MS in Cost Analysis from the Air Force Institute of Technology.

(E-mail address: [email protected])

Dr. Edward D. White is a professor of statistics in the Department of Mathematics and Statistics at the Air Force Institute of Technology. He received his MAS from The Ohio State University and his PhD in Statistics from Texas A&M University. Dr. White’s primary research inter-ests include statistical modeling, simulation, and data analytics.




Dr. Jonathan D. Ritschel is an assistant pro-fessor of cost analysis in the Department of Systems Engineering and Management at the Air Force Institute of Technology (AFIT). He received his BBA in Accountancy from the University of Notre Dame, his MS in Cost Analysis from AFIT, and his PhD in Economics from George Mason University. Dr. Ritschel’s research interests include public choice, the effects of acquisition reforms on cost growth in DoD weapon systems, research and development cost estimation, and economic institutional analysis.


Lt Col Brandon Lucas, USAF, currently serves as an assistant professor and director of the Graduate Cost Analysis Program at the Air Force Institute of Technology, Wright-Patterson AFB, Ohio. He holds an MS in Cost Analysis and a PhD in Economics. Lt Col Lucas has served in the budget, cost, and finance communities at base, center, and Air Staff levels.

(E-mail address: [email protected] )

407

July 2017

Mr. Michael J. Seibel retired as a senior cost analyst in the Cost Division of the Air Force Life Cycle Management Center at Wright-Patterson AFB, Ohio. During his 42-year career, he led or participated in numerous independent cost esti-mates, source selection cost panels, special studies, and program office support exercises. He was also very active in the cost research area. He holds an MS in Social and Applied Economics f rom Wr i g ht S t at e Un iver sit y a nd i s a n International Cost Estimating and Analysis Association member.

(Can be reached at: [email protected])

Estimating Firm-Anticipated DEFENSE ACQUISITION COSTSWITH A VALUE-MAXIMIZING FRAMEWORK

LTJG Sean Lavelle, USN

The author develops a value-maximizing framework to analyze defense contractor behavior within both risk-sharing and cost-plus contracts. He empirically observes that as development costs rise above estimates, the government tends to cut its order of the final product. Because the firm loses money from order cuts, irrespective of contract type, the author theorizes that cost-plus contracts will behave similarly to risk-sharing contracts if the government reliably cuts orders as development costs rise. For purposes of this article, the author uses a larger than typical dataset to empirically estimate the effects of contract incentives (final order size cuts in the case of this dataset) on firm value. With these effects estimated, he establishes a methodology for determining firm expectations of research and develop-ment costs based on the anticipated impact of cost overruns on firm value.

DOI: https://doi.org/10.22594/dau.16-762.24.03 Keywords: Cost Estimating, Defense Contract, Information Asymmetry, Value-Maximizing, Bidding



Estimating Firm-Anticipated Defense Acquisition Costs http://www.dau.mil

Publicly traded defense contractors bid and work on projects for one overriding reason: to increase their value to shareholders. When a new contract is awarded, investors look at the estimated value of the contract and decide whether to buy, sell, or hold their shares in the company. A higher valued contract might increase the stock price more greatly than a lower val-ued contract. During the life of the contract, as prices for research vary and quantities ordered increase or decrease, the estimated value of the contract will change as well. Rational investors should observe these fluctuations and make decisions about their holdings. Likewise, rational firms should only pursue those projects that will result in an increase in net value.

If we assume that the cost of research and development of a contract is predetermined and only known by the contractor, fluctuations during the

contract are determined simply by the initial bid. If the initial bid

is lower than actual cost, there will be cost over-runs. Alternatively, if we assume that contrac-tors have a great amount of control over t hei r costs of development, they should f ine-tune their cost performance throughout the project to maximize their value. As we develop our theory, only one of these assump-

tions must be true.

For cost-plus contracts, the government pays all cost overruns and the contractor is virtually unaffected. The only way that a contractor might be immediately impacted is if there is specific language inserted into the contract detailing a risk-sharing scheme, or if the govern-ment decides to cut the number of end-product units ordered in response to overruns. While risk-sharing contracts are difficult to empirically eval-uate due to a dearth of available data, changes in quantity ordered are well documented for any Major Defense Acquisition Program (MDAP). This reduction in quantity ordered will hurt the firm as most contracts have a research phase, resulting in a predetermined, negotiated rate for the cost of each manufactured unit. Production revenue can often be a significant portion of a government contractor’s cash flow in an MDAP.


July 2017

Whether we assume that the contractor can adjust its costs or that it bids with a fixed final cost in mind, the market will respond to new projects and fluctuations in value in the same manner. It should also not matter if the diminished contract value is due to an incentive scheme or changes in quantity ordered. This article will estimate the effect that new contracts and later amendments have on a company’s value. Once the effects are estimated, government contractors can then use them to determine optimal incentives for cost control and cost predictability.

We developed the model to take into account the market’s reaction to the ini-tial award amount, changes in development costs, and changes in the value of quantities ordered after development. Intuition tells us that the stock price will go up with a new contract award and will go down as less quantity is ordered. Our theory states that as development costs go up, the govern-ment will cut quantity ordered. Given this relationship, we expect that as development costs increase, firm value will decrease. We will compile evidence that development cost increases in a predominantly “cost-plus” environment will only affect firm value if the government changes quantity ordered in response.

After we estimate our model’s coefficients, we will develop a framework for designing optimal contract incentives. Since firms will not purposely lower their own value, we can find the ratio of quantity cuts to development cost overruns that forces the firm to either bid more closely to their estimations of cost or control costs below a certain desired level. We can understand this ratio, E1, as an elasticity where we measure the percentage change in quantity ordered for a percentage change in development costs.

Risk-Sharing ContractsCost overruns in defense acquisition have been a problem for as long

as the military has looked to industry to deliver innovative products. Most of the relevant literature views contracting for major, innovative defense projects through an insurance framework with moral hazard.

Cummins (1977) observes that prior to the 1960s, the government’s primary focus was on limiting contractor profits. To this end, cost-plus contracts, where the government bore all of the risk of cost overruns, were the pri-mary contract types. As awareness of the risk-sharing problem became better understood, the government shifted to greater use of contracts with incentives built in to share the risk between both parties. This shift of risk to the firm controlled cost overruns, but required a greater fixed fee to firms.

412

The basic framework of the risk-sharing contract is π = α - (1-y) C, where π is firm profit, α is a fixed fee, y is the fraction of cost overruns borne by the government, and C is the cost overruns. To ascertain a pareto-optimal contract, Cummins built upon a profit maximizing framework with contin-gent contracts and positive externalities gained by the firm from contract completion. In this article, we will use a value-maximizing framework. The effective difference is that a firm will likely need some market-desig-nated positive profit to undertake a project in our framework, whereas a firm would accept a contract with zero accounting profit in Cummins’ model.

Cummins showed that if we take the government’s objective to be lowering total cost vice cost overruns and firm profit, risk-sharing contracts can be ineffective. Hiller and Tollison (1978) provide empirical evidence to support this theory. As we increase the risk sharing in a contract, the firm requires a greater fixed fee. Any savings from increased firm attention to cost may be offset by the increased fixed fee. We could see lower cost overruns, but the same—or even greater—total cost. It is therefore much more important to focus on controlling total cost vice cost overruns. Our model will allow the government to calculate expected total cost prior to the awarding of a contract.

Weitzman (1980) takes the analysis of risk-sharing contracts in another direction. His research presents the risk-sharing contract equation and a utility maximization framework to determine an efficient sharing ratio between the principal (government) and the agent (contractor). Weitzman’s chief result is simple: if the sharing rate is high, the firm will bid higher and in turn make bids a more reliable indicator of final cost. We will obtain a similar result, but our “sharing rate” will be between development costs and quantity cuts, vice a contracted agreement on risk sharing. Additionally, we will show our sharing rate to impact total cost, not just cost overruns.

Goel (1995) builds upon Weitzman by including an auction framework. His research develops a model where the principal designates a sharing rate, y, and then allows various agents to bid on the contract. Under the model, the principal is able to coax agents to bid closer to their expectations of cost by increasing the amount of risk borne by the agent. Some differences between Goel and our article are a lack of bidding framework in this article, but the addition of a firm value perspective along with empirical investigation. Because the bidding framework does not affect the underlying result shared between Goel and Weitzman—that a higher sharing rate results in a bid that is more reflective of cost estimations—this article’s omission of a bidding framework should not impact its results.

413

The ModelIntuition tells us that markets will generally respond positively to any

mention of additional profit flowing to a publicly traded firm. Conversely, any news that suggests the possibility of a loss of profit should result in a nega-tive reaction. Our model defines how markets reward and punish defense contractors for different kinds of news. We will then test our model to see if the market encompasses new information in the manner we predict.

Rational markets should reward firms for securing a new contract. The firm should enjoy a larger increase in value if the contract is larger. Our initial model will look just at time period zero, T0, to determine how much of an effect the award has on stock price. Equation 1 shows this where P0 is the percentage change in stock price and IAt is the initially awarded contract amount divided by the value of the firm.

P0 = β1*IAt (1)

Next, we need an equation that explains the change in price for all further time periods. Firm value fluctuation during the life of the contract should primarily be affected by changes in the value of the contract. Defense con-tracts adjust through two main avenues: changes in quantity ordered and changes in development costs. Many MDAPs are of the cost-plus variety, meaning that an overrun in cost will be borne by the government, not the firm. Assuming our model’s firm is undertaking a cost-plus contract, its value should only be affected by a change in quantity ordered. Equation 2 shows this where Qt is the percentage change in quantity ordered.

Pt>0 = β2*Qt (2)

We can account for all time periods with Equation 3. In T0 of Equation 3, IAt will be equal to the initial amount of the contract whereas Qt will be zero. In all other time periods, IA will be zero whereas Qt will likely be nonzero numbers as long as the contract is still active.

Pt = β1*IAt + β2* Qt (3)

In our model, the government can decide Qt at any given time. If it requires less of the final product, Qt will be negative. If it requires more, Qt will be positive. We suspect that the primary reason that the government will change Qt is a change in Dt, or the percentage change in development costs. As development costs go up, the government must save costs elsewhere and cut Qt. Equation 4 gives us the relationship between Qt and Dt, where E1 is

414

the amount of Qt that the government will change for a given amount of Dt. (Note that E1 can be thought of as the elasticity of changes in quantity with respect to development cost fluctuations.) We will use this equation to empir-ically estimate E1 in the Results section of this article.

Qt = E1*Dt (4)

So that we can express our change in stock price equation only in terms controllable or estimable by the firm, we substitute Equation 4 into Equation 3 to get Equation 5.

Pt = β1*IAt + β2* E1*Dt (5)

The point at which our firm will either not bid on a project given known costs (or will avoid reaching if costs are controllable) is the point where the net change in value of the firm across the project is zero. Condition 1 gives us this mathematically, where i is the number of time periods.

Condition 1: (1+P0)* ∏i - 1t=1(1+Pt) = 1

If we assume a constant change in development costs over the life of the project, we can sum Equation 5 across all time periods. This allows us to calculate the total value change across the entire project to determine the optimal government reaction to a development cost increase. We use nota-tion P to denote the net percentage change in firm value for the duration of the project. If we take D to be the total development cost growth for the duration of the project and assume constant cost growth, we get each annual cost growth to be D1/i. The variable “i” is the number of years required for project completion.

P = [1+β1*IA] * [1+β2* E1* D1/i]i – 1 (6)

By solving Equation 6 for E1 where P = 0 (from condition 1), we can estimate the appropriate amount that quantity should be reduced if we know the gov-ernment’s opinion of the acceptable development cost growth of the program.

E1 = [D(-1/i)]*[ -1+(1/(β1*IA+1))1/i]/β2 (7)

If the government sets E1 equal to its result from Equation 7, the firm would lose net value from the project if it has greater cost growth than D. Because the firm will avoid losing net value, it will avoid having development cost growth greater than D. It can avoid such cost growth either by imple-menting cost controls where possible, or by bidding closer to the actual cost if it has a reasonable initial estimate.

415

Our predicted coefficient for all variables in all equations is displayed in Table 1. We will test Equations 3, 4, 5, and 6 empirically to estimate all relevant coefficients. We will then use those coefficients to develop our methodology for incentive creation and cost estimation from Equation 7.

Table 1 displays all of our empirically evaluated variables and our expected coefficients for each variable in each equation. The only vari-able that we have not previously discussed is SP. It represents the change in the Standard & Poor’s 500 Index (S&P 500) for the given time period. This variable will control for exogenous market variation.

TABLE 1. VARIABLE DESCRIPTION TABLE 1: VARIABLE DESCRIPTIONS

Variable Symbol Description Expected Coefficient

IA β1

The Initially Awarded Contract Amount Divided by the Market Cap of the Firm. Only nonzero during time

period of initial award.

Equation 3: PositiveEquation 5: PositiveEquation 6: Positive

Q β2

Change in Cost of Quantity Divided by Initial Contract

AmountEquation 3: Positive

D Eq 4: β3Eq 5: E1* β2

Change in Cost of Development Divided by Initial Contract Amount

Equation 4: NegativeEquation 5: Negative

Dq β3

Change in Cost of Development Divided

by Initial Contract Amountif Q Changed in Same

Time Period

Equation 6: Negative

Dn β4

Change in Cost of Development Divided by

Initial Contract Amount if Q did not Change in Same Time

Period

Equation 6: Insignificant

SP β5Percentage Change in

S&P 500 Positive

416

DataWe collected data from Selected Acquisition Reports (SARs) on 20

MDAPs that are published at least annually, sometimes quarterly. The 20 MDAPs were awarded to seven separate contractors between 2000 and 2014. The SARs give top-level summaries of changes in costs categorized by Support, Quantity, Engineering, Estimation, Economic, and Schedule.

We categorize Engineering, Estimation, and Schedule cost changes as developmental costs. Quantity changes will help us deter-

mine our Q variable, whereas Support and Economic changes will be their own category that we suspect

will have little impact on the model.

Hough (1992) discusses the difficulties associ-ated with using SARs to study cost overruns.

The most significant problem is that when incentive contracts are used, the costs borne by the contractor are not identified in the SAR. The SAR only identifies costs to the government, though it is intuitive that as the development cost grows in the SAR, a firm engaged in a risk-sharing contract will bear a

corresponding cost. It is therefore difficult to dissect the effect that the SAR-measured devel-

opment cost overruns have on firm value and the effects of the presumed collinearity that the SAR

measurement has with firm-borne costs. Since our data do not consist of completely cost-plus contracts,

we sanitize our results from this effect by testing for this collinearity in three ways.

Because our data will not exclusively consist of cost-plus contracts, we need to test whether or not Dt is impacting the stock price directly or indirectly, as hypothesized. To do this, we can compare the value of β2* E1, as estimated from Equations 3 and 4, and compare it to our empirical esti-mation from Equation 5. If the values are close to each other, then our data conform to our theoretical model. If there is a significant difference, then a change in Dt is having an effect on firm value independent of its impact on Qt.

The second test will be to simply regress both Qt and Dt against P t. This will allow us to control for either variable and determine which one is primarily driving Pt.

417

For a third way to test whether Dt is directly or indirectly changing firm value, we will look to Equation 6. We break Dt into Dqt and Dnt, where Dqt is Dt for all cases where Qt is nonzero and Dnt is Dt for all cases where Qt is zero. This way we break up the variable into two categories: one where Dt plausibly could have impacted Qt and one where it could have no impact.

Pt = β1*IAt + β2*Dqt + β3*Dnt (8)

We combined data from the last SAR for each year (usually from December) with the last stock quote of the corresponding company for the same year as reported by Google Finance. While it is unusual to rely on annual stock data for an event study, it is not possible in this instance to use daily, weekly, or monthly data. This is because the information included in a SAR flows to the market at varying times, while the SAR simply summa-rizes it. It is therefore problematic to decide when to designate the event when using more frequent market return data. This hurts the accuracy of this study, but does not negate its findings. For instance, Holthausen and Leftwich (1986) successfully used 300 days to define their event horizon in another event study. We additionally seek to mitigate this problem by including a larger dataset than is typical for this topic.

The predominant firm awarded each MDAP contract was used as the cor-responding company. All cost growths are noted in real dollars with the specified program’s start year as the baseline. Hough notes that simply using the real dollar values from the SAR is inadequate since initial cost estimates are based on certain inflation assumptions. To account for this, we discount the cost growth associated with economic factors.

To calculate production cost growth, we normalize to the contemporarily approved quantity, vice normalizing to the baseline-approved quantity. The selected calculation method is as reported in SARs.

In several instances, a contract was ongoing for several years and then was transformed into a new contract. Because the SAR data are very high-level and provide no amplifying information to ascertain estimated changes in quantity and development costs, we treated the change as the final year of the project. This slightly reduces the number of observations, but should not impair the validity of our findings.

Our data include 321 data points at the project level and, when aggre-gated into firm-level data, 113 data points. We perform our analysis with project-level data.

418

The following MDAPs appeared in our data:

• AESA – Active Electronically Scanned Array (Radar)

• SM-6 – Standard Missile-6 (Rocket Intercept Missile [RIM]-174 Standard Extended Range Active Missile)

• SDB II – Small Diameter Bomb II

• MH-60 – Military Helicopter-60 (Seahawk)

• Patriot – Missile

• AEHF – Advanced Extremely High Frequency (Satellite)

• ACS – Aegis Combat System (Integrated Command and Control/Weapons Control System)

• JSF – Joint Strike Fighter (Fighter, Strike, and Ground Attack Aircraft)

• Land Warrior – Integrated Soldier System (Weapon, Helmet, Computer, Digital and Voice Communications, Positional and Navigation System, Protective Clothing, Individual Equipment)

• EFV – Expeditionary Fighting Vehicle

• Stryker – Interim Armored Vehicle

• T-AKE – Auxiliary Cargo (K) and Ammunition (E) Ship, Military Ship Classification (MSC) Manned

• Bradley Upgrade – Infantry Fighting Vehicle

• Comanche – RAH-66 Helicopter

• F/A-18E/F – Aircraft Variants (Based on McDonnell Douglas F/A-18 Hornet)

• EA-18G – Boeing Growler (Electronic Attack Aircraft)

• CH-47F – Boeing Chinook (Twin-Engine, Tandem Rotor Heavy-Lift Helicopter)

• P-8A – Boeing Poseidon (Navy Maritime, Patrol, Reconnaissance Aircraft)

419

• FBCB2 – Force X X I Battle Comma nd Brigade a nd Below (Communications Platform to Track Friendly/Hostile Forces on Battlefield)

• Global Hawk – Unmanned Aircraft System

Summary statistics for our project-level data are summarized in Table 2.

TABLE 2. SUMMARY STATISTICS

Project-Level Data P SP IA D Q

Mean 0.144073 0.043694 0.013118 0.017552 -0.00057

Standard Error 0.013847 0.010968 0.004444 0.011839 0.002507

Median 0.145455 0.105877 0 0 0

Mode 0.535274 -0.10139 0 0 0

Standard Deviation 0.240243 0.189972 0.079493 0.211774 0.044844

Sample Variance 0.057717 0.036089 0.006319 0.044848 0.002011

Kurtosis 0.543315 0.128455 148.0833 264.9717 161.3609

Skewness 0.011312 -0.81439 10.99353 15.69839 -10.383

Range 1.314126 0.722675 1.175556 3.846937 0.912608

Minimum -0.59603 -0.40967 0 -0.22689 -0.67047

Maximum 0.718095 0.313007 1.175556 3.62005 0.242139

The key variables in our project-level data are summarized in Table 3. Strongly correlated variables are in bold and include Q with D, Q with Dq, and D with Dq. Only Q and D are both used in the same regression, and we should expect them to be negatively correlated as observed.

TABLE 3. PROJECT-LEVEL DATA CORRELATIONS

Variables IA Q D Dq Dn SP

IA 1

Q 0.0015 1

D -0.0147 -0.8073 1

Dq -0.0153 -0.8113 0.9953 1

Dn 0.0057 -0.0005 0.0996 0.0031 1

SP -0.1124 0.038 0.0527 0.056 -0.0232 1

420

Validity Test of DataSince our theory relies on the premise that development cost changes

impact firm value, primarily through their impact on order quantities, we must test our data to see that they conform to this assumption. We do this through three regressions, two of which will be discussed in this section, while the third will be discussed in the Results section. For the first test, we will split Dt into two variables: one that has value when Qt moves in the same time period, Dqt, and one that has value when Qt does not move in the same time period, Dnt, as in Equation 8. Table 4 defines and shows the results of this test regression.

TABLE 4. CATEGORIZED D REGRESSIONt

P =β *IA +β *Dq +β *Dn +β *SP +Constantt 1 t 3 t 4 t 5 t With Constant No Constant

SPt

.6584133(.000)

.7914403(.000)

IAt

.4269925(.004)

.675944(.000)

Dqt

-.1238065(.024)

-.0868289(.158)

Dnt

-.26261614(.641)

-.3981645(.530)

Constant .1432938(.000)

N/A

R2 .2825 .3647

Observations 300 300

Note: Parentheses contain P-Values.

Dqt is significant at the 95 percent level in our first model and significant at the 90 percent level in the second. The more interesting result is that Dnt is completely insignificant in both models, suggesting that a change in development costs without a change in quantity ordered has no impact on firm value. In other words, without a corresponding change in quantity ordered, variations in development costs are not associated with a change in firm value.

Our second test for data validity will be to regress both Dt and Qt against Pt to ascertain whether Dt becomes insignificant when controlled for Qt. We will use firm-level data for this regression, as project-level data result in insig-nificance for both variables. Table 5 shows the results of this regression.

421

TABLE 5. REGRESSION WITH Qt AS CONTROL VARIABLE

Pt=β1*IAt+β2*Qt+β3*Dt+β5*SPt+Constant With Constant No Constant

SPt

.0006096(.000)

.0006593(.000)

IAt

.2742662(.010)

.4170445(.000)

Dt

-.0101116(.918)

.0363141(.275)

Qt

.4029882(.003)

.4397591(.002)

Constant .072123(.000)

N/A

R2 .4487 .4975



As we can see, Dt, is completely insignificant when controlled for Qt. This indicates that our data are valid. We will discuss our final test in the Results section as we need to complete our primary analysis prior to completing the discussion on our final test. The test will also indicate that our data conform to our theoretical assumptions.

ResultsWe tested all regressions herein with OLS

(Ordinary Least Squares) and random effects and fixed effects models, with the firm as the panel variable and observed minimal variation from the basic regression. To simplify the reading of these results, we have only included the OLS version of each regression.

Our first OLS model (Table 2) looks at Equation 3, with the addi-tion of the SP variable (percent change in S&P 500 Index) and a constant in one of the models. Data are at the project level, with each year being a data point. Table 6 shows the results of this regression.

422

TABLE 6. EQUATION 3 REGRESSION

Pt=β1*IAt+β2*Qt+β5*SPt+Constant With Constant No Constant

SPt .6468396(.000)

.7822163(.000)

IAt .428198(.004)

.6746299(.000)

Qt .4995067(.052)

.4489935(.121)

Constant .1106027(.000)

N/A

R2 .2786 .3280



We can see that only in the model with a constant is our primary variable of interest, Qt, significant at the 95 percent level, though the point estimate is within 20 percent in either model. Its coefficient is positive as predicted, meaning that as the government cuts quantity ordered, the firm’s value decreases as well. We can also see that an increase in the S&P 500 and the initial contract award are correlated to an increase in firm value.With our next model, we try to tease out the relationship between a change in development costs and a change in quantity costs. Equation 4 from the model section of this article was Qt = E1*Dt. Our model in Table 7 is simply Dt regressed against Qt to find E1. With the assumption that the government adjusts Qt as a reaction to Dt, we understand E1 to be the ratio of Qt changes for a given Dt. In other words, the elasticity of Qt with respect to Dt.

423


Qt=E1*Dt+ Constant With Constant No Constant

Dt -.1709295(.000)

-.1699817(.000)

Constant .0024306(.103)

N/A

R2 .6516 .6487



Accepting the no constant model as our primary model because the con-stant is not significant at even the 90 percent level, we can understand the coefficient of Dt to mean that for every percent that an MDAP’s development costs increase from initial estimates, the government will order about 0.17 percent less quantity. Our 95 percent confidence interval for E1 lies between -0.1838 and -0.1562.

We also tested the Pearson correlation between the two variables and found it to be 0.8072, indicating a significant inverse relationship between them.

We can also analyze the relationship between Q and D by viewing them on a scatterplot. As we can see in Figure 1, which contains project-level data, the correlation appears negative.

FIGURE 1. PROJECT-LEVEL Q VS. D SCATTERPLOT

Perce

ntag

e Cha

nge i

n Dev

elopm

ent C

osts

Percentage Change in Quantity Ordered

0.2

0.15

0.1

0.05

0

-0.05

-0.1

-0.15

-0.2

-0.25

-0.3

-0.25 -0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2

Note: Omits one outlier data point from Joint Strike Fighter program.

424

Shifting our initial model from Table 6 to have Dt replace Qt, we would expect that the coefficient for Dt would simply be Qt coefficient multiplied by E1 whereas all other coefficients remain the same. The OLS model for Equation 5 is depicted in Table 8, and the far right column is our theory’s prediction of the model.


Pt=β1*IAt+β2*E1*D1+β5*SPt + Constant With Constant No Constant Predicted

SPt

.6589882(.000)

.7930062(.000) .6468396

IAt

.4268835(.004)

.676214(.000) .428198

Dt

-.1251678(.022)

-.0898093(.142) -0.08538

Constant .1121303(.000) N/A .1430959

R2 .2823 .3275 N/A

Observations 300 300 300


Whereas the no constant model has a higher R2, our constant model has all variables significant at the 95 percent level. For this reason, and because our model from Table 2 used the constant model as its primary one, we will use the constant model. In the far right, we can see our theoretical predictions for the OLS model with a constant and that all values are close to actual esti-mations (well within the 95 percent confidence interval). Our empirically estimated coefficient for Dt is -0.125 with a 95 percent confidence interval of -0.232 to -0.018. This places our theoretical prediction well within the limits of our actual estimations and lends significant credence to the valid-ity of both our data and theory.

All regressions including a constant were tested for heteroscedasticity using the Breusch-Pagan/Cook-Weisberg test. We were not able to reject the null hypothesis of homoscedasticity for any regression. We also observed the residual plot for every regression and found no clear evidence of any specifi-cation errors. Figure 2 shows the residual plot for equation 5. The plot is not uniformly distributed, but there is no clearly identifiable pattern suggesting omitted variable bias or another specification error. Various specifications all yield qualitatively similar plots.

425

FIGURE 2. EQUATION 5 RESIDUAL PLOT

.5

0

-.5

Resid

uals

Fitted Values

-.2 0 .2 .4 .6

ConclusionsOur theory, that even in a cost-plus contract, defense contractors’ firm

value f luctuates indirectly with development cost changes through the government’s quantity cutting response, presents a clear framework for building effective incentives to mitigate potential cost overruns. Our data show that as development costs rise from initial estimates, the quantity ordered by the government decreases. Because our theory and model show that a decrease in quantity ordered leads to a lowered firm value, develop-ment cost overruns that lead to less quantity ordered should have a similar effect. The only difference in a dollar of cost overrun and a dollar cut from the final order is the ratio between the two. If the government cuts 25 cents of the final order for every dollar of development cost overruns, the harm to the firm from development cost overruns will not be as strong as a 1:1 ratio. The government is the decider of this elasticity and can therefore determine how great the disincentive is for a firm to allow development costs to climb. It is hard to imagine the government determining at the outset of a contract

426

how much quantity they will cut based on development costs, but by estab-lishing a reputational E1, the government can effectively achieve the same objective. E1 will simply not be as flexible from contract to contract.

Our models all support the theory as we would expect. Changes in quantity ordered are positively correlated and changes in development costs are negatively correlated with changes in firm value. More convincingly, the magnitudes of these correlations are roughly the same as the ratio that the government chooses. Firm value increases about 0.51 percent per percent-age increase in end product purchases. Firm value decreases about 0.12 percent for every percentage increase in development costs. If our theory proved exactly correct, given our estimated ratio of quantity cost changes to development cost changes (~ -.17), firm value should decrease approximately 0.9 percent for every percentage increase in development costs. This value is only 25 percent away from our point estimate and is well within reason-able confidence intervals. Further, when we look at changes in development costs that occurred concurrently to changes in quantity ordered versus those that did not, our theory is further supported. Development costs with no quantity changes have no effect on firm value while those with quantity changes associated do have an effect.

If we take our theory and solve for the ratio of changes in quantity ordered to changes in development costs E1 = [D(-1/i)]*[ -1+(1/(β1*IA+1))1/i]/β2, the government can determine the optimal ratio to incentivize firms as desired. If the government wants the end cost to be below a certain amount and the firm can control costs, it must create incentives such that the firm will lose value on the project if it exceeds that amount. For a $70 billion firm bidding $20 billion on a project that will last 5 years, we might create a graphic as in Figure 3.

427

FIGURE 3. DEVELOPMENT COSTS VS. STRENGTH OF FIRM PENALTIES

Deve

lopme

nt Co

sts (B

illion

s)

E1

$60

$50

$40

$30

$20

$10

$0-0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0

If the government does not want total cost to exceed $40 billion, it should set the ratio of quantity ordered cut for a development cost overrun at approximately -0.19. Given our estimated values and the firm’s aversion to losing value, the firm will allow development costs to only grow an accept-able amount.

We create Figure 1 by calculating E1 for all reasonable values of D1. We can then calculate total cost for each value of E1 by adding the devel-opment cost effects and quantity-ordered effects to the initial bid.The government could also use Figure 3 to better predict final costs of research and development. All it needs to know is E1, and it can then ascer-tain a firm’s incentives to control costs.

If the government alternatively believes that the firm knows its costs, but has no control over them, it can seek to incentivize a realistic bid. The harsher the incentives (lower E1), the more closely the bid will reflect firm expectation of cost. If we assume a $70 billion firm that has known costs of $20 billion for development for a 5-year project, we can build Figure 4.

428

FIGURE 4. BID ACCURACY VS. STRENGTH OF FIRM PENALTIESBid

(% of

Tota

l Fina

l Cos

t)

E1 vs Bid

120%

100%

80%

60%

40%

20%

0%-0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0

E1

As we can see, the harsher the incentive, the closer the firm’s minimum bid gets to its actual estimate. We can rearrange our chart to give us the desired percentage of the total estimate that the government can reasonably expect all bids to at least reach.

We create Figure 4 by calculating E1 for every reasonable D1, given the firm’s expected development costs. We then infer the bid from D1.

This framework has several important implications for policy makers. If the government seeks to control costs or at least obtain an accurate estimation of the firm’s expectations of cost, it can use Equation 7 and our estimated coefficients to design an optimal contract. While this article looks at the ratio of quantity cuts to development cost overruns, we could easily calculate the profit lost from the quantity cuts and determine a more straightforward cost-sharing ratio with the same coefficients. A dollar of profit lost from quantity cuts should not impact the firm differently than a dollar of profit lost from a cost-sharing scheme.

The government might be encouraged to cancel projects at a lower threshold than is current policy. It might also be encouraged to establish firmer top-line budgets for projects, from development to production. The firm would then understand that as more funds were used in development, a smaller

429

share of the funds would go toward production. When a higher deterrent to cost overruns is established through a demonstrated willingness to cut quantity ordered, we should see more reasonable bidding, less cost overruns, and lower total cost of future projects.

With an E1 established by the government, the firm should bid in a predict-able manner, given its own expectations of the final cost of development. The government can then use the bid it receives to estimate the firm’s true cost expectations. For instance, if we know that a firm should bid 50 per-cent of its expected cost for an established E1, then the government should budget twice what is called for in the contract bid. This insight could allow the government to more accurately forecast expenses and improve contract stability. This stability could lead to lower costs to the taxpayer.

Development costs with no quantity changes have no effect on firm value while those with quantity changes associated do have an effect.

430

ReferencesCummins, J. M. (1977). Incentive contracting for national defense: A problem of

optimal risk sharing. The Bell Journal of Economics, 8(1), 168–185.Goel, R. K. (1995). Choosing the sharing rate for incentive contracts. The American

Economist, 39(2), 68–72.Hiller, J. R., & Tollison, R. D. Incentive versus cost-plus contracts in defense

procurement. The Journal of Industrial Economics, 26(3), 239–248.Holthausen, R., & Leftwich, R. (1986). The effect of bond rating changes on common

stock prices. Journal of Financial Economics, 17(1), 57–89.Hough, P. G. (1992). Pitfalls in calculating cost growth from Selected Acquisition

Reports (Report No. N-3136-AF). Santa Monica, CA: RAND.Weitzman, M. L. (1980). Efficient incentive contracts. The Quarterly Journal of

Economics, 94(4), 719–730.

431

Author Biography

LTJG Sean Lavelle, USN, is a naval flight officer serving with VP-26 on the P-8A Poseidon in Jacksonville, Florida. He holds a BS in Economics from the U.S. Naval Academy and a Master’s in Finance from Johns Hopkins University.

(E-mail address: : [email protected])

INFORMING POLICY through Quantification of the INTELLECTUAL PROPERTY

LOCK-IN Associated with DOD ACQUISITION

Maj Christopher Berardi, USAF, Bruce Cameron, and Ed Crawley

This article highlights a general lack of quantitative understanding surrounding trends in intellectual property lock-in associated with Depart-ment of Defense (DoD) acquisition. Important questions like, “How much does lacking rights in intellectual property cost DoD each year in lost competition?” remain unanswered. The analysis herein seeks to interpret DoD contracting trends in Federal Acquisition Regulation (FAR) 6.302-1(b)(2) exceptions to full and open competition as an indicator variable for intellectual property lock-in. By using data publicly available from USASpending.gov, this article revealed that approximately $6 billion in

contracts used FAR 6.302-1(b)(2) exceptions between FY08–FY15. Further analysis identified trends in the decreasing use of FAR 6.302-1(b)(2) exceptions, but no material trend in the amount of money obligated using exceptions. A key finding was prevalent use of FAR 6.302-1(b)(2) exceptions for service contracts as well as research and development contracts. This finding countermands conventional understanding of what is acquired using 6.302-1(b)(2) exceptions.

DOI: https://doi.org/10.22594/dau.16-767.24.03 Keywords: Intellectual Property, Data Rights, Technical Data, Competition, FAR 6.302-1(b)(2) Exceptions



Informing Policy through Quantification of the Intellectual Property Lock-In http://www.dau.mil

First and foremost, this article endeavors to analyze Department of Defense (DoD) acquisition data to capture metrics on either the increase or decrease in use of contracts awarded using Federal Acquisition Regulation (FAR) 6.302-1(b)(2) exceptions to other than full and open competition. Such contracts are used herein as an indicator variable for intellectual property lock-in. Intellectual property lock-in is defined as any contract in which the lack of rights to intellectual property is the basis for other than full and open competition. The choice of using FAR 6.302-1(b)(2) as an indicator variable comes with limitations, which are discussed at length. However, given the data publicly available, it remains the best available vari-

able. The methods utilized herein look for broad changes in DoD competition patterns with a specific focus on contracts

awarded using noncompetitive procedures due to FAR 6.302-1(b)(2) exceptions. These metrics and

resulting data are instrumental to inform future legislative and policy decisions.

Research QuestionsNone of the previous literature, with the

exception of Bera rdi, Ca meron, Stur teva nt, Baldwin, and Crawley (2016) as well as Liedke

and Simonis (2014), endeavored to either quan-tify the magnitude of intellectual property impact

on DoD acquisition or assert quantitatively whether there were any distinguishable trends. These avenues of investigation are imperative because policy mak-ers must understand the magnitude of the problem before making any cost-benefit decision about adopting or directing changes. Additionally, it is possible that

recent Better Buying Power initiatives (discussed in greater detail in the next section) are having a posi-

tive impact on the intellectual property trends within DoD acquisition. To address these fundamental questions,

this analysis turns to the Federal Procurement Data System (FPDS), which has been used by previous authors to successfully

analyze trends in DoD acquisition (Hunter et al., 2016; Hunter, Sanders, McCormick, Ellman, & Riley, 2015; Liebman & Mahoney, 2013). Although the FPDS database has limitations, it remains a sought-after data source because it contains up to 225 variables for each contract. This analysis uses two research questions to guide the investigation:


July 2017

Research Question 1

What is the magnitude of FAR 6.302-1(b)(2) exceptions on DoD acquisition?

Research Question 2

What trends, if any, are there in FAR 6.302-1( b)(2) exceptions on DoD acquisition?

Research Question 2(a)

What trends exist in the frequency of FAR 6.302-1(b)(2) J&As?

Research Question 2(b)

What trends exist in the products or services acquired using FAR 6.302-1(b)(2) J&As?

Background: Policy and StatuteIntellectual property regulations and statutes within the DoD are

among the most central issues in current policy discussions. This is evi-denced by the appearance of intellectual property initiatives in all three versions of Better Buying Power (BBP) (Carter, 2010; Kendall, 2013, 2015). The most recent iteration, BBP 3.0, outlines three strategies that confront the issue of intellectual property in DoD acquisition.

• The first, Remove Barriers to Commercial Technology Utilization, argues that the DoD should capture private sector innovation by using commercially available tech-nologies and products, but directs further analysis of the implications on intellectual property.

• The second strategy, Increase the Productivity of Corporate Independent Research and Development (IRAD), targets the misuse of IRAD funds by defense contractors on “de mini-mis investments primarily intended to create intellectual property” (Kendall, 2015) to secure a competitive advantage in future DoD contracts.

436

• The third strategy, Use Modular Open Systems Architecture to Stimulate Innovation, argues that the DoD must control relevant interfaces to ensure competitors with superior products are not occluded from competition due to intellec-tual property-restricted interfaces.

Additionally, in five1 of the last nine National Defense Authorization Acts (NDAA), Congress addressed rights to intellectual property in DoD acquisi-tion. Most recently, in the NDAA 2016, Congress directed the establishment of a Government-Industry Advisory Panel on rights in technical data for the purpose of “ensuring that such statutory and regulatory requirements are best structured to serve the interests of the taxpayers and the national defense,” which suggests further changes are imminent. However, with all the attention on statute and policy, no research is available that analyzes trends in DoD acquisition as a result of changes to intellectual property pol-icy. Basic questions such as, “How much does lacking rights in intellectual property cost the DoD each year in lost competition?” remain unanswered. The closest statistics are the annual measurements of competition levels in DoD acquisition (DoD, 2015; Hunter et al., 2016).

The 2015 version of the Performance of the Defense Acquisition System, released annually by the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD AT&L), argues that competi-tion is starting to increase. To substantiate this statement, the report uses a fractional measure of contracts competitively awarded by dollar amount. The most recent measures show that 58.3 percent of Fiscal Year2 14 (FY14) contracts, by dollar amount, were competitively awarded, which is up from 57 percent in FY13 (DoD, 2015). However, this methodology is sensitive to an outlier bias, where a few large contracts awarded competitively (i.e., contracts on the order of magnitude in the $100s of millions) overshadow the many smaller contracts awarded using other than full and open compe-tition. Independent analyses have also taken exception to the claim in the USD AT&L report by concluding, “the rate of effective competition (that is, competed contracts receiving at least two offers) has been largely unchanged in recent years” (Hunter et al., 2016); and “overall DoD competition rates ... have been largely steady in recent years” (Sanders, Ellman, & Cohen, 2015). Additionally, the USD AT&L analyses make no effort to break down the type of noncompetitive contracts, which makes it difficult to determine the root causes for trends in competition or to investigate trends in specific competition types.

437

However, independent analysis is possible. Contracts awarded noncom-petitively are documented using a contracting artifact referred to as a Justification and Approval, or J&A3 (Justification & Approval, 2016), which is a document released to the public when the DoD uses a procurement strategy other than full and open competition.4 The Federal Acquisition Regulation (FAR) does not enumerate all possible uses of J&As, but it does provide guidance on application of the regulation. In doing so, it provides situations in which the authority in FAR 6.302-1 may be appropriate. Of particular interest for this research is FAR 6.302-1(b)(2):

The existence of limited rights in data, patent rights, copy-rights, or secret processes; the control of basic raw material or similar circumstances make the supplies and services available from only one source (however, the mere existence of such rights or circumstances does not in and of itself justify the use of these authorities).

The research herein relies upon the example cited in FAR 6.302-1(b)(2) as an exception to full and open competition as the prime indicator of licensing issues with technical data, computer software, or intellectual property. This may include, but is not limited to: limited rights in technical data or other associated factors such as failure to have purchased a technical data pack-age, or failure to have taken delivery of a validated and verified technical data package. Although this FAR exception is an indicator of technical data licensing or intellectual property issues, it is a noisy indicator because the “similar circumstances” part of this FAR exception may apply even though the government has acquired unlimited rights to use the technical data or computer software. This is an acknowledged limitation of using FAR 6.302-1(b)(2) J&A as an indicator variable. However, to demonstrate that a FAR 6.302-1(b)(2) J&A is a reasonable indicator for technical data licensing rights or intellectual property issues, this article draws on Rogerson’s (1994) IRAD analysis technique to demonstrate a correlation between IRAD and FAR 6.302-1(b)(2) exceptions.

Independent Research and DevelopmentThe rights to technical data or computer software are generally deter-

mined by which party funded the development of a work or invention. These criteria are outlined in 10 U.S.C. §§ 2320 as well as 2321, and are promul-gated within the FAR and the Defense Federal Acquisition Regulation Supplement. As a general example, the contracting officer uses a funding test to determine licensing rights. If a contract is entirely publicly funded,

438

then the government should have unlimited rights to use technical data or computer software. Conversely, if a project is entirely privately funded, then the government should be entitled to limited rights or restricted rights. If the contract is funded using a mix of public and private funding, then the government should be entitled to government purpose rights. Table 1 is an example to demonstrate the general process, but does not cover all product types or exceptions to policy.

TABLE 1. DEFENSE RESEARCH & DEVELOPMENT EXPENDITURES BYPERFORMER AND FUNDING SOURCE

(FY84–FY14: ALL FY15 CONSTANT DOLLARS)

Performer Funding Source Sum % of Total

DoD DoD $523.4B 23.6%

Universities DoD (Contract R&D) $102.4B 4.6%

Nonprofit Firms DoD (Contract R&D) $47.8B 2.2%

For-Profit Firms DoD (Contract R&D) $1,348.9B 60.9%

For-Profit Firms DoD (IRAD) $93.1B 4.2%

For-Profit Firms For-Profit Firms (IRAD) $100.6B 4.5%

Source: All values except for the two IRAD values are from National Science Foundation’s (2016) WebCASPAR database and are obligations. The two IRAD numbers are from the Defense Contract Audit Agency (2016) and are incurred costs.

Assuming program offices and contractors correctly follow the funding test for determination of rights, it is reasonable to expect that the level of pri-vate IRAD funding should correlate to the level of FAR 6.302-1(b)(2) J&As. That is, if FAR 6.302-1(b)(2) J&As are a reasonable indicator of intellec-tual property issues, an increase in private funding should correlate to an increase in FAR 6.302-1(b)(2) J&A. To explore this potential correlation, Table 1 contains the sum of research and development (R&D) funding within the DoD by funding source and performer from FY84–FY14. Most of the combinations are self-explanatory, with the exception of “DoD (IRAD).” This is the proportion of For-Profit IRAD that is an allowable cost to the DoD, typically bundled in a firm’s overhead rate; whereas, the For-Profit Firm IRAD is the proportion of IRAD that was incurred, but was not allow-able as an indirect cost to the DoD. Figures 1a and 1b offer a visualization of the changes in IRAD spending from FY84-FY15.

439

FIGURE 1A. PERCENT OF TOTAL DOD R&D SPENDING BY PERFORMER AND/OR FUNDING SOURCE

FIGURE 1A. PERCENT OF TOTAL DOD R&D SPENDING BY PERFORMER AND/OR FUNDING SOURCE

100%

80%

60%

40%

20%

0%

1985 1990

1995

2000

2005

2010

Public IR&DPrivate IR&DUniversities

NonprofitFederalFor-Profit

FIGURE 1B. IR&D SPENDING TRENDS FROM FY84-FY14(CONSTANT FY15 DOLLARS)

FIGURE 1B. IR&D SPENDING TRENDS FROM FY84-FY14 (CONSTANT FY15 DOLLARS

Note. Public IRAD refers to "DoD (IRAD)" in Table 1 and Private IRAD refers to "For Pro�t Firms (IRAD)" in Table 1. (Source: Defense Contract Audit Agency, 2016)

12%

10%

8%

6%

4%

2%

0%

1985 1990

1995

2000

2005 2010

Public IR&DPrivate IR&D

Note. Public IRAD refers to “DoD (IRAD)” in Table 1 and Private IRAD refers to “For Profit Firms (IRAD)” in Table 1. (Source: Defense Contract Audit Agency, 2016)

440

It is likely that IRAD funds invested in a fiscal year would not impact FAR 6.302-1(b)(2) exceptions until, at the very least, the following fiscal year. To explore this potential relationship, a lagging correlation analysis is used. These results are outlined in Figures 2a and 2b, which show lagging Pearson product-moment correlations between both sum FAR 6.302-1(b)(2) J&A obligations (Figure 2a), and mean FAR 6.302-1(b)(2) J&A obligations (Figure 2b), to Private IRAD. Figures 2a and 2b illustrate which 7-year period of Private IRAD is the highest correlated with the Patent or Data Rights (PDR) J&A obligations from FY08–FY14. Clearly, this analysis indi-cates little correlation between the sum FAR 6.302-1(b)(2) J&A obligations and Private IRAD; however, a relatively strong positive correlation exists between mean annual FAR 6.302-1(b)(2) J&A obligations and Private IRAD. Although, this analysis does substantiate a correlation between private funding and FAR 6.302-1(b)(2) J&As, it is critical to point out that this does not suggest causality between the two variables—merely that there is a connection between these two variables; and while FAR 6.302-1(b)(2) contains provisions other than intellectual property, it is still a reasonable, albeit noisy indicator of intellectual property issues.

FIGURE 2A. LAGGING PEARSON PRODUCT-MOMENTCORRELATION COEFFICIENTS

FIGURE 2A. LAGGING PEARSON PRODUCT MOMENT CORRELATION COEFFICIENTS

Note. Sum FAR 6.3021(b)(2) J&A obligations' correlation with Private IRAD at lags 0 < n < 10.

2 40 6 8–1

0

1

Lag (years)

Corre

lation

Coe�

cient

Note. Sum FAR 6.3021(b)(2) J&A obligations’ correlation with Private IRAD at lags 0 < n < 10.

441

FIGURE 2B. LAGGING PEARSON PRODUCT-MOMENTCORRELATION COEFFICIENTS

FIGURE 2B. LAGGING PEARSON PRODUCT-MOMENT CORRELATION COEFFICIENTS

Note. Mean FAR 6.302-1(b)(2) J&A obligations' correlation with Private IRAD at lags 0 < n < 10.

2 40 6 8–1

0

1

Lag (years)

Corre

lation

Coe�

cient

Note. Mean FAR 6.302-1(b)(2) J&A obligations correlation with Private IRAD at lags 0 < n < 10.

MethodThis article uses the Cross-Industry Standard Model for Data Mining

(CRISP-DM) model to investigate the impact of FAR 6.302-1(b)(2) excep-tions in DoD acquisition (Shearer, 2000). Although designed primarily for data mining and machine learning applications, this model offers a struc-tured approach for the analysis of large data sets. The CRISP-DM model is broken down into six phases: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. However, to keep the discussion in this article parsimonious, the last two phases are omitted.

Phase One: Business UnderstandingThe data available in FPDS represent all prime unclassified federal gov-

ernment contract awards above the micropurchase threshold. Consequently, the data in FPDS come with the following limitations:

1. FPDS includes only data on prime contract awards. A sep-arate database (i.e., Federal Subaward Reporting System) tracks subcontract awards, but has historically been incom-plete (Moore, Grammich, & Mele, 2015).

442

2. FPDS reporting regulations require the disclosure of only unclassified contracts. The magnitude of spending this regulation omits from the overall FPDS numbers is unknown; some estimate it at as high as 10 percent (Center for Strategic & International Studies, 2017).

3. FPDS reporting regulations require disclosure of contracts above the micropurchase threshold, which is defined in FAR subpart 2.1 as $3,500 (although at the time of writing, FPDS still uses the past threshold of $3,000). Contracts below this amount are not captured in FPDS.

As with any publicly available database, any analysis is wholly dependent on the quality of the underlying data. Previous Government Accountability Office (GAO) studies commented on the inaccuracies and problems within FPDS (GAO, 2010), but those reports are over 10 years old and reflect, in part, problems with the migration of government acquisition data into the FPDS system (Center for Strategic & International Studies [CSIS], 2017; General Accounting Office, 2003). However, despite the limitations noted above, FPDS is one of the few publicly available exhaustive sources of data on government acquisition and is suitable for analyses aimed at identifying trends or making order-of-magnitude comparisons.

Phase Two: Data UnderstandingThis phase of the CRISP-DM process begins with the collection of data

and then transitions to identifying data quality problems. The goal of this phase is to build an understanding of the limitations in the data as well as to build an understanding of quality issues that may need to be remedied before the application of analytic models. This process concludes with a verification of data quality.

Two primary avenues allow users to access data from FPDS. The first is to download XML archives directly from the FPDS website by fiscal year. The second method is USASpending.gov, which is designed to aggregate federal government spending data and display it to taxpayers. USASpending.gov was ultimately selected and used for all the subsequent analyses herein.5

The data in USASpending.gov are available from FY00 through the current fiscal year. However, an inspection of fiscal years prior to 2008 revealed many missing data points and subsequently a lack of the necessary fidelity for inclusion in these analyses. In addition, the most recent fiscal year (FY16 at the time of writing) is excluded from this analysis because the DoD data in USASpending.gov are intentionally delayed by 90 days for national secu-rity reasons; and an additional 30- to 60-day delay is incurred to update the

443

data in FPDS after the initial 90 days, amounting to a 150-day lag in current fiscal year data. Subsequently, the analyses that follow will evaluate only fiscal years 2008–2015.

Across 8 fiscal years, over 12 million contract actions are executed within the DoD. For purposes of this analysis, all zero-dollar contract obligations (i.e., a contract action that neither adds nor subtracts funds) were removed from the data set. Zero-dollar obligations are used oftentimes to make administrative changes or no-cost changes to the contract; consequently, leaving them in the data set would skew later analyses that look at ratios of contract awards. This removal has no material impact on the overall dollars obligated, but does adjust the count of contracts across all fiscal years under study. After removing these contract actions, the remaining data set contains over 11 million contract actions totaling approximately $2.7 trillion (Table 2).

TABLE 2. DESCRIPTION OF DATA

Service Sum Count

DEPT OF THE ARMY $908,943,882,349 2,128,614

DEPT OF THE NAVY $741,787,602,050 1,797,791

DEPT OF THE AIR FORCE $496,212,601,573 836,929

OTHER DoD $311,293,701,289 562,164

DEFENSE LOGISTICS AGENCY $284,921,110,343 6,036,863

TOTAL $2,743,158,897,605 11,362,361

Phase Three: Data PreparationPhase Three involves selecting, cleaning, constructing, integrating, and

formatting data into the final data set before application of selected model-ing tools. All figures and tables displayed before this point are representative of the organic data with no data preparation applied.

Data selection. From the 225 available variables, only a small sub-sample is required to address the research questions.6 In total, 11 vari-ables were selected (Table 3), including data descriptions quoted directly from the USASpending.gov data dictionary (USASpending.gov, 2015). One could make an argument for including additional fields of data in the analyses, but computational memory limitations require a parsimonious set of variables for a data set in the tens of millions order of magnitude.

444

TABLE 3. DESCRIPTION OF DATA FIELDS

Field Name Description

descriptionofcontractreq A brief description of the goods or services bought (for an award) or that are available.

dollarsobligated The net dollar amount that is obligated or de-obligated by this transaction. If the net is a de-obligation, the amount will be negative.

fiscal year The federal government fiscal year, determined by the “Signed Date” provided by FPDS.

fundedbyforeignentity Indicates that a foreign government, international organization, or foreign military organization bears some of the cost.

maj_agency_cat The combination of two leftmost characters of the contracting agency code representing major federal organizations and departments, and their description.

modnumber An identifier that uniquely identifies a modification for a contract, agreement, or order.

principalnaicscode The principal North American Industry Classification System, or NAICS code that indicates the industry in which the contractor does business.

productorservicecode The code that best identifies the product or service procured. If more than one code applies, then the code that represents most of the ultimate contract value is reported.

psc cat The major category that the Federal Procurement Data System Product or Service Code for the record falls within.

reasonnotcompeted A code for the reason the contract was not competed – i.e., solicitation procedures other than full and open competition pursuant to FAR 6.3.

signeddate The date that a mutually binding agreement was reached. The date signed by the Contracting Officer or the Contractor, whichever is later.

Data cleaning. In 2015, Defense Logistics Agency (DLA) contract actions more than quadrupled over the previous fiscal year, while the total dollars obligated slightly decreased. This aberration is explained by a change in the way DLA accounts for contracts under the micropurchase threshold. To control for this change, the data for all DLA obligations in FY15 were cleaned to only those contract obligations that were over or under the micropurchase threshold (i.e., x < -$3,000 as well as x > $3,000). This cleaning step brings the number of DLA obligations in FY15 from 2,396,568 to 589,474, and is comparable to previous fiscal years.

445

Although this precipitous drop in obligations—a -75.4 percent change—may seem like a large impact, it ultimately reduced the total DLA obligations in 2015 by $967,630,808, or only -3.15 percent.

Finally, to enable comparisons across multiple fiscal years the dollarsob-ligated field was converted from nominal dollars to constant FY15 dollars using the annual average All-items Consumer Price Index for the respective fiscal years under study. All expressions of dollars obligated or total obliga-tions from this point forward are constant FY15 dollars unless otherwise noted in the analyses.

Data construction. From the selected 11 variables, an additional 2 were constructed. Psc_group and psc_simple data fields were constructed to mirror the CSIS taxonomy of Product Service Codes7 (PSC). Table 4 depicts 10 product categories and 5 service categories into which hundreds of PSC codes are aggregated, plus a category for R&D8 (Ellman & Bell, 2015).

The psc_group was created by mapping the productorservicecode field to the second level of the CSIS PSC taxonomy (i.e., Aircraft, Ships, Fuels, PAMS, etc.) and the psc_simple field was created by mapping the productorservice-code field to the first level of the taxonomy (i.e., Product, Service, or R&D). These additional fields enable categorical analyses that would not be pos-sible using only the data available in productorservicecode.

446

TABLE 4. PRODUCT SERVICE CODE CATEGORIES

Product Categories

1. Aircraft 6. Electronics & Communications (E&C)

2. Clothing & Subsistence (C&S) 7. Engines & Power Plants (E&PP)

3. Fuels 8. Ground Vehicles

4. Launchers & Munitions (L&M) 9. Missiles & Space

5 Ships 10. Other Products

Service Categories

1. Equipment-related Services (ERS)

2. Facilities-related Services & Construction (FRS&C)

3. Information and Communications Technology (ICT) services

4. Medical (MED) services

5. Professional, Administrative, and Management Support (PAMS)

Research & Development (R&D)

Phase Four: ModelingAs discussed in earlier phases, characteristics found in FPDS data and

subsequently USASpending.gov data may limit analyses aimed at identifi-cation of trends and order-of-magnitude comparisons. Consequently, Phase Four consists primarily of complex Boolean logic queries, visual modeling techniques, and regression models to answer the research questions set out at the beginning of this analysis.

Analysis of research question 1. This inquiry is most easily addressed by analyzing the reasonnotcompeted data field. In the most recent version of the FPDS data dictionary, the field “Other Than Full and Open Competition” is used as the designator for solicitation procedures other than full and open competition pursuant to FAR 6.3 or FAR 13 (USASpending.gov reti-tled this field as “Reason Not Competed”). This field provides 17 different codes, which correspond to FAR 6.301 or FAR 13. FPDS and subsequently USASpending.gov use the code “PDR” or the short description “Patent or Data Rights” to report a contract action justified pursuant to FAR 6.302-1(b)(2). Unfortunately, no information is provided on the specific justifica-tion—only the reference to FAR 6.302-1(b)(2). This makes segregating out the different exceptions in FAR 6.302-1(b)(2), for other than full and open

447

competition, impossible. Although reasonnotcompeted denotes the FAR exception used in the J&A, it is important to note that the unit of analysis here is the contract action authorized by the J&A, not the J&A itself. To ana-lyze the data, each contract action was organized into a group based on its reasonnotcompeted category, with null values omitted. Within the resulting groups, the dollarsobligated field is used to quantify the magnitude of the effect. The results are captured in Table 5, in which the Patent/Data Rights group is highlighted. This category denotes FAR 6.302-1(b)(2) exceptions.

This analysis suggests that FAR 6.302-1(b)(2) exceptions occluded over $6 billion in DoD contracts9 from full and open competition between FY08-FY15. However, this number may be misleading because of the way FPDS collects data. For each contract, an administrator is only permitted to select one field for reasonnotcompeted from the list of values outlined in Table 5. In practice, however, a J&A may contain more than one exclusion category, forcing the adminis-trator to make a judgment call on which category is most accurate. This nuance may explain the large number of “ONLY ONE SOURCE—OTHER” contract actions. Typically, a J&A for a con-tract would be “ONLY ONE SOURCE—OTHER” because of “UTILITIES” or “PATENTS/DATA RIGHTS.” This is because “PATENTS/DATA RIGHTS,” out-lined in FAR 6.302-1(b)(2), is a subsection of “ONLY ONE SOURCE—OTHER ,” a s outlined in FAR 6.302-1. Therefore, when admin-istrators are faced with selecting a single category that most accurately represents why an action was not sub-ject to full and open competition, they may select “ONLY ONE SOURCE—OTHER.” As a result, the aggregation of FAR 6.302-1(b)(2) exceptions in Table 5 are most likely only a lower bound of the true impact on DoD acquisition.

448

TABLE 5. COUNT, SUM, AND MEAN OF dollarsobligated GROUPED BY reasonnotcompeted, GROUPS SORTED BY SUM

Reason Not Competed Count Sum Mean

ONLY ONE SOURCE—OTHER 807,717 $600,738,669,601 $743,749

UNIQUE SOURCE 200,842 $209,486,641,145 $1,043,042

AUTHORIZED BY STATUTE 397,234 $99,108,891,481 $249,498

FOLLOW-ON CONTRACT 98,742 $74,653,183,136 $756,043

MOBILIZATION, ESSENTIAL R&D 34,166 $62,704,044,715 $1,835,276

INTERNATIONAL AGREEMENT 17,478 $61,851,393,409 $3,538,814

NATIONAL SECURITY 18,904 $31,207,522,647 $1,650,842

URGENCY 56,299 $30,980,330,505 $550,282

AUTHORIZED RESALE 39,693 $26,421,588,758 $665,649

SAP NON-COMPETITION 270,884 $8,763,887,633 $32,353

PATENT/DATA RIGHTS 13,150 $6,338,305,329 $482,000

UTILITIES 11,645 $4,830,634,587 $414,825

BRAND NAME DESCRIPTION 8,254 $1,824,756,424 $221,075

PUBLIC INTEREST 3,910 $1,821,008,661 $465,731

STANDARDIZATION 3691 $197,856,965 $53,605

UNSOLICITED RESEARCH 294 $144,568,046 $491,728

8AN 79 $49,346,244 $624,636

Note. SAP = Simplified Acquisition Threshold.

Analysis of research question 2a. The second question sought to identify trends, if any, in the use of FAR 6.302-1(b)(2) exceptions in DoD acquisition. To address this question, the data set was converted into a time

449

series using the signeddate as a date-time index. With the data set in this format, parallel lines of inquiry were used to identify two trends: ratios of dollars obligated and ratios of contracts awarded. For example, to determine ratio of dollars obligated, the data set was down-sampled into the 96 months that span fiscal year 2008 to 2015; Equation 1 sums the dollars obligated for each month from both the subset (α) and full set (β).

(1)

αmonth : βmonth =where:

α⊂ β,β ≠ 0,n = days in a given month

∑ n1 α n

∑ n1 β n

The result is two distinct time series of data. The first is of dollars obligated by month. The second is a count of contract awards, using only initial con-tract awards and not subsequent modifications, by month. Leaving contract modification data in the dataset would overstate the number of sole source contract awards supported by a FAR 6.302-1(b)(2) exception, since a J&A is typically not issued for each modificaiton. Using this time series trans-formation process, six different ratios were calculated:

1. The ratio of all contracts using a J&A to total number of contracts. Measured in terms of dollars obligated by contract and count of contracts (referred to henceforth as J&A to Total Awards).

2. The ratio of only contracts using FAR 6.302-1(b)(2) J&A to total number of awards. Measured in terms of dollars obligated by contract and count of contracts (referred to using the FPDS designator of “PDR” or Patent or Data Rights, which is henceforth referred to as PDR to Total Awards).

3. The ratio of contracts using a FAR 6.302-1(b)(2) J&A to contracts using any J&A. Measured in terms of dollars obligated by contract and count of contracts (referred to using the FPDS designator of “PDR” or Patent or Data Rights, which is henceforth referred to as PDR to J&A).

These two lines of inquiry illustrate whether the frequency of contracts awarded using a FAR 6.302-1(b)(2) exception is increasing or decreasing over the fiscal years under study. The results of this analysis are found in

450

Figure 3a and Figure 3b, which organize each of the ratios into a row with a column for ratio of obligation amounts (left column, denoted with a $ in the legend) and ratio of contract actions (right column). The red line on each graph in Figure 3a shows a 12-month exponentially weighted moving average (EWMA) for each time series ratio, which removes some of the irregular variations.

Additionally, to assist in quantifying the trend of data displayed in Figure 3a, Figure 3b illustrates an ordinary least squares (OLS) linear regression of the data. To compute this trend line, the dates on the x-axis were converted into numerical months ({Jan08,Feb08…Dec15}={1,2,…96}). Consequently, the x-axis in Figure 3b now corresponds to the number of months elapsed since the start of FY08. All other formatting and labeling factors remain constant between Figure 3a and Figure 3b.

451

FIG

UR

E 3

A. R

ATI

OS

OF

OB

LIG

ATI

ON

S (L

EF

T)

AN

D R

ATI

O O

F C

ON

TRA

CT

AW

AR

DS

(RIG

HT

)

90%

80%

70%

60%

50%

40%

30%

20%

JnA to Total Awds

JnA

to A

wds (

$)EW

MA (1

2-Mo

nth)

26%

24%

22%

20% 18% 16%

14% 12%

JnA

to A

wds

EWMA

(12-

Mont

h)

1.2%

1.0%

0.8%

0.6%

0.4%

0.2%

0.0%

PDR to Total Awds

PDR t

o Awd

s ($)

EWMA

(12-

Mont

h)

0.25

%

0.2%

0.15% 0.1

%

0.05%

0.0%

1.2%

1.0%

0.8%

0.6%

0.4%

0.2%

0.0%

3.0%

2.5%

2.0%

1.5%

0.5%

0.0%

PDR to Total JnA

PDR

to Jn

A ($

)EW

MA (1

2-Mo

nth)

PDR

to Jn

AEW

MA (1

2-Mo

nth)

PDR

to A

wds

EWMA

(12-

Mont

h)

2008

2010

2009

2011

2012

2013

2014

2015

2008

2010

2009

2011

2012

2013

2014

2015

452

FIG

UR

E 3

B. L

INE

AR

TR

EN

D L

INE

S FO

R E

AC

H R

ATI

O IN

FIG

UR

E 3

A

90%

80%

70%

60%

50%

40%

30%

20%

JnA to Total Awds

1.2%

1.0%

0.8%

0.6%

0.4%

0.2%

0.0%

PDR to Total Awds

3.0%

2.5%

2.0%

1.5%

0.5%

0.0%

PDR to Total JnA

26%

24%

22%

20% 18% 16%

14% 12%

0.25

%

0.2%

0.15% 0.1

%

0.05%

0.0%

1.2%

1.0%

0.8%

0.6%

0.4%

0.2%

0.0%

y=8.

17e–5

x+0.

426

r2 =0.

001

y=1.

11e–5

x+0.

0016

1r2 =

0.04

1

y=2.

54e–5

x+0.

0040

1r2 =

0.03

1

012

2436

4860

7284

96

Time (

mont

hs)

012

2436

4860

7284

96

Time (

mont

hs)

y= –

0.00

0575

x+0.

223

r2 =0.

332

y= –

1.5e

–5x+

0.00

184

r2 =0.

662

y= –

6.53

e–5x+

0.00

875

r2 =0.

576

453

Beginning with an analysis of Figure 4a, nearly all the ratios calculated from obligations indicate almost no trend in the data, with a possible exception of PDR to J&A. This suggests that, although the data are noisy, there is no meaningful change in the number of dollars obligated in any of the ratios over the past 8 fiscal years. Conversely, the ratios calculated from contract awards show a relative decline through the start of FY15, with a slight uptick in FY15. In summary, although the ratio of contract actions using J&As and FAR 6.302-1(b)(2) J&As declined over the past 8 fiscal years, a meaningful increase or decrease in the ratio of dollars obligated using FAR 6.302-1(b)(2) J&A was not noted.

FIGURE 4A. BOXPLOT OF INTERYEAR FAR 6.302-1(B)(2)OBLIGATION TRENDS

FIGURE 4A. BOXPLOT OF INTERYEAR FAR 6.302-1(B)(2) OBLIGATION TRENDS

Note. Red square denotes mean.

109

108

107

106

105

104

103

Oblig

ated A

moun

t ($)

2008 2009 2010 2011 2012 2013 2014 2015

Transitioning to Figure 4b, all the OLS regressions for the ratios of dollars obligated are slightly positive. However, PDR to J&A and J&A to Total Awards has a confidence interval that spans zero (95 percent confidence interval illustrated in Figure 4b by the shaded area on either side of the regression line). This analysis implies it is possible, although not necessarily probable, that the trend could be positive or negative. Conversely, the regres-sion results computed using the ratio of contract actions all have negative coefficients with confidence intervals that do not span zero. As a result, one can reasonably conclude that the frequency of both J&A relative to total awards, and PDR relative to both total J&As and total awards, declined from FY08 to FY15. This analysis suggests that some measures of trends in FAR 6.302-1(b)(2) use in DoD acquisition indicate a downward trend, or improvement; however, the results are confounded by other measures, which show no meaningful trend.

454

FIGURE 4B. LINE PLOT OF INTERYEAR FAR 6.302-1(B)(2)OBLIGATION TRENDS

2008 2009 2010 2011 2012 2013 2014 2015

$1.1B

$1.0B

$0.9B

$0.8B

$0.7B

$0.6B

1,800

1,600

1,400

1,200

1,000

800

600

400

Total Obligations

Average Obligation

Frequency (Secondary y)

To investigate further, Figure 5a and Figure 5b look at the interyear pat-terns in FAR 6.302-1(b)(2) J&A use. Specifically, Figure 5a shows the change in distribution of obligations by contracts using FAR 6.302-1(b)(2) J&A between FY08-FY15; with few exceptions, both the median and mean annual obligation amount increased year-over-year. Note the log scale in Figure 5a and the number of outliers in the boxplot with high order-of-magnitude obligations. Compare these results with those in Figure 5b, where the number of contracts is decreasing, but the annual obligation amounts trend erratically. These findings are important because they show that, although trends in FAR 6.302-1(b)(2) J&A use may be decreasing, FAR 6.302-1(b)(2) J&A contract obligations measured by either median or mean value are increasing.

455

FIGURE 5A. COUNT OF FAR 6.302-1(B)(2)J&A BY FISCAL YEAR AND PSC TAXONOMY LEVEL 1

1400

1200

1000

800

600

400

200

02008 2009 2010 2011 2012 2013 2014 2015

ProductsR&DServices

FIGURE 5B. MEAN OBLIGATION AMOUNT OF FAR 6.302-1(B)(2)J&A BY FISCAL YEAR AND PSC TAXONOMY LEVEL 1

2008 2009 2010 2011 2012 2013 2014 2015

$2.0M

$1.5M

$1.0M

$0.5M

$0.0M

Products R&D Services

456

Analysis of research question 2b. In addition to time series trends, it is important to look at “what” FAR 6.302-1(b)(2) J&As are buying to identify purchasing trends. To address this question, analyses focus on four of the data fields; psc_simple, psc_groups, principalnaicscode, and productorser-vicecode. Using these fields, it is possible to categorize each contract action as either a product or a service and attain a relative sense of the domain. This is done by employing a taxonomy that aggregates individual PSCs into broader baskets of goods. Table 6 outlines the sum obligations for the CSIS taxonomy at level 1 and level 2, as well as top 10 PSC and North American Industry Classification System (NAICS) codes for FAR 6.302-1(b)(2) J&As.

Analysis of the PSC codes reveals that FAR 6.302-1(b)(2) J&As are just as commonly used for services as they are for products. This is evidenced not only by the PSC taxonomy level 1 results, but also by all the PSC codes in Table 6 that begin with a letter. This same phenomenon is evident in the top 10 NAICS codes as well, where the 54-series category indicates a service. The two largest categories from level 2 of the PSC taxonomy are Professional, Administrative, and Management Support (PAMS), which is a service category; and Aircraft, which is a product category. Some of the service codes, “J016” for example, are logical given they denote the maintenance and repair of aircraft components. In this case, one could speculate the repair and manufacturing process of an aircraft component

is likely protected by a manufacturer’s intel-lectual property. However, less clear

457

is what intellectual property is the limiting factor in the codes that make up PAMS, for example R425, which is a generic code for engineering and support services.

The data in USASpending.gov do not offer any insight into why a FAR 6.302-1(b)(2) exception was used for any given contract. However, they do offer insight into product, service, and R&D trends (see Figure 5a and Figure 5b). In all except FY08 and FY09, more service than product contracts were authorized under a FAR 6.302-1(b)(2) J&A. Since FY09, the number of product FAR 6.302-1(b)(2) authorized contracts fell from a high of 1,204 per year to a low of 306 in FY15—a 75 percent drop. Although the drop seems precipitous, Figure 5b shows that the average product FAR 6.302-1(b)(2) J&A contract obligation amount increased since FY08. Consequently, one can reasonably conclude the DoD is using less product FAR 6.302-1(b)(2) J&As in contracts, but the average cost of each is rising. A similar conclusion can be reached for service FAR 6.302-1(b)(2) J&As, with the exception of the percentage decrease in frequency and increase in mean obligated dollars, which are of a lower magnitude. The exception is FAR 6.302-1(b)(2) J&A used for R&D, in which both the frequency and mean obligation amount increased over the past 8 fiscal years.

458

TABLE 6. PSC TAXONOMY LEVEL 1 AND TOP 10 PSC TAXONOMY LEVEL 2 PSC AND

NAICS CODES SORTED BY SUM OBLIGATED DOLLAR AMOUNT FOR PDR J&A

PSC Taxonomy Level 1 Count Sum

Products 5,533 $3,654,163,121

Services 7,098 $2,269,110,084

R&D 519 $415,032,124

PSC Taxonomy Level 2 Count SumPAMS 2,159 $1,593,098,494

Aircraft 1,449 $1,389,506,928

Missiles & Space 135 $693,343,385

Electronics & Communications 1,592 $632,919,208

ERS 4,359 $590,987,877

R&D 519 $415,032,124

Engines & Power Plants 452 $272,723,325

Launchers & Munitions 194 $269,714,770

Other 1,333 $167,492,024

Ground Vehicles 104 $125,907,354

PSC Code Count Sum

R414: Systems Engineering Services 672 $934,139,957

1410: Guided Missiles 62 $651,325,023

1510: Aircraft, Fixed Wing 220 $509,709,584

R425: Support -Professional Engineering 557 $339,143,920

J016: Maint/Repair/Rebuild of Equip 476 $215,031,480

1560 Airframe Structural Components 96 $199,453,727

1680: Misc. Aircraft Accessories 86 $161,457,768

1005: Guns Through 30 MM 47 $145,990,07

J016: Maint/Repair of Aircraft Components 356 $117,614,327

2510: Vehicle Cab Body Frame Structural 9 $117,345,238

NAICS Count Sum

336413: Other Aircraft Parts and Auxiliary 3,399 $1,311,566,685

336411: Aircraft Manufacturing 429 $1,084,203,683

541330: Engineering Services 1,044 $1,082,087,325

336414: Guided Missile and Space Vehi 62 $684,330,795

336412: Aircraft Engine and Engine Par 496 $226,071,643

334511: Search, Detection, Navigation 222 $208,272,040

541712: Research and Development in the 256 $203,694,628

332994: Small Arms, Ordnance, and Ord 57 $140,785,462

541512: Computer Systems Design Services 221 $139,646,028

541710: Research and Development in the 204 $133,528,895

Note. Above descriptions truncated to first 40 characters.

459

Future ResearchBeginning with Research Question 1, magnitude of impact, the anal-

yses outlined in this article established a lower bound for the impact of FAR 6.302-1(b)(2) exceptions at approximately $6 billion for the fiscal years under study. This number does set an order of magnitude for the impact, but as discussed earlier, it falls short because of the way FPDS ingests data and because of the “similar circumstances” provision included in FAR 6.302-1(b)(2) exceptions. The only way to reach a potentially more accurate estimate of the magnitude would be to take a random sample of J&As—approximately 100–200—from all exceptions to full and open categories and retrieve the soft copy J&As from the Governmentwide Point of Entry (GPE).10 After retrieval of all soft copy J&As in the random sample, each would need to be hand-coded to identify reason(s) not competed. Once all of the random sam-ples were coded, the total number of J&As could be resampled according to the distribution of coded J&As. Although this method may generate a more accurate bound, there would be no way to validate the new bound, and the time required to hand-code each J&A in the random sample is prohibitive. Future research is needed to identify a more accurate way to categorize exceptions to full and open competition at the contract level in FPDS.

This analysis was equally concerned with what was being purchased using FAR 6.302-1(b)(2) J&As. One of the key findings was the higher proportion of service contracts to product contracts in 6 of the 8 fiscal years examined. This finding is significant because FAR 6.302-1(b)(2) J&As are convention-ally considered a byproduct of intellectual property protections on physical design process artifacts (i.e., drawings, blueprints, technical data, etc.). Given the results, which counter this conventional understanding, it sug-gests more policy focus should be given to the numerous service contracts that are excluded from full and open competition due to FAR 6.302-1(b)(2) exceptions. Additional research into the justification for these artifacts of FAR 6.302-1(b)(2) is needed to inform future policy decisions.

Future research is needed to identify a more accurate way to categorize exceptions to full and open competition at the contract level in FPDS.

460

Lastly, analyses identified an increasing trend, both in terms of frequency and mean obligations, in the use of FAR 6.302-1(b)(2) exceptions for R&D. This is also an unexpected finding since the very nature of R&D suggests investigation of a novel product. Given the nature of R&D, it is difficult to understand how a contract could be authorized by FAR 6.302-1(b)(2) exceptions. These R&D contracts could be evidence of the “similar circum-stances” section in FAR 6.302-1(b)(2), evidence of an improper use of FAR 6.302-1(b)(2), evidence of a novel use of FAR 6.302-1(b)(2), or evidence of an error in the PSC taxonomy. Regardless, more investigation and research are needed into this finding.

Limitations/RecommendationsThe limitations previously outlined are not only confined to magni-

tude-of-impact estimates, but also to trend analyses. It is impossible to say that the trends discussed previously are, with any certainty, representative of the entire effect. These analyses must acknowledge the possibility that the trends previously discussed may not be a truly accurate representation of the actual trends in DoD acquisition. This difficulty in examining even the simplest questions in DoD acquisition is the most salient finding of this research. Trends are necessary to inform multibillion dollar DoD policy decisions, but gathering even the simplest trends results in a task mired by a complex web of policy and data interactions, confounded by limitations in publicly available databases and unnecessarily broad FAR language. Making a few changes in the way DoD contract data are collected should bring the community a step closer to much richer analyses and more gen-eralizable results:

• Allow for more than one FAR exception to full and open competition in FPDS per contract. This change to the data structure in FPDS should help more accurately define which exceptions are used in each J&A and permit more accurate analyses of trends in noncompetitive contracting types.

• Define or remove “similar circumstances” from FAR 6.302-1(b)(2). This phrasing in the FAR only adds ambi-guity to a section that predominantly concerns intellectual property and technical data. Furthermore, it is unnecessar-ily redundant given the qualifying language already in the FAR: “Use of this authority may be appropriate in situations

461

such as the following (these examples are not intended to be all-inclusive and do not constitute authority in and of themselves) (emphasis added).”

If FAR 6.301-1(b) is not meant to be all-inclusive, what is the necessity for a catchall statement in FAR 6.301-1(b)(2)? Removal of this language still maintains the purpose of FAR 6.302-1(b)(2) given the language in FAR 6.302-1(b).

ConclusionsThis article set out to capture metrics that would yield a better under-

standing of limitations in DoD acquisition due to intellectual property lock-in. In doing so, the analyses established a lower bound on the magnitude of impact at $6 billion and noted trends of decreasing FAR 6.301-1(b)(2) use, but no material trend in the amount of money obligated in contracts using a FAR 6.301-1(b)(2) exception. Furthermore, the prevalent use of FAR 6.301-1(b)(2) exceptions for service contracts was a key finding. Service contracts using FAR 6.301-1(b)(2) exceptions, measured in terms of frequency, were more common than product contracts; however, product contracts using FAR 6.301-1(b)(2) exceptions, in terms of dollars obligated, outpaced service contracts. This finding is salient because it countermands the conventional understanding of what types of contracts are authorized using a FAR 6.301-1(b)(2) exception to full and open competition.

462

ReferencesBerardi, C., Cameron, B., Sturtevant, D., Baldwin, C., & Crawley, E. (2016). Architecting

out software intellectual property lock-in: A method to advance the efficacy of BBP. Proceedings of the 13th Annual Acquisition Research Symposium (pp. 184–201). Monterey, CA.

Carter, A. B. (2010). Better buying power: Mandate for restoring affordability and productivity in defense spending. Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics.

Center for Strategic & International Studies. (2017). Methodology: The techniques, assumptions, and limitations underlying the analysis of defense contracting trends. Retrieved from https://www.csis.org/programs/international-security-program/defense-industrial-initiatives-group/methodology

Department of Defense. (2015). Performance of the Defense Acquisition System, 2015 annual report. Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics.

Ellman, J., & Bell, J. (2015). Analysis of defense products contract trends, 1990-2014 (Vol. HQ0034-12-). Retrieved from http://csis.org/publication/analysis-defense-products-contract-trends-1990-2014

General Accounting Office. (2003). Reliability of federal procurement data (Report No. GAO-04-295R). Washington, DC: U.S. Government Printing Office.

Government Accountability Office. (2010). Opportunities exist to increase competition and assess reasons when only one offer is received (Report No. GAO-10-833). Washington, DC: U.S. Government Printing Office.

Hunter, A. P., McCormick, R., Ellman, J., Sanders, G., Johnson, K., & Coll, G. (2016). Defense acquisition trends, 2015. CSIS Series on Strategy, Budget, Forces, and Acquisition. Washington, DC: Center for Strategic & International Studies.

Hunter, A., Sanders, G., McCormick, R., Ellman, J., & Riley, M. (2015). Measuring the success of acquisition reform by major DoD components. In Proceedings of the 12th Annual Acquisition Research Symposium (Vol. 1). Monterey, CA: Naval Postgraduate School.

Justification & Approval. (2016). In Defense Acquisition University Acquipedia. Retrieved from https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx?aid= 70a60a2f-c14b-4513-b32f-49afa3434999

Kendall, F. (2013). Implementation directive for Better Buying Power 2.0 - Achieving greater efficiency and productivity in defense spending. Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics.

Kendall, F. (2015). Implementation directive for Better Buying Power 3.0 - Achieving dominant capabilities through technical excellence and innovation. Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics.

Liebman, J. B., & Mahoney, N. (2013). Do expiring budgets lead to wasteful year-end spending? Evidence from federal procurement (NBER Working Paper No. 19481). Cambridge, MA: National Bureau of Economic Research. doi:10.3386/w19481

Liedke, E. J., & Simonis, J. D. (2014). Increasing competitive actions: A focus on technical data rights associated with non-commercial hardware items. Monterey, CA: Naval Postgraduate School.

Moore, N. Y., Grammich, C. A., & Mele, J. D. (2015). Findings from existing data on the Department of Defense industrial base: Guided missile and space vehicle

463

manufacturing example. In Proceedings of the 12th Annual Acquisition Research Symposium (pp. 347–365). Monterey, CA: Naval Postgraduate School.

Rogerson, W. P. (1994). Economic incentives and the defense procurement process. The Journal of Economic Perspectives, 8(4), 65–90. Retrieved from http://www.jstor.org.libproxy.mit.edu/stable/2138339

Sanders, G., Ellman, J., & Cohen, S. (2015). Competition and bidding data as an indicator of the health of competition and bidding data as an indicator of the health of the U. S. defense (Vol. HQ0034-12). Washington, DC: Center for Strategic and International Studies.

Shearer, C. (2000). The CRISP-DM model: The new blueprint for data mining. Journal of Data Warehousing, 5(4), 13–22.

USASpending.gov. (2015). In Data Downloads. Retrieved from https://www.usa spending.gov/DownloadCenter/Documents/USAspending.govDownloads DataDictionary.pdf

Endnotes1 FY07 NDAA, Pub. L. No. 109-364 § 802, 120 Stat. 2083 (2007); FY10 NDAA, Pub. L. No. 111-84 § 821, 123 Stat. 2190 (2010); FY11 NDAA, Pub. L. No. 111-383 § 824, 124 Stat. 4137 (2011), FY12 NDAA, Pub. L. No. 112-81 § 815,125 Stat. 1298 (2012); and FY16 NDAA, Pub. L. No. 114-92 § 813, 129 Stat. 726 (2016).

2 The federal government operates on a fiscal year that begins on October 1 and ends September 30 of the following year.

3 Justification and Approval is a document required to justify and obtain appropriate level approvals to contract without providing for full and open competition as required by the Federal Acquisition Regulation (Justification and Approval, 2016).

4 Other than full and open competition is defined as any sole source or limited competition contract action that does not provide an opportunity for all responsible sources to submit proposals.

5 To cross-validate the quality of data, the FPDS totals in each fiscal year were compared to the totals reported on the USASpending.gov website for the DoD. The total obligations in each FPDS fiscal year are within a tolerance of 0.02 percent of the totals listed on the USASpending.gov website. The relatively trivial disparities between the data reported on USASpending.gov and those in FPDS are most likely explained by updates to FPDS between USASpending.gov versions (the data used herein is from the USASpending.gov version [July 15, 2016]).

6 For a full list of variables, see the USASpending.gov data dictionary (USASpending.gov, 2015).

7 Product Service Codes (PSC) that represent products are 4-digit codes (e.g., 7110: OFFICE FURNITURE), whereas PSCs that represent services are alpha numeric 4-digit codes: (e.g., R425: ENGINEERING AND TECHNICAL SERVICES) codes.

464

8 The full CSIS PSC classification tables (for products, services, and R&D) are available on the CSIS GitHub data repository: https://github.com/CSISdefense/Lookup-Tables/blob/master/ProductOrServiceCodes.csv.

9 Of the $6 billion in contract actions, $126,630,720 are foreign military sales.

10 Governmentwide Point of Entry (GPE) means the single point where government business opportunities greater than $25,000, including synopses of proposed contract actions, solicitations, and associated information, can be accessed electronically by the public. The GPE is located at http://www.fedbizopps.gov. (48 CFR 2.101 Definitions)

465

Author Biographies

Maj Christopher Berardi, USAF, is a chief of staff of the Air Force Captains Prestigious PhD Fellows at the Massachusetts Institute of Technology (MIT). Prior to beginning his PhD, he served as both an acquisition and intelligence officer in the U.S. Air Force. Maj Berardi received a Master’s degree in Engineering and Management from MIT and a Bachelor of Science degree in Management from the United States Air Force Academy.


Dr. Bruce Cameron is a lecturer in Engineering Systems at MIT and a consultant on platform strategies. His research interests include tech-nology strategy, system architecture, and the management of product platforms. Dr. Cameron received his undergraduate degree from the University of Toronto, and a Master’s degree in Technolog y Policy and PhD in Engineering Systems from MIT.


466

Dr. Ed Crawley is a professor of Aeronautics and Astronautics and Engineering Systems at MIT. Professor Crawley received a Doctor of Science degree in Aerospace Structures from MIT. His early research interests centered on structural dynamics, aeroelasticity, and the development of actively controlled and intelligent structures. Recently, Dr. Crawley’s research has focused on the domain of the architecture and design of complex systems.


467

The Impact of a BIG DATADecision Support Toolon Military Logistics: MEDICAL ANALYTICS MEETS THE MISSION

Felix K. Chang, Christopher J. Dente, and CAPT Eric A. Elster, USN

Using big data and predictive analytics, more segments of the U.S. military will be able to create decision support tools that help them not only to carry out their missions more efficiently, but also to streamline their logistical requirements. Within the military’s medical community, the Surgical Crit-ical Care Initiative (SC2i) created one such tool that enables physicians to accurately assess the need for massive blood transfusions. To quantify the impact that tool could have on military logistics, SC2i developed a combat model that simulated a military campaign between NATO and Russian forces in eastern Poland and the Baltics. SC2i found that its tool would reduce NATO’s blood product consumption by 71,459 units, eliminating the need for 110 helicopter resupply missions and saving 25,740 gallons of fuel and 129,366 pounds of airlift capacity.

DOI: https://doi.org/10.22594/dau.16-769.24.03 Keywords: Predictive Analytics, Combat Modeling, Decision Science, SC2i

470 Defense ARJ, July 2017, Vol. 24 No. 3 : 468–487

The Impact of a Big Data Decision Support Tool on Military Logistics http://www.dau.mil

Military logisticians have long recognized the efficiency of using deci-sion support tools to streamline logistical systems. In the 1970s, the U.S. military began using automated linear-programing software to better stock and distribute military materiel. Usage of such tools gradually expanded to include those involved in the acquisition, maintenance, and distribution of parts for large and complex military systems. Hence, logisticians and those who were responsible for such systems came to take a leading role in devel-oping many of the military’s most robust decision support tools.

Logistical Benefits from Nonlogistical Decision Support Tools

If the U.S. military is to further streamline its logistics, logisticians should encourage other segments of the military to enhance their own capabilities through the greater use of big data-driven decision support

tools. Such tools can increase efficiency not only in opera-tional units, but also in the logistics that support them.

Any reduction in the volume and weight of sup-plies (particularly perishable ones) needed for

operational units to achieve their missions imparts added resilience to existing logis-

tical capacity.

Today, the cost and time needed to develop new decision support tools are steadily decreasing, especially when developed with machine-learning technology rather than human stat-isticians a lone. As a result, many segments of the military, which have not historically used big data or pre-

dictive analytics, can more easily do so now.

One of those segments has been the U. S . m i l it a r y ’s med ica l com mu n it y.

Traditionally, military medical researchers exclusively focused their energies on improving

the care of wounded warfighters. Now they have begun to use the growing amount of data from clinical

records, laboratory tests, and electronic medical monitors

471Defense ARJ, July 2017, Vol. 24 No. 3 : 468–487

July 2017

to create clinical decision support (CDS) tools that serve dual purposes. While the new tools still help physicians better treat their patients, what often goes unnoticed is that those tools also create substantial logistical benefits. The Surgical Critical Care Initiative (SC2i)—a leading U.S. military health research program in the development of CDS tools—conducted this study to describe the medical impetus behind one such tool and to quantify its clinical and logistical benefits (Buchman et al., 2016; Military Health System Communications Office, 2016).

Measuring the Benefits of a Clinical Decision Support Tool

Over the last decade, medical studies have demonstrated how mas-sive infusions of blood products can improve the clinical outcomes of patients with traumatic injuries (Allcock et al., 2011; Maciel et al., 2015; McDaniel, Etchill, Raval, & Neal, 2014; O’Keeffe, Refaai, Tchorz, Forestner, & Sarode, 2008). Those infusions are called massive transfusion protocols (MTP). They typically involve the infusion of large quantities of red blood cells (RBC), fresh frozen plasma (FFP), platelets, and cryoprecipitate in a fixed ratio.

Recognizing the clinical value of MTPs, military physicians actively used them to treat wounded warfighters during the latter stages of Operations Enduring Freedom and Iraqi Freedom (Beekley, Bohman, & Schindler, 2012, p. 25). Fortunately, sufficient blood products were on hand at the Level III military hospitals supporting those operations to handle the increased demand. Yet, that was largely because those operations produced relatively low casualty rates, particularly after 2007. Higher casualty rates would have quickly drained the blood product inventories of those hospitals, given their limited blood storage capacities and the perishability of the blood products themselves.

Further complicating matters, the medical assessment as to whether a patient needs an MTP is complex to make, even for well-trained physicians. It is an assessment that forces them to weigh multiple disparate factors related to a patient’s acuity and mechanism of injury. Many physicians get it wrong (Wijaya, Cheng, & Chong, 2016). They needlessly activate MTPs and waste blood products in the process.

To reduce that wastage, SC2i developed a CDS tool that enables physicians to quickly and accurately identify which patients require an MTP based on their individual anatomic and biological data. Atlanta’s largest trauma

472

center, which annually treats nearly a thousand patients with penetrating wounds, is now using SC2i’s MTP CDS tool in a prospective observation trial. A study at the trauma center concluded that such a tool could lower the MTP activations by as much as 17.9 percent while maintaining posi-tive clinical outcomes (Dente et al., 2010; Shaz, Dente, Harris, MacLeod, & Hillyer, 2009).

Such blood product savings clearly make blood banks more efficient (Haldiman, Zia, & Singh, 2014; O’Keeffe et al., 2008). They also provide the U.S. military with other benefits, particularly when it is involved in high-in-tensity combat. Keeping deployed Level III military hospitals, like the U.S. Army’s combat support hospitals (CSH), fully supplied with blood products requires a multitude of resources. Enabling physicians to more accurately decide whether an MTP is necessary means that fewer blood products would be wasted, which in turn means that military logistics would need to resupply CSHs less often, freeing up logistical resources that can support other medical or combat requirements.

Modeling Methods To fully understand the scale of the benefits from SC2i’s MTP CDS

tool, it must be put into the context of a military campaign. The campaign scenario that SC2i chose was based on one created for an unclassified U.S. Department of Defense (DoD) contingency planning study in the 1990s (Tyler, 1992a; 1992b). SC2i’s scenario focused on Russia, the top security concern of the last two chairmen of the U.S. Joint Chiefs of Staff (Shinkman, 2015; Stewart & Alexander, 2015). It envisioned a broad-front NATO military campaign to liberate eastern Poland and the Baltics after a Russian invasion. That scenario turned out to be particularly relevant, as variations of it have recently been used as the foundation for NATO’s Sabre Strike exercise and DoD-sponsored wargaming (Sharkov, 2016; Shlapak & Johnson, 2016).

SC2i developed that scenario into a combat simulation to estimate the vol-ume and rate of casualties that a campaign would produce. Those results were then fed into a casualty relevance model to determine the number of daily casualties that would likely require an MTP. Finally, those casualty figures enabled SC2i to assess the difference in daily blood product usage with and without its MTP CDS tool, and ultimately the tool’s logistical and operational benefits.

Combat simulation. Like the original DoD study, SC2i employed Lanchester’s square law force-exchange framework to power its combat

473

simulation (Davis, 1990; Johnson, 1990; Kaufmann, 1992, pp. 57–59). That framework is embodied in the equations below:

Bt =[( r1/2

b1/2B–

(

R e(rb)1/2t +( r1/2

b1/2B–

(

R e-(rb)1/2t ].5

Rt =[( b1/2

r1/2R–

(

B e(rb)1/2t +( b1/2

r1/2R–

(

B e-(rb)1/2t ].5

B = NATO combat powerR = Russian combat powerb = NATO combat effectivenessr = Russian combat effectivenesst = Combat days from the start of the campaign

SC2i did, however, update three of the DoD study’s assumptions to better reflect the modern combat conditions of its scenario (Table 1). Using those updates, SC2i’s combat simulation calculated the number of likely casualties to be generated each combat day (i.e., a day in which all forces are engaged) over the course of its campaign scenario. In total, the simulation estimated that NATO would suffer 33,806 casualties, of whom 6,938 would be killed in action (KIA) and 26,868 wounded in action (WIA), before it achieved victory over its Russian foe (Table 2).

TABLE 1. COMBAT SIMULATION ASSUMPTIONS

Assumption U.S. Department of Defense, 1992

Surgical Critical Care Initiative, 2017

Rationale

NATO and Russian division equivalents modeled in the combat simulation

24 15

Standing militaries in NATO and Russia are smaller today than they were during the Cold War

Combat power ratio between NATO and Russian forces

1.4 1.1

Technological gap between NATO and Russian forces has narrowed, because the pace of Russian military modernization has oustripped NATO’s over the last decade

Percentage of enemy forces destroyed to achieve victory

100% 50%

Combat formations are rendered ineffective well before they are completely destroyed

474

TAB

LE 2

. C

OM

BA

T SI

MU

LATI

ON

RE

SULT

S

Co

mb

at D

ay (

t)N

ATO

Co

mb

at

Pow

er (

Bt)

Rus

sian

Co

mb

at

Pow

er (

Rt)

NA

TO

Cas

ualt

ies

NA

TO K

illed

in

Act

ion

NA

TO W

oun

ded

in

Act

ion

Rus

sian

C

asua

ltie

sR

ussi

an K

illed

in

Act

ion

Rus

sian

W

oun

ded

in

Act

ion

016

.50

15.0

00

00

00

01

16.3

514

.67

1,75

43

60

1,3

94

3,2

86

674

2,6

122

16.2

114

.35

1,7

143

52

1,3

62

3,2

55

66

82

,58

73

16.0

614

.02

1,6

77

34

41,

33

33

,227

6

62

2,5

65

415

.93

13.7

01,

63

83

36

1,3

02

3,1

99

65

72

,54

25

15.7

913

.39

1,6

01

32

91,

272

3,17

26

51

2,5

21

615

.66

13.0

71,

56

33

21

1,24

23

,14

56

45

2,5

00

715

.53

12.7

61,

527

313

1,2

143

,119

64

02

,479

815

.40

12.4

51,

48

93

06

1,18

33

,09

36

35

2,4

58

915

.28

12.1

41,

45

42

98

1,15

63

,06

86

30

2,4

38

1015

.16

11.8

41,

417

29

11,

126

3,0

44

62

52

,419

1115

.04

11.5

41,

38

12

83

1,0

98

3,0

206

202

,40

012

14.9

311

.24

1,3

46

276

1,0

702

,99

86

152

,38

313

14.8

210

.94

1,3

1026

91,

04

12

,974

610

2,3

64

1414

.71

10.6

41,

276

262

1,0

142

,95

36

06

2,3

47

1514

.61

10.3

51,

240

25

49

86

2,9

32

60

22

,33

016

14.5

010

.06

1,20

624

89

58

2,9

115

97

2,3

1417

14.4

09

.77

1,17

224

19

31

2,8

91

59

32

,29

818

14.3

19

.48

1,13

82

34

90

42

,87

15

89

2,2

82

1914

.21

9.2

01,

104

227

87

72

,85

25

85

2,2

67

2014

.12

8.9

21,

07

12

208

51

2,8

34

58

22

,25

22

114

.04

8.6

31,

03

72

138

242

,816

578

2,2

38

22

13.9

58

.35

1,0

03

206

797

2,7

99

574

2,2

25

23

13.8

78

.08

97

119

97

722

,78

25

71

2,2

1124

13.7

97.

80

93

819

374

52

,76

65

68

2,1

98

25

13.7

17.

52

90

618

672

02

,75

05

64

2,1

86

2613

.64

7.2

58

7317

96

94

2,7

35

56

12

,174

Tota

l33

,80

66

,938

26,8

68

77,4

92

15,9

04

61,

588

475

Casualty relevance model. SC2i’s model then iteratively narrowed down which among those estimated casualties would likely be candidates for a massive transfusion. First, the model distinguished between all those casualties who were either KIA or returned to duty (RTD) before they were treated at a hospital, from those WIA who were treated at a hospital, since only hospitalized patients could receive a massive transfusion. To ascertain how many casualties were likely to be KIA, RTD, and treated WIA, SC2i sought data from a historical campaign wherein its high-intensity combat mirrored what NATO could expect to encounter during its campaign scenario. Hence, SC2i selected the experience of the U.S. First and Third Armies in eastern France and Germany during World War II. Both armies fought over terrain and were exposed to the type and volume of munitions that NATO forces would likely face in eastern Poland and the Baltics.

From 1944 to 1945, the U.S. First and Third Armies suffered 752,396 casualties. Of those, 152,359 casualties were KIA (or 20.2 per-cent of the total) and approximately 150,000 (or 19.9 percent of the total) were RTD. The remaining 450,037 (or 59.8 percent of the total) were WIA who were treated at field hospitals (Holcomb, Stansbury, Champion, Wade, & Bellamy, 2006; Reister, 1975, p. 4). Using those historical per-centages, SC2i’s model determined the number of treated WIA each combat day of its scenario.

Second, the model sought to ascertain which among those treated WIA would have received wounds severe enough to prompt physicians to consider using a massive transfusion. To do so, SC2i examined the impact that casu-alty-causing agents could have in producing such wounds. Again, SC2i turned to the experience of the U.S. First and Third Armies. Military records from the two armies revealed that of their combined 217,070 wounded, 24.6 percent suffered wounds from small arms, 60.2 percent from shell fragments, 3.2 percent from blasts, 4.9 percent from bombs, 1.2 percent from burns, and 5.9 percent from all other casual-ty-causing agents (Beyer, Arima, & Johnson, 1962; Beyer et al., 1962, p. 77). Judging that small arms, shell fragments, blasts, and bombs would be the most relevant agents, SC2i used their historical incidence to better estimate the number of likely MTP candidates.

476

To improve on that estimate, SC2i studied the likely severity of wounds based on their physical locations. Once again, SC2i sought data from a historical campaign. This time, however, it had to choose one in which American troops were not only engaged in sustained high-intensity combat, but also equipped with body armor commensurate with that of their poten-tial modern contemporaries. SC2i chose the experience of the U.S. Eighth Army during the last year of the Korean Conflict. Of the U.S. Eighth Army’s casualties who wore body armor, 14.2 percent suffered wounds to the head, 2.7 percent to the neck, 4.7 percent to the chest, 4.0 percent to the upper back, 9.2 percent to the lower back, 1.6 percent to the abdomen, 34.6 percent to an upper extremity, 28.4 percent to a lower extremity, and 6.0 percent to the genitalia (Herget, Coe, & Beyer, 1962, p. 733). SC2i then asked physicians from Walter Reed National Medical Center and the Uniformed Services University of the Health Sciences to assess the likelihood that wounds in those areas of the body would be severe enough to warrant the consideration of a massive transfusion. The combination of historical data and physicians’ assessments enabled SC2i to further refine its model. Ultimately, it found that 11,556 wounded—about one-third of the total casualty population—would be likely MTP candidates.

Logistical benefits model. With an estimate of likely MTP candidates each combat day, SC2i’s model could determine the daily demand for blood products. It could also determine the difference in that demand with and without the use of SC2i’s MTP CDS tool, and thus calculate the blood prod-uct savings that the tool could generate. However, before the model could do so, it required a few programmed assumptions. It assumed that NATO, following doctrine, would assign one CSH to each of its five three-division corps (Lewis et al., 2010, p. xi). It also assumed that NATO would adopt SC2i’s MTP blood product ratio of 16.5 units of RBC, 9.8 units of FFP, 0.9 apheresis of platelets, and 7.2 units of cryoprecipitate as its standard massive transfusion protocol, rather than the U.S. Armed Services Blood Program (ASBP)’s suggested blood product ratio (Departments of the Army, Navy, & Air Force, 2011, p. 43). SC2i did so to ensure that its model could make a clear-cut comparison of blood product demand with and without the use of its tool.

Moreover, SC2i’s model had to contend with the fact that blood products are consumed by not only wounded who require MTPs, but also those who do not. To account for the latter’s use, SC2i assumed that they would consume the ASBP’s suggested ratio of 3.0 units of RBC, 1.6 units of FFP, and 0.15 apheresis of platelets per patient.

477

SC2i’s model also had to consider the inventory of blood products that each CSH would initially carry with it in-theater. The model assumed that each CSH would deploy with an inventory of 300 units of RBC, 100 units of FFP, 24 apheresis of platelets, and 100 units of cryoprecipitate, which is consistent with the recommendations found in Emergency War Surgery (Department of the Army, 2013). Of course, that does not always happen. When the 31st CSH deployed to Iraq in 2010, it stocked only 180 units of RBC, 160 units of FFP, and 90 units of cryoprecipitate (Luschinski, 2011).

Even so, given the high casualty rate of its combat simulation, SC2i’s model expected that all of the deployed CSHs would quickly exhaust their blood product inventories. As in the past, CSHs would call upon in-theater blood donors to replenish some of their blood supplies. SC2i estimated that such “walking blood banks” would provide 1,200 units of RBC every combat day across all five CSHs. NATO would have to source the remaining blood product shortfall from outside the theater, most likely from the ASBP’s blood reserve in the continental United States (Armed Services Blood Program, n.d.).

The task of resupplying the CSHs with fresh blood products would then fall on U.S. military logistics. Transporting such perishable supplies from the United States to CSHs near the frontline requires a considerable effort. It begins with the packaging of blood products into ASBP-standard ship-ping containers. The containers are then assembled into groups of 120 and placed on pallets—those carrying RBC weighing 5,400 pounds and those carrying other blood products weighing 4,680 pounds (Departments of the Army, Navy, & Air Force, 2007, p. 41; 2011, pp. 45–46). U.S. Air Mobility Command (AMC) would then transport those pallets into theater airheads where blood distribution detachments would divide the pallets into smaller blood product shipments and route them to corps-level airfields. From there, UH-60 MEDEVAC helicopters would fly the shipments on blood product

As the technology needed to capture and analyze big data becomes more widely available and at ever-lower cost, even military elements not directly engaged in logistics can create new decision support tools—tools that not only enable them to more efficiently carry out their missions, but also reduce their logistical footprints.

478

resupply missions to individual CSHs, as they have done in every U.S. mil-itary campaign since the Persian Gulf Conflict (Cholek & Anderson, 2007; Department of Defense, 1992, p. 463; Department of the Army, 2005, p. H-6).

Since UH-60 pilots are generally required to hold a 20- to 30-minute fuel reserve, a UH-60 operating at its maximum range would be expected to consume about 85 percent of its 360-gallon internal fuel load (IHS Jane’s, 2008; M. Crivello, personal communication, December 26, 2014). Since not all blood product resupply missions require UH-60s to operate at their maximum range, SC2i’s model estimated that the average blood product resupply mission would consume only 65 percent of their internal fuel load.

According to the ASBP, each UH-60 MEDEVAC helicopter can carry up to 50 standard shipping containers (Departments of the Army, Navy, & Air Force, 2011, p. 44). While that might be true for empty helicopters, MEDEVAC helicopters operating in combat would be loaded with armor, racks, and medical equipment. Given the large number of wounded that would require battlefield evacuation during a high-intensity conflict, UH-60 crews are unlikely to reconfigure their helicopters to fly an occasional blood product resupply mission. Moreover, a veteran pilot revealed that the most a combat-configured UH-60 MEDEVAC helicopter could carry is 30 standard shipping containers (Ginn, Ferencz, & Marble, 2008; M. Crivello, personal communication, May 7, 2015). Hence, that is the number SC2i’s model used.

By linking the blood products’ savings that SC2i’s MTP CDS tool could produce with the resources that would have been needed to transport the products to CSHs in the field, SC2i’s model quantified the logistical benefits that would accrue from the tool over the course of its campaign scenario.

Modeling ResultsImpact on blood product usage. Given that SC2i’s MTP CDS tool was

principally designed to support medical decision making, it is no surprise that the tool improves the treatment of critically injured patients. It allows physicians to quickly and accurately determine whether such patients require an MTP, enabling them to reap the full clinical benefit from an early MTP activation. In so doing, the tool also reduces the number of unneeded MTP activations and the associated waste in blood products. SC2i’s model estimates that if NATO’s five CSHs use the tool, they would waste 71,459 fewer units of blood products over the course of SC2i’s campaign scenario. Those blood products would include 34,144 units of RBC, 20,399 units of FFP, 1,907 units of platelets, and 15,009 units of cryoprecipitate (Table 3).

479

TAB

LE 3

. C

ASU

ALT

Y R

ELE

VA

NC

E A

ND

LO

GIS

TIC

AL

BE

NE

FIT

S M

OD

ELS

RE

SULT

S

Co

mb

at

Day

NA

TO

Wo

und

ed

in A

ctio

n

MT

P-

Rel

evan

t W

oun

ded

in

Act

ion

Blo

od

Pro

duc

t U

sag

eC

linic

al a

nd O

per

atio

nal S

avin

gs

fro

m S

C2i

MT

P C

DS

Too

l

Wit

hout

SC

2i M

TP

CD

S To

ol

(uni

ts)

Wit

h SC

2i M

TP

CD

S To

ol

(uni

ts)

Red

Blo

od

Cel

l (un

its)

Fres

h Fr

ozen

P

lasm

a (u

nits

)

Pla

tele

t (u

nits

)C

ryo

-p

reci

pit

ate

(uni

ts)

Blo

od

Pro

duc

t C

ont

aine

rs

UH

-60

B

loo

d P

rod

uct

Res

upp

ly

Mis

sio

ns

Jet-

A F

uel

(gal

lons

)

Air

lift

Wei

ght

C

apac

ity

(po

und

s)

11,

39

46

00

19,5

38

15,8

30

1,7

721,

05

89

97

7915

95

1,17

06

,73

52

1,3

62

58

62

1,6

28

18,0

05

1,73

11,

03

49

776

115

45

1,17

06

,52

23

1,3

33

573

21,

140

17,5

95

1,6

94

1,0

129

574

515

25

1,17

06

,43

84

1,3

02

56

020

,626

17,1

63

1,6

54

98

89

272

715

05

1,17

06

,35

45

1,27

25

47

20,1

36

16,7

52

1,6

179

66

90

711

145

51,

170

6,1

35

61,

242

53

419

,63

516

,33

11,

579

94

38

86

94

141

51,

170

5,9

67

71,

214

52

219

,16

015

,93

21,

54

29

21

86

678

138

51,

170

5,8

38

81,

183

50

918

,65

815

,511

1,5

04

89

88

46

61

134

51,

170

5,6

709

1,15

64

97

18,1

96

15,1

23

1,4

69

87

78

26

45

131

51,

170

5,5

41

101,

126

48

417

,70

814

,713

1,4

31

85

58

06

29

127

51,

170

5,3

7311

1,0

98

472

17,2

33

14,3

131,

39

58

33

786

1312

55

1,17

05

,29

512

1,0

704

60

16,7

7013

,92

51,

35

98

1276

59

812

20

05

,16

613

1,0

41

44

816

,29

513

,526

1,3

23

790

745

82

119

51,

170

5,0

37

141,

014

43

615

,84

713

,14

91,

28

97

7072

56

711

65

1,17

04

,914

159

86

424

15,3

71

12,7

50

1,2

52

748

705

51

113

51,

170

4,7

85

169

58

412

14,9

22

12,3

731,

218

728

68

53

610

95

1,17

04

,617

179

31

40

114

,473

11,9

96

1,18

470

76

65

2010

65

1,17

04

,48

818

90

43

89

14,0

25

11,6

191,

149

68

76

45

05

103

51,

170

4,3

59

198

77

37

713

,574

11,2

41

1,11

56

66

62

49

010

00

04

,23

620

85

13

66

13,1

40

10,8

761,

08

26

46

60

475

97

00

4,1

07

21

824

35

412

,69

010

,49

81,

04

76

265

84

60

94

00

3,9

782

279

73

43

12,2

42

10,1

22

1,0

136

05

57

44

58

95

1,17

03

,76

52

37

723

32

11,8

209

,76

79

81

58

65

54

31

88

51,

170

3,7

2624

745

32

111

,38

49

,40

19

47

56

65

34

168

55

1,17

03

,59

72

572

03

1010

,96

19

,04

69

155

47

51

40

28

15

1,17

03

,42

326

69

42

98

10,5

25

8,6

80

88

25

274

93

88

785

1,17

03

,30

0

Tota

l26

,86

811

,556

34,1

44

20,3

99

1,9

07

15,0

09

3,0

5611

025

,74

012

9,3

66

480

Impact on military logistics. Beyond the blood product savings it could generate, SC2i’s MTP CDS tool also yields a number of operational benefits. SC2i’s model estimates that the tool’s use would enable UH-60 MEDEVAC helicopters to fly 110 fewer blood product resupply missions during SC2i’s campaign scenario. Instead, those UH-60s could fly medical

evacuation missions. Given a helicopter’s maximum airlift capacity, that means existing corps-level UH-60 resources

could evacuate up to 770 more wounded off the battlefield.

Should no additiona l wounded require evacuation, SC2i’s model

estimates that the fewer num-ber of blood product resupply

missions would save a total of 25,740 gallons of Jet-A fuel. Since NATO AH-64 attack helicopters a lso consume Jet-A fuel and have a similar fuel capac-it y a s the UH-60, they could use the saved fuel to

fly an additional 110 close-air support missions.

The broader logistical bene-fits are equally meaningful.

The lower need for blood products would also reduce

the need for AMC fixed-wing aircraft to ferry blood prod-

ucts in-theater. That would free up space and weight aboard

AMC f lights that NATO could use to carry other military sup-

plies. SC2i’s model estimates that the lower blood product demand

would mean that the A MC would have to transport 3,056 fewer blood

product shipping containers. Those con-tainers have a combined weight of 129,366

pounds. That is equivalent to the combined

481

weight of 308 AGM-114 missiles, 1,463 Hydra 70 rockets, and 92,400 rounds of 30mm ammunition—enough to arm 77 of the additional AH-64 close-air support missions that SC2i’s MTP CDS tool could enable U.S. Army aviation units to f ly (Figure).

FIGURE. SUMMARY OF CLINICAL & LOGISTICAL BENEFITS

FROM SC2I’S MTP CDS TOOLS

Blood Product Savings Jet-A Fuel Savings Close-Air SupportMissions

MEDEVAC Wounded

Close-Air SupportOrdnance

Airlift WeightCapacity Savings

34,144 Red Blood Cell20,399 Fresh Frozen Plasma

1,907 Platelets15,009 Cryoprecipitate

25,740 gallons 770 110

129,366 pounds

92,400 30mm1,463 Hydra 70308 AGM-114

Conclusions/RecommendationsThis study illustrated the scale of the benefits that could accrue from

using SC2i’s MTP CDS tool during a military campaign. Although military medical researchers primarily designed the tool to help physicians make faster and more accurate decisions as to whether to activate an MTP and thus improve patient survival and recovery, it served a dual purpose. The tool’s developers also sought to lower the unneeded expenditure of scarce blood products, streamlining the logistical requirements of CSHs in the field.

Past efforts to streamline logistics have largely been led by logisticians and those who manage large and complex military systems. This study demon-strates the degree to which all military organizations can make meaningful contributions to streamlining logistics by using big data and predictive analytics to improve their own operations. More elements of the military can and should become involved. As the technology needed to capture and analyze big data becomes more widely available and at ever-lower cost,

482

military elements not directly engaged in logistics can create new decision support tools—tools that not only enable them to more efficiently carry out their missions, but also reduce their logistical footprints.

SC2i’s model quantified what one decision support tool could do to ease the burden on military logistics by reducing waste in a single supply category—blood products. The U.S. military can develop and deploy many more tools in the future. They and the predictive analytics that underlie them should be encouraged to flourish.

AcknowledgmentsWe would like to acknowledge the important contributions of Michael

J. Crivello, Dr. Frederick Lough, and Dr. Benjamin Kyle Potter in the Department of Surgery at the Uniformed Services University of the Health Sciences and Walter Reed National Military Medical Center.

483

ReferencesAllcock, E. C., Woolley, T., Doughty, H., Midwinter, M., Mahoney, P. F., & Mackenzie,

I. (2011). The clinical outcome of UK military personnel who received a massive transfusion in Afghanistan during 2009. Journal of the Royal Army Medical Corps, 157(4), 365–369.

Armed Services Blood Program. (n.d.). Armed Forces Blood Program Facts [Fact sheet]. Retrieved from http://www.med.navy.mil/sites/nmcp/Dept/Shared Documents/BloodBank/BloodProgramFactSheet.pdf

Beekley, A. C., Bohman, H., & Schindler, D. (2012). Combat casualty care: Lessons learned from OEF and OIF. In E. Savitsky & B. Eastridge (Eds.), Modern warfare. Washington, DC: Department of the Army, Office of the Surgeon General.

Beyer, J. C., Arima, J. K., & Johnson, D. W. (1962). Enemy ordnance materiel. In J. C. Beyer (Ed.), Wound ballistics. Washington, DC: Department of the Army, Office of the Surgeon General.

Buchman, T. G., Billiar, T. R., Elster, E., Kirk, A. D., Rimawi, R. H., Vodovotz, Y., & Zehnbauer, B.A. (2016). Precision medicine for critical illness and injury. Critical Care Medicine, 44(9), 1635–1638.

Cholek, C. B., & Anderson, M. A., Sr. (2007). Distribution-based logistics in Operation Iraqi Freedom. Army Logistician, 39(2), 60.

Davis, P. K. (1990). Aggregation, disaggregation, and the 3:1 rule in ground combat. Santa Monica, CA: RAND.

Dente, C. J., Shaz, B. H., Nicholas, J. M., Harris, R. S., Wyrzykowski, A. D., Ficke, B. W., Vercruysse, G. A., … & Ingram, W. L. (2010). Early predictors of massive transfusion in patients sustaining torso gunshot wounds in a civilian level I trauma center. Journal of Trauma, 68(2), 300–303.

Department of the Army. (2005). Theater hospitalization (FM 4-02.10). Washington, DC: Author.

Department of the Army. (2013). Battlefield transfusions. In Emergency war surgery (4th ed.). Washington, DC: Office of the Surgeon General & Borden Institute.

Departments of the Army, Navy, & Air Force. (2007). Operational procedures for the Armed Services Blood Program elements (TM 8-227-11, NAVMED P-5123, AFI 44-118). Washington, DC: Author.

Departments of the Army, Navy, & Air Force. (2011). Armed Services Blood Program: Joint Blood Program handbook (TM 8-227-12, NAVMED P-6530, AFH 44-152_IP). Washington, DC: Author.

Department of Defense. (1992). Medical support (Appendix G). In Conduct of the Persian Gulf War: Final report to Congress. Washington, DC: Author.

Ginn, R.V.N., Ferencz, A., & Marble, S. (Eds.). (2008). In their own words: The 498th in Iraq, 2003. Falls Church, VA: Office of Medical History, Office of the Surgeon General, Headquarters, U.S. Army Medical Command.

Haldiman, L., Zia, H., & Singh, G. (2014). Improving appropriateness of blood utilization through prospective review of requests for blood products: The role of pathology residents as consultants. Laboratory Medicine, 45(3), 264–271.

Herget, C. M., Coe, G. B., & Beyer, J. C. (1962). Wound ballistics and body armor in Korea. In J. C. Beyer (Ed.), Wound ballistics. Washington, DC: Department of the Army, Office of the Surgeon General.

484

Holcomb, J. B., Stansbury, L. G., Champion, H. R., Wade, C., & Bellamy, R. F. (2006). Understanding combat casualty care statistics. Journal of Trauma, 60(2), 397–401.

IHS Jane’s. (2008). Sikorsky S-70 (H-60) - US Army UH-60Q DUSTOFF MEDEVAC Conversion. Jane’s Aircraft Upgrades. Retrieved from https://my.ihs.com/Janes?th=JANES&callingurl=https://janes.ihs.com

Johnson, R. L. (1990). Lanchester’s square law in theory and practice. Fort Leavenworth, KS: U.S. Army Command and General Staff College, School of Advanced Military Studies.

Kaufmann, W. W. (1992). Assessing the base force. Washington, DC: Brookings Institution.

Lewis, M. W., Bower, A., Cuyler, M. T., Eden, R., Harper, R.E., Morganti, K. G., … Valdez, R. S. (2010). New equipping strategies for combat support hospitals. Santa Monica, CA: RAND.

Luschinski, A. (2011). Laboratory services in austere environment: Camp Dwyer, Afghanistan. Society Scope, 14(2), 9.

Maciel, J. D., Gifford, E., Plurad, D., de Virgilio, C., Bricker, S., Bongard, F., … & Kim, D. (2015). The impact of a massive transfusion protocol on outcomes among patients with abdominal aortic injuries. Annals of Vascular Surgery, 29(4), 764–769.

McDaniel, L. M., Etchill, E. W., Raval, J. S., & Neal, M. D. (2014). State of the art: Massive transfusion. Transfusion Medicine, 24(3), 138–144.

Military Health System Communications Office. (2016, March 7). Precision medicine research paving the way for smarter, more effective treatment. Health.mil. Retrieved from http://www.health.mil/News/Articles/2016/03/07/Precision-medicine-research-paving-the-way-for-smarter-more-effective-treatment

O’Keeffe, T., Refaai, M., Tchorz, K., Forestner, J. E., & Sarode, R. (2008). A massive transfusion protocol to decrease blood component use and costs. Archives of Surgery, 143(7), 686–691.

Reister, F. A. (1975). Medical statistics in World War II. Washington, DC: Department of the Army, Office of the Surgeon General.

Sharkov, D. (2016, May 29). US kicks off 10,000-strong drill in Eastern Europe. Newsweek. Retrieved from http://www.newsweek.com/us-kicks-strong-drill-eastern-europe-464678

Shaz, B. H., Dente, C. J., Harris, R. S., MacLeod, J. B., & Hillyer, C. D. (2009). Transfusion management of trauma patients. Anesthesia and Analgesia, 108(6), 1760–1768.

Shinkman, P. D. (2015, March 4). Top military minds list top threats facing the U.S. U.S. News and World Report. Retrieved from http://www.usnews.com/news/articles/2015/03/04/ash-carter-martin-dempsey-list-top-3-threats-facing-the-us

Shlapak, D. A., & Johnson, M. W. (2016). Reinforcing deterrence on NATO’s eastern flank: Wargaming the defense of the Baltics. Santa Monica, CA: RAND Corporation.

Stewart, P., & Alexander, D. (2015, July 9). Russia is top U.S. national security threat: Gen. Dunford. Reuters. Retrieved from http://www.reuters.com/article/us-usa-defense-generaldunsmore-idUSKCN0PJ28S20150709

Tyler, P. E. (1992a, February 17). Pentagon imagines new enemies to fight in post-Cold-War era. New York Times, pp. A1, A8.

485

Tyler, P. E. (1992b, February 17). 7 Hypothetical conflicts foreseen by the Pentagon. New York Times, p. A8.

Wijaya, R., Cheng, H. M., & Chong, C. K. (2016). The use of massive transfusion protocol for trauma and non-trauma patients in a civilian setting: What can be done better? Singapore Medical Journal, 57(5), 238–241.

486

Author Biographies

Mr. Felix K. Chang is the chief strategy officer of DecisionQ. He is also an assistant professor in the Department of Surgery at the Uniformed Services University of the Health Sciences and Walter Reed National Military Medical Center. He is also a senior fellow at the Foreign Policy Research Institute. He previously served as a senior planner and an intelligence officer in the U.S. Department of Defense. Mr. Chang earned his MBA from Duke University and an MA and BA from the University of Pennsylvania.


Dr. Christopher J. Dente is a professor in the Department of Surgery at Emory University’s School of Medicine and deputy chief of Surgery for Emory at Grady Memorial Hospital. He earned his MD from the Medical College of Pennsylvania and completed his general surgical training at Wayne State University/Detroit Medical Center. Dr. Dente also completed a trauma/surgical crit-ical care fellowship at Grady Memorial Hospital.


487

CAPT Eric A. Elster, USN, is currently chair-man/professor of the Department of Surgery, Uniformed Services University of the Health Sciences and Walter Reed National Military Medical Center. He was last deployed as director of Surgical Services at NATO’s military hospital in Kandahar, Afghanistan. He has published over 140 scientific manuscripts and received numerous research grants. CAPT Elster earned his MD from the University of South Florida, completed a surgery residency at National Naval Medical Center, and received an organ transplantation fellowship at National Institutes of Health.


BEYOND INTEGRATIONReadiness Level (IRL):A Multidimensional Framework to Facilitate the

INTEGRATION OF SYSTEM OF SYSTEMS

Maj Clarence Eder, USAF (Ret.), Thomas A. Mazzuchi, and Shahram Sarkani

Integration Readiness Level (IRL) can be an effective systems engineering tool to facilitate integration of systems. Further research in systems integration, analysis of integration data, and the development of systems architecture framework can help enhance IRL principles for systems integration use in Department of Defense (DoD). IRL was developed to support Technology Readiness Level, but DoD never implemented IRL. Expanding the use of IRL can address growing integration challenges of DoD acquisition programs. DoD Space Systems are prime examples of System of Systems that can help identify attributes for an integration framework that can enhance IRL assessment.

DOI: https://doi.org/10.22594/dau.16-766.24.03 Keywords: Technology Readiness Level (TRL), Systems Integration, DoD Space Systems



Beyond Integration Readiness Level http://www.dau.mil

Integration Readiness Level (IRL) was introduced to help understand the maturity of integrating one system into another (Sauser, Gove, Forbes, & Ramirez-Marquez, 2010). The need to expand the use of IRL is becoming increasingly more relevant in U.S. Department of Defense (DoD) acquisition as program managers aim to develop and acquire weapon systems with ever increasing multiple capabilities and interfaces. Likewise, understanding

integration feasibility early in the program is beneficial in man-aging and planning for the success of overall System of

Systems (SoS) integration.

Throughout the years, the DoD acquisition com-munity has implemented several systems

engineering processes and tools to help meet budgetary requirements while still trying to produce the best weapon systems available. Also, DoD Space Systems acqui-sition has increased in actual systems costs from the initial estimates while the

capability of the systems decreased from original intent. According to a Government

A c c o u n t a bi l i t y O f f i c e (G AO, 2 0 1 1) report, “the total estimated costs for major

space programs increased by about $13.9 billion from initial estimates

for fiscal years 2010 through 2015, which is an increase

of about 286 percent ” (p. 4). This needs to be

managed better within DoD, and additional

tools are needed to understand future impacts to system delivery.

D o D a l s o i m p l e -mented initiatives

in place to expedite t h e d e p l o y m e n t o f

capabilities into opera-tions as part of the Urgent

Wa rf ighter Needs effor ts.


July 2017

Based on a GAO (2012) report, various practices have been implemented by program offices in order to meet challenges to deliver capabilities within short timelines. Having additional integration tools available to help program/product teams understand the feasibility of weapon systems deployment on aggressive timelines could prevent schedule delays.

The DoD acquisition issues of uncontrollable cost growth and the need to expedite the deployment of capabilities into operations trigger the drive to improve systems engineering processes and tools that program man-agers can depend on when making program decisions. In order to make decisions about systems and technology availability, the DoD acquisition community adopted the use of Technology Readiness Level (TRL) in 2002 (DoDI 5000.02, 2017). TRL provides measurement for explaining the matu-rity of a system based on the technology used for that system.

To further the use of TRL, IRL was introduced as an integration tool to complement TRL as illustrated in Table 1 (Sauser et al., 2010). IRL was developed to align with the TRL definitions, but it was never officially imple-mented by DoD to help with integration assessment. Other readiness levels such as System Readiness Level (SRL) and Test Readiness Level were also introduced, but not officially recognized by the DoD acquisition community. Although not implemented, the use of IRL could become a necessary tool to help reduce integration risks of complex systems. Integrating SoS is becoming more complex, and the current definitions of IRL do not allow it to be independent of the TRL process, which could be one reason why IRL is heavily scrutinized in current systems engineering literature. IRL was developed to support the limitations of TRL in evaluating complex inter-faces. According to London, Holzer, Eveleigh, & Sarkani (2014), while TRL is used for “considering discrete technology elements, IRL is only limited to connecting these technology links between components” (p. 380).

Understanding the feasibility of integration early in the process can help temper the expectations needed throughout the development and deployment of new systems.

492

TABLE 1. INTEGRATION READINESS LEVEL (IRL) AND TECHNOLOGY READINESS LEVEL (TRL) LEVELS

Level TRL IRL1 Basic principles observed

and reportedAn interface between technologies has been identified with sufficient detail to allow characterization of the relationship

2 Technology concept and/or application formulated

There is some level of specificity to characterize the interaction between technologies through their interface

3 Analytical and experimental critical function and/or characteristic proof of concept

There is compatibility between technologies to orderly and efficiently integrate and interact

4 Component and/or breadboard validation in laboratory environment

There is sufficient detail in the quality and assurance of the integration between technologies

5 Component and/or breadboard validation in relevant environment

There is sufficient control between technologies necessary to establish, manage, and terminate the integration

6 System/subsystem model demonstration in relevant environment

The integrating technologies can accept, translate, and structure information for its intended application

7 System prototype demonstration in relevant environment

The integration of technologies has been verified and validated with sufficient detail to be actionable

8 Actual system completed and qualified through test and demonstration

Actual integration completed and mission-qualified through test and demonstration in the system environment

9 Integration is mission-proven through successful mission operations

Execute a support program that meets operational support performance requirements and sustains the system in the most cost-effective manner over its total life cycle

Note. As defined by Sauser et al. (2010)

Based on Sauser et al. (2010), IRL was introduced as, “a metric for system-atic measurement of the interfacing of compatible interactions for various technologies and the consistent comparison of the maturity between inte-gration points” (p. 18). The concept itself could be more useful, but it is currently limited to support TRL concepts (Table 1). The main concept of IRL is to be used to evaluate integration of two technologies that were evaluated by the TRL process.

The principles of IRL can be used and enhanced to facilitate the integration of SoS. As indicated by Ge, Hipel, Kewei, & Chen (2013), “SoS is defined as large-scale integrated systems that are heterogeneous and independently operable on their own, but are networked together for a common goal” (p. 363).

493

Many commonly used SoS terms such as interoperable and synergistic are important factors that need to be considered when integrating SoS (Madni & Sievers, 2014). The challenges of SoS integration are constant in DoD acquisition as technology continues to improve and new capabilities emerge as better options for the warfighter. Understanding the feasibility of integration early in the process can help temper the expectations needed throughout the development and deployment of new systems.

Problem FocusThe focus of the research is to understand the integration issues and

challenges of major DoD Space Systems so that they can be used to further analyze facilitation of SoS integration. As challenges continue when inte-grating new capability into a family of systems, the need for understanding the overall effort is necessary for upfront planning. The components that make up SoS usually have different purposes, and the different organiza-tions associated with those purposes typically use different terminologies and concepts. Thus, SoS integration must be able to address the differences in semantic and syntactic between their organizations. Identifying these issues ahead of time helps highlight areas that need additional focus on IRL assessment. DoD Space Systems have to satisfy integration requirements that may include affordability, reliability, security, resilience, and adapt-ability (Madni & Sievers, 2014).

Understanding systems integration itself is a very challenging task due to several factors that impact the integration processes. Systems integration can be interpreted differently, and although integration is a commonly used word, it is not clearly defined by both industry and academia on its applica-tion (Djavanshir & Khorramshahgol, 2007). As explained by Tien (2008),

“integration can occur through functional, physical, organizational, and temporal dimensions, which can also include management and processes of the actual system” (p. 393). Elements involved in integration tend to be multidimensional, which can be more complicated than evaluating tech-nologies. Also, most current systems integration efforts focus on products instead of the process, which can lead to inadequate analyses when trying to understand the impacts of integrating components (Magnaye, Sauser, & Ramirez-Marquez, 2010). Integration of SoS is complicated, which goes beyond merely assembling components.

The focus of the research is to develop a framework to help address SoS integration issues using IRL principles. The subjectivity of TRL and IRL processes has been widely criticized through several journals.

494

Arguably, the literature has oversimplified facets of system readiness and maturity. The risk and effort required for higher readiness levels does not factor into the current assessment of TRL and IRL (Kujawski, 2013). However, the process of conducting an initial assessment of the system prior to the development and integration is beneficial in understanding overall feasibility. Defining feasibility of integration early enough in the process can impact decisions that could avoid integration pitfalls later in the process.

This research provides a tool to facilitate the integration of SoS; however, it does not focus on the development of systems integration requirements. Integration requirements are critical, but this research assumes that the stakeholders have agreed to the requirements before planning the scope that allows integration activities to be executed. Requirements need to be clearly defined to the lowest level and the assumption is to have clear requirements before conducting integration assessment(s). The proposed IRL assess-ment process will be iterative based on the changes in scope (if any) by the stakeholders. Understanding enterprise-wide integration requirements is necessary before developing a specific architectural framework to help with the integration process (Bolloju, 2009). Understanding integration requirements processes can help with traceability of derived requirements during the integration planning and implementation stages.

TheoryIRL can be an effective systems integration assessment tool, and given

the right multidimensional framework, it can facilitate the integration of SoS. Utilizing other integration variables and expanding the current notional definitions of IRL can significantly impact the assessment of SoS integration. IRL was also proposed as an intermediate step by mak-ing it part of a matrix function with TRL in order to determine the SRL (McConkie, Mazzuchi, Sarkani, & Marchette, 2013). When IRL is used as a function of SRL, IRL could be overlooked from being a significant inde-pendent assessment value, and the IRL level may be influenced by what is needed as the SRL value. Current IRL definitions can be useful as an input to the framework as opposed to an intermediate assessment conducted merely to understand technology or system maturity.

Other acquisition practitioners may determine that integration readiness can be assessed as part of the DoD acquisition community’s Technology Readiness Assessment (TRA) process (the official process to determine TRL score), but this process does not capture the purpose of integration. TRL was never intended to measure integration maturity (Magnaye et al., 2010).

495

A system with mature technology does not automatically equate to having a high IRL when interfacing with another system with mature technology.

DoD Space Systems continue to provide examples of complex SoS. With very limited opportunities to do operational tests and analyses for satellite sys-tems and rocket launches, space systems provide a platform to incorporate the latest technologies and processes to attain successful operational sys-tems. As part of space system complexity, space systems also factor in the need for expediting the insertion of cutting-edge technology while adapting to evolution of new technology and reducing qualification timelines (Pittera & Derrico, 2011). IRL can be used to assess the integration of these types of complex SoS given a rigorous process that accounts for other variables.

Literature ReviewUse of the IRL process has been documented through abundant lit-

erature with different strategies of application. The most common use is an interim tool to support TRL and/or SRL processes. TRL metrics were initially developed from seven levels promulgated by the National Aeronautics and Space Administration in the late 1990s, and then to the nine levels that were adopted by DoD in 2002 (Mankins, 2009). Assessment of TRL became more necessary as technology assessment moved into complex SoS, and DoD acquisition programs needed tools to help make decisions in complex environments (McConkie et al., 2013). The original intent of IRL was to provide a systematic analysis of integrating different

496

technologies and understanding the maturity between the points of integration (Pittera & Derrico, 2011). IRL was focused on being a tool to understand the maturity of technologies used to integrate the systems as opposed to merely a tool to integrate two or more systems.

When the goal is to incorporate the use of IRL into systems integration processes, systems integration strategies and challenges must be identified. Systems integration is a strategy advocated in order to achieve sustainable development—it is very diverse and can be interpreted in many different ways (Vernay & Boons, 2015). System integration strategies have been explained in several journals, which provided different perspectives and helped narrow the focus of this research from supporting technical/system maturity to readiness of integration.

According to Gagliardi, Wood, Klein, and Morley (2009), “severe integration problems can arise due to inconsistencies, ambiguities, and gaps in how quality attributes are addressed in the underlying systems” (p. 12). One of the main reasons for this is not identifying the right quality attribute that supports the integration activities early in the process. This is where a set of quality attributes derived from the DoD Space Systems integration issues can be identified and leveraged to help with systems integration.

Application of systems integration processes can be enhanced when additional systems, processes, and organizations are involved in order to integrate SoS. Generally, SoS can be defined as a collection of interoperating components and systems producing results that are not achievable by each of the individual systems. Adding capabilities to an already complex system epitomizes the definition of SoS (Zhang, Liu, Wang, & Chen, 2012). To fully realize the analysis of SoS, engineering and systems architecture must be used to address allocation of functionality to component interaction as opposed to merely dealing with individual components (Ender et al., 2010).

To understand the properties of SoS, DoD Space Systems will be used as examples of complex SoS. DoD satellites are complex SoS that cannot be accessed for upgrade or changes after they have been placed into orbit. Conducting technical demonstration also requires significant funding and resources for these types of complex SoS (Dubos & Saleh, 2010). Thus, it is imperative that integrating them successfully the first time becomes a high priority.

To understand the comprehensive processes involved in the integration of complex SoS, one must understand how complex SoS are managed. Literature is replete with architectural framework proposals to design, develop,

497

and manage these types of systems (Suh, Chiriac, & Hölttä-Otto, 2015). Many of these larger systems involved multiple organizations as stake-holders, and according to Rabelo, Fishwick, Ezzell, Lacy, and Yousef (2012), “large scale organizations, which have significant operations management needs, seek out and execute complex projects that involve a very large number of resource and schedule requirements” (p. 112). In addition to the wide range of factors involved in complex SoS, recent SoS development requires intensive information technology software, which also triggers the use of an architectural framework (Piaszcyk, 2011).

Architectural frameworks are developed to illustrate operations and functions to help clearly define roles of entities within complex SoS. Understanding and implementing the right architecture and component design will improve performance, resulting in faster integration processes and reduced complexity (Jain, Chandrasekaran, Elias, & Cloutier, 2008). The resulting data from this research will provide the components needed to develop an architectural framework. This framework will help explain how major attributes will contribute to the assessment of enhanced IRL and will help illustrate the different factors needed for consideration during evaluation of IRL at certain, specific timelines.

Major attributes to support the integration assessment framework will be finalized based on the results of the data collection. As explained by Jain et al. (2008), “attribute is a property or characteristics of an entity that can be distinguished quantitatively or qualitatively by human or automated means; the attributes of a system or architecture are used to describe the properties of the system or system architecture in a unique distinguishing manner.” Attributes are usually basic elements related to requirements, which can be interpreted that a requirement consists of multiple attributes (Lung, Xu, & Zaman, 2007).

498

For attributes to be an effective tool for integration, they need to be inte-grated into the framework and optimized to balance complexity and simplicity. Organizations can adopt these attributes when integrating capa-bilities into the family of systems. Complex systems can have unpredictable behaviors, and using multiple attributes can help measure the performance of these behaviors by quantitatively and qualitatively weighing the individ-ual attributes (Gifun & Karydas, 2010).

The initial list of attributes that will be considered for this research is derived from discussions with space systems integration experts, journal articles on integration variables (Jain, Chandrasekaran, and Erol, 2010), and key integration process areas (Djavanshir & Khorramshahgol, 2007). Table 2 shows the list of attributes considered for purposes of this research.

TABLE 2. LIST OF ATTRIBUTES

System Availability Architecture and Design Commercial-off-the-Shelf (COTS)

Complexity Configuration Management Concept of Operations

Documentation Functionality Hardware/Software Verification and Validation

Interface Control Information Assurance Internal/External Organizations and Coordination

Processes and Procedures Planning and Scheduling Programmatic and Risks

Quality Assurance Requirements Management Resources (Funding)

Risk Analysis and Management Semantic Consistency Standards

Schedule Control Strategic Planning Technology Insertion

Testing Training

These potential attributes will be used to help define each integration issue data set that is aligned to a specific integration variable. Some of these attributes may overlap or become part of another attribute. Collecting DoD Space Systems data will help identify and provide the criteria for the major attributes associated with each data set.

GoalsThe goal of this research was to collect data on six major DoD Space

Systems integration issues over the past 16 years, align the data into items that impact integration, group the items into major attributes, and use the

499

attributes to develop a systems integration architectural framework to assess integration feasibility. After the integration items were grouped into major attributes, a survey was solicited from experienced systems engineers, systems integrators, space integrators, and DoD space acquisition person-nel. The goal of the survey was to verify the derived integration attributes and help determine how the attributes are weighted when used in the sys-tems integration assessment framework. Understanding the feasibility of integration ahead of time would help identify risks and mitigation strategies.

The goal of integrating desired capabilities into a family of DoD space/satellite systems is defined notionally in Figure 1. The desired capability of DoD Space Systems is usually integrated into a sensor, which becomes the payload of a satellite. The payload is integrated into a satellite bus and both systems become the satellite system. The satellite system is integrated into ground systems for communications and data exchange. The ground systems are the satellite operations centers, ground antennas, tracking stations, and the locations that process the mission data transmitting from the satellite system.

FIGURE 1. NOTIONAL DOD SPACE SYSTEMS ARCHITECTURE AND MAJOR

INTEGRATION POINTS

Family ofSatellite Systems

GroundSystems

Satellite/BusPayload/Sensor

Launch Vehicle

End Products &Capabilities

DesiredCapabilities

500

Due to its uniqueness, the integration of desired capabilities into the pay-load is usually the most challenging effort in development and integration of space systems. After the payload has been developed and integrated into the satellite bus, it is rigorously tested on the ground. From there, the sat-ellite system is integrated onto a launch vehicle (its ride to get into orbit). The satellite system is also integrated into the family of systems to ensure interoperability with existing systems and produce the final product/data to its users. The launch vehicle is integrated into the ground systems and the satellite system. The ground systems enable communication with all systems throughout launch, deployment, and operations of the satellite system. The desired end state or successful SoS integration includes suc-cessful launch and deployment of a satellite system in orbit that meets (or exceeds) performance of all systems involved.

The goal is to develop a systems integration assessment framework tool based on the attributes identified through data analysis. It is imperative that assessment of complex systems must address common interfaces, interop-erability, reliability, and operations (Bhasin & Hayden, 2008). Methods and processes in systems engineering are used for defining requirements and provide the framework to identify a generalized architecture (Bhasin & Hayden, 2008). One of the major goals is to understand how integra-

tion issues can be addressed early in the acquisition process or planning stages to reduce the number of integration chal-lenges and complexities later in the systems engineering process (Jain et al., 2010).

Cost for systems integration can increase substantially, and implementing processes to understand enterprise-wide systems integration require-ments must be considered before deciding on a solution a nd a rchitecture (Bol loju,

2009). Based on findings by Kruchten, Capilla,

501

and Duenas (2009), architectural elements are the primary constituents of describing “components and connectors, while nonfunctional properties shape the final architecture” (p. 36).

To improve integration of SoS, the understanding of integration issues must first be identified. Table 3 defines the process.

TABLE 3. PROCESS FROM DATA COLLECTION TO INTEGRATION

ASSESSMENT FRAMEWORK DEVELOPMENT

1. Collect high level integration issues from six major DoD major Space Systems from 1999–2014

2. Identify the integration variable that is impacted by each integration issue

3. Group the integration variables based on common focus areas

4. The integration variables will define the parameters for the final set of attributes

5. Analyze how each attribute is distributed throughout all six major space systems in the 16 years of data sets

6. Complete regression analysis of the attributes and determine weights of each attribute

7. Validate attributes and the type of integration variable associated with each attribute by completing a survey of systems integration experts in DoD

8. Analyze survey results using regression and correlations

9. Compare both data analysis and survey analysis; identify final weight of each attribute

10. Based on the analysis, develop a systems integration framework tool with weighted attributes to help facilitate SoS integration process

Once the integration assessment framework is developed, it can be used to help facilitate SoS integration. The output of the framework is the IRL score for the entire SoS, and as each major attribute matures, the IRL score improves. The framework can be used throughout the design, development, and deployment of the SoS. The stakeholders or designated Subject Matter Experts (SMEs) provide upfront analysis and agree on what constitutes the initial IRL level for each of the major attributes. The initial IRL levels for each attribute will be the inputs to the framework. The initial IRL definition for each attribute will be identified from 1 to 9 (using Table 1 as a reference) once the criteria are defined for each major attribute.

Data Collection and ExplorationMajor Space Systems

The integration data will be collected from six major DoD Space Systems. As defined in DoD Instruction 5000.02 (2017), the six space systems are considered “Major Defense Acquisition Programs (MDAP) Acquisition

502

Category I (ACAT I), which are defined as overall program procurement cost of over $2.79 billion (using Fiscal Year 2014 constant dollars)” (p. 49). The major space systems are: (a) Advanced Extremely High Frequency (AEHF) Satellite Communications (SATCOM); (b) Evolved Expendable Launch Vehicle (EELV); (c) Global Positioning Satellite (GPS); (d) National Polar-Orbiting Operational Environment Satellite System (NPOESS); (e) Space Based Infrared Systems (SBIRS); and (f) Wideband Global SATCOM (WGS).

For 16 years from 1999–2014, major integration reporting data from the Director, Operational Test and Evaluation (DOT&E), with additional GAO supporting documents, were collected (DOT&E, n.d.), which explained major integration issues encountered by the six major space programs for each year. The issues identified in the reports are explained, but additional background information was gathered from the U.S. Air Force Space & Missile Systems Center (SMC) program offices. Additional interviews of SMC personnel to gather background information were also conducted for better understanding of some of the issues. These space acquisition programs have encountered challenges in planning, designing, developing, integrating, testing, and deploying different systems throughout the 16-year reporting period. Each major issue identified per year was counted as one issue (i.e., if one issue persisted through 4 years, it will be counted as four issues). Table 4 presents the description and data collection summary of each of the six DoD space systems.

TABLE 4. SIX MAJOR SPACE PROGRAMS: 1999–2014

Description Data Collection Summary

1. AEHF (Advanced Extremely High Frequency) Satellite Communications (SATCOM) is a communications satellite SoS that provides secure and survivable communications capability to U.S. military during any level of conflict. The AEHF capability ensures continuous secure communication during all levels of conflict for warfighters.

There were 79 major integration issues identified for AEHF SATCOM during the 16-year span. Some of AEHF major integration issues included: testing processes for digital processors, integrating nuclear hardening and shielding, electric propulsion, phased array antenna, and nuller spot beam.

2. EELV (Evolved Expendable Launch Vehicle) is an SoS launch vehicle and service provided to DoD and commercial customers. It was developed to provide launch capability that reduces the cost of alternative launch options while meeting mission assurance requirements.

There were 24 major integration issues identified for EELV during the 16-year span. Some of the major issues were data availability to the stakeholders for test verification processes, documentation of flight analysis plan, and integration of several design processes.

503

TABLE 4. SIX MAJOR SPACE PROGRAMS: 1999-2014 (continued)

3. GPS (Global Positioning System) is a major SoS, providing data that both DoD and civilian users are dependent upon to precisely determine their velocity, time, and position. The GPS constellation calls for 24 satellites that provide real-time and highly accurate positioning and location data. It can function through all weather and provides passive data to worldwide users.

There were 71 major integration issues identified for GPS during the 16-year span. Some of the major issues were communications between cross-link systems, development and integration of GPS Military-Code and civil signal capabilities, integration of receivers into major platforms (i.e., ships and aircraft), and refinement of end-to-end test strategy.

4. NPOESS (National Polar-orbiting Operational Environmental Satellite System) was an enhanced weather SoS program that was administered by three different government organizations: DoD’s Air Force; Department of Commerce (DoC)’s National Aeronautics and Space Administration (NASA); and DoC’s National Oceanic and Atmospheric Administration (NOAA). NPOESS provided the platform to acquire satellite SoS to collect and disseminate environmental data for the three organizations. Although development planning and integration started in 1999, the program was eventually cancelled in 2011. There are many valuable integration lessons learned and information to be derived from the program throughout the years.

There were 18 major integration issues identified for NPOESS during its existence. Some of the major issues included field terminal integration and interoperability, integration of quality environmental data record, and integration challenges of program sensor design and algorithm development.

5. SBIRS (Space Based Infrared System) is a remote sensing satellite SoS that supports DoD by providing unprecedented timely and improved infrared data quality.SBIRS’s top missions of improving missile warning and missile defense have been realized through complex development and integration processes. It has also greatly enhanced the other mission areas of technical intelligence and battlespace characterization.

There were 46 major integration issues identified for SBIRS during the 16-year span. Some of the major integration issues included defining multiple test strategies, development of Modeling & Simulation systems to support operations of integration and testing, ground software development and integration, and flight software integration into satellite.

6. WGS (Wideband Global SATCOM) SoS enables communication to U.S. military and Coalition partners and allies during war and all levels of conflicts. Its wideband capability provides greater bandwidth to transfer data during conflicts.

There were 33 major integration issues identified for WGS during the 16-year span. Some of the major issues included integration complexity of X-band and Ka-band cross-banding for the on-board satellite, integration technical issues on orbital placement, frequency reuse, and launch services integration, interoperability complexity of control software development.

504

Attributes Derived from Data CollectionTo understand the impact of each major integration issue, each issue

was aligned with an integration variable to best describe the major area impacted by the issue. The integration variables were then grouped together based on similar focus areas and aligned with their contributing attri-bute. Interviews with integration experts and additional research were conducted to scope the right number of attributes from the initial list. The definition of each potential attribute continued to be refined based on the different groups of integration variables supporting them. After identifying the integration issues and integration variables, the list of con-tributing attributes was finalized. Each integration variable was assigned to a major attribute, which also helped define the criteria for that attribute. A discussion of the resultant list of seven final attributes follows.

Availability. To enable integration activities, Availability is critical to understanding the benefits to be derived from accessibility of supporting hardware, software, documentation, and expertise. Integration of new systems into legacy systems may be made more difficult by improper or insufficient support documentation or the unavailability of expert per-sonnel/training expertise. This attribute is critical in identifying these supporting entities. The items derived from the integration issues that sup-port this attribute are access to high TRL supporting systems, subsystems and components, and access to supporting documentation and expertise.

The availability of the right supporting systems is critical to the success of the overall system delivery. To help address Availability, the reuse of existing infrastructure and the use of COTS items are identified before integration activities. To meet time commitments, the reuse of systems, subsystems, and components from the current infrastructure are highly encouraged with the possibility of replacing them with COTS items on future updates (Tyree & Akerman, 2005).

Complexity. Complexity is the attribute that manages the technical risks of integration. The items derived from the integration issues that support this attribute are managing low technical maturity of system design, development, and integration, and determining complex interfaces. According to Jain et al. (2008), “complexity can be defined as the degree to which a system or component has a design or implementation that is difficult to understand and verify” (p. 211). The complexity of systems integration has significant influence on driving schedule delays (Dubos, Saleh, & Braun, 2008). Complexity affects the degree of complication of several factors such

505

as the number of interfaces and intricacies of different components (Jain et al., 2008). Although it has been widely studied, there is no defined mathe-matical model for systems related to Complexity (Haghnevis & Askin, 2012).

Interoperability. Interoperability is the attribute that will address compatibility, connectivity, and communication between systems, sub-systems, and components. The items derived from the integration issues that support this attribute are understanding compatibility and interface issues, addressing semantic/syntactic issues, and managing communica-tions issues between systems. Interoperability assessments can be very challenging in SoS because of the different testing involved to verify their functionality (Lin et al., 2009). According to Madni and Sievers (2014),

“interoperability is the ability of distinct systems to share semantically compatible information and then process and manage the information in semantically compatible ways, enabling users to perform desired tasks” (p. 342). Interoperability can be improved, which means that the metrics to measure it can be defined. However, Interoperability is still a complex and broad topic, and its condition may not be easily quantified (Rezaei, Chiew, & Lee, 2013).

Management. Management is the attribute most critical to planning and developing integration strategies. The items derived from the inte-gration issues that support this attribute include leading coordination between stakeholders, providing guidance and directives, determining scope, managing requirements, and developing and implementing policies and agreements. Management ensures that the stakeholders’ derived in-tegration requirements are met throughout the integration process. Man-aging stakeholders’ decisions is critical in shaping the final design of the system (Booch, 2006). Management must be able to navigate through the challenges of diminishing budgets, changing politics, and evolving tech-nologies to support integration activities.

The Management methods for integration address issues of philosophy, operation, and collaboration (Tien, 2008). Management addresses re-quirements traceability. As indicated by Piaszcyk (2011), “traceability is used to evaluate the impact of top-level requirements changes” that can help eliminate requirements creep (p. 325).

Processes. Processes is the attribute that enables the development, documentation, and execution of all integration activities. The items derived from the integration issues that support this attribute are processes that develop, document, and execute the following: testing, verification and validation, configuration management, training, modeling and simulation

506

(M&S), M&S activities, information assurance, mission assurance, com-munications and decision making, security, safety, quality assurance, and manufacturing and assembly. Processes implemented during integration must be clearly defined and documented. Many organizations’ efforts in key process areas for integration did not have much success due to ill-defined Processes (Djavanshir & Khorramshahgol, 2007). Processes for integration include standards, procedures, and algorithms associated with integration activities (Tien, 2008).

The goal is to identify necessary Processes needed for successful integration and the impacts to the integration requirements. Great examples from the space integration data also explain how the integration issues are impacted due to the different test and verification Processes that have been imple-mented. Using M&S as part of the integration process will help minimize risks. Computer-based M&S early in the process can help thrust systems engineering methods throughout the integration process (Ender et al., 2010).

Resources. Resources is the attribute that enables integration of SoS through funding and providing the right integration tools. The items derived from the integration issues that support this attribute are sufficient funding to support all integration activities, providing the right amount of trained personnel, and providing the right tools and facilities throughout the integration activities. Resources planning throughout SoS integration is critical to overall success. During the early phase of integration, informa-tion is scarce, but SMEs are potential sources of information with valuable inputs to address life-cycle cost of development and overall integration of a system (Tien, 2008).

Schedule. Schedule is the attribute that manages Schedule impacts throughout the integration activities. The items derived from the integra-tion issues that support this attribute are understanding timeline goals (i.e., need dates, expected deliveries, and milestones), understanding sched-uling parameters, and understanding critical path and impacts to Schedule delays. Schedule can be impacted by several of the attributes mentioned; however, an independent assessment is needed to understand how different changes impact scheduled milestones and activities. Need date and allowed time of integration must be established early on with analysis for fluctu-ations and critical path. Usually, the more time given for integration, the better the IRL score, which can be attributed to system maturity through time and processes.

507

Schedule delay usually increases the overall cost of the system and may impact the current requirements (i.e. launching a satellite at a certain month/year as required). Based on Dubos et al. (2008), “schedule slippage plagues the space industry and is autonomic with the recent emphasis on space responsiveness; the GAO has repeatedly noted the difficulties encoun-tered by the DoD in keeping its acquisition of space systems on schedule, and they identified the low TRL of the system/payload under development as the primary culprit driving schedule risk and slippage” (p. 386).

Space Integration Data Using Derived AttributesSome of the attributes derived from the data may slightly impact the

other attributes based on how they are interpreted. Therefore, the stake-holders need to establish clear definitions (as summarized previously), understand the distinctions between attributes, and document assumptions being made for each major attribute. Each integration issue is aligned with an integration variable and then grouped with one of the seven major attri-butes based on their focus areas. Tables 5a to 5f capture the top issues for each attribute from each space system. They show how a specific integration issue is aligned to an integration variable and how it becomes a subset to one of the seven attributes. Tables 5a to 5f also provide the number of years that the issue impacted each of the space systems.

508

TABLE 5A. TOP ISSUES FOR EACH ATTRIBUTE IN AEHF SATCOM

AEHF Satellite Communications (SATCOM)

Integration Issue Integration Variable

Contributing Attribute

# of Yrs Impacted

Nuclear hardening and shielding integration into system

System Design Maturity

Complexity 6

Lack of terminal synchronization within system

Communications between Systems

Interoperability 4

Planning of space, mission control, and user segment integration & synchronization strategy

Strategy Development

Management 3

Testing anti-jam capabilities; test processes to evaluate AEHF capabilities

Test Process Processes 3

Fidelity of available system test simulator

Supporting System Availability

Availability 2

Schedule issues by development contractor to resolve first-time test article and test fixture problems

Schedule Impacts Schedule 1

Insufficient budget to fund additional filters to meet High-Altitude Electro Magnetic Pulse (HEMP) testing and certification

Insufficient Funding

Resources 1

TABLE 5B. TOP ISSUES FOR EACH ATTRIBUTE IN EELV

Evolved Expendable Launch Vehicle (EELV)



# of Yrs Impacted

Design issues on reliability, logistics, supportability, multiple payload interfaces, and information assurance


Complexity 2

Lack of contractual requirement for test reporting

Contracted Agreement and Policies

Management 2

Insufficient processes to obtain data of system qualification tests for verification

Verification Processes

Processes 2

Lack of additional development and integration funding to satisfy government and industry launches


Resources 1

Availability of modified ground system


Availability 1

Integration of legacy upper stage to the new launch vehicle design

System Interfaces and Compatibility

Interoperability 1

No major integration issue identified that the Schedule attribute impacted

509

TABLE 5C. TOP ISSUES FOR EACH ATTRIBUTE IN GPS

Global Positioning System (GPS)



# of Yrs Impacted

Ongoing issues with the integration of GPS receivers into different platforms (ships, aircraft, etc.)

Communication between Systems

Interoperability 8

Availability of M-code and civil signal capabilities through development and integration


Availability 7

Planning and refinement of integration and test strategy; insufficient rigorous end-to-end planning

Planning Test Strategy

Management 7

Configuration management process issues on operations control system development and integration

Configuration Management Process

Processes 3

Schedule delays impacting integration, tests, and supporting resources

Schedule Impacts

Schedule 3

Design issues on Control Software Integration


Complexity 1

Funding availability to system development and integration causing event delays


Resources 1

TABLE 5D. TOP ISSUES FOR EACH ATTRIBUTE IN NPOESS

National Polar-Orbiting Operational Environment Satellite System (NPOESS)



# of Yrs Impacted

Design issues with the quality of environmental data record on algorithm, sensor performance, quality control of interface data processing segment

System Design Complexity

Complexity 2

Insufficient interoperability of field terminals and ground systems

System Interoperability

Interoperability 2

Funding issues to support interoperability of high rate data in X Band and low rate data in L Band


Resources 1

Inconsistent agreement and definitions of different decision makers’ roles and responsibilities

Stakeholder Agreements

Management 1

Lack of documentation process of adequate threshold definitions in data terminals

Documentation Processes

Processes 1

Lack of available space environment sensors that can be used for integration, test, and upgrades

Supporting Subsystem Availability

Availability 1

No major integration issue identified that the Schedule attribute impacted

510

TABLE 5E. TOP ISSUES FOR EACH ATTRIBUTE IN SBIRS

Space Based Infrared System (SBIRS)



# of Yrs Impacted

Insufficient definition of integrated test strategy to meet current schedule, ground architecture, and overall systems operational requirement

Strategy Development

Management 6

Lack of tools and resources to support M&S development of operational integration and testing

Insufficient Tools and Resources

Resources 3

Availability of ground software delivery for integration (delayed in development)

Supporting Software Availability

Availability 3

Degraded test data from Sensor Test Integration Lab due to Infrared Starer sensor and telescopic assembly process

Assembly Process

Processes 3

Communication issues of space control and telemetry, and flight software integration in satellite

Communications between Systems

Interoperability 3

Compressed schedules to accreditation of operational test scenarios increased software integration issues

Schedule Impacts

Schedule 2

Software instability in tracking, telemetry, and control functions

Software Design Maturity

Complexity 1

511

TABLE 5F. TOP ISSUES FOR EACH ATTRIBUTE IN WGS

Wideband Global SATCOM (WGS)



# of Yrs Impacted

Complexity of designing and integrating effective cross-banding between X-band and Ka-band onboard the satellite


Complexity 4

Lack of software tools to support development of system configuration control element

Insufficient Tools Resources 2

System interoperability issues of WGS and transmission system

System Interoperability

Interoperability 2

Insufficient documentation processes to support development and integration

Document Process

Processes 1

Ill-defined requirement definitions to separate legacy and new systems on orbit

Strategy Development and Requirements Management

Management 1

Insufficient schedule assessment to account for development and integration

Schedule Impacts

Schedule 1

Lack of upgraded automation software operations center networks


Availability 1

The result of the data collection based on the attributes gathered from the six major DoD Space Systems is summarized in Table 6. The total number of major integration issues collected from the six DoD space systems for the 16 years is 271 with a total of 35 for Availability, 51 for Complexity, 64 for Interoperability, 37 for Management, 52 for Processes, 16 for Resources, and 16 for Schedule.

512

TAB

LE 6

. SU

MM

AR

Y O

F IN

TEG

RA

TIO

N D

ATA

: SIX

DO

D M

AJO

R S

PAC

E S

YST

EM

S (1

99

9–2

014

)

TOTA

L (M

ajo

r In

teg

rati

on

Issu

es/Y

ear)

A

vaila

bili

tyC

om

ple

xity

Inte

rop

erab

ility

Man

agem

ent

Pro

cess

esR

eso

urce

sSc

hed

ule

1999

32

22

20

2

200

04

28

44

20

200

13

74

35

10

200

22

108

38

31

200

33

79

18

30

200

44

76

15

10

200

55

96

58

02

200

62

54

43

01

200

70

14

31

13

200

81

06

32

33

200

94

03

33

11

2010

20

22

21

1

2011

11

12

10

1

2012

00

01

00

0

2013

00

00

00

0

2014

10

10

00

1

TOTA

LS3

55

16

43

75

216

16

Tota

l Maj

or In

teg

rati

on

Issu

es27

1

513

Data AnalysisRegression analysis was completed to assess the current data and how

the attributes impacted the different years. The summary of analysis is highlighted in the Normal Probability Graph shown in Figure 2.

The resulting numbers are provided in Tables 7a-d, with Table 7d providing the regression equation of the relationship between the attributes and the 16 years defined.

FIGURE 2. NORMAL PROBABILITY GRAPH OF 7 ATTRIBUTESDERIVED FROM COLLECTED DATA

PERC

ENT

DATA-5.0 -2.5 0.0 2.5 5.0 7.5 10.0 12.5

VariableAvailabilityComplexityInteroperabilityManagementProcessesResourcesSchedule

1

5

10

20304050607080

90

95

99

514

TABLE 7A. SUMMARY STATISTICS

Variable N Mean Standard Deviation Minimum Minimum

Availability 16 2.1875 1.6008 0 5

Complexity 16 3.1875 3.6555 0 10

Interoperability 16 4 2.9212 0 9

Management 16 2.3125 1.4477 0 0

Processes 16 3.25 2.8402 0 8

Resources 16 1 1.1547 0 3

Schedule 16 1 1.0328 0 3

TABLE 7B. ANALYSIS OF VARIANCE

Source DF P-ValueRegression 7 0.1177

Schedule 1 0.9536

Resources 1 0.3814

Processes 1 0.2031

Management 1 0.5817

Interoperability 1 0.7945

Complexity 1 0.1569

Availability 1 0.1393

TABLE 7C. MODEL SUMMARY

S R-sq R-sq(adj)3.6838 68.07% 40.13%

TABLE 7D. REGRESSION EQUATION

Year = 15.140 − 0.070 Schedule − 2.183 Resources + 2.341 Processes − 0.5640 Management − 0.290 Interoperability − 1.5063 Complexity − 2.162 Availability

The highest mean is Complexity, which also had the highest standard deviation. The lowest average is 1.0 for both Resources and Schedule, but with a lower standard deviation on Schedule. The null hypothesis in this case is that all the attributes have a relationship with Year. Based on the

515

analysis, the P-Value for each attribute is greater than 0.05; therefore, all attributes have an impact on the Year. For statistical measure, R-squared helps explain how close the data are to the fitted regression line. The good-ness of fit (R-squared) shows that it is 68.0 percent, which is just below the acceptable 75 percent value. This shows that there is correlation between the variables and the Year. A modified version of the R-squared value is the Adjusted-R-squared, which accounts for the number of predictors in the model. In this case, the Adjusted-R-squared significantly drops the R-squared value. The Regression Equation helps with predicting attributes for the upcoming years.

Validation through SurveyThe integration data were very useful in identifying critical attributes

to support a framework for facilitating SoS integration. SoS integration is complex, and the ability to develop a systems integration assessment framework tool to determine integration feasibility is necessary for system integration planning. Although the previously described seven attributes can help improve the integration feasibility assessment, they need to be validated by integration experts.

Therefore, a survey was conducted to help validate the usefulness of the seven major integration attributes. The survey lasted 4 weeks and targeted 159 systems integration experts with varying experiences in systems engi-neering, DoD acquisition, space systems, and/or complex SoS. From the 159 integration experts surveyed, 88 responded, which helped validate the attributes derived from the data of the six major space systems (Table 4). The 88 participants were systems engineers, DoD acquisition professionals, and SMEs who primarily have backgrounds in space systems.

Integration of new systems into legacy systems may be made more difficult by improper or insufficient support documentation or the unavailability of expert personnel/training expertise.

516

The survey asked the participants to list specific systems on which they worked (current and prior experience), their specific field (systems engi-neering, DoD acquisition, systems integration etc.), and years of experience in their chosen field of work. Once their credentials had been provided, they were asked to rank the attributes from one to seven—one being the most important; the respondents were then asked if they agreed with the seven attributes. They were also given the opportunity to provide additional attributes, and to explain if they agreed or did not agree with the list of attributes and the integration items supporting each attribute. They were asked to provide examples on how useful an SoS integration framework using IRL principles would be in helping with integration activities. Finally, a set of questions was asked to provide examples based on their experience on major integration risk areas, integration timelines, and potential use-fulness of IRL.

Summary of Survey ResultsThere were a total of 18 questions in the survey; some questions, how-

ever, were for clarification of responses to prior questions. The survey participants were a very experienced group, with an average of 19.4 years of experience in their respective field. Of the 88 participants, only seven said they were not in complete agreement with the attributes; however, their con-cerns were all addressed throughout the research process and through the updated definition for each attribute. The concerns were: exclusion of man-agement guidance (addressed under Management); exclusion of concept of operations, which is developed as part of the scope of integration (also a Management task); exclusion of M&S tools (addressed under Processes); exclusion of overall planning (addressed under Management); exclusion of requirements (which is outside the scope of this research, but addressed in Problem Focus); and exclusion of risk areas (technical risks were addressed in Complexity and overall program risks are the output of the systems inte-gration assessment framework). Most of the survey participants agreed that given the right supporting tools, an enhanced IRL assessment would be useful. Based on the results of the initial research (literature reviews and interviews with system integration experts), data collection, and expert survey, the final criteria for each attribute are summarized in Table 8.

517

TABLE 8. CRITERIA FOR EACH ATTRIBUTE USING DERIVED INTEGRATION VARIABLES

Integration Variables (Criteria to Major Attributes) Major Attributes

Access high TRL supporting systems, subsystems, and components (i.e., COTS, operational ground systems, operational simulators)

AvailabilityAccess overall expertise and supporting documentation to execute integration activities

Access supporting hardware and software to execute integration activities (i.e., COTS, Hardware/Software reuse)

Manage technical risks and low maturity of system design, development, and integration

ComplexityDetermine complex interfaces between systems, subsystems, and components

Manage low TRL supporting systems, subsystems, and components

Understand compatibility and connectivity of systems, subsystems, and components; inputs into integrated systems produce desired outputs

InteroperabilityManage communications between systems, subsystems, and components; address semantic/syntactic issues

Develop and document interface control documents

Plan and develop integration strategies and priorities

Management

Lead coordination, execution, and communication between stakeholder organizations

Provide guidance and directives to integration activities

Determine scope of integration activities and concepts of operation

Manage requirements changes (i.e., eliminate/minimize requirements creep)

Manage support to integration activities and changes (i.e., supply chain)

Develop and document policies and agreements with all stakeholders

Document overall integration activity processes (i.e., overall communication and decision-making processes)

Processes

Develop, document, and execute risks/mitigation processes

Develop, document, and execute configuration management processes

Develop, document, and execute information assurance and mission assurance processes

Develop, document, and execute standards, procedures, and algorithms

Develop, document, and execute M&S activities

Develop, document, and execute test, verification, and validation processes

Develop, document, and execute security and safety considerations

Develop, document, and execute Training processes

Develop, document, and execute manufacturing and assembly processes

Allocate sufficient funding to support all integration activities

ResourcesProvide trained personnel to support integration activities

Provide the right tools and facilities to support integration activities

Meet timeline goals (i.e., need dates, expected deliveries, milestones etc.)

ScheduleUnderstand schedule impacts, parameters, and critical path

Plan for schedule changes and impacts of delays

518

All 88 participants ranked the seven attributes in order. Table 9 captures the first and last five survey entries. The ranking of the attributes is distributed by having the No. 1 ranked attribute equal to 7/7 (1.0). Values for the subse-quent ranking: No. 2 is 6/7 (or 0.857), No. 3 is 5/7 or (0.714), No. 4 is 4/7 (or 0.571), No. 5 is 3/7 (or 0.429), No. 6 is 2/7 (or 0.286), and No. 7 is 1/7 (or 0.143)

TABLE 9. SURVEY RESULTS: YEARS OF EXPERIENCE AND RANKING OF ATTRIBUTES (PART I)

# Years of Exp. Availability Complexity Interoperability Management Processes Resources Schedule

1 39 1.000 0.857 0.714 0.571 0.429 0.286 0.1432 35 0.143 0.714 0.571 0.286 1.000 0.857 0.4293 40 0.571 0.714 0.857 1.000 0.429 0.286 0.1434 34 0.714 0.857 1.000 0.429 0.571 0.286 0.1435 16 0.857 0.143 0.286 0.714 0.571 1.000 0.429• • • • • • • • •• • • • • • • • •• • • • • • • • •

84 26 1.000 0.286 0.857 0.143 0.571 0.714 0.42985 27 0.286 0.429 0.571 1.000 0.857 0.714 0.14386 14 0.571 0.857 0.714 1.000 0.143 0.286 0.42987 35 0.286 0.429 0.143 1.000 0.571 0.857 0.71488 8 1.000 0.571 0.143 0.857 0.429 0.286 0.714

Survey AnalysisWeights of the attributes will be determined using the years of expe-

rience and the ranking of each attribute. The Years of Experience is set up where the values are a lot higher than the value of ranking the attributes. To reduce the skewness of the dependent variable (Years of Experience), the log function will be used for the regression analysis. The log function will help normalize the Years of Experience relative to the scores given to attri-bute rankings. The results of the survey showed that almost all participants were in agreement with the derived attributes.

The summary of analysis is highlighted in the Normal Probability Graph in Figure 3. The resulting numbers are provided in Tables 10a to 10d, with 10d providing the Regression Equation for the Log Years and the attributes. The ranking of each attribute differed with each person, but there was a clear emphasis on the two highest attributes (Management and Resources) as indicated in Table 10a as well as two of the lowest ranked attributes in Schedule and Interoperability. Based on the analysis, the P-Values for all attributes are slightly greater than 0.05; therefore, the null hypothesis for

519

each attribute impacting the Log Years of Experience is true. The goodness of fit (R-squared) is a very low 18.84 percent, which is expected due to the ranking of each variable where no variable has the same value per sample.

FIGURE 3. NORMAL PROBABILITY PLOT FOR ALL ATTRIBUTES BASED ON SURVEY

PERC

ENT

DATA-5.0 0.0 0.5 1.0 1.5

VariableAvailabilityComplexityInteroperabilityManagementProcessesResourcesSchedule

0.1

5.0

20.030.040.050.060.070.080.090.095.0

99.0

99.9

10.0

1.0

TABLE 10A. SUMMARY STATISTICS

Variable N Mean Standard Deviation Minimum Minimum

Availability 88 0.59569 0.2618 0.143 1

Complexity 88 0.55038 0.27935 0.143 1

Interoperability 88 0.44165 0.27522 0.143 1

Management 88 0.69642 0.28298 0.143 1

Processes 88 0.58276 0.26971 0.143 1

Resources 88 0.69963 0.25802 0.143 1

Schedule 88 0.43672 0.25645 0.143 1

520

TABLE 10B. ANALYSIS OF VARIANCE

Source DF P-Value

Regression 7 0.0161

Availability 1 0.0883

Complexity 1 0.0716

Interoperability 1 0.0503

Management 1 0.1185

Processes 1 0.0697

Resources 1 0.109

Schedule 1 0.0876

TABLE 10C. MODEL SUMMARY

S R-sq R-sq(adj)0.283784 18.84% 11.74%

TABLE 10D. REGRESSION EQUATION

Log Years = −5.925 + 1.776 Availability + 1.857 Complexity + 2.051 Interoperability + 1.601 Management + 1.888 Processes + 1.654 Resources + 1.758 Schedule

Comparing ResultsEvaluating the mean and standard deviation for each attribute for both

the integration data analysis and the survey analysis will determine the weight for each variable to be used in the integration feasibility framework. Tables 11a to 11c illustrate the derived weight of each attribute.

Based on the percentage for each attribute, there are a lot of disconnects between the integration data analysis and the expert survey analysis on how the attributes should be weighted. The closest numbers for the Data and Survey are Availability at about 12–14 percent and Management at about 13–17 percent. Using the standard deviation, the High and Low for the Means have also been calculated. The percentages for the High and Low values show significant variances. In this case, using the average of two Means will provide a better estimate to use as the final weight of each attribute (Table 11c).

521

Systems Integration Assessment Framework

The integration requirements are addressed through an architectural framework, which according to Jain et al. (2010), “includes both the physi-cal and functional architectures of the system” (p. 279). Based on the data analysis, subsequent attributes, and the survey analysis to validate the weights of the attributes, a systems integration assessment framework to determine an enhanced IRL value has been developed. When the models of a system are related through transformation, the collective frames can be referred to as a framework (Lin et al., 2009). In this case, the items that were derived from the integration data (and identified by the stakeholders) were used to support the major attribute.

TABLE 11A. MEAN AND STANDARD DEVIATION FOR INTEGRATION DATAAND SURVEY RESULTS

Comparing Analyses

Integration Data, Mean

Integration Data, Standard Deviation

Survey, Mean

Survey, Standard Deviation

Availability 2.188 1.601 0.596 0.262

Complexity 3.188 3.656 0.550 0.279

Interoperability 4.000 2.921 0.442 0.275

Management 2.313 1.448 0.696 0.283

Processes 3.250 2.840 0.583 0.270

Resources 1.000 1.155 0.700 0.258

Schedule 1.000 1.033 0.437 0.256

TABLE 11B. PERCENTAGE OF EACH MEAN AND STANDARD DEVIATION FOR INTEGRATION DATA AND SURVEY RESULTS

Weight Integration Data, Mean

Integration Data, High

Integration Data, Low

Survey, Mean

Survey, High

Survey, Low

Availability 12.9% 12.0% 25.7% 14.9% 14.6% 15.7%

Complexity 18.8% 21.7% -20.5% 13.7% 14.1% 12.8%

Interoperability 23.6% 21.9% 47.2% 11.0% 12.2% 7.9%

Management 13.7% 11.9% 37.8% 17.4% 16.6% 19.5%

Processes 19.2% 19.3% 17.9% 14.6% 14.5% 14.8%

Resources 5.9% 6.8% -6.8% 17.5% 16.3% 20.8%

Schedule 5.9% 6.4% -1.4% 10.9% 11.8% 8.5%

522

TABLE 11C. DERIVED WEIGHT OF EACH ATTRIBUTE

Average Weight Mean Total Final Weight

Availability 27.8% 13.9%

Complexity 32.6% 16.3%

Interoperability 34.6% 17.3%

Management 31.1% 15.5%

Processes 33.7% 16.9%

Resources 23.4% 11.7%

Schedule 16.8% 8.4%

In architectural framework, understanding how the problem is framed is the most important step because it helps define the subsequent steps. Frameworks and standards define what should be modeled as opposed to which models should be used and how these models are related to one another (Zalewski & Kijas, 2013). According to Martin (2008), “problem framing techniques have been found to be useful in avoiding rework and maximizing results” (p. 306). Architectural approaches include conveying options, change, and implications, as well as identifying easier traceability by providing agile documentation (Tyree & Akerman, 2005). The change, based on initial IRL scores to enhanced IRL scores, supports this approach. Architecture analysis includes a key foundation that provides consistency, data completeness and transformation, lack of ambiguity, and flexibility to support iterative processes (Ge, Hipel, & Chen, 2014). Static Analysis is used in which, according to Ge et al. (2013), helps leverage “static architec-tural models for analyzing and comparing the associations captured from the data elements” (p. 370).

The resulting systems integration assessment framework to facilitate SoS integration is illustrated in Figure 4.

In the framework, one of the initial steps is to determine the IRL value of the integration items supporting each attribute by using the current IRL definitions as guidelines. Tables 12a and 12b provide guidance on how initial IRL for each attribute is defined.

523

FIG

UR

E 4

. SY

STE

MS

INTE

GR

ATI

ON

ASS

ESS

ME

NT

FRA

ME

WO

RK

Inte

rope

rabi

lity

(17.3

%)

Proc

esse

s(1

6.9%

)

Com

plex

ity(1

6.3%

)

Mana

gem

ent

(15.

5%)

Avai

labi

lity

(13.

9%)

Reso

urce

s(1

1.7%

)

Sche

dule

(8.4

%)

Inte

grat

ion

Requ

irem

ents

New

Capa

bilit

ies

Fam

ily of

Syste

ms

Inte

grat

ion

Activ

ities

Initi

al IR

LAs

sess

men

tSc

ope o

fIn

tegr

atio

n

STAK

EHOL

DER

DECIS

IONS IR

L Ass

essm

ent G

uideli

ne to

ea

ch A

ttribu

te us

ing IR

L pr

incipl

es is

sum

mar

ized i

n Ta

bles 1

2a an

d 12b

Conn

ectiv

ity; C

ompa

tibilit

y;Co

mm

unica

tion b

etwe

en sy

stem

s

Deve

lop, D

ocum

ent,

and E

xecu

te

integ

ratio

n acti

vity p

roce

sses

Tech

nical

Risks

, Com

plex i

nter

faces

, Lo

w TR

L sup

porti

ng sy

stem

s

Plann

ing; D

eter

mine

scop

e,str

ateg

ies an

d prio

rities

; Guid

ance

& Di

recti

ves;

Mana

ge Re

quire

men

ts

High

TRL S

uppo

rting

syste

ms;

Acce

ss to

expe

rtise

and d

ocum

enta

tion;

Acce

ss su

ppor

ting H

W/S

W

Fund

ing; P

erso

nnel;

Right

tools

and f

acilit

ies

Timeli

ne go

als; S

ched

ulepa

ram

eter

s; Im

pacts

of D

elays

Valid

ate

IRL S

core

;ide

ntify

risk

ar

eas;

dete

rmine

optio

ns to

adjus

t sco

pe if

nece

ssary

ENHA

NCED

IRL S

CORE

524

TABLE 12A. GUIDLINES TO INITIAL IRL FOR EACH ATTRIBUTEUSING IRL PRINCIPLES (PART I)

IRL Level

IRL Principles IRL Availability IRL

ComplexityIRL Interoperability

1 Interface has been identified

All currently available and unavailable supporting integration items (systems, subsystems, components, expertise, and documentation) are identified

Complexity and technical risk areas of design, development, and integration are identified

Interfaces and connectivity areas of systems, subsystems, and component items are identified

2 Interaction has been specifically characterized

Available and unavailable items are characterized

Technical risk areas are characterized

Interaction between items is characterized

3 There is compatibility

Initial plans to obtain unavailable items are compatible with overall strategy

Initial mitigation strategies are compatible to technical risk areas

Interoperability strategies are compatible with integration activities

4 Sufficient detail in quality and assurance

Plans to obtain unavailable items provide sufficient details

Sufficient detail for mitigation strategies is defined

Interoperability strategies have sufficient details

5 Sufficient control

Plans are delivered and have sufficient control

Proposed mitigation strategies are delivered and have sufficient control

Proposed strategies are delivered and have sufficient control

6 Accept, translate, and structure information

Proposed plans are accepted and executed

Mitigation strategies are accepted

Proposed strategies are accepted and executed

7 Verified and Validated

Plans are verified and validated

Mitigation strategies are verified and validated

Proposed strategies are verified and validated

8 Mission-Qualified

All items are delivered and mission-qualified

Mitigation strategies are mission-qualified

Proposed strategies are mission-qualified

9 Mission-Proven

All items are mission-proven

Mitigation strategies are mission-proven

Proposed strategies are mission-proven

525

TABLE 12B. GUIDLINES TO INITIAL IRL FOR EACH ATTRIBUTEUSING IRL PRINCIPLES (PART II)

IRL Level

IRL Definition

IRL Management

IRL Processes

IRL Resources

IRL Schedule

1 Interface has been identified

Management strategies, scope, requirements, and priorities are identified

All integration activity processes are identified

All resources and funding risks to support integration are identified

Detailed integration schedule and schedule risks are identified

2 Interaction has been specifically characterized

Key management decisions are characterized

Processes are characterized

All resources and funding risks are characterized

All schedule items and risks are characterized

3 There is compatibility

Current strategies are compatible with management decisions

Processes are documented and are compatible with management strategies

Initial mitigation strategies to funding risks are compatible with integration strategy

Initial mitigation strategies to schedule risks are compatible with integration strategy

4 Sufficient detail in quality and assurance

Management strategies have sufficient detail in quality and assurance

Processes are sufficient detail in quality and assurance

Mitigation strategies have sufficient detail in quality and assurance

Mitigation strategies have sufficient detail in quality and assurance

5 Sufficient control

Management strategies are implemented with sufficient control

Processes are documented with sufficient control

Mitigation strategies are delivered with sufficient control

Mitigation strategies are delivered with sufficient control

6 Accept, translate, and structure information

Management strategies, goals, and scope are accepted and executed

Processes are accepted and executed

Mitigation strategies for funding risks are accepted and executed

Mitigation strategies for schedule risks are accepted and executed

7 Verified and Validated

Management strategies are verified and validated

Processes are verified and validated



8 Mission-Qualified

Management strategies are mission-qualified

Processes are mission-qualified

Proposed strategies are mission-qualified

Mitigation strategies are mission qualified

9 Mission-Proven

Management strategies are mission-proven

Processes are mission-proven

Mitigation strategies are mission-proven

Proposed strategies are mission-proven

526

Approaches to determine IRLs include: (a) individual estimation by an SME; (b) group discussion estimation through meeting or conference; and (c) indi-vidual-group estimation where SMEs complete independent estimations and then discuss them with a group for a consensus decision (Tan, Ramirez-Marquez, & Sauser, 2011). Some of the current techniques that can be used to formulate the estimate of IRL assessment also include educated guess (not enough knowledge and not enough time), analogy (comparing prior work), and standards (developed within different organizations) (Tan et al., 2011). Using the framework for systems integration will be an iterative process.

The process of using the Systems Integration Assessment Framework, consisting of 11 steps, is as follows:

1. SoS integration begins with the concept of integrating new capabilities to either an existing Family of Systems or a new SoS.

2. Integration requirements are written clearly and approved by the stakeholders.

3. The Scope of Integration (i.e., cost, schedule, performance, integration planning, Concept of Operations etc.) are deter-mined by the stakeholders.

4. Based on the Scope, the integration activities can either pro-ceed (if integration has been done before) or assess the overall feasibility of the Scope of Integration.

5. To begin understanding the overall feasibility of integration, the stakeholders and designated SMEs determine IRL score for all possible integration items supporting each major attribute.

6. Using IRL principles (as described in Tables 12a and 12b), determine the initial IRL value for each attribute.

527

7. Use the derived weighted percentage to determine the enhanced IRL score for the SoS (IRLSoS). The resulting equa-tion becomes:

8. The Enhanced IRL score will determine integration feasibility.

9. Stakeholders evaluate and validate the enhanced IRL score, identify risk areas, and determine options to adjust the scope of integration, if necessary.

10. Determining integration feasibility is an iterative process, and this framework allows the stakeholders to look at options to improve the scope when new information is obtained.

11. Assessment of IRL can also be tied to major events and reviews throughout the development and deployment processes. It allows program managers to make adjustments to their programs based on the results of the IRL assessment.

ConclusionsBased on the research, integration data collected from the six major

DoD Space Systems, and expert survey, seven attributes were derived in order to develop a framework that can facilitate SoS integration. A systems integration assessment framework tool provides an enhanced IRL score that will help determine integration feasibility in planning and facilitating integration activities. Based on a survey of experienced systems engineers, systems integrators, and DoD space acquisition personnel, the attributes were necessary to help plan and implement integration processes. The attri-butes were validated through the expert survey, and the results proved that other attributes were more important than those initially defined.

The attributes with their derived weights are the primary entities for the integration assessment framework tool. The tool can be used as many times as new information is provided, and adjusting the expectations and scope can help with understanding the feasibility of the IRL score. Calculating the enhanced IRL score is an iterative process with the thought that the IRL

IRLSoS = (IRLAvailability x 0.139) + (IRLComplexity x 0.163) +

(IRLInteroperability x 0.173) +(IRLManagement x 0.155) + (IRLProcesses x 0.169) +

(IRLResources x 0.117) + (IRLSchedule x 0.084)

528

score could improve through time due to additional information and adjust-ments to each attribute. Although the current DoD process of deploying space system capabilities for operational use does not require assessment of integration maturity, the result of this research should help quantify the feasibility of integrating SoS using IRL principles.

Limitations and Future WorkAlthough several processes have been provided to assess TRL and IRL,

there is still some subjectivity to these analyses. Every program has some uniqueness on how to integrate its systems, but the past data and past expe-rience can help put some rigor in the analysis. IRL has effective principles if it complements other analyses and helps with program manager decisions. Results can offer areas of emphasis as opposed to detailed solutions. For IRL to be used effectively, high-level assumptions need to be clear, upfront, and agreed to by stakeholders.

When grouped with other readiness metrics, IRL may have some applica-tions in DoD research and development (Ross, 2016), but IRL itself does not have the capacity to critically assess research and development efforts. It also does not evaluate cost and schedule (Sauser et al., 2010). A more quan-titative algorithm may be required for the assessment of IRL for complex systems. Without rigorous criteria, assessment for the determining IRL could lead to inaccurate analysis. Assessment of TRL and IRL sometimes can lead to oversimplification (Ramirez-Marquez & Sauser, 2009).

Although some of IRL may be subjective, there are several analyses using IRL principles that can be done to develop processes and help with program managers’ decisions on SoS integration. Throughout DoD acquisition and large complex programs, integration challenges will continue to persist, and the need for additional analysis and tools to overcome those challenges is real and necessary.

529

ReferencesBhasin, K., & Hayden, J. (2008, June). Architecting network of networks for space

system of systems. Paper presented at 2008 IEEE International Conference on System of Systems Engineering, Monterey, CA.

Bolloju, N. (2009). Conceptual modeling of systems integration requirements. IEEE Software, 26(5), 66–74.

Booch, G. (2006). On architecture. IEEE Software, 23(2), 16–18.Department of Defense. (2017). Operation of the Defense Acquisition System (DoDI

5000.02). Washington, DC: Author.Director, Operational Test & Evaluation. (n.d.). Annual reports. Retrieved from http://

www.dote.osd.mil/annual-report/index.htmlDjavanshir, G., & Khorramshahgol, R. (2007). Key process areas in systems integration.

IT Professional, 9(4), 24–27.Dubos, G., & Saleh, J. (2010). Risk of spacecraft on-orbit obsolescence: Novel

framework, stochastic modeling, and implications. Asta Astronautica, 67(1–2), 155–172.

Dubos, G., Saleh, J., & Braun, R. (2008). Technology readiness level, schedule risk, and slippage in spacecraft design. Journal of Spacecraft and Rockets, 45(4), 836–842.

Ender, T., Leurck, R., Weaver, B., Miceli, P., Blair, W., West, P., & Mavris, D. (2010). Systems-of-systems analysis of ballistic missile defense architecture effectiveness through surrogate modeling and simulation. IEEE Systems Journal, 4(2), 156–166.

Gagliardi, M., Wood, W. G., Klein, J., & Morley, J. (2009). A uniform approach for system of systems architecture evaluation. Crosstalk, 22(3–4), 12–15.

Ge, B., Hipel, K., & Chen, Y. (2014). A novel executable modeling approach for system-of-systems architecture. IEEE Systems Journal, 8(1), 4–13.

Ge, B., Hipel, K., Kewei, Y., & Chen, Y. (2013). A data-centric capability-focused approach for system-of-systems architecture modeling and analysis. Systems Engineering, 16(3), 363–377.

Gifun, J. F., & Karydas, D. M. (2010). Organizational attributes of highly reliable complex systems. Quality and Reliability Engineering International, 26(1), 53–62.

Government Accountability Office. (2011). Space acquisitions: DoD delivering new generations of satellites, but space acquisition challenges remain (Report No. GAO-11-590T). Washington, DC: U.S. Government Printing Office.

Government Accountability Office. (2012). Urgent warfighter needs: Opportunities exist to expedite development and fielding of joint capabilities (Report No. GAO-12-385). Washington, DC: U.S. Government Printing Office.

Haghnevis, M. & Askin, R. (2012). A modeling framework for engineered complex adaptive systems. IEEE Systems Journal, 6(3), 520-530.

Jain, R., Chandrasekaran, A., Elias, G., & Cloutier, R. (2008). Exploring the impact of systems architecture and systems requirements on systems integration complexity. IEEE Systems Journal, 2(2), 209–223.

Jain, R., Chandrasekaran, A., & Erol, O. (2010). A systems integration framework for process analysis and improvement. Systems Engineering, 13(3), 274–289.

Kruchten, P., Capilla, R., & Duenas, J. (2009). The decision view’s role in software architecture practice. IEEE Software, 26(2), 36–42.

530

Kujawski, E. (2013). Analysis and critique of the system readiness level. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 43(4), 979–987.

Lin, C., Lu, S., Fei, X., Chebotko, A., Pai, D., Lai, Z., … Hua, J. (2009). A reference architecture for scientific workflow management systems and the view SOA solution. IEEE Transactions on Services Computing, 2(1), 79–92.

London, M. A., Holzer, T. H., Eveleigh, T. J., & Sarkani, S. (2014). Incidence matrix approach for calculating readiness levels. Journal of Systems Science and Systems Engineering, 23(4), 377–403.

Lung, C., Xu, X., & Zaman, M. (2007). Software architecture decomposition using attributes. International Journal of Software Engineering and Knowledge Engineering, 17(5), 599–613.

Madni, A., & Sievers, M. (2014). System of systems integration: Key considerations and challenges. Systems Engineering, 17(3), 330–347.

Magnaye, R., Sauser, B., & Ramirez-Marquez, J. E. (2010). System development planning using readiness levels in a cost of development minimization model. Systems Engineering, 13(4), 311–323.

Mankins, J. (2009). Technology readiness and risk assessments: A new approach. Acta Astronautica, 65(9–10), 1208–1215.

Martin, J. (2008). Using architecture modeling to assess the societal benefits of the global earth observation system-of-systems. IEEE Systems Journal, 2(3), 304–311.

McConkie, E., Mazzuchi, T., Sarkani, S., & Marchette, D. (2013). Mathematical properties of system readiness levels. Systems Engineering, 16(4), 391–400.

Piaszcyk, C. (2011). Model Based Systems Engineering with Department of Defense architectural framework. Systems Engineering, 14(3), 305–326.

Pittera, T., & Derrico, M. (2011). Multi-purpose modular plug and play architecture for space systems: Design, integration, and testing. Acta Astronautica, 69(7–8), 629–643.

Rabelo, L., Fishwick, P., Ezzell, Z., Lacy, L., & Yousef, N. (2012). Ontology-centered integration for space operations. Journal of Simulation, 6(2), 112–124.

Ramirez-Marquez, J., & Sauser, B. (2009). System development planning via system maturity optimization. IEEE Transactions of Engineering Management, 56(3), 533–548.

Rezaei, R., Chiew, T., & Lee, S. (2013). A review of interoperability assessment models. Journal of Zhejiang University: Science, 14(9), 663–681.

Ross, S. (2016). Application of system and integration readiness levels to Department of Defense research and development. Defense Acquisition Research Journal, 23(3), 248–273.

Sauser, B., Gove, R., Forbes, E., & Ramirez-Marquez, J. E. (2010). Integration maturity metrics: Development of an integration readiness level. Information Knowledge Systems Management, 9(1), 17–46.

Suh, E. S., Chiriac, N., & Hölttä-Otto, K. (2015). Seeing complex system through different lenses: Impact of decomposition perspective on system architecture analysis. Systems Engineering, 18(3), 229–240.

Tan, W., Ramirez-Marquez, J., & Sauser, B. (2011). A probabilistic approach to system maturity assessment. Systems Engineering, 14(3), 279–293.

Tien, J. (2008). On integration and adaptation in complex service systems. Journal of Systems Science and Systems Engineering, 17(4), 385–415.

Tyree, J., & Akerman, A. (2005). Architecture decisions: Demystifying architecture. IEEE Software, 22(2), 19–27.

531

Vernay, A., & Boons, F. (2015). Assessing systems integration: A conceptual framework and a method. Systems Research and Behavioral Science, 32(1), 106–123.

Zalewski, A., & Kijas, S. (2013). From principles to details: Integrated framework for architecture modeling of large scale software systems. E-Informatica Software Engineering Journal, 7(1), 45–52.

Zhang, Y., Liu, X., Wang, Z., & Chen, L. (2012). A service-oriented method for system-of-systems requirements analysis and architecture design. Journal of Software, 7(2), 358–365.

532

Author Biographies

Maj Clarence Eder, USAF (Ret.), currently works as a senior principal systems engineer at the Air Force Space and Missile Systems Center (SMC), Milita r y Satellite Communication Systems Directorate. Maj Eder holds a PhD in S y s t em s E n g i ne er i n g f r om T he G e or ge Washington University, and has over 21 years of experience in DoD space, intelligence, missile defense, and aircraft programs as a program manager, systems engineer, and space operator. In addition to his PhD, he holds a BS in Mechanical Engineering from the University of Hawaii, and an MBA from Wright State University.


Dr. Thomas A. Mazzuchi is a professor of E n g i ne er i n g M a n a gem ent a n d S y s t em s Engineering in the School of Engineering and Applied Science at the The George Washington University. He received a BA in Mathematics from Gettysburg College; and an MS and DSc, both in O p er a t ion s R e s e a r c h , f r om T he G e or ge Washington University. He has also served as chair of the Department of Operations Research, cha i r of t he Depa r t ment of En g i neer i n g Management and Systems Engineering, and as interim dean of the School of Engineering and Applied Science.


533

Dr. Sha hra m Sa rk a ni is a professor of E n g i ne er i n g M a n a gement a nd S y s t em s Engineering (EMSE), and faculty advisor and a ca dem ic d i rector of EMSE Of f- Ca mpus Programs at The George Washington University (GWU). Dr. Sarkani joined the GWU faculty in 1986, where his previous administrative appoint-ments included chair of the Civil, Mechanical, and Environmental Engineering Department and interim associate dean for Research, School of Engineering and Applied Science. Sarkani holds a PhD in Civil Engineering from Rice University, and BS and MS degrees in Civil Engineering from Louisiana State University.


EFFECTIVENESS TESTand EVALUATION of Non-lethal Weapons in Crowd Scenarios: METRICS, MEASURES, AND DESIGN OF EXPERIMENTS

Elizabeth Mezzacappa, Gordon Cooke, Robert M. DeMarco,

Gladstone V. Reid, Kevin Tevis, Charles Sheridan,

Kenneth R. Short, Nasir Jaffery, and John B. Riedener

This article discusses test and evaluation methods for benchmarking and comparison of non-lethal weapons (NLW, including munitions and devices) intended for use in crowd management situations. Several types of weapons of different modalities were tested, including a fielded acoustic non-lethal device, simulated dismounted infantry directed energy weapon, a simulated long distance directed energy weapon, and a projectile weapon. The work demonstrates that NLW effectiveness of any modality can be quantified into standard metrics and statistically analyzed. Relative effectiveness can then be compared among weapons of different technologies, platforms, and energies. The resulting information can be used for Analysis of Alternatives and data-driven trade-space studies. This testing can be applied to a variety of operational scenarios and should be conducted on current NLW inventory to establish benchmark performance and effectiveness. Once benchmark performance is established, new NLWs can be evaluated to determine if improvements are significant enough to warrant investment.

DOI: https://doi.org/10.22594/dau.16-768.24.03 Keywords: Non-lethal Weapons (NLW), Effectiveness, Test and Evaluation, Human Subjects



Effectiveness Test and Evaluation of Non-lethal Weapons in Crowd Scenarios http://www.dau.mil

A soldier relates this story about an experience with crowds:

It was about 2006/2007 time frame. The people of northern Iraq were heading south in the pilgrimage. I don’t remember the exact date. The bad guys were planting IEDs [Improvised Explosive Devices] all over the place, killing the pilgrims by the hundreds, but they kept coming. We found an IED and started to cordon it off. We had to block the pilgrims’ path long enough for EOD [Explosive Ordnance Disposal] to blow the bomb … but the people wanted to take their chances with the bomb. We had to push them back and that made them mad. It was no less than 3,000 people at that time. It was hard, and I was worried they would attack …. They just hated the idea that we blocked the path. We asked the crowd to bring up the Imam or oldest in the crowd and they brought someone forward. We told him through the “terp” [interpreter] what was going on and he didn’t even care—he wanted to keep marching. When the EOD blew the IED, the crowd got very angry

and they started chanting and approaching the cones and concertina wire. I thought for sure we

were going to be run over. It was only two teams of us at that time … It was a feeling of the will to die for their beliefs. I saw all kinds of faces—young, old, men, women. All had the same look. Let us go on—we want to make it south … They were capa-ble of overrunning us easy. We could have fought for a few seconds, but it was way too many of them.

This soldier’s report of an interaction with crowds (Mezzacappa et al., 2011) illustrates a few of the factors and doc-trinal guidance for this type of mission (Headquarters, Department of the Army [HQDA], 2003; HQDA, 2005, 2006). In the passage, readers can see differences in cultural behaviors, religious motiva-tions, the importance of conversations between the interpreter (“terp”) and the Imam, and an appreciation of the group dynamics and emotions involved with interaction with crowds. Complex crowd interactions are not a novel situation for the military. However, novel weaponry,


July 2017

that is, non-lethal weapons (NLW), have been proposed to increase a sol-dier’s options in these crowd missions, and other current and near future operations (Burgei, Foley, & McKim, 2015, pp. 30–34; National Research Council, 2003; Tafolla, Trachtenberg, & Aho, 2012). Recent Department of Defense (DoD) directives and instructions attest to the importance of this class of weapons (DoD, 2012, 2013).

NLW have been used successfully in-theater, for example, as detailed in Gudmundsson’s (n.d.) “Kosovo Incident Case Study: Use of Non-lethal Weapons.” In April of 2000, members of the NATO-led multinational task force providing security to inhabitants of Kosovo were trapped by a crowd of 50–75 Serbs. The crowd was protesting the arrest of a Serbian man. Militants within the crowd had incited the crowd to violence, and attempts by the former leaders to de-escalate the crowd had failed. Those in the back of the crowd who were on higher ground threw rocks and bottles, while those in the front pushed forward brandishing sticks. In addition, rioters came from behind and attacked from the rear. Sponge grenades were then used against the crowd. The NLW had the intended effect of causing some to fall down, some to run away, and to break up the organized behavior of the crowd. Ultimately, U.S. forces were able to escape the crowd without any soldiers killed and without killing any civilians. This incident clearly demonstrated the operational utility of NLW.

For many reasons, though, acquisition processes for all NLW have been considered to be problematic (Government Accountability Office [GAO], 2009; LeVine & Montgomery, 2002). Principally, the core issue can be identified as a lack of methods to test and evaluate the effectiveness of NLW against intended human targets (GAO, 2009; Mezzacappa, 2014; North Atlantic Treaty Organisation, 2009). A search of articles in the Defense Technical Information Center, ProQuest, and Google Scholar did not reveal any reports of test and evaluation of NLW against crowds (aside from those authored by the Target Behavioral Response Laboratory), and very little empirical work on NLW effectiveness testing in general.

The present experiment was funded by the Joint Non-lethal Weapons Program (JNLWP), the U.S. DoD authority that has oversight of NLW activities (DoD, 2013). The objective of the study was to identify behav-ioral response metrics to be used in determining NLW effectiveness. The results identify metrics, measures, and a design of experiments and analytic methods for assessing NLW effectiveness within a crowd-control force paradigm. Given the current push (DoD, 2015) for an “institutionaliza-tion of scientific test design and statistical rigor” (Warner, 2013, p. iii) in the

538

test and evaluation community, we propose that the experimental results present a sound quantitative method of evaluating effectiveness of any NLW in crowd-control force interactions. First, a brief introduction to crowd research is presented. Next, the special processes for NLW acquisition will be detailed. After these two overviews, a general framework and method for effectiveness testing for NLW against crowds are presented. The method can be described as an infusion of academic social psychological methods into the developmental and operational test and evaluation procedures.

BackgroundCrowd Research

One of the foremost current crowd researchers today has described crowds as the “elephant man” of social science—“something strange, some-thing pathological, something monstrous. At the same time, they are viewed with awe and fascination” (Reicher, 2001, p. 182). Perhaps because of the difficulty of conducting research on crowds and perhaps the entrenched and fear-inducing notions of “mob mentality,” there is little direct research, especially experiments, done on crowds. What little there exists is summa-rized in Reicher’s chapter and in other sources (Kenny, et al., 2001; Miller, 2000; Reicher, 2001; Silver, 2002).

Early information-gathering attempts by the NLW community included convening of a special panel of crowd subject matter experts (SMEs). The Human Effects Advisory Panel (Kenny et al., 2001) assembled leading researchers and practitioners in crowd control science to identify what is known and not known about crowd behavior. The panel presented evidence that many of the common notions about crowd protests and riots have been found to be inaccurate. Following an extensive section on a history of riots, the report lists stereotypes that are commonly held about crowds, but have been debunked by scientists. In summary, the findings are that the mem-bers of crowds come with different motivations and seldom act in unison, that they come in groups, and that they are not anonymous to one another. Moreover, crowds are not particularly distinguished by violence and disor-der, they do not have emotional reactions that are out of the ordinary, and the members do not lose their minds, nor is there concrete evidence that violence is precipitated by a “spark” or “flashpoint.” These ideas have been well researched in the literature, particularly in the work of Clark McPhail (McPhail, 1991; McPhail & Tucker, 1990; Schweingruber & McPhail, 1999).

539

Crowds and Non-lethal WeaponsMore recently, Mezzacappa and colleagues conducted in-depth reviews

of the literature relevant to crowd-control force interaction with non-le-thal weapons (Mezzacappa, 2009; Mezzacappa et al., 2011). As part of the review of the literature, they examined results from an online question-naire on soldiers’ accounts of crowd encounters and any NLW use in those encounters. The review and responses reveal the complex and frightening dynamics involved in crowd-control force encounters. The following points were highlighted for the field operations: (a) crowd control is more up close and personal; conventional training does not prepare one for verbal and physical abuse; (b) it is important to understand crowd motivations—they help you to control the situation; and (c) it is critical to communicate with the crowds, leaders, and members to get the information you need and to influence the behavior of the crowd. Moreover, reports from Somalia, Haiti, Brčko, Kosovo, and Iraq show that military operations with crowds vary greatly in terms of crowd motivation and U.S. mission (Mezzacappa, 2009).

Crowd control is considered to be one of the core capabilities for NLW (Bedard, 2002) and has been a driving factor in NLW initiatives (Albertson, Murphy, Jackson, & Jones, 2000; Kenny et al., 2001; Shuttleworth, 1998). Therefore, it is incumbent upon the acquisition community to develop methods of evaluating the effectiveness of NLW, specifically against crowds. Given the human complexity of crowds, proposed methods must be able to incorporate behavioral science research designs to obtain valid measures of NLW performance. All NLW though, must undergo atypical acquisition processes as discussed in the next section.

The Acquisition Process of Non-lethal Weapons The Joint Non-lethal Weapons Directorate (JNLWD) has created an

acquisition process unique to NLW. The DoD realized early on that acqui-sition of this class of weapons required inclusion of expertise not normally involved in the acquisition process. Medical personnel, in particular, are needed to assist in evaluations of risk of significant injury in order to sup-port the characterization of non-lethality (LeVine & Montgomery, 2002). To address the requirements of non-lethality, DoDI 3200.19 (DoD, 2012) requires that non-lethal acquisition programs undergo review by the Human Effects Review Board (HERB) to assess the adequacy of the information on human effects, the identified potential risks, and mitigation strategies to mitigate the risks. In addition to this review, which focuses on the “non-le-thal” safety aspect, an additional scientific assessment (Burgei, Foley, & McKim, 2015) reviews the human effects characterization, modeling, and engineering efforts during the development phase of NLW.

540

Another critical component of the NLW acquisition is the DoD Non-lethal Weapons Human Effects Team, which consists of the Human Effects Center of Excellence (HECOE) and the JNLWD Human Effects Office (Simonds, 2014). The DoD NLW Human Effects Team’s primary mission is to oversee NLW Human Effects Characterization—a formal process mandated by DoDI 3200.19 (DoD, 2012) for fully describing the compendium of physiological and behavioral effects knowledge associated with a given NLW. It estab-lishes the baseline human effects understanding of NLWs, identifies risk and data gaps in human effects knowledge, and facilitates presentation and communication of its human effects (JNLWD, 2016). The team provides sup-port to science and technology, research and development (R&D), assists in transition and integration, and provides combat and material development support (Simonds, 2014).

It is important to note that the JNLWD, HERB, and HECOE serve in a con-sultative role for the Services and are a clearinghouse for information about NLW, but they do no test and evaluation. The testing that HECOE conducts is primarily in human effects in support of modeling and simulation, or M&S (JNLWD, 2016; Simonds, 2014). As such, the efforts by HECOE are

541

in support of development of NLW in that their efforts are not configured to be able to answer the particular quantitative performance questions demanded by the current acquisition process (Director, Operational Test and Evaluation [DOT&E], 2015; Warner, 2013). The JNLWD Human Effects Office is tasked with ensuring the Services have the human effects support that they need for the NLW acquisition process, but the Services are respon-sible themselves for conducting proper acquisition process procedures and subsequent decisions.

In response to this situation, this article was written to provide guidance for the acquisition workforce involved with NLW. Since 2004 and with JNLWD support, the Tactical Behavior Research Laboratory (TBRL) has conducted developmental research, and test and evaluation on a range of NLW energies, including blunt impact (Short, 2006; Short, Reid, Cooke, & Minor, 2010), light and laser (Cooke, 2007; Cooke, Mezzacappa, Yagrich, & Riedener, 2010), and acoustic stimuli (Riedener, 2007; Riedener et al., 2007a, 2007b ). The approach taken by the TBRL is based on DoDD 3000.03E guid-ance (DoD, 2013) that NLW “Deter, discourage, delay, or prevent hostile and threatening actions” (p. 2). Because the word “behaviors” can be substituted for the word “actions,” one can readily surmise that the primary science involved in NLW effectiveness research and testing is behavioral science.

Test and Evaluation of NLWThe importance of behavioral science. Over the centuries behav-

ioral science has invented methods for the metrics, measures, and analyses of scores of human behaviors, such as deterrence, discouragement, delay, and prevention.

The challenge for crowd NLW testers is to infuse experimental social psy-chological methods into developmental and operational test and evaluation methods. In essence, the methods of the crowd-control force paradigm for NLW test and evaluation stretch behavioral science methodologies to configure developmental and operational test and evaluation methods to accommodate the human factors related to crowd testing.

The diminished importance of replication. One point of tension between the two approaches (behavioral science versus test and evalua-tion) is the idea of replication; that is, that test conditions will ideally match conditions in-theater (DOT&E, 2015). In the case of test and evaluation for crowds, clinging to this notion may lead to dismissal of test and eval-uation of NLW and crowds as impossibly difficult. The test matrix of the bewildering array of human factors and varieties of crowd-control force scenarios would be endless. Not only would the costs render this type of

542

testing prohibitive, especially in the age of DoD funding austerity, but this insistence on replication limits the ability of the results to validly generalize to another possible control force—crowd conditions.

From the perspective of behavioral science, the key for testers of crowd NLW is to induce essential components of the crowd-control force sce-nario. Well-designed NLW experiments are studies that recreate critical social psychological forces that cause a person to engage or disengage with a control force (Cooke et al., 2010; Mezzacappa, Cooke, & Yagrich, 2008; Mezzacappa, 2014). Essentially, the crowd control force scenario consists of a plurality of people in conflict with a control force, where the control force wishes to control in some manner the movements or actions of crowd members—to deny, deter, suppress, etc. If a study can evoke this, essential dynamic results can be generalized to many other different variants of the crowd-control force encounters.

Similarities between testing of lethal and non-lethal weapons. Along these lines, it may be instructive to compare the testing of non-lethal and lethal weapons. Many variables can affect the performance of a typical lethal weapon, from operator skill and state, temperature, humidity, wind to name a few. Nevertheless, the primary tests of lethal weapon effective-ness occur in the controlled laboratory, with little concern for replication of operational realism. Limited lethal weapon testing against a real target is only used as a final demonstration to confirm the earlier controlled test-ing (except for anti-personnel weapons, which are never tested against an actual human target). In these testing conditions, lethal effectiveness can be assessed by measuring the effect that the weapon has on a standard target. For example, tank round effectiveness at penetrating armor is assessed by measuring depth of penetration into rolled steel. Small arms effectiveness at wounding soft tissue is assessed by measuring cavity volume and depth of penetration into a gel block calibrated to replicate soft tissue. In a similar manner, an NLW’s performance is assessed in the laboratory by its impact on its target, that is, how well it denies, deters, and suppresses target actions.

For lethal weapons, these laboratory results are combined with other con-trolled tests (such as round dispersion) to derive higher level metrics such as Probability of Kill (PK). PK is based on past scientific findings about how the direct measures interact with other measures of weapon characteristics. PK also is based on how the standard targets used for testing correlate to the variety of targets found on the battlefield.

543

It should be noted that lethal weapons have an advantage of over a century worth of scientific study compared to NLW. Scientific formulae, such as Gávre and deMarre’s, for penetration of ammunition into armor were already being discussed at the end of the 19th century (Maw & Dredge, 1891). The U.S. Navy was studying penetration at the turn of the century through the work of then Navy Lieutenant Cleland Davis. These findings were being taught in ordnance textbooks at the beginning of the last century (Alger, 1915; Hayes, 1938; U.S. Naval Institute, 1910). Advances in the study of lethal weapons effectiveness were incorporated into design manuals throughout the middle of the 20th century and were well established in textbooks at the end of that century (Headquarters [HQ], U.S. Army Materiel Command, 1962; HQ, U.S. Army Materiel Command, 1963; U.S. Army Ordnance Corps, 1957; U.S. Military Academy, 1999). The science of NLW effectiveness is nascent by comparison. A search of the literature reveals that there were a few years of interest around the 1950s and again in the 1970s. This interest was revived at the start of the 21st century, but overall, our understanding of NLW is over a century behind the lethal counterparts.

Examination of the well-accepted methods of lethal weapons effectiveness shows that fidelity to operational conditions is not a priority. Comparative test and evaluation of NLW may also adopt a similar approach using a similar rationale. While a myr-iad of variables are involved, we suggest that the critical metrics of NLW performance are best assessed within the confines of a test-ing facility, with experimental controls over as many variables as is pos-sible. This is not to say that in crowd scenarios the cultural, h i e r a r c h i c a l , r e l i g i ou s , g roup dy na m ic cha ra c-teristics do not mat ter i n det er m i n i n g crowd behavior. The point is that milestone decisions for acquisition can be based on results from l a b or at or y t e s t a nd evaluation without the empha sis on high-f i-d e l i t y c r o w d - c o n t r o l force replications. Fidelity

544

to operational conditions is not a priority in comparative test and eval-uation of lethal weapons; the same may be said for comparative test and evaluation of NLW.

Solely focused on performance. It is also critical to recognize that the behavior of the crowd is of interest only as far as it relates to the performance of the weapon. Researchers must, however, understand enough about human behavioral science to conduct test and evaluation so that results lead to valid conclusions about weapon performance (e.g., controlling extraneous variables, randomization, blinding, counterbalanc-ing, etc.) (Leedy & Ormrod, 2016). Within the acquisition process, these performance metrics are compared among candidate devices, against a benchmark, or against requirements. The task at hand is to generate a procedure that can be standardized and replicated. The goal is to derive a laboratory procedure so that, all things being equal, the device that per-forms best in the laboratory, under controlled experimental conditions, can be predicted to perform best in-theater.

Collaborative programs of test and evaluation and research. Behavioral science approaches make the NLW crowd-testing effort more tractable, but not perfect. As will be seen later in this article, the method presented can provide quantitative information upon which to make sound milestone decisions. In addition, this information can be aggregated with other information from demonstrations, subject matter expertise, or first-hand accounts of experiences to support milestone decisions on NLW. A particularly productive collaboration may be one between the laboratory scientists and M&S researchers of crowd behavior and NLW (Mezzacappa, 2014; Reid et al., 2014). Because of the great interest DoD has in developing NLW, the acquisition community needs to start to address feasible yet fruitful approaches to test and evaluation of NLW in crowd-control force operations.

The Present Study: A Methodology for Crowd NLWEffectiveness Evaluation

A design of experiments. The current study, which was funded by the JNLWP, examines the approach methods by which effectiveness can be evaluated. Like effectiveness test and evaluation of lethal weapons, an understanding of the key metrics of effectiveness and metrics of perfor-mance as well as statistical means of analysis must be developed to provide rational support for acquisition decisions (DOT&E, 2015; Warner, 2013).

545

The current study demonstrates the construction of a research design that yields quantitative metrics and statistical design of experiment analyses (Montgomery, 2012). These are the required characteristics for research and testing for NLW (JNLWD, 2016; Office of Naval Research, 2014).

Cross modality test and evaluation. Earlier NLW effective-ness testing efforts typically investigated one modality or type of NLW (e.g., light, sound, blunt impact). The metrics developed in these single modality studies were then used to develop approaches that could be used to evaluate and compare effectiveness among different types of weapons (Mezzacappa, 2014; Mezzacappa et al., 2012). That is, these methods can now be used to benchmark and compare NLW of different energies (e.g., light, sound, blunt impact, etc.). In essence, these methods can be used to generate standardized laboratory metrics that are necessary for creation of a framework for evaluating effectiveness of NLW (North Atlantic Treaty Organisation, 2004, 2009).

The present study was undertaken to explore testing methodologies that can be used in comparing effectiveness of NLW of different modalities. As such, the endeavor must be understood as a methodological study, that is, the work seeks to establish an approach—a methodological framework—in which to evaluate NLW. The results speak to the ways in which NLW can be evaluated. Readers should not construe the results of this experiment as representative of the performance of any simulated NLW. Instead, it is our hope that readers are able to see that the methods presented here are appli-cable to evaluation of NLW throughout the developmental and operational test phases of the acquisition process.

Crowd-control force paradigm. The present testing occurred within a given NLW scenario—a control force facing a crowd. The acquisition-rel-evant questions addressed were:

• What are appropriate NLW effectiveness metrics?

• Can these metrics be analyzed statistically?

• Can these metrics and ana lyses be used to benchmark and compare effectiveness among weapons of different modalities?

546

MethodScenario Configuration

All procedures were approved by the local Human Research Ethics Boa rd (A R DEC IR B #10-0002, “Effectiveness Testing for Crowd Management with Non-Lethal Weapons”). The general paradigm was established by earlier work (Cooke et al., 2010; Mezzacappa, Cooke, & Yagrich, 2008). The testing situation was configured based on NLW count-er-personnel tasks taken from the Joint Non-lethal Effects Capabilities Based Assessment. Seven small crowds participated in a scenario facing non-lethal and simulated NLW. The tasks that were chosen for examination from those capabilities identified by the JNLWD (2016) were: “Deny access into/out of an area to individuals (open, single/few/many)” and “Deny access into/out of an area to individuals (confined, single/few/many).”

Research participants (“subjects”) were incentivized with monetary rewards to approach a target; however, approach was punished by NLW fire (i.e., they were hit by NLW). Each directed-energy and projectile-weapon condition was tested four times (four trials); the acoustic hailer and the no-weapon conditions were tested twice (two trials). Exposure to the acous-tic hailer was limited to protect the subjects against potential permanent auditory damage.

Overview of ScenarioSubjects targeted an area protected by a single control-force defender at

a fixed location at the center of the protected area (Figures 1 and 2). Subjects earned money for scoring points during the test by successfully approaching the protected area and throwing “rocks” into a target. The control force pro-tected the area using either foam projectiles, directed energy, or an acoustic hailer (only one weapon type per trial). Type of weapon was not revealed to subjects before the trial. Subjects could lose money by being hit by the control force during the test. Subjects were paid $20/hour for participa-tion (Cooke et al., 2010; Mezzacappa, Cooke, & Yagrich, 2008). During the experiment, a computer recorded each subject’s location, orientation, and locomotion through the test bed through motion-capture and video cameras.

547

FIGURE 1. CROWD BEHAVIOR TEST BED CONFIGURATION

X Axis (meters)

Y Axi

s (m

eter

s)

Target

-4

-2

0

2

4

6

-4 -2 0 2 4

Origin

Notional start line used

Actual start/quit Line

Note. Subjects started in the area marked “Actual Start/Quit Line” and traveled to the target area during the trial. On projectile and dismounted energy weapon trials, the control force stood midline at the target. The long-range energy and acoustic weapons were located on the test bed.

548

FIGURE 2. CONTROL FORCE SCENARIO

Note. Top: Subjects moving toward the target from the starting safe zone area. Bottom: Subjects moving back toward the safe zone after challenging the control force (middle of target) at the end of the trial.

549

ParticipantsParticipants were recruited from the general population to participate

in an investigation on “Crowd Movement.” Fifty-two men and women par-ticipated in one of 7 experiment days. Subjects were healthy local residents or Picatinny Arsenal employees over the age of 18. To safeguard partici-pants, subjects had to pass a hearing test, be free of reactive airway disease or sensitivity to loud sounds or hearing damage, and have no history of adverse effects to brief, loud noises.

Weapon ConditionsDuring counterbalanced experimental trials, the crowd faced one of

four possible types of weapons. Subjects were instructed that they would receive weapons fire if they crossed over the line between the safe zone and the targeting area. Each weapon condition was run four consecutive times.

Acoustic hailer. An acoustic hailer is a loud sound played over a cur-rently fielded loudspeaker; an American Technology Corporation Medium Range Acoustic Device (MRAD), was used against the crowd on acoustic trials. The “warble” tone that is preprogrammed into the MRAD loud-speaker was used. Following on from previous experimentation (Riedener, 2007; Short et al., 2009), sound levels were set to deliver decibel levels of 115dB(A) in the field of exposure. For protection, hard barriers of the testbed prevented participants from approaching the MRAD any closer than 50 feet, thus ensuring safety in the exposure to 115dB(A).

The U.S. Army Public Health Center has established time limits on expo-sures for several levels of noise (U.S. Army Center for Health Promotion and Preventive Medicine, 1999). These standards limited exposure to the 115dB(A) level (about as loud as heavy construction equipment) in this experiment to a duration of no more than 28.14 seconds. Therefore, the MRAD was under computer control for the onset and offset of sounds so that the trial total exposure time was 28.14 seconds. In order to have at least two full trials of an acoustic hailer, all trials were set to be 14 seconds (28.14 sec/2 trial = ~14 per trial). The MRAD was located so that the greatest acoustic stimulation occurs at the target that the subjects were approaching. The computer control turned the sound on when subjects crossed over the line into the zone for targeting during encounters, and turned off when all subjects had left the targeting zone.

Projectile weapon. A commercially available foam projectile gun was used by the control force. The foam projectiles are tipped with Velcro mate-rial that sticks to fabric so that impacts on subjects can be recorded. The sponge or foam ends were coated with chalk powder that marked contact

550

with subjects. To facilitate visualization of the chalk powder, subjects were given white tee shirts to wear. For every hit, the subjects lost $2.50, and they could be hit multiple times per trial.

Long-range directed energy weapon. No control force person-nel were visible to the crowd under this weapon condition—simulating a directed energy weapon that is located about 1 kilometer away. A com-puter-controlled feedback system was developed to simulate the primary perceived characteristics of a long-range directed energy weapon, that is, an impact with severe consequences with no detectable source. During these trials, subjects were targeted by computer when they entered into the desig-nated fire zone. Firing was computer-controlled and automatically initiated when a subject’s location indicated he or she was in the designated area for fire. Subjects wore a commercially available alert system that emitted blinking lights, sound, and vibrations when triggered by radio signal. When subjects were hit by the weapon, they did not experience physical damage; however, $10 was taken away for every hit. Subjects could be hit only one time during each encounter.

Dismounted infantry/soldier-carried directed energy weapon. In this weapon condition, the characteristics of the weapon fire were iden-tical to those in the long-range version of the weapon, except that subjects faced a single, gun-wielding control-force person. The crowd could therefore observe the control-force person’s target selections. Simulation of a directed energy fire impact event was subsequently executed by computer com-mand. A computer-controlled feedback system was developed to simulate the primary perceived characteristics of a soldier-carried directed energy weapon. That is, in this weapon condition, crowd members could experience a weapon impact with severe consequences without a visible projectile.

No-weapon baseline condition. Subjects first targeted the protected area without a control-force defender present, and no weapon was fired. The conditions were otherwise identical to those with weapons. Because previ-ous work showed that there was little variation in response to no-weapon baseline conditions, this condition was run two consecutive times to con-serve time and monetary resources.

Dependent VariablesCrowd measures: centroid and leading edge. Effectiveness mea-

sures were assessed using the location and movement of the crowd as a whole, not the specific individuals that make up the crowd. Following on from previously published work (Cooke et al., 2010), the movement and

551

location of the “Leading Edge” (front of the crowd) and the “Centroid” (the geometric middle of the crowd) were derived from location and time data (Figure 1).

Effectiveness metrics. The concept of effectiveness in the crowd-con-trol force scenario was explored with several simple questions (Table 1). These effectiveness questions were then translated into mathematical terms using the individual and aggregate crowd measures.

TABLE 1. EFFECTIVENESS QUESTIONS: CROWD-CONTROL FORCE SCENARIO

1. Does the non-lethal weapon make the crowd stay away in a safe area? (% Suppressed)

Ultimately, in a crowd-deny access task, the best outcome is to have the crowd stay away, that is, to suppress approach. This direct metric is based on the percentage of the crowd members that did not cross over the line separating the safe zone and the area where they would be targeted by non-lethal weapons. Motion capture data were used to identify subjects who did not leave the safe zone during the trial. For each crowd and for each trial, the count of these subjects was divided by the total number of participants in the crowd to arrive at the percentage of subjects whose approach behavior was suppressed. This metric is perhaps the most straightforward of all the effectiveness metrics proposed.

2. Does the non-lethal weapon make a person think long and hard about facing the weapon? (Hesitancy)

If subjects did cross out of the safety zone, the time at which each subject crossed the line could be determined. For each crowd and for each trial, the time until crossing was averaged over all the subjects who crossed the line. Delayed approach (hesitancy) may be reasonably associated with cognitive decision making regarding cost/benefit ratio. No delay may be reasonably associated with high motivation to confront the control force. Hesitancy or latency was calculated for each individual who chose to approach. This delay was calculated as the time from start of trial until the time when the individual crossed the line to leave the safety zone to approach the protected area.

3. When a crowd is on its way to a protected area, by how much does the non-lethal weapon slow down the crowd? (Approach Speed)

For each crowd and for each trial, the movement of the centroid of the crowd was derived. For these analyses, the approach speed of the centroid toward the target was used.

4. For how long does a crowd stay within the non-lethal weapon’s range? (Time Under Fire)

For each crowd and for each trial, the movement of the centroid of the crowd was derived. For these analyses, the amount of time the centroid is located in the targeted area was used.

5. How close does the crowd get to the non-lethal weapon? (Closest Distance)

For each crowd and each trial, the movement of the leading edge was derived. For these analyses, the closest distance from the protected area was used. Because the leading edge is the part of the crowd most likely to interact with the control force, it is the most appropriate aggregate crowd metric for these analyses.

552

All NLW performance was compared to a no-weapon baseline where approach was not opposed. That is, if the crowd approaches just as quickly and as closely to a protected area when a weapon is there as opposed to when there is no weapon, then the weapons can be judged as ineffective.

Data Analysis Analyses were carried out to address the questions:

• Is the NLW better than nothing?

• Is any single NLW more effective than the other weapons tested?

Data were analyzed using multivariate repeated measures linear regression methods (Montgomery, 2012). First, omnibus regressions compared the no-weapon baseline condition with each of the NLW on all five effectiveness measures (% Suppression, Hesitancy, Approach Speed, Closest Approach, Time Under Fire) with the experimental design Condition (No-Weapon vs. Weapon) x Trial (1, 2, 3, 4). These analyses were run separately for the acoustic, dismounted directed energy, long-range directed energy, and projectile weapon. The no-weapon condition was run twice. Therefore, to enable comparisons against the weapons with four trials, data from Trials 1 and 2 were entered again for Trials 3 and 4. The baseline data were miss-ing from the first crowd run; the means from the other six crowd runs were used as substitutions.

Second, those NLW that were found to be significantly different from base-line were run again in another similar omnibus regression comparing performance among the weapons. Regression analyses were run on change scores derived from the difference in effectiveness metrics from baseline and under the weapon conditions, with the design Weapon (Dismounted Directed Energy, Long-Range Directed Energy, Projectile) x Trial (1, 2, 3, 4).

ResultsGraphics/Images – Overview

Figures 3–7 show each NLW’s performance on each of the effectiveness metrics. The figures are discussed with respect to the questions that eval-uate an NLW’s effectiveness (Table 2).

553

FIGURE 3. MEAN PERCENTAGE OF CROWD SUPPRESSED

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 2 3 4

Perc

enta

ge o

f Cro

wd Su

ppre

ssed

Trial

None Acoustic Long-Range Directed Energy Dismounted Directed Energy Projectile

Note. Mean and standard error of percentage of crowd members who choose not to venture into the line of fire, that is, approach behavior is suppressed. The higher the percentage, the more suppression the weapon exerts. Note that the data for the acoustic hailer completely overlap the data for the no-weapon condition.

554

FIGURE 4. MEAN DELAY TO APPROACH/HESITANCY


-2

0

2

4

6

8

10

12

1 2 3 4

Late

ncy/

Hesti

ancy

, sec

onds

Trial

Note. Mean and standard error of time until people decide to start approaching protected area over trials for each weapon. The higher the point, the longer the person takes until starting approach. Note that the data for the acoustic hailer completely overlap the data for the no-weapon condition.

FIGURE 5. MEAN SPEED OF APPROACH


0

0.5

1

1.5

2

2.5

1 2 3 4

Spee

d, m

eter

s per

sec

Trial

Note. Mean and standard error of speed of crowd approach on each of the trials for each of the weapons. The higher the point is, the faster the approach toward the protected area.

555

FIGURE 6. MEAN CLOSEST DISTANCE ON APPROACH


0

1

2

3

4

5

6

7

8

9

10

1 2 3 4

Dista

nce t

o Pro

tecte

d Are

a, m

eter

s

Trial

Note. Mean and standard error for closest distance of crowd to protected area on each trial for each weapon. The higher the point, the farther away the crowd stayed from the protected area. Note that the data for the acoustic hailer almost completely overlap the data for the no-weapon condition.

FIGURE 7. MEAN TIME IN LINE OF FIRE


0

2

4

6

8

10

12

1 2 3 4

Time i

n lin

e of fi

re, s

econ

ds

Trial

Note. Mean and standard error for time crowd spends in the line of fire for each weapon on each trial. The higher the point, the longer the crowd stays within range of the weapon.

556

TABLE 2. EFFECTIVENESS QUESTIONS: NON-LETHAL WEAPONS (NLW)

1. Does the NLW convince the crowd to stay away in a safe area?

Figure 3 shows the average percentage of subjects in a crowd who chose not to approach the protected area on each trial for each weapon. The acoustic hailer condition is identical to the no-weapon condition; no crowds were suppressed to any degree in either of these conditions. The most suppression was recorded under the long-range directed energy condition (60%) while the dismounted directed energy and projectile conditions recorded less suppression.

2. Does the NLW make a person think long and hard before facing the weapon?

Figure 4 shows the average amount of time that passes for each NLW before a subject starts toward the protected area for each of the trials. Again, the delay associated with the acoustic hailer is again identical to that of the no-weapon condition; that is, there is no delay in starting toward the protected area. Subjects showed the greatest hesitation in starting toward the target during the long-range directed energy condition, with the dismounted directed energy and projectile weapons showing less of a hesitation by the subjects in the crowd.

3. When a crowd is on its way to a protected area, by how much does an NLW slow down the crowd?

Figure 5 shows the average speed at which the crowd moves toward the protected area for each of the trials and each of the weapons, calculated as the speed of centroid of mass of crowd. The graph shows that the crowd moves fastest when there is no weapon protecting the area. The crowd moves slowest when faced with the projectile weapon and the control force, at least when first challenged by the weapon.

4. How close does the crowd get to the NLW?

Figure 6 shows on average how close crowds got to the protected area for each trial for each NLW, calculated for the leading edge of the crowd. Crowds reached closest to the protected area when there was no weapon protecting it and when the area was protected with an acoustic hailer. Crowds stayed farthest away when faced with the long-range directed energy weapon.

5. For how long does a crowd stay within an NLW’s range?

Figure 7 shows the average amount of time the crowd stays within range of each of the NLW on each trial, calculated as time the centroid of mass of the crowd stayed within the range of the weapon. Crowds spent the longest time within firing range during the no-weapon and acoustic hailer condition. The least amount of time spent in firing range was when the crowd was faced with the long-range directed energy weapon.

Statistical Analyses: Is the NLW Better Than Nothing?Acoustic hailer vs. no weapon. Univariate tests showed no statisti-

cally significant differences between effectiveness measures under acoustic vs. no-weapon conditions (Table 3).

557

Dismounted directed energy weapon vs. no weapon. Univariate tests showed statistically significant differences between the effectiveness measures of Approach Speed and Closest Approach under dismounted directed energy weapon vs. no-weapon conditions. When faced with the control force wielding this directed energy weapon, the crowd approached more slowly and stayed farther away compared to when there was no weapon present (Table 3).

Long-range directed energy weapon vs. no weapon. Univariate tests showed statistically significant differences on all effectiveness mea-sures. When faced with this directed energy weapon, more subjects were suppressed, subjects hesitated more, approached more slowly, stayed farther away, and spent less time within firing range compared to when there was no weapon present (Table 3).

Projectile weapon vs. no weapon. Univariate tests showed statis-tically significant differences on all effectiveness measures except Time Under Fire where there was a statistical trend toward significance. When faced with this projectile weapon, more subjects were suppressed, subjects hesitated more, approached more slowly, stayed farther away, and spent slightly less time within firing range compared with when there was no weapon present (Table 3).

Statistical Analyses: Is any single NLW more effective than the other weapons tested?

The acoustic hailer failed to show a difference from the no-weapon con-dition; therefore, this weapon was left out of these analyses. Change scores from no-weapon baseline were calculated for each weapon and each trial for each crowd. These change scores were entered into an omnibus multivariate repeated measures regression analysis. Effectiveness measures were com-pared simultaneously among the dismounted directed energy, long-range directed energy, and projectile weapons.

Multivariate-level analyses indicated a significant overall difference in the effectiveness measures among the weapons. Univariate analyses indi-cated that the NLW differed significantly in % Suppression and Hesitancy (Table 4). Within-subject contrasts indicated that compared with the projec-tile weapon condition, the long-range directed energy weapon was associated with greater % Suppression and greater Hesitancy (both F(1,6)=6.14, p<.05). Within-subject contrasts also indicated that compared with the dismounted directed energy weapon condition, the long-range directed energy weapon was associated with slightly greater % Suppression and slightly greater Hesitancy (both F(1,6)=4.58, p=.08).

558

TABLE 3. DIFFERENCES IN EFFECTIVENESS METRICS COMPARED TO NO WEAPON

Metric of Effectiveness

Acoustic Hailer

Dismounted Directed Energy Weapon

Long-Range Directed Energy Weapon

Projectile Weapon

Approach Speed * F(1,6)=19.31, p<.01 F(1,6)=7.52, p<.05 F(1,6)=47.79,

p<.0001

Closest Approach * F(1,6)=7.26, p<.05 F(1,6)=26.98,

p<.005F(1,6)=27.04, p<.005

% Suppression * * F(1,6)=15.45, p<.01 F(1,6)=7.02, p<.05

Hesitancy * * F(1,6)=11.85, p<.05 F(1,6)=6.59, p<.05

Time Under Fire * * F(1,6)=48.91,

p<.0001F(1,6)=5.65, p=.055)

* Denotes that there was no statistically significant difference in this metric of effectiveness between this NLW device and no weapon in the test bed.

TABLE 4. MULTIVARIATE-LEVEL ANALYSES OF METRICS OF EFFECTIVENESS

Omnibus Multivariate-Level Analysis F(10,18)=3.08, p<.05

Univariate-Level Analyses

% Suppression F(2,12)=3.97, p<.05

Hesitancy F(2,12)=3.97, p<.05

Learning EffectsAcross all the statistical analyses, there were significant effects of Trial.

These findings indicate that as the crowds learned about the vulnerabili-ties of the control force and the control-force weapons, they adjusted their countermeasures to overcome penalties. In some cases, the countermeasure was complete avoidance of the control force, that is, behavioral suppres-sion. As a result, in these cases effectiveness of NLW increased with trials. From Figures 3–7, one can see that the learning effect is especially pro-nounced for the simulated long-range directed energy weapon. As the trials progressed, this weapon showed increasing effectiveness in deterring approach toward the protected area. This finding has implications for tactics, techniques, and procedures for using similar existing and future directed energy systems, such as the Active Denial System (LeVine, 2009).

DiscussionSummary of Statistical Results

This test demonstrated that common human performance metrics could be used to compare across weapon systems of varying modalities. The results indicate that the acoustic hailer was ineffective, and overall

559

the long-range dismounted directed energy weapon was the most effective weapon. When faced with the long-range directed energy weapon, the sub-jects were less likely to attempt to approach the protected area and took longer to decide to approach than when faced with the dismounted directed energy weapon or the projectile weapon. Crowds modified their tactics in dealing with the control force as they learned what countermeasures were effective against receiving penalties; therefore, in some cases the effectiveness of the weapons increased as the crowd better understood the capabilities of the weapons.

A Test and Evaluation Methodology To reiterate, the purpose of this experiment was to demonstrate a

methodology for NLW effectiveness testing in a crowd scenario. The results from analyses of the simulated weapons can in no way be generalized to these classes of weapons, or any other weapons. The sole exception is the results for the acoustic hailer. Indeed, the results presented here replicate the results from previous reports, which examined a number of different acoustic stimuli at different wavelengths (Riedener, 2007; Riedener et al., 2007a, 2007b; Short et al., 2009; VanMeenen, 2006). This design and anal-ysis of experiments offers several benefits for the acquisition community for the development, and test and evaluation of NLW.

Benefits of this MethodQuantitative analysis of alternatives. As the Results section

showed, statistical analyses can provide the basis for quantitative evalu-ations of whether or not an NLW performs better than no weapon at all, and can identify superior performance among a set of devices. In addition, as illustrated in Figures 3–7, a design of experiments approach allows for quantitative Analyses of Alternatives (AoA) that includes effectiveness parameters. For example, Figure 3 indicates that by Trial 4, an average of 60 percent of the crowd was suppressed by the simulated long-range directed energy weapon, compared to an average 5 percent of the crowd that was suppressed by the projectile weapon. Analysts can then use these metrics to examine trade spaces involving cost and effectiveness, and other considerations. Simply put, the decision becomes whether or not the average 55 percent greater suppression of the crowd is worth the costs and effort of acquiring the directed energy weapon. Similarly, as can be seen in Figure 7, in the first trial the acoustic weapon manages to keep the crowd to an average of 0.5 meters from the protected area, whereas the crowd is held to an average of 7 meters beyond that. AoAs can include assessments of the cost of the additional average of 7 meters of stand-off performance improvements afforded by the projectile.

560

Quantitative benchmarking and requirements generation. The ability to measure quantitative measures of NLW performance cre-ates the ability of the acquisition community to benchmark performance against which candidate devices can be evaluated. For example, the 7.5 meter stand-off afforded by the projectile may set the target stand-off distance for the next generation of NLW to be double or triple this bench-mark. Numerical evaluation allows the measurement of improvement. Moreover, quantitative methodologies can put into perspective the cur-rent requirements as identified and set by the JNLWD (Office of Naval Research, 2014). One can use the present methodology to both assess the current capabilities and better understand the challenges of meeting the requirements set by decision makers.

Developmental engineering and specifications relevant to NLW effectiveness. One of the purposes of the present experiment was to demonstrate a method of test and evaluation that allows for comparisons of weapons of different modalities. Beyond comparison of the effectiveness of entire systems, comparisons of the effectiveness of specific components can be undertaken with the methods discussed herein. Finer resolution weapon characteristics (e.g., platforms, fire control, soldier interfaces) can also be tested for their impact on NLW effectiveness. Requirements gen-eration and benchmarking then can also be set for critical factors such as accuracy, reload time, slew rate, or impact area—all of which are known to affect the effectiveness of weapons in general.

Research on the underlying psychological mechanisms of NLW effectiveness. Without question, crowd behavior, targeted or not by NLW, necessarily involves more variables that can be manipulated, controlled, eliminated, recorded, or observed in any study, no matter how large or comprehensive. The proposed methodology provides a robust paradigm in which psychosocial and cultural variables can be explored for their effects on NLW effectiveness. Within these research designs, psychosocial variables can be varied with subsequent effectiveness measures compared among the different conditions. Thus, this is a paradigm that can be used to explore how different NLW of different modalities may fare against crowds of differing characteristics.

Perceived Limitations on NLW Effectiveness TestingTo our knowledge, this is the first effort to produce a procedure of gen-

erating empirical performance metrics capable of meeting the DoD test and evaluation definition of quantitative performance of NLW against crowds. Its shortcomings are recognized. The prime shortcoming is that there may

561

be limitations on generalization of results, the issue that recurs in every controlled experiment done in a laboratory. Restrictions arising from safety, logistical, financial, and ethical concerns necessarily constrain the ideal scientific testing procedures. However, we propose that some commonly perceived limitations are found to be, under closer scrutiny, far less debil-itating for the acquisition process. These perceived limitations are briefly discussed here because they may be perceived to apply to all laboratory-con-trolled NLW effectiveness testing.

Motivation and NLW effectiveness. A more particular criticism can be directed toward motivation levels. Logically and ethically, it is impossible to create in the laboratory the varied and extremely high levels of motivation that are assumed to be found in natural crowds encountered in-theater. In the laboratory, we control degree of motivation through monetary rewards and penalties. In real life, motivation is controlled by unknown forces, be they political, religious, or mundane. Crowds may be annoyed at, hostile to, or indifferent toward control forces. The criticism is then how can findings from the relatively placid New Jersey population extrapolate to other populations in other quite different cultures?

In examining this criticism, we again must restrict the discussion to issues relevant to DoD acquisition test and evaluation. The core question: “How does a target’s motivation relate to a weapon’s performance—the ability to deny, disable, move, or suppress his, her, or their behavior?” Very simply, it is expected that the more motivated a target is, the lesser is the ability of the NLW to deny, disable, move, or suppress that target.

562

Note that in this relationship, the kind of motivation (e.g., religious, finan-cial, appetitive) appears to matter less than degree of motivation (e.g., high, moderate, low).

If then, degree of motivation is the construct of interest, one may be able to see how extrapolation from the laboratory to theater can be valid. From the NLW operator’s point of view, denying, disabling, moving, or suppressing a highly motivated religious pilgrim may not be very different than denying, disabling, moving, or suppressing a highly motivated greedy capitalist. Manipulating degree of motivation is certainly more feasible than manipulating type of motivation. Within the bounds of research ethics and fundings, it may be possible to induce high levels of motivation within the laboratory.

It is recognized, however, that it is unlikely the laboratory can induce the highest levels of motivation that compel in-theater civilians. Even so, the findings of laboratory testing are valuable for NLW as a screening method. That is, if an NLW fails to show effectiveness against neutral crowds from New Jersey suburbs, it will most certainly fail against an in-theater hostile crowd. This proposition has been dubbed “The NJ Screening Test for Non-lethal Weapons” (Mezzacappa, 2014).

Institutional review boards forbid live-fire NLW testing. Readers might argue that NLW cannot be tested and evaluated through live-fire. They might protest that because of the ethical issues of targeting people with real NLW, Institutional Review Boards (IRB) and Human Research Protection Offices will refuse to review and approve NLW protocols. While it is beyond the scope of this article to fully discuss the IRB issues relevant to NLW, a short response to that statement is that the statement is not true. Experiments using NLW fires against intended human targets can be proposed to, and approved by, IRBs if the risks of the experiment to the subjects are outweighed by the scientific benefits from the study. This approval can be possible only if: (a) a complete set of safety data is available for the NLW, (b) risks to targets are fully described, (c) risks are properly mitigated within the research procedures, (d) all risks are fully and clearly described to potential subjects, and (e) subjects’ consent to participate are freely given. Resistance on the part of IRBs has not been the real obstacle to NLW effectiveness research. Rather, we have found in our decade-long experience, the difficulty in running live-fire NLW testing is the inability to receive from vendors comprehensive safety data. Without complete risk data, investigators cannot: (a) present to potential subjects a complete description of the risks they will be accepting with exposure to

563

the NLW, and (b) properly mitigate the risks. It is this lack of safety data, not a lack of safety, that renders NLW experimentation unacceptable to IRBs. When IRBs can ensure that all relevant information can be presented to a subject for consideration, and that risks are adequately mitigated, then IRBs can approve live-fire testing with NLW.

ConclusionsAgain, the results here should not be construed as evaluation of the

specific weapons tested. The results should be used as guidelines for test and evaluation of all NLW. This experiment demonstrates derivation of quantitative metrics and statistical effectiveness analyses of NLW per-formance. Using similar methods, relative effectiveness can be compared among weapons of different technologies, platforms, and energies. Similar testing paradigms, adapted to the particular weapon and concept of opera-tion, will support the acquisition of NLW from concept through deployment. The resulting information can be used for AoA and data-driven trade studies.

As operational scenarios and desired results become part of the NLW’s testable requirement, test beds can be designed to capture the effectiveness metrics that represent effectiveness for that weapon within that specific sce-nario or task. Once a non-lethal system is proven effective in the laboratory setting, then correlations to actual operational performance and effective-ness can be made from field reports. As the science of NLW effectiveness marches forward, there will be an expected evolution of improved test bed designs and metrics used to quantify the effectiveness of the NLW tested.

Therefore, the primary initial recommendation is to gather benchmark data on NLW effectiveness in the manner presented in this article. These methods enable the DoD acquisition community to gather benchmark data on NLW effectiveness, then to build upon and improve an item. Using test data will not only ensure that improvements are being made, but that the improvements are significant enough to warrant acquisition of a new item, thus giving the warfighter enhanced capability.

AcknowledgementsThis experiment was funded by the Joint Non-lethal Weapons Program

(JNLWP).

564

ReferencesAlbertson, M., Murphy, R., Jackson, A., & Jones, M. (2000, April). Civil disturbances:

Incorporating non-lethal technology, tactics, techniques and procedures (CALL Newsletter No. 00-7). Fort Leavenworth, KS: Center for Army Lessons Learned.

Alger, P. R. (1915). The groundwork of practical naval gunnery. Annapolis, MD: U.S. Naval Institute.

Bedard, E. (2002). Nonlethal capabilities: Realizing the opportunities. Defense Horizons, 9, 1–6.

Burgei, W. A., Foley, S. E., & McKim, S. M. (2015). Developing non-lethal weapons. Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics.

Cooke, G. M. (2007). Effectiveness of light on targeting (Report No. TBRL-MCSE-TR-07001/ARQES-TR-08001/AD-E403 156). Picatinny Arsenal, NJ: Target Behavioral Response Laboratory.

Cooke, G., Mezzacappa, E., Sheridan, C., DeMarco, R., Tevis, K., Reid, G., … Riedener, J. (2010, December). Topology and individual location of crowds as a measure of effectiveness for non-lethal weapons. In Proceedings of the 27th Army Science Conference, Orlando, FL.

Cooke, G., Mezzacappa, E., Yagrich, K., & Riedener, J. (2010, November). Effects of lasers on driving. In Directed Energies Professional Society, Proceedings of Thirteenth Annual Directed Energy Symposium, Bethesda, MD.

Department of Defense. (2012). Non-Lethal Weapons (NLW) human effects characterization (DoDI 3200.19). Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics.

Department of Defense. (2013). DoD executive agent for non-lethal weapons (NLW), and NLW policy (DoDD 3000.03E). Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics.

Department of Defense. (2015). DOT&E Test and Evaluation Master Plan (TEMP) guidebook. Washington, DC: Office of the Director, Operational Test and Evaluation.

Government Accountability Office. (2009). Defense management: DoD needs to improve program management, policy, and testing to enhance ability to field operationally useful non-lethal weapons (Report No. GAO-09-344). Washington, DC: U.S. Government Printing Office.

Gudmundsson, B. (2002). Kosovo incident case study: Use of non-lethal weapons [White paper]. Quantico, VA: U.S. Marine Corps, Joint Non-lethal Weapons Directorate.

Hayes, T. J. (1938). Elements of Ordnance: A textbook for use of cadets of the United States Military Academy. New York: J. Wiley & Sons.

Headquarters, Department of the Army. (2003). Peace Ops: Multi-service tactics, techniques, and procedures for conducting peace operations (FM 3-07.31). Langley AFB, VA: Air Land Sea Application Center.

Headquarters, Department of the Army. (2005). Civil disturbance operations (FM 3-19.15). Washington, DC: Author.

Headquarters, Department of the Army. (2006). Counterinsurgency (FM 3-24). Washington, DC: Author.

Headquarters, U.S. Army Materiel Command. (1962). Research and development of materiel, engineering design handbook, elements of terminal ballistics (part one),

565

introduction, kill mechanisms and vulnerability (AMC Pamphlet [AMCP] 702-160). Washington, DC: Author.

Headquarters, U.S. Army Materiel Command. (1963). Research and development of materiel, engineering design handbook, elements of armament engineering (part two), ballistics (AMCP 702-107). Washington, DC: Author.

Joint Non-lethal Weapons Directorate. (2016). Joint non-lethal weapons program science and technology strategic plan. Quantico, VA: Joint Non-lethal Weapons Directorate.

Kenny, J. M., McPhail, C., Waddington, P., Heal, S., Ijames, S., Farrer, N. D., … Odenthal, D. (2001). Crowd behavior, crowd control, and the use of non-lethal weapons. University Park, PA: Institute for Non-lethal Defense Technologies.

Leedy, P., & Ormrod, J. (2016). Practical research: Planning and design (11th ed.). Hoboken, NJ: Pearson.

LeVine, S. (2009). The Active Denial System: A revolutionary, non-lethal weapon for today’s battlefield. Retrieved from National Defense University website: http://jnlwp.defense.gov/

LeVine, S., & Montgomery, N. (2002, July-August). Non-lethal weapon human effects: Establishing a process for DoD program managers. Program Manager, 31(4), 50–54.

Maw, W. H., & Dredge, J. (1891). Modern French artillery, ballistic efficiency of quick firing Hotchkiss guns (No. LXII). Engineering, 51, 393–395.

McPhail, C. (1991). The myth of the madding crowd. New York: A. de Gruyter. McPhail, C., & Tucker, C. (1990). Purposive collective action. American Behavioral

Scientist, 34, 81–94.Mezzacappa, E. (2009). Crowd dynamics and military interaction with non-lethal

weapons and systems (JNLWD11-006). Picatinny Arsenal, NJ: Target Behavioral Response Laboratory.

Mezzacappa, E. (2014). Effectiveness testing of non-lethal weapons. Journal of Defense Modeling and Simulation, 11, 91–101.

Mezzacappa, E., Cooke, G., Merenda, B., Jaffery, N., Galonski, L., Hedderich, E., … Riedener, J. (2011). Crowd dynamics and military interactions review updated (Report No. JNLWD12-004). Quantico, VA: Joint Non-lethal Weapons Directorate.

Mezzacappa, E., Cooke, G., Reid, G., DeMarco, R., Sheridan, C., & Riedener, J. (2011, March). Mathematical capture of human crowd behavioral data for computational model building, verification and validation. In Proceedings of the 20th Annual Behavior Representation in Modeling & Simulation (BRiMS) Conference, Sundance, Utah.

Mezzacappa, E., Cooke, G., Reid, G., DeMarco, R., Sheridan, C., & Riedener, J. (2012, March). Data-driven modeling of target human behavior in military operations. In Proceedings of the 21st Annual Behavior Representation in Modeling & Simulation (BRiMS) Conference, Amelia Island, FL.

Mezzacappa, E., Cooke, G., & Yagrich, K. (2008, December). Network science and crowd behavior metrics. In Proceedings of the 26th Army Science Conference, Orlando, FL.

Mezzacappa, E. S., Sheridan, C., DeMarco, R., Tevis, K., Reid, G., Short, K., … Riedener, J. (2012). Tactical checkpoint – hail/warn and suppress/stop. Journal of Directed Energy, 4, 255–274.

566

Miller, D. (2000). Introduction to collective behavior and collective actions. Prospect Heights, IL: Waveland Press.

Montgomery, D. (2012). 15.4 repeated measures. In D. Montgomery, Design and analysis of experiments (8th ed.). Hoboken, NJ: Wiley.

National Research Council. (2003). An assessment of non-lethal weapons science and technology. Washington, DC: The National Academies Press.

North Atlantic Treaty Organisation/Science & Technology Organisation. (2004). Non-lethal weapons effectiveness assessment (RTO TR-085). Brussels, Belgium: Author.

North Atlantic Treaty Organisation/Research & Technology Organisation. (2009). Non-lethal weapons effectiveness assessment development and verification study (RTO TR-SAS-060). Neuilly-sur-Seine Cedex, France: Author.

Office of Naval Research. (2014). Joint non-lethal weapons program fiscal year 2015 non-lethal weapons technologies (Broad Area Announcement ONRBAA14-008). Arlington, VA: Author.

Reicher, S. (2001). The psychology of crowd dynamics. In M. A. Hogg & R. S. Tindale, Group processes. Hoboken, NJ: Wiley Blackwell Publishers.

Reid, G., Cooke, G., DeMarco, R., Weaver, C., Riedener, J., & Mezzacappa, E. (2014, April). Mathematical capture of human data for computer model building and validation. In Proceedings of 23rd Annual Conference on Behavior Representation in Modeling and Simulation (BRiMS 2014), Washington, DC.

Riedener, J. (2007). Acoustic weaponization: Deterrence on target (Report No. TBRL-AW-TR-07003). Picatinny Arsenal, NJ: Target Behavioral Response Laboratory.

Riedener, J., & Mezzacappa, E. (2012, March). Data-driven modeling of human behavior in military operations. In Proceedings of the 21st Annual Behavior Representation in Modeling & Simulation (BRIMS) Conference, Amelia Island, FL.

Riedener, J., Short, K., Mezzacappa, E., Cooke, G., Sheridan, C., Jaffery, N., … Yagrich, K. (2007a). Acoustic weaponization: Deterrence on target low frequency (Report No. TBRL-AW-TR-07004). Picatinny Arsenal, NJ: Target Behavioral Response Laboratory.

Riedener, J., Short, K., Mezzacappa, E., Sheridan, C., Jaffery, N., Jones, R., & Yagrich, K. (2007b). Acoustic weaponization: Deterrence on target, contingency (Report No. TBRL-AW-TR-07005). Picatinny Arsenal, NJ: Target Behavioral Response Laboratory.

Schweingruber, D., & McPhail, C. (1999). A method for systematically observing and recording collective behavior. Sociological Methods and Research, 27(4), 451–498.

Short, K. B. (2006). Blunt impact as deterrent: Human approach-avoidance behaviors and other stress responses studied within a paintball gaming context. In G. T. Shwaery, J. G. Blitch, & C. Land (Eds.), Enabling technologies and design of nonlethal weapons (Vol. 6219), Proceedings of SPIE Optics + Photonics 2016 Conference & Exhibition (pp. 62190H1–12), San Diego, CA.

Short, K., Reid, G., Cooke, G., & Minor, T. R. (2010, November). Can repeated painful blunt impact deter approach toward a goal? Proceedings of the 27th Army Science Conference, Orlando, FL.

Short, K., Riedener, J., Mezzacappa, E., Sheridan, C., DeMarco, R., Tevis, K., …Yagrich, K. (2009). Convoy protection against aggressive acts. Picatinny Arsenal, NJ: Target Behavioral Response Laboratory.

567

Shuttleworth, A. (1998, December). Crowd control and the human effects of non-lethal weapons. Paper presented at Jane’s Non-Lethal Weapons Conference, London, UK.

Silver, M. (2002). Tactics, training, and procedures for the warfighter reacting to crowd dynamics (Report No. AFRL/HE/BR/TR-2002-0149). Brooks AFB, TX: Air Force Research Laboratory/Human Effectiveness Directorate.

Simonds, J. (2014). Non-lethal weapons human effects. Brooks AFB, TX: Human Effects Center of Excellence.

Tafolla, T. J., Trachtenberg, D. J., & Aho, J. A. (2012). From niche to necessity: Integrating nonlethal weapons into essential enabling capabilities. Joint Forces Quarterly, 66, 3rd Qtr, 71–79.

U.S. Army Center for Health Promotion and Preventive Medicine. (1999). Dosimetry and risk assessment (TG 181). Aberdeen Proving Ground, MD: Author.

U.S. Army Ordnance Corps. (1957). Ordnance engineering design handbook, artillery ammunition series (section 2), design for terminal effects (ORDP 20-245). Washington, DC: Author.

U.S. Military Academy. (1999). Mechanical design [ME402 class text]. West Point, NY: Department of Mechanical Engineering.

U.S. Naval Institute. (1910). Ordnance and gunnery: A text-book prepared for the use of the midshipmen of the United States Naval Academy. Annapolis, MD: Author.

VanMeenen, K. S. (2006). Suppression: Sound and light interference with targeting In G. T. Shwaery (Ed.), Enabling technologies and design of nonlethal weapons (Vol. 6219), Proceedings of SPIE Optics + Photonics 2016 Conference & Exhibition (pp. 62190J1-11), San Diego, CA.

Warner, C. (2013). Report on the test science roadmap. Washington, DC: Office of the Director, Operational Test and Evaluation.

568

Author Biographies

Dr. Elizabeth Mezzacappa is currently a sci-entist with the U.S. Army Armament Research, Development and Engineering Center Tactical Behavior Research Laboratory (ARDEC TBRL); she also serves as Adjunct faculty with the Army’s Armament Graduate School. Her research inter-ests include human dimensions of military operations. Dr. Mezzacappa holds a BA in Biology a n d Ps y c holo g y f r om t he Un iv er s it y of Pennsylvania; and a PhD in Medical Psychology from the Uniformed Services University of the Health Sciences.


Dr. Gordon Cooke is currently the laboratory chief of the ARDEC TBRL where he has served as a research engineer and principal investigator. His research interests include the performance of humans in military environment scenarios. He is a graduate of the U.S. Military Academy at West Point and former combat engineer officer. Dr. Cooke holds a PhD in Biomedical Engineering from Stevens Institute of Technology.


569

Mr. Robert M. DeMarco is currently a measure-ment and automation specialist who serves as a lead software developer for the ARDEC TBRL. He is a Certified LabVIEW Developer, with over 15 years of experience programming. He also serves as an adjunct professor in the New Jersey Institute of Technology, Biomedical Engineering Department, where he teaches virtual instru-mentation. He holds a BS in Computer Engineering and an MS in Biomedical Engineering from the New Jersey Institute of Technology.


Mr. Gladstone V. Reid is currently a biomedical instrumentation engineer supporting human behavioral research at the ARDEC TBRL. He recently served as laboratory chief of the ARDEC TBRL and has developed over 30 instrumented Test Beds over his career at the New Jersey Institute of Technolog y, the Department of Veterans Affairs, Rutgers University, and ARDEC TBRL. He holds a BS in Electrical Engineering and an MS in Biomedical Engineering from the New Jersey Institute of Technology.


570

Mr. Kevin Tevis is currently the laboratory chief of the Radiographic Laboratory, but has served in many diverse roles/positions over his 10-year career at ARDEC. From 2007 to 2014, Mr. Tevis performed both lead engineering efforts and principal investigator duties at the ARDEC TBRL. Mr. Tevis holds a master’s degree in Mechanical Engineering from Stevens Institute of Technology and a Master’s degree in Business Administration from Florida Institute of Technology.


Mr. Charles Sheridan is currently a research assistant at the ARDEC TBRL, serving as the link between the research staff and the human research participants. He has been responsible for the recruitment, scheduling, and administration of the informed consent process, subject protection, and general interaction with over a thousand subjects for the laboratory. He holds a BA in English from Fairleigh Dickinson University.


571

Dr. Kenneth R. Short is currently the Provost, U.S. Army Armament Graduate School and a scientist at the ARDEC TBRL. His research interests include behavioral and neurochemical consequences of stress, especially in traumatic or military contexts, and relation to subsequent anxiety. Dr. Short holds a BA in Psychology from Swarthmore College; and an MA in Experimental Psychology and PhD in Neuroscience from the University of Colorado.


Mr. Nasir Jaffery is currently a quality engineer for the ARDEC TBRL, serving as experiment test lead, assisting in behavioral data collection and analysis, and participating in institutional review board approval processes. He has contributed to several armament programs and International Organization for Standardization 9000 audits. He holds a BS in electrical engineering from the State University of New York at Buffalo and an MBA from the Florida Institute of Technology.


572

Mr. John B. Riedener is currently the FAST- 7ATC Science & Technolog y Advisor to the ARDEC Atlantic 7th Army Training Command in Grafenwoehr, Germany. In 2016, he was labo-ratory chief for the ARDEC TBRL, with oversight of dozens of human subjects research projects. He holds a BA in Computer and Information Science from the New Jersey Institute of Technology and an MS in Systems Engineering from the Stevens Institute of Technology.


573

T h e D e f e n s e A c q u i s i t i o n Professional Reading List is intended to enrich the knowledge and under-standing of the civilian, military, contractor, and industrial workforce who participate in the entire defense acquisition enterprise. These book recommendations a re desig ned to complement the education and training vital to developing essen-tial competencies and skills of the acquisition workforce. Each issue of the Defense Acquisition Research Journal will include one or more reviews of suggested books, with more available on our Website http://dau.mil/library.

We encourage our readers to submit book reviews they believe should be required reading for the defense acquisition professional. The books themselves should be in print or gen-erally available to a wide audience; address subjects and themes that have broad applicability to defense acquisition professionals; and pro-vide contex t for the reader, not prescriptive practices. Book reviews shou ld be 450 words or fewer, describe the book and its major ideas, and explain its relevancy to defense acquisition. Please send your reviews to the managing edi-tor, Defense Acquisition Research Journal at [email protected].


Featured Book Destructive Creation: American Business and the Winning of World War II Author: Mark R. WilsonPublisher: University of Pennsylvania PressCopyright Date: 2016Hardcover: 392 pages ISBN: 9780812248333Reviewed by: Dr. Benjamin Franklin Cooling, Professor of National Security Studies, The Eisenhower School, National Defense University


July 2017

Review:In some circles, a popularized dogma is that the real winners of World

War II were private business and industry. Just ask popularizer Art Her-man, Freedom’s Forge (2012), and now academic Mark Wilson. Wilson best invites criticism of all parties and concludes that self-serving corpo-rate memory won. The touch of anti-statism that permeates all America particularly enabled public relations to build a compelling alternate his-tory or dogma, approximating more the South’s Lost Cause interpreta-tion after the Civil War or Weimar Germany’s after World War I. Wilson has contributed previously to The Business of Civil War and knows how to ferret out details and synthesize analysis from published and unpub-lished sources. This present work tells a tale stretching from two world wars to the Cold War, with World War II as the high table for interpreting the private sector as savior of the Free World. He also suggests how the struggle between public and private sectors for heroes, villains, and “who won” has continued to shape economic and political development even today. Arguably, his most useful chapter is his last, styled “reconversion.” Here, Wilson builds into a tight discussion of privatizing resourcing in the early Cold War, thereby fulfilling and perpetuating America’s capital-ist dream. Yet it was World War II that “offers sobering lessons about the power of economic elites to shape American politics.”

Wilson’s narrative invites reflection on how “business and government were reluctant, contentious, and even bitter partners” in the war effort. In fact, Wilson shows World War II public-private partnering as a con-tinuation of battles between corporate and government, capital and la-bor, regulation and free enterprise as well as in-fighting between politi-cians, military, and private elite personalities and philosophies dating back to the Progressive Era. Wilson recounts the story from shadows of the Great War (World War I) that conditioned how the United States planned for the next one, to building the arsenal (even before Pearl Har-bor’s casus belli), through what Wilson styles “one tough customer” or the government’s exacting price constraints and tight regulation in the second conflagration while inducing product competition yet also pro-viding massive public investment to get the job done. Wilson also dips into unsavory wartime labor unrest—strikes and seizures that contradict the notion that all America put shoulders to the common weal in patri-otic unity. Wilson’s book is hard hitting, but balanced, detailed without being pedantic, and eminently stimulating.

Aside from what Wilson suggests World War II teaches us (maybe less



than he thinks), does his book yield anything useful for acquisition pro-fessionals today? No doubt the system of public-private partnering in World War II was made more robust by political statist philosophy, mar-ginal private weapons-production capability until underwritten by cen-tral government financing, and innovative infrastructure provisions. “Ar-senic and red tape” as one long-lost postwar account called it, or Bruce Catton’s classic War Lords of Washington (1948), reveals in more colorful prose the machinations of top officials (and those have not gone away). Moreover, the times dictated private capacity to augment, not replace, government in-house capabilities and basic innovative fiscal creation of warfighting, not merely deterrent capability. The choice of quantity over quality superimposed on the exigency of fighting a hot war was an industrial/mobilization age emergency measure (or “expedient corpo-ratism”) and reflected best, perhaps, by the penultimate public/private endeavor—the Manhattan Project. Wilson really wants us to focus on the continuum of liberal progressive-protectionist conservative friction; the battle of free enterprise versus state socialism is still with us. Only at the end does the reader realize the basic deference and dependency that befell the government’s national security/common defense respon-sibilities; by the time of Eisenhower, McNamara has reached its epitome today. Wilson’s story is not an explanation resting upon the evolving na-ture of war, technological change, or even the fundamental tautological difference between industrialized world wars and the atomic Cold War. The reader must do the work of extrapolation. Wilson’s book does stand at the threshold of better explaining the migration of the Arsenal of De-mocracy to the Military-Industrial Complex and National Security State.


New Research in DEFENSE ACQUISITION

Academics and practitioners from around the globe have long con-sidered defense acquisition as a subject for serious scholarly research, and have published their findings not only in books, but also as Doctoral dissertations, Master’s theses, and in peer-reviewed journals. Each issue of the Defense Acquisition Research Journal brings to the attention of the defense acquisition community a selection of current research that may prove of further interest.

These selections are curated by the Defense Acquisition University (DAU) Research Center and the Knowledge Repository. We present here only the author/title, abstract (where available), and a link to the resource. Both civil-ian government and military Defense Acquisition Workforce (DAW) readers will be able to access these resources on the DAU DAW Website: https://identity.dau.mil/EmpowerIDWebIdPForms/Login/KRsite. Nongovernment DAW readers should be able to use their local knowledge management cen-ters and libraries to download, borrow, or obtain copies. We regret that DAU cannot furnish downloads or copies.

We encourage our readers to submit suggestions for current research to be included in these notices. Please send the author/title, abstract (where avail-able), a link to the resource, and a short write-up explaining its relevance to defense acquisition to: Managing Editor, Defense Acquisition Research Journal, [email protected].


THE ECONOMICS OF ENTERPRISE TRANSFORMATION: AN ANALYSIS OF THE DEFENSE ACQUISITION SYSTEMMichael J. Pennock

Abstract: Despite nearly 50 years of attempts at reform, the U.S. defense acqui-

sition system continues to deliver weapon systems over budget, behind schedule, and with performance shortfalls. A parade of commissions, pan-els, and oversight organizations has studied and restudied the problems of government acquisition with the objective of transforming the defense acquisition enterprise, yet the resulting legislative and procedural changes have yielded little, if any, benefit. Thus, the obvious question is, “Why has acquisition reform failed?” Three potential contributors were identified in the literature: misalignment of incentives, a lack of a systems view, and a lack of objective evaluation criteria. This dissertation attempts to address each of these problem areas.

First, the author considers the issue of incentivization in the context of defense technology policy. A frequent criticism of defense acquisition pro-grams is that they tend to employ risky, immature technology that increases the cost and duration of acquisition efforts. To combat this problem, the



Department of Defense rewrote its acquisition regulations to encourage a more evolutionary approach to systems development. Nominally, this requires the use of mature technologies, but studies have revealed that acquisition programs continue to use immature technologies in spite of the new policies. To analyze this issue, the defense acquisition cycle was modeled as a stochastic process. Then, assuming that each acquisition program serves a diverse set of stakeholders, game theory was applied to show that the stable solution is to employ immature technology. It turns out that there is a tragedy of the commons at work in which the acquisition program serves as the common resource for each of the stakeholder groups to achieve its objectives. Since there is no cost to using the resource, there is a tendency to overexploit it. The result is an outcome that is worse than if there had been a coordinated solution. Thus, the rational actions of stake-holders will lead to a contradiction of acquisition policy. Consequently, if the Department of Defense expects adherence to its evolutionary acquisition policy, it must either strictly enforce technology maturity requirements or else realign incentives with desired outcomes. Second, the author evaluates cost and performance implications of the most recent defense acquisition transformation initiative—evolutionary acquisition. Proponents suggest that evolutionary acquisition will lower acquisition program costs, shorten delivery times, and improve the performance of fielded systems through the use of shorter and more incremental acquisition cycles. Supporting arguments focus on the impact of evolutionary acquisition on individual programs, but fail to consider the defense acquisition enterprise as a system.

To address this shortcoming, the author analyzes the impact of evolutionary policies through the use of a discrete event simulation of the entire defense acquisition system. It was found that while there should be an increase in the performance of fielded systems under evolutionary acquisition policies, the cost of operating the defense acquisition system as a whole does not inherently decrease. This is because the shorter acquisition cycles created by evolutionary polices mean that the overhead costs of each acquisition cycle are incurred more frequently. If these overhead costs do not decline sufficiently, the net cost to operate the acquisition system rises. This finding demonstrates the importance of considering the entire acquisition system before implementing a new policy. Finally, the author addresses the lack of objective evaluation criteria by developing a method to value acquisition process improvements monetarily. This is accomplished through the com-bination of price indices and options analysis.


July 2017

Since the U.S. government is a nonprofit entity, traditional cash flow-based valuation methods are not applicable. Instead, the use of price indices cap-tures the changes in the government’s buying power induced by acquisition reforms. This may be converted into an equivalent, augmented budget stream that allows traditional investment evaluation tools to be applied. An additional advantage of the buying power method is that it captures the impact of the economies of scale inherent in the production of military systems. The augmented budget stream serves as the basis for applying options analysis, which properly accounts for the risk-mitigating effects of staging. A comparison of this new method with more traditional methods reveals that only considering cost savings can significantly undervalue acquisition improvement opportunities, and even small improvements can have large returns.

Citation:Pennock, M. J. (2008). The economics of enterprise transformation: An analysis of the

defense acquisition system (Order No. 304645429). Available from ProQuest Dissertations & Theses Global. Retrieved from https://search.proquest.com/docview/304645429?accountid=40390

ISSUES WITH ACCESS TO ACQUISITION DATA AND INFORMATION IN THE DEPARTMENT OF DEFENSE: DOING DATA RIGHT IN WEAPON SYSTEM ACQUISITIONMegan McKernan, Nancy Young Moore, Kathryn Connor, Mary E. Chenoweth, Jeffrey A. Drezner, James Dryden, Clifford Grammich, Judith D. Mele, Walter T. Nelson, Rebeca Orrie, Douglas Shontz, & Anita Szafran

Abstract: Acquisition data and information are the foundation for decision mak-

ing, management, and oversight of weapon system acquisition programs. They are critical to initiatives to improve defense acquisition, such as Better Buying Power. The Department of Defense as a whole gathers a wide variety of acquisition information and stores it in multiple, sometimes incompatible systems, most of which are built for reporting, not analysis. Large busi-nesses have similar problems, and the concept of master data management may have lessons for both. The authors review 21 key acquisition-related

582582


data information systems and their origins and uses, and identify how acquisition data might be improved. They also summarize background on acquisition data; review commercial practices in data management; and offer findings and recommendations to further improve acquisition data quality, access, and use.

Citation:McKernan, M., Moore, N. Y., Connor, K., Chenoweth, M. E., Drezner, J. A., Dryden, J., …

Szafran, A. (2017). Issues with access to acquisition data and information in the department of defense: Doing data right in weapon system acquisition. Retrieved from: https://www.rand.org/pubs/research_reports/RR1534.html

A LOW DISHONEST DECADE ..: SMART ACQUISITION AND DEFENCE PROCUREMENT INTO THE NEW MILLENNIUMJohn Louth

Abstract: Smart acquisition was the change programme introduced at the end

of the twentieth century charged with transforming the effectiveness of defence procurement within the United Kingdom (UK). The initiative was rolled-out as a cornerstone of the Blair government’s strategic defence initiative from 1998 onwards, and today represents the management phi-losophy, public sector organisational structures, and UK industrial strategy for delivering defence equipment. This research seeks to understand the manner and extent of changes to defence procurement derived from the smart acquisition initiative, viewed as a ‘technology’ through which gov-ernment exercises power. Accordingly, understanding smart acquisition develops and deepens our knowledge of the nature of government itself. The author offers, in chapters 1 and 2 initially, an introduction to smart acquisition, its background, and historical antecedence. He then discusses the methodology employed for interrogating the phenomenon as an auto/ethnographical study of UK defence practices. Chapter 3 details the fac-tors that drove defence reorganisation, whilst chapter 4 derives smart acquisition as rational and benign managerial change. Chapter 5 critiques this perspective by unveiling smart acquisition as a neoliberal construct through which government procures and cements assemblages of regimes of control and socialisation, legitimised through managerial narratives and

583

governmentalist forms. Consequently, a revised critical analytical model of smart acquisition embracing governmentalist notions is provided in chapter 6. Chapter 7 introduces a specific defence procurement project team and describes its transformation strategy and emerging business model. In chapter 8, the project team is superficially revealed as a rational change agent embedding and embracing management reform. Chapter 9 critiques this, presenting the team as a constructed governmentalist regime, an expression of control, socialisation, and surrender of agency. Chapter 10 concludes the research by observing that smart acquisition is a complex set of understandings, and a multiplicity of forms and discourses.

Citation:Louth, J. (2010). A low dishonest decade ..: Smart acquisition and defence

procurement into the new millennium (Order No. 1780172723). Available from ProQuest Dissertations & Theses Global. Retrieved from https://search.proquest.com/docview/1780172723?accountid=40390

DELUSION AND DECEPTION IN LARGE INFRASTRUCTURE PROJECTS: TWO MODELS FOR EXPLAINING AND PREVENTING EXECUTIVE DISASTERBent Flyvbjerg, Massimo Garbuio, & Dan Lovallo

Abstract: “Over budget, over time, over and over again” appears to be an appropri-

ate slogan for large, complex infrastructure projects. This article explains why cost, benefits, and time forecasts for such projects are systematically overoptimistic in the planning phase. The underlying reasons for forecast-ing errors are grouped into three categories: delusions or honest mistakes, deceptions or strategic manipulation of information or processes, or bad luck. Delusion and deception have each been addressed in the management literature before, but here they are jointly considered for the first time. They are specifically applied to infrastructure problems in a manner that allows both academics and practitioners to understand and implement the suggested corrective procedures. The article provides a framework for analyzing the relative explanatory power of delusion and deception. It also suggests a simplified framework for analyzing the complex principal-agent relationships that are involved in the approval and construction of large infrastructure projects, which can be used to improve forecasts. Finally,


the article illustrates reference class forecasting, an outside view de-bias-ing technique that has proven successful in overcoming both delusion and deception in private and public investment decisions.

Citation:Flyvbjerg, B., Garbuio, M., & Lovallo, D. (2009, Winter). Delusion and deception

in large infrastructure projects: Two models for explaining and preventing executive disaster. Available from ProQuest Dissertations & Theses Global (Order No. 215864135). Retrieved from https://search.proquest.com/docview/215864135?accountid=40390.


Defense ARJ Guidelines FOR CONTRIBUTORS

The Defense Acquisition Research Journal (ARJ) is a scholarly peer-reviewed journal published by the Defense Acquisition University (DAU). All submissions receive a blind review to ensure impartial evaluation.

IN GENERALWe welcome submissions from anyone involved in the defense acqui-

sition process. Defense acquisition is defined as the conceptualization, initiation, design, development, testing, contracting, production, deploy-ment, logistics support, modification, and disposal of weapons and other systems, supplies, or services needed for a nation’s defense and security, or intended for use to support military missions.

Research involves the creation of new knowledge. This generally requires using material from primary sources, including program documents, policy papers, memoranda, surveys, interviews, etc. Articles are characterized by a systematic inquiry into a subject to discover/revise facts or theories with the possibility of influencing the development of acquisition policy and/or process.

We encourage prospective writers to coauthor, adding depth to manuscripts. It is recommended that a mentor be selected who has been previously published or has expertise in the manuscript’s subject. Authors should be familiar with the style and format of previous Defense ARJs and adhere to the use of endnotes versus footnotes (refrain from using electronically embedded endnotes), formatting of reference lists, and the use of designated style guides. It is also the responsibility of the corresponding author to furnish any required government agency/employer clearances with each submission.

587

MANUSCRIPTSManuscripts should reflect research of empirically supported experi-

ence in one or more of the areas of acquisition discussed above. The Defense ARJ is a scholarly research journal and as such does not publish position papers, essays or other writings not supported by research firmly based in empirical data. Empirical research findings are based on acquired knowl-edge and experience versus results founded on theory and belief. Critical characteristics of empirical research articles:

• Clearly state the question,

• define the research methodology,

• describe the research instruments (e.g., program documenta-tion, surveys, interviews),

• describe the limitations of the research (e.g., access to data, sample size),

• summarize protocols to protect human subjects (e.g., in sur-veys and interviews) , if applicable,

• ensure results are clearly described, both quantitatively and qualitatively,

• determine if results are generalizable to the defense acquisi-tion community,

588


• determine if the study can be replicated, and

• discuss suggestions for future research (if applicable).

Research articles may be published either in print and online, or as a Web-only version. Articles that are 5,000 words or less (excluding abstracts, references, and endnotes) will be considered for print as well as Web pub-lication. Articles between 5,000 and 10,000 words will be considered for Web-only publication, with an abstract (150 words or fewer) included in the print version of the Defense ARJ. In no case should article submissions exceed 10,000 words.

Book ReviewsDefense ARJ readers are encouraged to submit book reviews they believe

should be required reading for the defense acquisition professional. The reviews should be 500 words or fewer describing the book and its major ideas, and explaining why it is relevant to defense acquisition. In general, book reviews should reflect specific in-depth knowledge and understanding that is uniquely applicable to the acquisition and life cycle of large complex defense systems and services.

Audience and Writing StyleThe readers of the Defense ARJ are primarily practitioners within the

defense acquisition community. Authors should therefore strive to demonstrate, clearly and concisely, how their work affects this community. At the same time, do not take an overly scholarly approach in either content or language.

FormatPlease submit your manuscript with references in APA format (author-

date-page number form of citation) as outlined in the Publication Manual of the American Psychological Association (6th Edition). References should include Digital Object Identifier (DOI) numbers when available. The author(s) should not use automatic reference/bibliography fields in text or references as they can be error-prone. Any fields should be converted to static text before submission. For all other style questions, please refer to the Chicago Manual of Style (16th Edition).

Contributors are encouraged to seek the advice of a reference librarian in completing citation of government documents because standard formulas of citations may provide incomplete information in reference to government works. Helpful guidance is also available in The Complete Guide to Citing Government Information Resources: A Manual for Writers and Librarians (Garner & Smith, 1993), Bethesda, MD: Congressional Information Service.

589

July 2017

Pages should be double-spaced in Microsoft Word format, Times New Roman, 12-point font size and organized in the following order: title page (titles, 12 words or fewer), abstract (150 words or fewer to conform with formatting and layout requirements of the publication), two-sentence sum-mary, list of keywords (five words or fewer that do not appear in the title of the manuscript), reference list (only include works cited in the paper), and author’s note or acknowledgments (if applicable). Manuscripts submitted as PDFs will not be accepted.

Figures or tables should not be inserted or embedded into the text, but submitted as a separate file in the original software format in which it was created. For additional information on the preparation of figures or tables, refer to the Scientific Illustration Committee, 1988, Illustrating Science: Standards for Publication, Bethesda, MD: Council of Biology Editors, Inc. Restructure briefing charts and slides to look similar to those in previous issues of the Defense ARJ.

The author (or corresponding author in cases of multiple authors) should attach a signed cover letter to the manuscript that provides all of the authors’ names, mailing and e-mail addresses, as well as telephone and fax numbers. The letter should verify that the submission is an original product of the author(s); that all the named authors materially contributed to the research and writing of the paper; that the submission has not been previously published in another journal (monographs and conference pro-ceedings serve as exceptions to this policy and are eligible for consideration for publication in the Defense ARJ); and that it is not under consideration by another journal for publication.

COPYRIGHTThe Defense ARJ is a publication of the United States Government and

as such is not copyrighted. Because the Defense ARJ is posted as a complete document on the DAU Website, we will not accept copyrighted manuscripts that require special posting requirements or restrictions. If we do publish your copyrighted article, we will print only the usual caveats. The work of federal employees undertaken as part of their official duties is not subject to copyright except in rare cases.

Web-only publications will be held to the same high standards and scru-tiny as articles that appear in the printed version of the journal and will be posted to the DAU Website at www.dau.mil.

590


In citing the work of others, please be precise when following the author-date-page number format. It is the contributor’s responsibility to obtain permission from a copyright holder if the proposed use exceeds the fair use provisions of the law (see the latest edition of Circular 92: Copyright Law of the United States of America and Related Laws Contained in Title 17 of the United States Code, Washington, DC: U.S. Government Printing Office). Contributors will be required to submit a copy of the writer’s permission to the managing editor before publication.

We reserve the right to decline any article that fails to meet the following copyright requirements:

• The author cannot obtain permission to use previously copy-righted material (e.g., graphs or illustrations) in the article.

• The author will not allow DAU to post the article in our Defense ARJ issue on our Internet homepage.

• The author requires that usual copyright notices be posted with the article.

• To publish the article requires copyright payment by the DAU Press.

SUBMISSIONAll manuscript submissions should include the following:

• Cover letter

• Author checklist

• Biographical sketch for each author (70 words or less)

• Headshot for each author saved as a 300 dpi (dots per inch) JPEG or Tiff file no less than 5x7 inches with a plain back-ground in business dress for men (shirt, tie, and jacket) and business appropriate attire for women. All active duty military should submit headshots in Class A uniforms. Please note: low-resolution images from Web, PowerPoint, or Word will not be accepted due to low image quality.

• One copy of the typed manuscript, including:

° Title (12 words or less)

591

° Abstract of article (150 words or less)

° Two-line summary

° Keywords (5 words or fewer—do not include words appear-ing in the manuscript title)

° Document double-spaced in Microsoft Word format, Times New Roman, 12-point font size (5,000 words or less for the printed edition and 10,000 words or fewer for online-only content excluding abstracts, figures, tables, and references).

° Copyright release form

A ll forms a re ava ilable at our website: w w w.dau.mil/libra r y/a rj. Submissions should be sent electronically, as appropriately labeled files, to the Defense ARJ managing editor at: [email protected].

Defense ARJ PRINT SCHEDULE

The Defense ARJ is published in quarterly theme editions. All submis-sions are due by the first day of the month. See print schedule below.

Author Deadline Issue

July January

November April

January July

April October

In most cases, the author will be notified that the submission has been received within 48 hours of its arrival. Following an initial review, submis-sions will be referred to peer reviewers and for subsequent consideration by the Executive Editor, Defense ARJ.


Contributors may direct their questions to the Managing Editor, Defense ARJ, at the address shown below, or by calling 703-805-3801 (fax: 703-805-2917), or via the Internet at [email protected].

The DAU Homepage can be accessed at: http://www.dau.mil

DEPARTMENT OF DEFENSE

DEFENSE ACQUISITION UNIVERSITY

ATTN: DAU PRESS (Defense ARJ )

9820 BELVOIR RD STE 3

FORT BELVOIR, VA 22060-5565

593

January

1

CALL FOR AUTHORSWe are currently soliciting articles and subject mat-ter experts for the 2017 Defense Acquisition Research Journal (ARJ) print year. Please see our guidelines for contributors for submission deadlines.

Even if your agency does not require you to publish, consider these career-enhancing possibilities:

• Share your acquisition research results with the Acquisition, Technology, and Logistics (AT&L) community.

• Change the way Department of Defense (DoD) does business.• Help others avoid pitfalls with lessons learned or best practices from your project or

program.• Teach others with a step-by-step tutorial on a process or approach.• Share new information that your program has uncovered or discovered through the

implementation of new initiatives.• Condense your graduate project into something beneficial to acquisition professionals.

ENJOY THESE BENEFITS:• Earn 25 continuous learning points for

publishing in a refereed journal.• Earn a promotion or an award.• Become part of a focus group sharing

similar interests.• Become a nationally recognized expert

in your field or specialty.• Be asked to speak at a conference

or symposium.

We welcome submissions from anyone in-volved with or interested in the defense ac-quisition process—the conceptualization, initiation, design, testing, contracting, pro-duction, deployment, logistics support, mod-ification, and disposal of weapons and other systems, supplies, or services (including con-struction) needed by the DoD, or intended for use to support military missions.

If you are interested, contact the Defense ARJ managing editor ([email protected]) and provide contact information and a brief description of your article. Please visit the Defense ARJ Guidelines for Contributors at http://www.dau.mil/library/arj/p/ARJ-Guidelines.

Defense ARJ and Defense AT&LOnline-only for individual subscribers

NEWOnline presence for easier use on mobile and desktop devices:https://www.dau.mil/library/defense-atlhttps://www.dau.mil/library/arj

PLEASE SUBSCRIBEor resubscribe so you will not miss out on

accessing future publications.

Send an e-mail to [email protected] and/or [email protected], giving the e-mail address you want

us to use to notify you when a new issue is posted.

Type “Add to LISTSERV” in the subject line.

Also use this address to notify us if you change your e-mail address

S U R V E Y

Please rate this publication based on the following scores:

5 —Exceptional 4 — Great 3 — Good 2 — Fair 1 — Poor

Please circle the appropriate response.

1. How would you rate the overall publication? 5 4 3 2 1

2. How would you rate the design of the publication? 5 4 3 2 1

True Falsea) This publication is easy to readb) This publication is useful to my careerc) This publication contributes to my job effectivenessd) I read most of this publicatione) I recommend this publication to others in the acquisition field

If hand written, please write legibly.

3. What topics would you like to see get more coverage in future Defense ARJs?

4. What topics would you like to see get less coverage in future Defense ARJs?

5. Provide any constructive criticism to help us to improve this publication:

6. Please provide e-mail address for follow up (optional):

Defense ARJ

New Online Subscription

Defense AT&L

Cancellation

Change E-mail Address

Last Name:

First Name:

Day/Work Phone:

E-mail Address:

Signature: (Required)

Date

ver 01/03/2017

PLEASE FAX TO: 703-805-2917

The Privacy Act and Freedom of Information Act In accordance with the Privacy Act and Freedom of Information Act, we will only contact you regarding your Defense ARJ and Defense AT&L subscriptions. If you provide us with your business e-mail address, you may become part of a mailing list we are required to provide to other agencies who request the lists as public information. If you prefer not to be part of these lists, please use your personal e-mail address.

FREE ONLINES U B S C R I P T I O N

S U B S C R I P T I O N

Thank you for your interest in Defense Acquisition Research Journal and Defense AT&L magazine. To receive your complimentary online subscription, please write legibly if hand written and answer all questions below—incomplete forms cannot be processed.

*When registering, please do not include your rank, grade, service, or other personal identifiers.

Articles represent the views of the authors and do not necessarily reflect the opinion of DAU or the Department of Defense.

We’re on the Web at: http://www.dau.mil/library/arj

Current. Connected. Innovative.