Observations on the Theoretical Basis of Cost-Effectiveness

Observations on the Theoretical Basis of Cost-EffectivenessAuthor(s): M. C. Heuston and G. OgawaSource: Operations Research, Vol. 14, No. 2 (Mar. - Apr., 1966), pp. 242-266Published by: INFORMSStable URL: http://www.jstor.org/stable/168253 .

Accessed: 09/05/2014 19:31

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

INFORMS is collaborating with JSTOR to digitize, preserve and extend access to Operations Research.

http://www.jstor.org

This content downloaded from 195.78.109.69 on Fri, 9 May 2014 19:31:40 PMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=informs

http://www.jstor.org/stable/168253?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


OBSERVATIONS ON THE THEORETICAL BASIS

OF COST-EFFECTIVENESS*

M. C. Heuston and G. Ogawa

TRW Systems, Redondo Beach, California

(Received June 14, 1965)

This paper presents some observations on the theoretical foundation of cost-effectiveness analysis. It describes the results of continuing research to develop a comprehensive and rigorous description of the important elements of cost-effectiveness as used by the aerospace industry for military and commercial systems planning. The primary objective is to utilize basic mathematical and statistical theory to construct the rules, properties, and hypotheses that are needed to satisfy the contractual requirements imposed by various government customers.

T HIS paper presents some observations on the fundamental concepts of the theory of cost-effectiveness analysis. These observations are the

results of continuing research aimed at developing a theoretical foundation suitable for application by engineers and operations analysts to industrial planning and control problems. The realm of practical interest lies in those decision problems arising out of the selection of (1) a preferred mission, (2) a preferred system, and (3) an optimum equipment design. The systems of interest run the gamut of most major missioll equipment and types of subsystems, but in particular those involved in military weapons and future generation space vehicles.

Cost-effectiveness is an interdisciplinary subject. It is complex in its more sophisticated theories and at the same time disarmingly simple in its basic concept. Its application is as extensive and different as its use for force structure planning by the Department of Defense and for the planning of manufacturing processes by value engineers. Its interdisciplinary approach stems from the contributions of well established theories of engineering, economics, and mathematics.

The following observations represent an examination of the important elements of cost-effectiveness and a synthesis of a variety of existing concepts into a composite theory that will be more useful to industrial planners. Suggestions are made concerning the nature of cost and effectiveness models and the properties those models should satisfy. In the case of the effec-

* This paper was presented at the joint meeting of the Western Section of the Operations Research Society of America and The Institute of Management Science in Los Angeles, California, April 29, 1965.

242



Theoretical Basis of Cost-Effectiveness 243

tiveness model, an example is presented of the use of a type of mathematical theory for the identification of the functional form of the effectiveness curve that satisfies these properties.

The following sections are organized into several parts. The second section brings to the reader's attention several documents that provide ideas and requirements for cost-effectiveness. The third section presents the general foundation for the concept, includinig a general definition and description of essential elements. The next section describes the fundamental concept of the cost model. The fifth section describes some of the theoretical elements of the effectiveness model, and the final section provides some concluding remarks and some ideas about cost-effectiveness criteria.

GENERAL BACKGROUND

NOT TOO many years ago cost-effectiveness was a term used in a formal way by a relatively few individuals, primarily economists and system analysts within the Department of Defense and its supporting contractors. Since McNamara's term as Secretary of Defense, it has been used in an ever- increasing circle by individuals in both government and private industry. The term, 'cost-effectiveness' is now applied for all phases of system and component design, development, production, and operation. It is used now not only for long-range planning but also for short-range cost control of the smallest of component hardware.

Currently, the emphasis by the Department of Defense (DOD) on the Preliminary Definition Phase has provided the major stimulant for eost- effectiveness analyses. However, DOD interest in cost-effectiveness ex- tends to other phases as well. It may be emphasized that all phases from the conceptual to the operational present planning problems that are amenable to the approach and techniques of cost-effectiveness.

There are many documents that have been published by both government and contractors on cost-effectiveness. Most deal with applications to specific systems and can be studied only as parts of larger system evaluations. There are others, however, that present the concept, techniques, and theory, which individuals who practice or manage cost-effectiveness studies will find useful. The titles of five important documents by DOD or published under direct sponsorship by DOD are shown in references 1 through 5. Two books, one by HITCH AND MCKEEN,[6] and the other by PECK AND SCHERER,[7] will provide excellent orientation. In addition, there are several papers and reports by contractors that should be in any bibliography and are listed in references 8 through 17. Other pertinent documents are mentioned throughout this paper. For a differently oriented bibliography, including reports dealing with special applications and practical estimating problems, see the bibliography in reference 4.



244 M. C. Heuston and G. Ogawa

THE CONCEPTUAL FOUNDATION

THIS SECTION summarizes the broad concept of cost-effectiveness in terms of its definition, tasks, and factors. The applicable theories from contribut- ing formal disciplines are mentioned in the appropriate following sections. A more extensive discussion of the concept as it is used by the Air Force and Air Force contractors can be obtained in references 3 through 5.

The development of most theories begins with a definition of the subject. Likewise, a theory of cost-effectiveness must start with a simple but comprehensive definition. One that has already proved useful may be expressed as follows: Cost-effectiveness analysis is an analytical technique for evaluating the broad management and economic implications of alternative choices of action with the objective of assisting in the identification of the preferred choice.

The words used in this definition are commoin, but some of them have a special connotation in reference to the broad range of management planning and control problem now utilizing this analysis. For example, 'analytical technique' means a procedure or procedures utilizing a logical sequence of repeatable and verifiable steps that separate and examine the parts and interrelations of the performance and costs of systems, subsystems, components, and processes. As another example, 'evaluating' means to ascer- tain the value or measure in quantifiable terms. 'Broad management and economic implications' means those grounds for inference that might be drawn from the effect upon the financial budget, profit return, market posi- tion, organizational structure, manpower, schedules, over-all performance vis-a-vis the enemy, competitor or environment, and limited resources of one kind or another. 'Alternative courses of action' means those proposed plans (e.g., competitive military missions, alternative equipment configura- tions, facility designs, maintenance policies, labor utilization schedules, production schedules, and operating rates) that offer the decision maker (often the customer of the cost-effectiveness study) an opportunity to select one that best satisfies some criterion.

The specific method by which the evaluation is handled rests in a theoretical sense on (1) the nature of the criteria, (2) the nature of the decisions or choices of action to be made, (3) in the degree of uncertainty inherent in the alternatives evaluated, and (4) in the internal structure or composition of the alternatives. In a practical sense, the techniques and sequence of steps rest on the time schedule and financial budget allowed for the study. A complete description of these conditions is beyond the scope of this short paper. The expansion of the application of cost-effectiveness to many levels of decision problems and types of systems not originally conceived as a part of the traditional total military systems makes it extremely difficult to discuss these in the abstract.




The evaluation process, that is, the procedures by which the alternatives are compared and the preferred alternative is determined, identified, and selected, takes on many forms. In general, however, one of the simple and more common evaluations concerns the comparative analysis of alternative total systems. Many examples can be given, such as, the selection of the preferred system from (1) a force of Minuteman, Polaris, and some future ballistic missile, (2) a force of a future ballistic missile with solid fuel or liquid, and (3) a force of a future missile operating from alternative basing concepts. The usual criterion involves the selection of the least cost alternative, given that all alternatives evaluated have been designed to achieve the same mission performance (e.g., the same expected number of megatons delivered successfully by the force under a predicted enemy threat). The alternative system that is estimated to cost the least is identified as the preferred system.

Another common evaluation, particularly in preliminary design, concerns an analysis of the trade-off between design or program characteristics (e.g., reliability of booster and spacecraft versus reliability of spacecraft experimental equipment; total vehicle reliability versus delivery time) and the selection of the best performance vehicle for the money available or the price to be charged. More explicitly the task involves the search for the maximum performance (i.e., effectiveness) subject to a budget constraint.

A third type of evaluation particularly common in the design of spacecraft concerns the comparison of several designs each of which has a somewhat different primary and/or secondary performance capability. It is often impossible or impractical for design engineers to design several spacecraft that can be considered as alternatives with identical capabilities. The evaluation usually consists of dividing the appropriate cost of each design by some single measure of its effectiveness or capability, or vice versa, and selecting that alternative design that shows the best ratio. The determination of a measurement of effectiveness is especially difficult for those cases where the spacecraft is a carrier for numerous experimental equipments of unequal priority, unequal payoff to the customer, different types of outputs, and different reliabilities.

The mathematical relation between cost and effectiveness has been formulated and described by many authors for many specific evaluation problems and for many specific systems. Very few have attempted to present a general formulization that might be applicable to all kinds of problems. One valuable exception is the concept presented by Fox, where among other things, he derives cost-effectiveness schedules where cost or effectiveness are random variables.["8] An approach where effectiveness is defined as 'reliability' has been applied to missiles and particularly to the growth of their reliability over time. This relation and the problems of




determining the relation was performed as a NASA study contract. '91 A different approach can be found in a paper by BROOKS. [201 A mathematical basis for the formulation of a 'figure of inerit' in design optimization problems is proposed in his paper.

For most of these types of evaluations and approach, it may be stated that the common major tasks of all analyses are as follows: (1) Problem Definition, (2) Development of Appropriate Arrays, (3) Definition of Alternatives, (4) Development of a Cost Model, (5) Development of an Effectiveness Model, and (6) Final Synthesis. These represent the basic sequence of steps for contractor performance for most Conieeptional and Preliminary Definrition Phase Studies. For a slightly different version, particularly oriented toward Air Force weapon systems, see reference 4.

The first three steps essentially deal with the scope and nature of a cost- effectiveness study. Their character embodies problems both in management of the study and the setting up of the analysis from the conceptual and work schedule point of view. The Problem Definition step includes several tasks. The first usually comprises defining the primary and secondary missions of the system under study. In many cases, this means converting the broad definition of the mission requirement (which may be in narrow engineering terminology) into the context of a total operational program. The remaining tasks usually comprise (1) defining the criteria for selection, (2) scheduling analytical engineerinig tasks for the study, and (3) budgeting available study hours and costs.

The second major step, Development of Appropriate Arrays, includes, (1) the development of an array (sometimes called a matrix) that explicitly classifies all appropriate functional accounting elements, and (2) the development of a similar array structuring the effectiveness determinants and interrelations by subsystem and function. The first step, incidentally, is an essential preparatory step to the development of a cost model in step four. The Definition of Alternatives includes the development of descrip- tions of the major characteristics of each alternative plan in sufficient engineering detail to serve as inputs for the estimation of costs and the measurement of performance.

The next three steps are analytical (from the statistical point of view) and computational in character. The Development of the Cost Model includes (1) the collection of appropriate empirical cost and characteristic data, (2) the development of statistical relations based on these empirical data, (3) development of the structure of the model, and (4) the incorpora- tion of the empirical relations. This task is described in more detail in the next section. The Development of the Effectiveness Model includes (1) the examination of subsystem reliability and other parametric performance characteristics, and (2) development of the structure of the model. This




task is also described in greater detail later in this document. The Final Synthesis step includes (1) the estinmation of costs and effectiveness imeas- urements based on given inputs, (2) the identification of critical parameters, (3) the trade-off of costs and effectiveness, (4) the perturbation of uncertain parameters as a part of a sensitivity analysis, and (5) preparation of the final report.

The basic factors of any cost-effectiveness analysis that can be broadly classified irrespective of the type of study or specific hardware fall under two major headings; i.e., System Factors and Progranm Factors. The System Factors refer to characteristics of units of hardware and software, while Program Factors refer to the over-all employment of these units over some period of operation. These categories are subdivided further into static and dynamic.

The purpose for listing 'factors' is to provide a means of visualizing possible independent variables that determine the estimated costs and effectiveness. There are several methods of classifying these factors. They could be divided according to the types of specific hardware involved in the study (i.e., B-52, lMtinuteman, MORL, Satellite Surveillance, Reentry Vehicle, etc.). They could also be segregated according to the following categories: (1) type of use of the study (i.e., long-range planning for Pre- PDP or PDP or short-range planning for Value Engineering), and (2) type of study (i.e., total system, subsystem, or component).

In the following list, only general examples of possible variables are provided for each category.

A. System Characteristics 1. Hardware

(a) Static Characteristics, which describe basic engineering features or design parameters, such as weight, length, diameter, speed, and thrust, of all mission and support equipment.

(b) Dynamic Characteristics, which describe performance under certain conditions of stress and nonstress, such as malfunction frequency or reliability, survivability, launch rate, flying hours, of all mission and support equipment.

2. Facilities (a) Static Characteristics, which describe square footage, depth, acreage,

etc. (b) Dynamic Characteristics, which describe survivability, maintain-

ability, etc. 3. Software

(a) Static Characteristics, which describe such items as the number of people required to operate, maintain, and support the system, the logistic support, and training requirements.

(b) Dynamic Characteristics, which describe the utilization of people




(i.e., man-hour limits and shift schedules), survivability, performance under stress, etc.

B. Program Characteristics 1. Number of hardware items required for test, training, etc., to perform

the mission over a period of time. 2. Delivery Schedules for Research, Development, Test and Evaluation

and Production, Activation of Operational Units, and Production Rate. 3. Organizational Structure. 4. Period of Time for Development and Production.

Conceptually, in summary, cost-effectiveness analysis must comprise the following: (1) alternatives to evaluate, (2) a criterion for judging which alternative is best, and (3) suitable parametric models for predicting the costs and effectiveness of each alternative.

THE COST MODEL

THE THEORY of the cost model in cost-effectiveness rests basically on two aspects. One concerns the matter of deciding what is to be costed and charged to the system. This is essentially a task of 'listing' and 'defining' the appropriate categories of total system costs. The other concerns the technique of cost 'prediction.' A proper approach to both of these matters is critical to the development of a cost model for any practical study. It must be emphasized that a good cost model is the only way to provide valid and credible estimates with a high degree of confidence for cost-effectiveness analyses.

The decision about an appropriate expense or cost of the system under study is more difficult than individuals unfamiliar with cost-effectiveness analysis are at first willing to concede. The results of the analysis can be easily biased by the exclusion and inclusion of inappropriate cost categories. Certain types of costs (e.g., 'sunk' costs, 'incremental' costs, and 'discounted' costs) if incorrectly handled, may also invalidate a good cost model.

The exclusion of costs is a common error of industrial contractors en- gaged in cost-effectiveness analyses. In their defense, it should be said that it is difficult if not impossible for many of them to estimate costs that may be incurred by the system which are beyond their immediate contractual obligation. Nevertheless, these costs are essential to proper evaluation of the alternatives in terms of the customer's decision and of the ultimate use of the proposed system.

The inclusion or exclusion of costs for cost-effectiveness analyses is often a matter of judgment. The rule, if there is one at all, for defining the mean- ing of 'total system costs,' might be stated as follows: 'include estimates of the costs of all economic resources that will be expended by the customer at




large (e.g., DOD, NASA; USAF, USA, and USN) in order to fulfill the mission of the system.' In some cases, total national resources irrespective of the budget structure of the customer are appropriate. Usually, the summation of all costs as derived via the cost model reflects the total monetary, time-phased impact of the systems on the customer's budget. Special difficulties arise in deciding what is appropriate and what time scale is appropriate when the customer is at some lower organizational echelon. The nature of the study, the criteria used for final selection of the preferred alternative, and the use to which the customer is going to put the study all have a bearing in a practical way on how the scope of the cost model is defined. No simple rule can adequately tell a practitioner how to weigh these considerations.

The general scope of the model can be described in a systematic manner in terms of a standard array sometimes called a cost matrix or a 'chart of accounts.' This array for computational and analytical convenience classifies all general cost elements horizontally and system elements vertically. In the case of practical study, the array that may be developed could be very complex and include the terminology of the specific hardware subsystems and accounting elements appropriate to the contractor and the customer. However, for the purposes of a general theory for this paper, the array must be as simple as possible. In summary form, it appears as in Table I. The "X's" indicate the accounts for which cost estimates will usually be computed. Table II is a continuation of Table I.

There are, of course, many classification schemes in existence, most of which reflect acceptable concepts of total system costs. It is important, whatever scheme is used, that it serve (1) as an adequate check list to avoid serious omissions and double countings and (2) as an adequate device for identifying critical and sensitive cost parameters as a part of the analytical portion of the cost-effectiveness analysis.

A brief definition of the vertical categories (i.e., system elements) in the array is given in the following list.

(a) Total System includes all equipment, consumables, personnel, procedures, and facilities required to accomplish a mission over time.

(b) Prime Mission Equipment includes all major equipment required to perform the primary mission, which may be airborne vehicle equipment for military aircraft and missile systems.

(c) Associated Support Equipment includes all equipment used in support of the primary mission, which may be aerospace ground equipment (including operational ground equipment and mechanical ground equipment) for military aircraft and missile systems.

(d) Real Property Installed Equipment includes items on nonfixed facilities, such as water control, electrical systems, sewage, and heating.




(e) Prime Mission Personnel includes all personnel associated with the primary missions, which may be air crew and ground crews for military aircraft and launch crews for military missile systems and space systems.

(f) Associated Support Personnel includes personnel providing direct and indirect support to PMP, which may be maintenance, supply, administration, and base support personnel for military aircraft and missile systems.

The horizontal or cost elements (which include labor, material, over- head, and fee) are as follows:

(a) Total Cost includes all costs incurred in performance of the mission from concept through deactivation, including salvage.

(b) Research, Development, Test, and Evaluation (RDT&E) includes cost of developing the system elements to the point where they are ready for regular production. For example, the subcategories may include design, test, program management, test vehicle fabrication, engineering, and quality control.

(c) Initial Investment includes costs of producing or acquiring the system elements. The subcategories may include fabrication of complete items and spares, tooling, sustaining engineering, technical data, contractor training, operational site acquisition and construction, and others.

(d) Operations includes the pay of operational personnel, the cost of maintain- ing the hardware and replacing it, and several other subcategories.

TABLE I

THE GENERAL COST ARRAY RESEARCH, DEVELOPMENT, TEST, AND EVALUATION COSTS

(Illustrative Example)

Cost elements

System elements Grand RDT & E

Total Total Design Test Other

Total System CT CD -

Hardware CH CDLH -

Prime mission equipment (b) - x x x x Prime mission equipment spares - x x x x Associated support equipment (c) x x x x Associated support equipment - x x x x

spares Real property installed equipment x x x x

(d) Other x x x x

Software CS CDS Prime mission personnel (e) _ x x x Associated support personnel (f) . x x x




The general cost matrix may be described in the following manner. The first row of aggregations (horizontal sumimations) can be written:

CTr CD+CP+CO, (1)

where

CT-grand total cost for a proposed plan of action, CD= total costs incurred for RDT & E,

Cp=-total costs incurred for initial investment, Co= total costs incurred for operations over a period of time.

Also, for vertical summation:

CT-E=C+CS, (2)

where

CE-grand total cost for the hardware subsystems, Cs= grand total cost for the software subsystems.

TABLE II

Tim, GENERAL COST ARRAY INITIAL INVESTMENT AND OPERATIONS COSTS


Cost elements

System elements Initial investment Operations

Total Fatbrinca- Other Total Pay and Mainte- Other tion ~~allowance nance

Total system CP co -

Hardware Cp - - Co l -

Prime mission equipment x x x x - x x (b)

Prime mission equipment x x x x - x x spares

Associated support equip- x x x x x x ment (c)

Associated support equip- x x x x _ x x ment spares

Real property installed x x x x - x x equipment (d)

Other x x x x _ x x Software Cps x x Cos x x x

Prime mission personnel (e) x x x x x x Associated support per- x x x x x x

sonnel (f)




For many studies, each vertical element can be divided by year. The final computational product of the model permits an examination of 'stream' of expenditures over a period of time.

In general, any lettered total or subtotal (cl), may be represented as the summation of its subordinate variables over all appropriate i's and j's:

c, = 7i Ej cij; (i=1, 2, *. j,n) 3 (j= 1, 2, *.. ) 3m)

where ci= costs for some ith cost subelement and the jth subsystem subelement.

The estimate of the cost of the subelements, cij, can be expressed as follows:

cii==F(xi, X2, ...*, Xk; aO, al, ..., Ck),

where x1, ... , xk are observable numerical parameters that describe the hardware and program and where the aj are parameters to be estimated. For a general academic discussion, see reference 36.

The cij may be either linear or nonlinear functions in the aq. A linear model typically has the form,

c= ao+aix+* * +ak Xk+E, (4)

where e denotes the totality of all errors between c and the linear model for c.

There are several methods of obtaining good estimations of the a1 in linear models, and there are many properties under consideration when an estimation is classified as being good. These methods include minimum mean square regression, 231 minimum variance regression, [28] ordinary least squares, [24,25,26] weighted least squares, [27] and maximum likelihood. [28]

The method used in a particular case will depend on the model chosen, the assumptions on the et (independent or correlated), the assumptions on the xi (random or nonrandom), and the available computational methods.

More generally, whenever a cij is nonlinear in the a1, then iterative techniques of nonlinear least squares may be used to minimize

Q- .1 [ci-F(xii, X2V, . . .*, Ski; ao, ai, .. * , ak)] (5)

and obtain estimates of the ax . Such techniques are from time to time explained in professional journals in statistics, numerical analysis, or computational techniques. As an example, SPECKMAN AND CORNELL[291 discuss estimation in a one-parameter exponential model and give references to other work.

The cost model represents the integration of the individual subelement regression equations into a complex computational scheme. In the prac-




tical world-particularly for (1) large numbers of elements, (2) large numbers of input system characteristic data, and (3) large numbers of alternatives to be costed-this scheme is usually programmed for electronic computer operation. In the simpler types of studies, however, manual operation may be sufficient.

THE GENERAL EFFECTIVENESS MODEL

THE CONCEPT of effectiveness describes the over-all performance of the system, subsystem, or component being studied in a cost-effectiveness analysis. For military systems, this concept includes degradation of normal peace- time performance for conditions imposed by the enemy threat, or, more specifically, military combat. For commercial systems, particularly space systems, the concept of effectiveness deals primarily with the reliability of the hardware under the natural physical environment in which the systems must operate over a period of time.

The effectiveness model in cost-effectiveness analysis serves as a tool in parametric form to facilitate the systematic estimation of a single quantitative measurement for each of all alternative systems or plans under study. Also because of its parametric form, the model serves as a convenient means by which the 'sensitivity' to changes in the values of the parameters can be evaluated.

The procedures for estimating this single measure are very difficult for certain types of systems. For military combat systems with a single mission (e.g., Minuteman anid the B-52), it is relatively straight forward to measure effectiveness in terms of the capability of alternative systems to deliver weapons against enemy targets in the face of enemy attacks. On the other hand, for systems that have multiple anid rather obscure and intangible mission objectives, the estimating process is much more difficult. The estimation of effectiveness for many military and commercial space systems, for example, is complicated by (1) joint missions, many of which have no tangible output suitable for quantitative measurement and (2) unique characteristics, some of which represent advanced scientific experimental components with little or no reliability data.

Like the cost model, the effectiveness model can be described in terms of its elements and the methods for predicting the numerical value of those elements. The kinds of detailed elements that can be used are as numerous as there are kinds of systems and kinds of cost-effectiveness problems. The basic elements by necessity must over-simplify a complex and very diverse real world. All that is required of these elements is that they usefully ex- press the basic analytical relation between the elements and the performance, or more specifically, the capability of the system to achieve the desired output.




Elements that have been suggested by other authors for effectiveness include (1) Availability, (2) Survivability, (3) Reliability, (4) Penetrability, and (5) Lethality. These were conceived to be applicable specifically to military aircraft and missile systems. Although useful for many studies, they are not entirely applicable to the broad range of cost-effectiveness analyses of military and commercial spaceeraft that are currently of im- portance to industrial contractors.

For spacecraft, special elements are required for the effectiveness model. 2 ' Two categories, Availability and Performance, seem suitable for very simple cases. The latter should be interpreted to encompass Reli- ability, Survivability (to action by the enemy or competitors, regardless of where the action is consummated), and, if necessary, other related quantifiable conditions affecting successful operation of the hardware, such as Penetrability and Lethality, if the hardware is military. For cost-effectiveness studies dealing with commercial systems, subsystems, and components in the developmental phase, Availability and Reliability are commonly used to reflect ultimate performance capability. For definitions of availability and reliability, see references 30 and 31. For application to spacecraft, see reference 32.

With reference to availability, it should be pointed out that it can be defined to include the probability of timely delivery to the customer of developmental or production systems. In some cost-effectiveness studies, the alternatives are not all equally available at a given point in time. It may be infeasible for prospective manufacturers, for example, to deliver all alternatives to meet the same total fabrication and test schedules, even if the cost for accelerated schedules is immaterial. A three-shift work schedule may not solve a schedule problem if, for example, there are certain physical limitations in material supplies.

The use of these concepts in a matrix type array has advanitages for showing the interrelations of all the effectiveness elements and total system elements. A general effectiveness array is suggested in Tables III, IV, and V. Each table categorizes major effectiveness elements horizontally and system elements vertically. The system elements are the same subelements as used in the vertical stub of the cost array. However, unlike the cost array, neither horizontal nor vertical subelements (i.e., within Development, Production, and Performance) are additive for computational purposes. Within the major elements, the subelements are generally treated as multiplicative. For many types of spacecraft studies, each individual subelement can be expressed as a probability of successful operation.

Algebraically, the basic structure of an effectiveness model may be written for simpler cases as follows:

?D,P,O= U j7 TjL- eij (6)




where ED, P The composite effectiveness of a system, subsystem, or com-

ponent for either Development, Production, or Operations. The partition of total system into three categories of Develop- ment, Production, and Operation represents an attempt to reflect varying ultimate goals to which the hardware under

study is put by the custonmer. This partition, of course, con- flicts with the cradle to grave concept commonly applied to military system problems. However, for commercial systems, particularly for spacecraft that may have only a single shot experimental goal, or for Value Engineering problems that concentrate on the production phase, this division of the conventional cradle to grave effectiveness is essential.

U= Some appropriate unit of measure for computational convenience.

TABLE III

THE EFFECTIVENESS ARRAY FOR DEVELOPMENT PHASE PROBLEMS


Effectiveness elements

Performance

System elements Composite Initial reliability Growth effec tiveness reliability Avail- Other for develop- ability

ment Engi- Test Feed Qnality

neering re- back assur- estirmate sults ance

Total system &D x x x x x x

Hardware &DH X X X X X x

Prime mission equipment (b) &DH1 x x x X X X

Prime mission equipment &DH2 x x x X X X

spares Associated support equipment &DH3 x x x X x x

(c) Associated support equipment &DH4 x x x X X X

spares Real property installed equip- &DH5 x x x X X X

ment (d) Other PDH6 X X X X X X

Software &DS X X X X X X

Prime mission personnel (e) 8DS1 x x x X x x

Associated support personnel &DS2 X x x x x x

(I)




eij Effectiveness of the ith-jth subelement within the D, P, or 0 major element.

i = Columns of effectiveness elements. j=Rows of system elements.

To carry the description one step forward with reference to the categories in Table III, IV, and V, the general structure of the model for any system development phase problem as an example may be expressed as follows:

&DH= Hig 1 , j- 1 eDHij, eI)H -=RPij RA4j Roij)

RP-Reliability of performance (Probability of Success), RA- =Probability of Availability,

TABLE IV

EFFECTIVENESS ARRAY FOR PRODUCTION PHASE PROBLEMS



Performance

System elements Composite Initial effectlveness reliability Fabrica ] Availa- Other

for tion_____~ Quality Assembly bility production rels- assur- relia-

Pur- Off- ability ance bility chased the: blt parts shelf

Total System 8p x x x x x x x Hardware 8PH X X X X X x x

Prime mission equsp- x x x x x x x x ment (b)

Prime mission equip- x x x x x x x x

ment spares Associated support x x x x x x x x

equipment (c) Associated support x x x x x x x x

equipment spares Real property installed x x x x x x x x

equipment (d) Other x x x x x x x x

Software gPs x x x x x x x Prime mission personnel x x x x x x x x

(e) Associated support per- x x x x x x x x

sonnel (f)




Ro-Probability associated with other categories related to a specific problem.

The specific methods of determining the value of the e's will not be dis- cussed here. For cost-effectiveness analyses used for long-range planning where the hardware has not been developed, the determination of the e's

becomes a problem of predicting the a priori probabilities of reliability based on some of the same statistical techniques required for the cost model. For short-range cost-effectiveness studies, experimental data may be available. Substantial research already has gone into the reliability subject, particularly for those latter cases where experimentation and engineering improvement can be made.[21]

There is one special type of evaluation between cost and effectiveness that has been of interest as far as questions concerning the mathematical foundation of an effectiveness model are concerned. In the third section of this paper, the concept of the trade-off between two design characteristics

TABLE V

THE EFFECTIVENESS ARRAY FOR OPERATIONAL PROBLEMS (Illustrative Example)


Availability Performance System element Composite____- ______-___- ManOte

effectiveness Main- for Sched- Storage Launch Orbit Threat tenance

operations uled relia- relia- relia- degrad- delivery bility bility bility ation

Total System so x x x x x x x Hardware 80H X X X X X X X

Prime mission equip- x x x x x x x x ment (b)

Prime mission equip- x x x x x x x x ment spares

Associated support x x x x x x x x equipment (c)

Associated support x x x x x x x x equipment spares

Real property installed x x x x x x x x equipment (d)

Other x x x x x x x x Software SOB x x x x x x x

Prime mission personnel x x x x x x x x

(e) Associated support per- x x x x x x x x

sonnel (f)




was briefly described where an indifference curve of constant effectiveness played an important part in identifying the optimum combination. The mathematical properties of this type of curve or isoquant have been useful to assist in the identification and development of similar curves for specific space vehicle planning problems.

One clue to placing the concept of effectiveness on a sound mathematical foundation was found in the mathematical basis of the economic theory of preference fields as described in the text by WOLD AND JUREEN.[33] A part of this text deals with preference functions, which represent a method of partially ordering sets of n-variables.* The authors claim that effectiveness can be treated in a similar manner. The sets of n-variables, in the case of effectiveness, can be the parameters that characterize a system. These sets of vectors can be considered, for convenience, points in a euclidean space of n-dimensions. The desired partial ordering can be accom- plished with the aid of hypersurfaces, as follows: Any two points P' and P" are defined as 'equivalent' or 'indifferent' when P' lies in the same hypersurface as P"; P' is 'greater than' or is 'preferred to' P", when it lies above the surface containing P", and, P' is less than or 'disfavored to' P" other- wise. These concepts will be clarified and made more rigorous in the following paragraphs, although no claim is made that a complete solution is presented.

It is proposed that effectiveness be a function defined by the following characteristics:

(a) Its domain of definition is that part of the euclidean space of n-dimensions satisfying the condition that a point is in that set when all its components (with respect to some reference system) (i.e., its independent variables) are all nonnegative.

(b) Its range, that is the values the function takes (or the independent variable), is also nonnegative.

(c) When it is set equal to some constant, the resulting 'contour' will define a hypersurface. This implies that it has all partial derivatives, which are continuous, up to and including the third.

(d) Its first partial derivatives are all positive. (e) It is strictly quasi-concave, a property that can be shown to be equivalent

to the 'law of diminishing marginal returns.'

The first and second postulates are basically for convenience, since the origin for the space can be changed by a simple translation to meet this

* By definition, a set S or elements x, y, z, *-- is said to be partially ordered if, and only if it admits an ordering relation or rule of precedence, indicated by the symbol <, between some elements in S which is transitive (if x<y, and y<z, then x_z), antisymmetric (if both x_cy and y<x, then x=y), and reflexive (x_x, holds for any x). A simple example of a partially ordered set is the set of all subsets in the plane with the ordering relation of set inclusion.




requirement. Design parameters can always be defined so that they are never negative. The second postulate allows various designs to be compared by points corresponding to them on a linear scale, the real numbers being simply ordered. In other words, effectiveness is a function

y=&(xl, ,. Xn)7 (7)

where the xi's (for i-1, , n) satisfy the condition that xi >0. In terms of geometry, g is possibly some kind of surface imbedded in an n+1 dimensional space. When n= 2, 8 can be a surface in three-space as shown in Fig. 1.

The last three postulates are nmore interesting, and their roles will be stated. The necessity for the expression 8(xi, * , x) K, for any constant K, to represent a hypersurface was already indicated. The hypersurface is used as a means of introducing a partial ordering into the points in the space. It is shown in texts on differential geometry that this is possible only when g has at least all first, second, and third partial derivatives, which are continuous. In addition, for 8 to represent a hypersurface, conditions of integrability, called Cadozzi's equations, must be satisfied. The latter equations are quite complicated, and will not be exhibited here. They can be found in standard books on differential geometry.1341 The differentiability requirement stated above makes meaningful the next postulate that all first partial derivatives are positive, and it excludes such functions as, in two dimensions 8(x1, x2) =x2. It is a real valued function satisfying all requirements so far, and all its partial derivatives exist, and are continuous. Moreover,

W/Ox2 = 1 > O, (8)

but 8 is completely independent of xi, and allows no 'trade-off.' In- cidentally, differentiability in one variable and derivative not zero, plus continuity are required in order to obtain a parametric representation in terms of n-I variables for the hypersurface. A simple function satisfying the last postulate is

(Xl, X2) = xl+x2, since (9)

8&/OX1l- /aX2-1> O. (10)

The final postulate introduces still more structure into the hypersurfaces, and excludes such simple surfaces as planes. The law of diminishing marginal returns states that "as one system or input is substituted for another with all other inputs and 8 held constant, the terms on which the substitution can be made will become less and less favorable." N] First of all, if x1 and x2, as well as & are held constant, then

d&-= (0/Oxi) dx1+ (O&/Ox2) dX2=O, (11)

and dx2/dx= - (a&/4X2)1(a&1axi) (12)




is always negative by postulate d. The law of diminishing marginal returns then requires that

dx2 dxl dx2 j ~~~(13)

decreases when xi increases, as indicated in Fig. 2. It can be shown that this is equivalent to ? being strictly quasi-concave. This means that for any two points x' and x" with coordinates (xl', .., xXn) and (xi", *., xn"),

that &[Ox/+ ( 1-0)x"] > &(xf) (14)

holds for any 0, O< 0<0< 1, and that the quantity holds only at the end points x or x". This rules out such simple functions as the linear ones, since the function is not greater at the intermediate points, if the end-points lie on the line. This is a stronger condition that quasi-concavity which requires that the point set of x's for which (x1, *, X) > KK, for any constant K, be convex. (A set is said to be convex when: any two points x' and x" are in the set, and any intermediate point, Ox'+(1-0)x" is also in the set).

Thus a design represented by the coordinates (x1', *. , x.') x' is preferred to the design represented by (xi", **, x" )x" , if 8(x') >8(x"). This means geometrically that the point x', and all the points in the hyper-

Curve g(x11x2)~C Z Surface y = F(x,x2)

Plane y = C

xl

Domain of definition of &

X2

Fig. 1. Graphical representation of &.




surface passing through it, lies at a greater distance away from all axes than does the hypersurface passing through x".

It is to be noted that no claim is made that an effectiveness function is unique. There may be many functions that will perform the required function of placing designs in order of preference. The selection of this function is dependent on the particular design problem.

This section will be closed with an example, purposefully simplified, of an effectiveness function applied to a multiflight manned spacecraft program, which can be seen to satisfy the proposed postulates. Two flight plans-illustrated below-are to be evaluated, each of which involves two space borne experiment packages labeled 'A' and 'B,' given reliability figures for these packages and the booster-spacecraft.

FLIGHT PLAN I

Flight No. Experiment

I 2

A I5 I5 B I5 I5

FLIGHT PLAN II

Flight No. Experiment

I 2 3

A IO 10 IO B 10 IO IO

x 2

d tB Convex Region%'

d x ' x

I I /x

l 1

F1 fni sh

iFig. 2. Graph of function satisfying the law of diminishing return.




The numbers appearing in the tables are the hours each experiment is scheduled for each space flight, and in both schedules experiments A and B each are allocated a total of 30 hours. If the equipment for experiment A fails, for example, on the first flight in Plan I, then the astronaut will elect to perform 30 hours of experiment B, not wasting his time; and the second flight will be rescheduled for 30 hours of experiment A.

The measure of effectiveness will be defined as the expected value of the time spent successfully by an astronaut in conducting assigned experiments, with the provision that in the event one of the experiment equipments should fail, he is free to continue with the remaining experiments to the maximum allocated time. The effectiveness function is somewhat difficult to write explicitly, but the expected number of hours can be computed using an aid shown in Fig. 3, which shows every possible event involving the experiments and the booster-space vehicle combination. Beginning at the point marked '0,' the first event is the operation of the booster with reliability R or its failure R'. If, for example, the booster has not failed,

B

B' B A

A/~~~~~ ~~ B' BB Bl~~~~~~~~~~~~~~~B

R' R/ R Bi R

Al

B R' R B P, B' B

Fig.R3. Logical diagram for a two-flight program R

0

Fig. 3. Logical diagram for a two-flight program.




experiment A will operate or fail, A'. If A has succeeded, then B can operate with reliability R, or fail B'.

If the assumption is made that the experimental equipment and the spacecraft are mutually independent, then the measure of effectiveness can be written as sums and products of reliability and unreliability parameters. Thus it can be seen that postulates (a) and (b), as well as (d), are satisfied

by 8. If 8 is set at 45 hours, for instance, then the result 'indifference' curve (in the language of economics) is displayed in Fig. 4. Also the region above and to the right of this curve is a convex regioni.

This measure was applied when RA RB = 0.8 and RB 0.9. The calcula- tions show that 8 = 46.0 houirs for Flight I and = 48.3 hours for Flight II. Clearly the second is to be preferred.

THE COST-EFFECTIVENESS MODEL

THE COST-EFFECTIVENESS model is essentially the final comiparative analysis based on the output of the cost model and effectiveness model. The

1.0

_ .0 hr LU

45 hr

c~ 0.5

z

00

-0.5 0.9 1.0

BOOSTER RELIABILITY, RB

Fig. 4. Indifference curves showing reliability trades.




algebraic structure of this final comparison depends primarily on the nature of the criteria that are developed and defined for the final selection of the best alternative plan, system, or component. It will also depend to some extent on the nature and complexity of the alternatives involved in the analysis (e.g., whether the alternatives are total systems, components, or manufacturing processes).

The problem of developing the criterion is basically one of explicitly setting forth the 'test of preference' by which alternative A1 is determined to be better than alternative A2 ... An . Over-all the objective of the decision maker, whether in government or commerce, is to maximize the worth of his proposed plan. In the abstract, this means simply the maxi- mization of his returns or gains less his expenditures. The practical analytical difficulty is that the worth of proposed systems (military and many commercial) and components cannot easily be measured in dollar terms.

The errors of adopting the wrong criteria have been stressed in several documents. For example, there are the errors of (1) ignoring the absolute scale of either effectiveness or costs (i.e., using a ratio of costs to effectiveness over a wide range), (2) setting the wrong objective or scale of effectiveness, (3) ignoring uncertainty, (4) ignoring effects on other operations, and (5) adopting an over-determined test. 35] All of these are important considerations regardless of whether the cost-effectiveness analysis is of the large scale system selection type or of the Value Engineering type.

Some of the criteria used in previous studies were summarized by the Industrial Advisory Committee. [3, 4, 5] These are presented in the following outline. It might be pointed out that all of them make use of a ratio of costs to some measure of effectiveness. Although this procedure is objec- tionable, in the sense that it ignores the absolute scale, in many cost effectiveness analyses it is the only convenient way of comparing the alternatives when neither cost nor effectiveness can be held constant.

EXAMPLE OF

AREA OF ENDEAVOR COST-EFFECTIVENESS CRITERION

Nonmilitary: Building Dollars per square foot Air Passenger Dollars per passenger mile Freight Dollars per ton mile Computer Dollars per bit Communications Dollars per message unit Electricity Dollars per kilowatt hour Gas Dollars per cubic foot Public highways Dollars per mile Farming Dollars per acre




Military: Launch vehicles Dollars per pound payload in orbit Satellites Dollars per hour of successful operation in

orbit Missiles Dollars per kill Interceptors Dollars per intercept

ACKNOWLEDGMENT

THE AUTHORS wish to acknowledge the valuable assistance and helpful criticisms of PROF. T. H. GAWAIN, Naval Post Graduate School, Monterey, California, and Consultant to the TRW Systems Group, and D. E. ROBISON AND D. FELBER of TRW Systems Group.

REFERENCES

1. DEPARTMENT OF DEFENSE (DOD) Directive No. 3200.9, "Project Definition Phase," 26 February 1964.

2. DIRECTOR OF DEFENSE RESEARCH AND ENGINEERING, DOD, "Guide to Project Definition Phase," (draft) 1964.

3. FINAL REPORT OF TASK GROUP IV, Weapon System Effectiveness Industry Advisory Committee, Air Force Systems Command, AFSC-TR-65-4, Vol. I, "Summary, Conclusion and Recommendation, Cost-Effectiveness Optimi- zation," January 1965.

4. , Weapon System Effectiveness Industry Advisory Committee, Air Force Systems Command, AFSC-TR-65-4, Vol. II, "Task and Analysis Methodology, Cost-Effectiveness Optimization," January 1965.

5. , Weapon System Effectiveness Industry Advisory Committee, Air Force Systems Command, AFSC-TR-65-4, Vol. III, "Technical Supplement, Cost-Effectiveness Optimization," January 1965.

6. C. J. HITCH AND R. MCKEAN, Economics of Defense in the Nuclear Age, Har- vard University Press, Boston, 1960.

7. M. J. PECK AND F. M. SCHERER, The Weapon Acquisition Process, An Eco- nomic Analysis, Graduate School of Business Administration, Harvard Uni- versity, Boston, 1962.

8. J. P. KORNFELD, Cost-Effectiveness as a Method of Evaluating Design Changes, Boeing Airplane Co., Aero-Space Division, Seattle, Wash. (Oct. 10-12, 1960).

9. A. S. GOLDMAN, "Introduction to the Economic Theory of System Develop- ment in Operational Support," RM 62 TMP-39, TEMPO, General Electric Co., Santa Barbara, Calif. (1 May 1962).

10. , "Risk and Uncertainty in Cost-Effectiveness and Budgeting," SP-255, TEMPO, General Electric Co., Santa Barbara, Calif. (January 1964).

11. M. C. HEUSTON, "Concepts of Cost Analysis for the Defense Industry," AATM No. 3, Missile and Space System Division, Douglas Aircraft Co., Santa Monica, Calif. (April 22, 1963).

12. I. NATHAN, "A Figure-of-Merit Cost-Effectiveness Evaluation Model," 1964 Systems Division, General Precision Aerospace, Wayne, N.J. (Sept. 14-18).

13. F. H. TRINKL, "The Art of Applying Cost-Effectiveness Analyses to the Selec-



266 M. C. Heuston and G. Oaawa

tion of R&D Projects: Application to Military Space Systems," OASD (Comptroller), DOD, (Oct. 9, 1964).

14. E. S. QUADE, "Cost-Effectiveness: An Introduction and Overview," P-3134, The Rand Corp., Santa Monica, Calif. (May 1965).

15. W. GRODOWITZ, "Some Uses and Limitations of Cost-Effectiveness Analysis in Military Decision Making," Office of the Director of Aerospace Programs, HQ USAF, Washington, D.C. (Oct. 1964).

16. M. C. HEUSTON, "The Data Problem and Solutions in Cost-Effectiveness Analysis," TRW Systems Group, Redondo Beach, Calif. (30 Sept. 1964).

17. -, "Generalized Cost-Effectiveness Procedures," Vol. 17, Texas Instru- ments (30 July 1964).

18. P. D. Fox, "A Theory of Cost-Effectiveness for Military Systems Analysis," Opns. Res. 13, 191-201 (1965).

19. -, "Cost/Reliability Research," Vol. I, TRW Space Technology Labora- tories, Thompson Ramo Wooldridge, (18 May 1964).

20. J. E. BROOKS, "Cost-Effectiveness Formulation for Standardized Space Guid- ance System Study," TRW Space Technology Laboratories (April 1964).

21. A. J. MONROE, "Achievements of the Weapon System Effectiveness Industry Advisory Committee (WSEIAC)," TRW Space Technology Laboratories, presented at the 6th New York Conf. on Electronic Reliability, 21 May 1965.

22. L. A. AROIAN, "The Elements of Reliability for Space Vehicles," TRW Space Technology Laboratories No. 9300.6-187 (11 September 1961).

23. H. CRAMER, Mathematical Methods of Statistics, Princeton University Press, 1945.

24. F. B. HILDEBRAND, Introduction to Numerical Analysis, McGraw-Hill, New York, 1956.

25. H. SCHEFFE, The Analysis of Variance, Wiley, New York, 1959. 26. U. GRENANDER AND M. ROSENBLATT, Statistical Analysis of Stationary Time

Series, Wiley, New York, 1959. 27. D. J. BORDELON, "Best Linear Estimate of Missile Location Based on Diverse

Simultaneous Tracking Data," SIAM Rev. 4, 91-93 (1962). 28. M. KENDALL AND A. STUART, The Advanced Theory of Statistics, Vol. 2, "Infer-

ence and Relationship," Hafner, New York, 1961. 29. J. A. SPECKMAN AND R. G. CORNELL, "Estimation of a One-Parameter Expo-

nential Model," J. Am. Stat. Assoc. 60, 560 (1965). 30. G. H. SANDLER, System Reliability Engineering, Prentice-Hall, Inc., New York,

1963. 31. IGOR BOZOVSKY, Reliability, Theory and Practice, Prentice-Hall, New York,

1961. 32. A. J. MONROE, "Mathematical Mission Models for Space Programs," TRW

Paper presented to Aerospace Corp. (April 1965). 33. A. WOLD AND L. JUREEN, Demand Analysis, Wiley, New York, 1963. 34. L. P. EISENHART, An Introduction to Differential Geometry with the Use of

Sensor Calculus, Princeton University Press, 1940. 35. E. S. QUADE, "Analysis for Military Decisions," Rand Report R-387-PR,

(November 1964). 36. J. JOHNSTON, Statistical Cost Analysis, McGraw-Hill, New York, 1960.



Documents

Observations on the Theoretical Basis of Cost-Effectiveness