IEEE TRANSACTIONS OX EXCaNKEHIXi; ΛΓΑΧΑΠΕΝΠ'.ΝΤ VOL, EM-12, NO. 3 SEl'TEMOER. 1065

An Approach to Research and Development Effectiveness

A. B. N U T T

Abstract—The major portion of this paper consists of a descrip­tion of the rationale and use of R D E (Research and Development Effectiveness), a computerized planning program developed in-house and designed to utilize analytical techniques in the manage­ment of R & D resources in the Air Force Flight Dynamics Labora­tory, RTD, AFSC. For the three years (including this year) that the program has run, it has been to a degree experimental in that changes have been made to the measure of effectiveness model during the program evolution. Further changes at this time are not contemplated, so that next year time for preparation of input data will be markedly decreased. This will be possible by merely requiring the marking up of already prepared input plots rather than constructing entirely new ones. The capability to achieve detailed insights into the program at short notice h a s given rise to the initiation of development of an overall Management Informa­tion System for the Laboratory, which will complement the RDE program by providing past resource consumption trend data to key into the planning data obtained from the RDE program.

^ H E R D E PROGRAM was evolved and is used jyrimarily as an aid to Laboratory management in

* allocating the annual Laboratoiy research budget, although ancillary benefits have been derived as will be shown. This discussion of the R D E iirogram will be pref­aced by a brief descrii)tion of the Flight Dynamics Lab­oratory and its mission and research program to show the environment in which the program is used. Finally, the results of program usage and impact of the program on Laboratory management and management thinking, including an outline of actions planned in connection with future use of the program, wiW be covered.

The Flight Dynamics Laboratory is one of the eight Laboratories of the Research and Technology Division of the AFSC. Tts primary mission consists of acting as the AF focal point in its areas of assigned technical re­sponsibility and conducting exploratory and advanced development in these areas leading to the solution of tech­nical problems and creation of applicable design criteria for future AF"" flight vehicles. The primary areas of re­sponsibility are flight vehicle flight control, aerodynam­ics, structures; equipment, and dynamics. The Laboratory also provides technical support as reciuired to AF systems offices, other product Divisions of AFSC such as BSD, SSD, ASD, and ESD, and also, where certain of its unique development facilities are concerned, provides support to other government agencies and industry.

Mamisciipt received August 27, 1065. Presented at the National Aeronautics Electronics Conf., Dayton, Ohio, M a y 12, 1065.

The author is with the Air Force Flight Dynamics Lab., Wright-Patterson AFB, Ohio.

The Laboratory is authorized 7G2 personnel; of this number, 418 are ])rofessional scientists or engineers. The annual budget is 34.6 million dollars, of which 16.5 mil­lion are expended in exploratory development. The R D E program concerns itself with the allocation of the explor­atory development funds and about 70 percent of the scientific and engineering manpower. The Laboratory is located in 33 buildings totaling 600 000 square feet in the Ai'ca 'Έ^' complex at Wright-Patterson AFB. The actual Laboratory comprises 350 000 square feet of this space, the remainder being devoted to offices and administrative areas.

The technical i)rogram is included in two Department of Defense Program Elements and is comprised of 22 projects covering over 200 research tasks.

Based on the largely vehicular design criteria objectives of the Laboratory I'csearch program, any resource allo­cation program deΛ'eloped must consider:

1) future AF systems needs, relative importances and proba])ility of development;

2) the state-of-the-art in various areas of the Labor­atory's technical responsibilities;

3) the degree of support afforded each system and area of technology by each technical activity or task;

4) the most effective research approach (in-house or contract) ;

5) the relationship of progress in each technical activ­ity to the resources level applied; and

6) the cost of technical alternatives relative to limited resources.

Figure 1 shows in capsule form the research budget allocation problem faced by the Flight Dynamics Labo­ratory. I ts similarity to the capital budgeting problem in industry is apparent [2] .

As can be seen, the number of possible task resource

Available Resources

Five Divisions

200 Tasks -

Advanced Development-

Programs

• •Al locat ion^ Proce.ss^

i Optimum

Allocations to Each Task

30-40 -Advanced

Systems

,^Technical Disciplines

Fig. 1. A F F D L research allocation problem.

CONCEPTUAL SYSTEMS

Τ

Γ - >

1 0 4 I E E E T R A N S A C T I O N S O N E N G I N E E R I N G M A N A G E M E N T

allocations, based on all ]^ossihlc interrelationships 1)C-twccu the varial)lcs concerned, conld run into the thou­sands . Howevei*, as has been shown, th.c basic considei*a-t i o n s in resource allocation for the Laboratory are re la-t i A ' c l y few in nunil)er. The solution to the ]UObleni then is to struetui'e management judgement in such, a way as to take i n t o account tlie abo\'C Ijasic considca'ations for budget allocation, while g iv ing unbiased a n d ec|ual con­sideration to resource allocations to each research ae t iv -i t y or task. To organize thi.- data a n d rapidly consider all tliese interactions, the data handling cai)ability of the digital conijiuter is used. Figure 2 is a grajihic rej)rt^-sentation of the RDTC ])iOgram. Tlic top block of tlie diagram, t h e AF Mission ^ la t r ix , is used as a basis for the ent i re j^rogram. Since the Falioratory's job consists of iproviding technology to fulfill t he AF nc^eds, these needs must l)e expressed quantitatively if tlu* Laboratory is to be able to evaluate quantitatively its i.bility to sup|)ort the AF.

Figtire ?>, the m i s s i o n matrix itself, expresses tlic tech­nical 3ieeds of tlie Al ^ in its ro le of supportiiig tlic Natiojial Objectives. I t says in effect tliat t l ie AF conducts its four basic types of operations in three kinds of Avar

environments and that technical i iceds exist in each of these twelve mission elements, lilach element η,,- of the matrix is givcji a Λτϋηο representing technological im­provement required in support of this mission elcme^it relative to a similar value for each other e l e m e n t of the matrix. This Avas done b y stai'f planners a t lleadc|uarters, AF, using the military, economic, geopolitical, and in­telligence data aATiilablc to them. Thc}^ utilized a ma t l i -ematical technique known as the Churchma]i , Arnoff, Ackoif method for determination of relative va lue [3]. The A'ahies were then normalized so t l i a t ^ = 1. The matrix is re-evaluated ])y a higher headquarters plans oiiice eacli time the program is run to reflect the latest tlnnking in t e rms of the changing A v o r l d environ­ment.

Referring again to Fig. 2, the technology block rep­resents the research tasks of the Laborator3^ To relate tlie s u p p o r t each proΛήdes to tlie AF mission, the future conceptual AF flight veliicle s^^stems, as defined in the AFSC Long Range Teclmological AVar Plan a n d the technical goals of the Laboratoiy as defined in the Research a n d Technology Division Long Range Plan, are used as links. A relatii^e value of importance for each of these s y s t e m s and goals is determined using again the Churchman, Arnoff, Ackoff metliod by proceeding as fol­lows

SEPTEMBER

A1"R FORCE MISSION MATRIX

TECHNICAL GOALS

TECHNOLOGY (Sensitivity of Progross to Resource Application)

1 · 2.

3. FiLT. 2.

Task Selection Relative Values of Resources Technology Profiles

R D E i)r{)m-ain (liairiain.

c ; 1 ' : x 1 ' : h a l W A l l

L I M I T E D WAR

COLD WAR

C O M B A T DRER ATIONS η Μ )1ΐ2 Ν 13

R E C O N . )l -2 1 /?·:·:

LOGISTICS i);; 1 η^3

SHOW OF F O R C E η 42

Fit^. ο . Air Force ^Fission ^ l a t r i x .

3) ^Liltiply each R'^si by the corresponding element of the matrix and siun to get .1 / values of

4) Normalize, so tliat ^ η^Ε^Ι = 1, giving a relative value for each System in providing improved capabilities for the A F in carrying out each element of the AF Mission ALatrix.

Similarly, for Ρ number of teclmical goals, obtain n , / f i r G p = 1? giving a similar relative value for each

technical goal. To determine the measurement of the effecti\'eness or

value of each research task, the following factors must be considered:

1) Determine the relative value (expressed as R%'^) for each System ά\ thru S„, in support of ]Mission ^Litrix element

2) Repeat this determination for each element of the matrix resulting in il/ sets of values of 7?J*j.

rate of technical progress, s tatus witli respect to state-of-the-art, relative importance of task applicability to systems, number and relati\ 'e importance of future S3^stems supported, number and relative importance of technical goals supported, timeliness of effort, relati\'e

1965 NUTT: Κ & D EFFECTIVENESS 105

contributions to systems and technical goals, and rela­tionship between progress achieved and resources re­quired.

These factors are coml^ined into llie following math­ematical model:

where

RDF. Alodel Objective Function CoeiHcient current Level of Probability of Success in task-Le\*el i.

i = task resource level Change in this Probability of Success during

At tlie reference time period Dr task Applicability Factor Μ numt)er of systems j system

m timeliness function contribution of task to a system \'ahie of system j

CL, Confklence Level for task resource lev^el i ACL, change in Confidence Level over the reference

At t ime period L number of technical goals 1: technical goal Qk contribution of task to a technical goal Ih. technical goal eiTectivit\\

This model is the value calculation block referred to in Fig. 2 . In an overall sense, the model says tha t the measure of effectiveness V i of each task resource ΙΟΛΌΙ is comprised of two ])arts. One is the effectiveness the task has in supi)ort of development of future AF flight velii-cles, represented by the term to the left of the ])lus sign, and the other is the effectiveness the task has in support of the achievement of the technical goals of the Labora­tory, represented by the other term.

The Confidence Level (CL) referred to in the model structures technical lUOgress on a given task from idea conception and problem definition to acqui.sition of ade­quate technology. These values, from 0 .1 to 1.0, are pre­defined for the task engineer in 1 0 discrete steps. The probability of success (P..) is defined as the probability (from 0 .1 to 1.0) that , in a normal system development cycle, the task technology at a given confidence level could be successfully incorporated into a system develop­ment wdtliout further state-of-the-art development. For all tasks, a Ps of 0 . 8 is equated to a CL of 0 .8 and is de­fined as the point \vhere technical feasibility of a tech­nology is demonstrated.

In actual operation of the R D E program, each task engineer plots the progress of his task, based on the actual work and resources required to progress from one

confidence level to another, on a graph of CL and P.,- vs. years. A typical i)lot is shown in Fig. 4 . Three curves are l)lotted, one using programmed resources (dollars and scientific and engineering manj:)Ower), one using one-half the programmed manpower, and one using twice the man­power. For the latter two, the h\u(\< in dollars ai'e com­mensurate with the manpower and facilities scheduled to be made available. The contract/in-liouse man|)Ower re­sources mix used is a function of the ability of the per­sonnel and the I'ccjuirements of the nature of the re­search to be performed. The t a sk resource level referred to in the model and shown at the end of the curve plots in Fig. 4 is level 1, 2 , or 3 corresponding to one-half the ] ) i O g i ' a m m e d manj)ower, j nOgrammed manj^ower, and twice i)rogrannned manj)ower, respectively. By a subrou­tine in the computer input program, three intermediate resource levels are also computed, giving the program six ])ossible resource levels to be chosen from for each task during tbe allocation process.

The task Applicability Factor in tlie model relates the I 'elative importance of each task (from 0.1 to 1.0) in terms of its ai^plication importance to a system on whicli it would be used. This factor is applied l)y the Assistant Chief for Research and Technology of each Division in the Lal)oratory for each of his Division \s tasks.

The timeliness function (valued from zero to 1.0) in tlie model is of tra])ezoidal shape and evaluates each task in terms of wdiether or not it achieves technical feas­ibility in time to be aj^ilied to the systems and goals it supports. Each of the systems and technical goals has a desired initiation of acquisition and completion date, re­spectively. A trapezoidal form is used to ])rovide a pen­alty for l)eing too early or too late.

The contribution shown in the model of a task to a system or technical goal is determined l)y reference to a predefined scale from zero to 1.0, which goes from remote association to absolutely essential in the case of .systems and from minor contribution to ])otential breakthrough in the case of technical goals. The A^alue of the systems and the value (technical effectivity) of the technical goals are determined by their relationship to the mission inatrix as previously described.

Referring again to Fig. 2 , the block labeled ]\iathemat-ical Program reju'esents the linear program put into the computer for solution of the budget allocation problem. I t utilizes: 1 ) the measure of effectiveness of each re­source level of each task as computed by the mathemat­ical model; 2 ) the dollar and lnanl)OΛver resource level of each task as stipulated l)y the task engineer (represented by the lower left hand block of the diagram in Fig. 2 ) ; and 3 ) the total annual resources in dollars and man­power made available to the Laboratory by higher head­quarters (lower right hand block of Fig. 2 labeled Re­source Constraints).

The input matrix is as follows:

lOG IEEE TKAXSACTIOXS OX EXGIXEERIXG ^LVXAC.EMEXT

CONFIDENCE lEVEL DISPLAY CHART

SKPTEMBUR

PROJECTAASK/EFFORT NR 139907 TECHNICAL GOALS 1, h, 5,

ο

ϋ

8

1.0-

. 9 .

. 8 .

.7

. 6 -

.5-

I

.21.0

I

/ 5 6 9 10

, / /

\λ / /

/ -

66 67 68 69 70 71 72 73 7U 75

1.0

.9

.8

.6

-9

FISCAL YEARS

(ShCTf progress Made by the End of Each F.Y. For Each Resource Level)

Plot the Estimated C.L. as of 1 July 196^ (This Point is Common to All Resource Levels)

• Indicate With a Ο the Estimated Confidence Level at 1 July 196U (This is not to be part of the plot)

F i - 4.

7. 8. 9, 10 , 1 1 , 12 ,

1$

ο

a

CO

I

-FILL IN APPROPRIATE P XALE AND SHOW SYSTEM DATES ν/ΠΉ AN "X" ON THE .8 Ps LEVEL

BRANCH APPROVAL .

DIVISION APPROVAL

ΛΙΑΧΙΛίΙΖΕ A'ALUi: OF

SUBJECT T O

a,X, + a,X, + . . . a..A,, < C O N T R A C T S + S U P P O R T S

h,X, + IKX, + · . . KXr < ICNGINKLRS (CONTRACT)

c,X, + c,X, + . . . c,,X. < E N G I N E E R S

(IN-HOUSE)

X, + X, + · · · A . = 1

A \ . . 5 + Xr-. + Xn = 1 ( a p p r o x i m a t e l y 200 un i t a ry c o n s t r a i n t e q u a t i o n s )

In the mati 'ix, A",, re})resents single resource levels for each t a s k . is t h e m e a s u r e of effectiveness for each r e ­source level of eaeh t a s k as comini ted us ing the m a t h e ­m a t i c a l model descril^ed. This c o m p u t a t i o n is j-jcrformed in t h e compute r by an in}nit su l^Out ine . a,, I 'epresents t h e dol la r cost of each resource level of each t a s k anrl 6» a n d r,, t he scientific a n d eng inee r ing m a n p o w e r cost lor con­t r a c t effort and in -house effort, r espec t ive ly , for each resource level of each t a s k . VT t h en is t he measure of effccti\ 'eness for t he t o t a l Laboratory p r o g r a m and is to be maximized b y choosing t h e p rope r solution set of t a s k s , where the use of t h e r e c o m m e n d e d a l locat ion will r esu l t in dollar a n d m a n p o w e r expend i tu res less t h a n or equa l to those m a d e a v a i l a b l e to and budge ted by the Lab­o r a t o r y . Since each t a s k h a s six possible resource levels a n d only one level is to l:)e chosen, t h e u n i t a r y cons t r a in t equa t i ons shown y)revcnt m o r e t h a n a single resource level select ion per t a s k . Since t h e Laboratory h a s a})out 200 ta.sks, t he actual m a t r i x will \m\Q a b o u t 1200 terms in t h e

1965 NL'TT: R .S : D EFFECTIVENESS

FLOW C H A R T OF ' T . D E " I N P U T PPJCPARATION

(.Vo/c: Tyi)e or use black ink or heav\' pencil in preparing Forms I, l b H b I '

Τ l i ίίΤ IV V

107

Prepare Resource iSclu'dule SUNIUIARV Work Shoe I

Read Atch ^2

Prepare C()nlideiii.'e Level Work 1 )es('i*ipi ion Sheer ' (in Alch )

Pre pi Ire Prepare Coniiclonee bevel PavoiT Display —> List Char t

iin At Ml (in AU'ii --••i))

Head Alch Clieck for coi-rcct ta<k Nr ,

ΤΗΙ·: SIM:CIFK; T A S K / S E I I T A S K Xl MBKR IS INDICATKI) ON ('OXFIDLVXCE LLOVKL DLSL'L.W CIIAKTJ ΑΤΙ ;Η

Read Alch Η

JICFCV la: 1\ημτ. AlanpowiM*

vs. Contract Dollars (h-a])h in Alch ^ 2 Atcdi ;^o—AFFDL

F\'icilitics Atch 12—Sample

Input

LIIUR TO: Alch ^5 A F F D L Facilities Alch . G Adv'd Dov. Proiiriims Resource Schcdetle SumuKuy

Work Shoot in Atch -2 Alch List of TDO's Atcii System Descriptions Atch ^Ίϋ Contribution to

S\'siem Scale Atch 712 Sample TNJMIT

Prepare Forward Form .Miister of AFFDL-l.-) I, 11, HI , Punch > IV, cc λ' to Card F D P Transcri])t throu ;!"!

chaiuiols

UCFZ ΊΟ: Atch /.T List of Ti:)0's Alch ,-'S System Descriptions Atch -0 C^>utril)utions to Tech. Dev. Obi, Scnlo Atch •••10 Contrihutions to System Scale' Confidenco Level Display Chart in Atch 3 Limilod War Evaluation Scale in Atch Atch F 12 Sample Input

* Where task/subtask ohjoctivos remain imchan.irod FV GG Work Dcscri])tion Sheets can bo used as rol'oronce to decrease })roi)aratioii time,

Yvz. 5.

first 4 equations and one constraint equation of six terms each for each task, resulting in a 204 I'ow by 1200 column matrix. To cite AYeingartner [3]

This mod(il ΛΝΊΐΐ select among independent alternativos tliose task resource levels whoso total measure of etfi'cti\(UIOSS is maxi­mum, but whose total rosotirce cousumjUioii is within the budget limitation. The problem of indivisibilities is solved iu the sense that the linear programming solution imj^liciily looks at all com­binations of resource levels of tasks, not just one resource 1( vol of one task at a time, to scdect that set whoso total measure of ef­fectiveness is maximum. Furthermore, the \u:>IH'r limit of unity on each Xn~z> ' ' * Xn guarantees that no more than one of any re­source level of any task will be included in the final program. T h e omission of such a limitation would clearly lead to allocating the entire budget to mull ipks of the "b(-st" resource levi^ls.

The computing program used is the C E I R LP 90, per­formed on an IBAI 7094 computer. Programmers assigned to the R D E program in the Digital Comi^utation group, Deputy for Studies and Analysis, SEG are Thomas Du-vall and Lt. Robert Jurick. The latter replaced Chester Wolfe, now^ with the Avionics Laboratory, R T D .

The basic solution to the problem consists of a recom­mended dollar and manpow^er allocation for each of the set of selected tasks. In addition, the output format is designed to print out the allocation by Project, Ta.sk, and DiA'ision within the Laboratory, All input data is printed out for checking purposes, as are the computed measures

of effectiveness for each resource level of each task. (The fiow chart furnished each task engineer as a guide to ]n-eparation of input instructions is SHOAVN in Figure δ.) I t is of interest to note tha t in the last program run t h a t of 236 tasks in the program, 135 were recommended for resource suimport.

Out])ut formats are also designed to print out priority lists of selected and nonselected tasks. This is ]^ossil)le since the i) iOgram solution includes a computation of the value or •'im]')licit price' ' [4] of a unit of each kind of resource (dollars, contract direction, and in-house man-powe^O in t e r m s of effectiveness units. By subtracting the resource ^'cost" from the measure of effectiveness of a given task resource level, a net '^profit" or "loss'' can 1)0 determined. This computation is performed by the computer in preparing the task priority list. In addition, since the program runs for five consecutive years , tech­nology and S3^stem profiles are printed out, showing graphically for each system and technical goal the in­crease in confidence level by year for each task support­ing each system and technical goal. By noting the desired year of initiation of system acquisition, a determination can be quickly made from the profile as to \vhether or not the Laboratory technical program is supporting a given technical goal or future system on a timely basis.

A sensitivity analysis was performed on the program to determine the frequency of task resource level selection

1 0 8 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT SEl'TKMBKU

as a function of random changes in A-alues of their effec-tivencss as measured by tlie model. The R D E linear pro­g r a m was adjudged to be basically stable with only a smal l percentage of marginally selected activities being affected hy deviations in effectiveness values \o].

Tlio distribution of effort in preparation ( f ini)uis to tlie })rograin is as follows:

1") calculMtion of relative A*alues of sy<t(Miis and tech­nical goals—Senior Staff Personnel;

2» stijuilaiion of task applicai)ility ftictors—Assistant Division Chiefs;

3) task resource estimates—Task Engineers (Sec for­mat. Fig. 6) ;

4» task jnOgross ])lots—Task h^ngineers iRef. Fig. 4Ί ; 5> launch card transcript foi*ms—Task Engineers (S(^e

fo rmat . Fig. 7) : t) i review of input data—Branch cl' Division INIanagc-

nient: 7) analysis of output data—All levels of Eahoratory

Management; S) objective review of engineers' evaluation of task

values—Advanced Systems Analysis Group and Selected System Project Offices.

For the three years (including this year) tha t the p r o ­gram has run, it has been to a degree ex]^erimental in that changes liavc been made to the measure of effective­ness model during the program evolution. Further changes at this time ai'c not contemiilated, so tha t next year time for preparation of input data will be markedly' deceased. Tins will be ])ossible I:)y merely requiring tlie marking up of already pre))ared input plots rather than constructing entirely new ones. This will also perform a feedback func­tion in that the task engineers' pas t progress Avill be re­flected on the marked uji progress plot sheet. This will resemble Fig. 4 Avith the curves removed and a \)omt on the 66/67 vertical line showing predicted progress for the current fiscal year. The starting points for the two pre­vious fiscal years will remain on the plot.

The specific impact Λvhich the program has exerted on the Laboratory allocation i) iOcess to date has been as follows:

1) Total (exploratory development fund allocations to Di\'isions ha\'e been made based on the results of the R D E program output,

2) Those tasks recommended for funds decreases in order that the Laboratory might absorb an overall ex­ploratory development budget decrease have been deter­mined as a result of a special R D E ]')rogram run, using the decreased Laboratory budget as a new funds con­straint in the linear program and the actual planned tasks^ funds as coefficients. In these cases, Laboratory top management is furnished a list of tasks recommended

for decreases whose original sum total fund allocation totals about 50 i)ercent mere than the recommended cut. Management then selects the final tasks from the set of those submitted. This is to permit management judgment to be exercised, since certain tasks cannot be decreased in funding due to policy or other reasons, despite their rclatiA'c cost ineffectiveness. ]\ianagemcnt, howcAxr, is able to focus its attention on a much smaller area than would be the case if the entire Laboratory task list had to be reΛdewed. A similar p rocedu re is followed for budget increases.

To achieve maximum use of tlie program in its role as an aid to budget allocation, full management confidence will be required. This will be done by reviewing in depth the input data covering a randomly chosen set of selected and nonselccted tasks and comparing the recommenda­tions of the FY 66 R D E program with those of top Lalior-atory management. Previous reviews of this type re\'ealed a sufficient number of improperly prepared inputs to war­rant only limited use of the program as an allocation aid, as described prcA ' iously.

The program has proved to be an aid to planning a t all leΛ\•els, from task engineer to Laboratory Director, in that it closes the looj) from higher echelon guidance, to Λvork performed, to resu l t s achieved and estimated for the future. As Fig. 5 indicates, the program provides each task engineer con.<iderable background data to aid him in his program input ]n*eparation or '^jilanning," namely, the list and descri])tions of future Air Force weapon systems, the list of future facilities planned by the Laboratory, the list of. Laboi'atory Technical Devel-o])mcnt Olijectives, and the list of ])lanned Laboratory Advanced Develojnnent Programs.

Since so much data is assembled in conduction of the program, the resultant information retrieval capabilities liaA'e proved to l)e most useful in answering various ques­tions Avith regard to the Laboratory technical program. In addition, since reruns can be made using different ceil­ing budget figures, detailed effects on the program due to budget changes can be determined and j'lrintcd out by task Avith less than twenty-four hour's notice. Appendix I is a list of management studies conducted by the plans group during the first part of F Y 66 using R D E da t a ; Appendix II is a list of studies wliich RDE either has furnished in past fiscal years or has the capability to furnish for management use either internally or in re­sponse to higher headquarters request.

The capability to achieve detailed insights into the program at short notice has given rise to the initiation of development of an oΛ'erall Management Information Sys­tem for the Laboratory which λνίΐΐ complement the R D E program by proΛ'iding past resource consumption t rend data to key into the planning data obtained from the R D E program.

1965 ΝΙ'ΤΤ: R Λ: D KFFKCTIVKNKSS 109

Ta.sk Xn. __

Engineer

E E S O U R C E S C H E D U L E S U ^ O E V R Y λΥΟΙΙΚ SME i r r K D E F Y G7 Oct 05

FV

liosourcc Level 1

Cuntracl Dollar.^

Sui)p()rt 1 )c)llar.s

Total IJollar.s (JODO'.s)

l' η <i i η 0 vr ΛI a η y e; i r.<— C^ontract Monii()i'in<i'

Enjiineer Manyears— In-House Work

llesource E e v e l 2

Con trad Dollars

Support l)(.)llars |

Total Dollars (lODO's) j

Funtls—Other Afreneies* |

ADP ^lanpower**

OS 09 70 74

En^;ineer Ylanyears--Contraet Monitoring:

IC η μ; i η ee r Λ1 a η >- ea rs— In-House Work

liesouree Level ο

Contract Dollars

Support Dollars

Total Dollars (iOOO's) |

Enμ;ineer Manyear.s— ' Contract ]Monitorin^ |

En<!;ineer ^lanyears— i In-Hou.se Work i

Insert appropriate code letter in parenthesis after final amount in blocks A. Army B. N a v v C. NASA D. A D P (AVrile out title and number here) VJ. U S A F other than exploratory and Adv.anced Development F . Other—(Write out source here) Show that part of total manpower (contract ά in-house) e.xpended on Advanced Development Program if applicable. If not, leave blank

Fi<r. 6.

n o IEEE TRAXSACTIOXS ON EXGIXEEUIXG MANAGEMENT SliPTHMHKR

1 ^

ΐ \ ΐ:ττ: κ D ΕΙ·

The RDV] piOgi'mu does not eover expHeitly the aUo-cation of resoiii-ees for sevei'al major aspects of tlic Laboratory mission. Such items, as sdcction of new facili­t ies , i-esources in sui)port of current operational Air Force systems and certain Laboratory advanced development juOgrams (except for manpowiM'), and special ju'ojects directed at random times l)y higher li(\ad(ituirters are all outside the dii'ect scope of liDE except as i n d i c a t r d . These items, however, ai'c either few enough (in the case of facilities') to be handled by the direct exercise of m a n ­agement judgment or are of such a nattu'c that, s ince they are unpredictable in occurrence, they cannot 1)0 planned for in dc^tail.

The basic rationale for opei'ation of the ])rogram is that it should not lOijuire anything in the way of plan­ning that should not be done in a n y event. For i n s t ance , in the normal intuitive ])lanning cytde, project eng ineers in exploratoiy development should consider the needs of future high ])riority advanced Air Foi'ce sy s t ems , the aA'ailability of facilities that will be i-eciuii'cd for the re.-icarch, tht^ cui*rent s ta te -of - the-ar t of the technology under consideration, and all the o t h e r Λ'al·ial)les included in the RDE model. Sujiervisory Ic^-els above them should consider similar factors, but in a less detailed fashion. The bias that a project engineer shows in recommending his program for ado])tion in an intuitive planning en-v i iOnment is also considei'cd in the operation of liDE. Here this is handhnl l)y having the task eng ineers ' evalu­ations of the importance of their tasks also evaliuited by objective outside organizations. Jn the case of Rl^ly. these ai'c engineers from the Advanced System Anal\'sis Group, which is not ]xirt of the Flight Dynamics Labora­t o r y , and engineers from s e l e c t i M l advanced .-systems project offices. AVhere deviations of more than 0.2 from the selected A'alue are found, these differences are brought to the attention of the Laboratory Division of­fice concerned foi* adjudication witli the outside ol)jective eΛ'aluator. Relative to biases at Division lev(d, a study w a s made of t h e distribution of task apjilicability A'alues for the five Laboratory Divisions, and it was foimd that the curΛ'es were practically identical.

Finally, a few words should ])e said about the devel­opment of such a program. Since it is so closely tailored t o t he needs of the organization for which it is designed, care m u s t be taken to ensure that it I'cfiects as closely a s possible t h e judgment of the nnmagement involved. The evolution of the j^rogram reciuires thorough planning p r io r to execution and usage and should, in its initial stages, be used in parallel with the normal planning techniques used by the oi-ganization concerned.

The success the program h a s had is due in large part to the con t inu ing a n d pos i t ive interest of the Laboratory t o p management during de \ 'c lopment of the jMOgram. AVithout such an attitude, a program such as th i s is doomed to failure.

FECTIVKXESS 111

APPEXDIX I

SL'M.\L\KY LIST—FY 06 R13E STUDU:S TO DATE

1) Low ])riority task list—l)y Divisions. 2) Task selections for ^lay, 196Γ) l^udget cut. 3.) Reconnuended task allocation list for 1.4 million

F Y ()7 ]:5udg( t increase. 4) System/Task l^ayoff ])ackages—12 advtniced s y s ­

tems. ή) System/Task Payoff package—SST (for F A A K G) Information for (luarterly input to A F S C Lin^ited

^\^ar Book. 7) Identification of Laboratory task sup])Oi't to

]\IHTY advanccil systiMU (for Advanced D(.*\'el()[)-nient l^lan backup data).

8) hlentification of V \ ) \ j techniral areas in Avhicli othei' agencies (indir-try and go\'ernni(-nt) were doing significant amounts of \voik .

9) In])ut to the F D L AutomatiMl MaiiagiMuent In­fo rma t ion System Study,

10) Prediction of new Advanced Development Plans ( F D D .

11) Prediction of |)lamn'(i FDL usage of cm'rent and future FDL and existing other agency facilities.

12) Evaluation of Sh S I'c^view of FDL task contribu­tions to future systems.

13) Preparation of FDL uumpower vs. contract dollar cm've.

14) Tn])ut to cui'rent FDL manpower utilization sttnly. 15) Ojuimized F Y 66 and F Y 67 FDL (^\])loratoi'y d e -

A'eloj)inent program (comiiuter runs).

ArPEXDLX π

LxEOHMATiox RE'rKn:vAL S U M M A R Y — R l ) h ] OUTPUT

A. The following information will l)e availal,)le for eacli task: 1) Resource Rcijuircnients—5 years.

a) iMigr.s—C; Engrs—IH ; Fund.^—Contr ])lus Sujit.

b) 6 Resource Levels (using in-hou.<e/conti'act mix) keyed to authorized Ceiling M / Y , \ Ceiling Λ1/Υ, Twice Ceiling M / Y .

2) Rate of progress for varying resource levels. 3) Change in Advanced Systems Suppoit witli varia­

tion in Resource Levels. 4) Timeliness of task technology Avitli respect to Air

Force Advanced Systems requirements (FY 65-75).

5) Advanced Systems and AFFDL technical goals supported l)y each task, and level of the im­portance of this support.

6) Work required for each ΙΟΛΧΙ of task progress and resources required to accomplish work.

7) Progress achieved with past year funding.

il2 IEEE TRAXSACTIOXS OX EXGIXEERIXG MAXAGEMEXT SEPTEMBER

S) Ideiuificatioii of tasks rcquirino; support by other A F F D L tasks.

9) Brief deseri|)tion of task state-of-the-art. 10) Iiuproveuient afforded each advanced system sup-

])orted by task. 11) SPO coorcUnation of iinj^ortant task contributions

to systems. 12) Tvisk of task technical aj^proach. 13) Amoimt of work in task area l)y othei' organiza­

tions.

B. The folhnving assembled data will be collected:

1) Applicability of task technologies to general sys­tems support.

2) Effect on the Technical Program of additions or subtraction of resources to or from tlie Labora­tory's allocation.

3) Task Cost Effectiveness ]n'iority list. 4) Optimal resource allocation for given Laboratory

resources. Γ)) Advanced Development Pi'ogranis Lhtiation Dates

—Xew and Proposed.

C. Tn addition to the above, the Plans Gi'oup has the capal)ility to extract such data as:

1) Tasks wliich contril)ute to the various sectors of the corridor of flight.

2) l^elative Cost Effectiveness of new tasks and Ad­vanced DeΛ'elopment Plans as com]-)ared to the current AFFDL Technical Program.

3) Cost of providing accelerated support to specific advanced systems.

4) Cost of providing accelerated support to specific Laboratoi'v technical .o;oals.

5) Relative importance of tcclmical goals with re­spect to support of Advanced Systems.

6) Tasks limited l)y state-of-the-art progress. 7) PelatiA'o importance of AFSC preferred advanced

systems, 8) Facility requirements summary. 9) Task list flagged for Management Attention (5-

year nonselection). 10) Sunnnary of funds received from other agencies.

. \ C κ NOW Ε Ε I) G Μ Ε Χ Τ

Capt. Rol.)ert Rea, formerly of the Planning Staff. Flight Dynamics Lab., now with Clark Abt Associates, Boston, iviass., was responsible for the initial studies and devel­opment of the program. Lt. Tom Synnott, formerly an instructor in the Systems ]\ranagement Dept., Air Force Inst, of Technology, now Avitli the L"i"nited States Trust Co., Nevr York, N. Y., acted as consultant and deserves major credit for development of the mathematical tech­niques involved [1].

REFEREXCES

[1] E . ΙΓ. Rea and T. W. Synnott, 'Trojoet RDK, a framework for the f'onun-etionsion and analvsis of Research and Deve lop-mc]\t FffeciivfMiess," T M 63-22, Air Force Flight Dvnamics Lab., AVricht-Patrerson AFB, Ohio, O.totior 1963.

Γ2] IL IT. Weingarlnor, Μatlunvntiral Programming and ihe An(du.^is of Capilal Budqetiug i\-(d)Iems. Enulewood Cliffs, N . J.: Prenfice-Hall, 1963.

[3] R. L. Ackoff, Sciendfic Method; Optituizing Applied Research Deci^<:ions. New York: Wiley, 1962.

[41 G. G. Dantzig. Linear Programming anrl Extensions. Prince­ton. N . J.: Princeton Univ(^*sity Press, 1958.

[51 L. B. Chester, "Analysis of ihe eff(M*t of variance on linear proirrammina: problems," M.A. tliesis. Air Force Inst, of Teclmol.^iiv, Wrioht-Patterson A F B , Ohio, Seiilember 1964.