Technical Report 504 L

A DECISION SUPPORT FRAMEWORK

FOR DECISION AIO DESIGNERS

Ruth H. Phelps, Stanley M. Halpin, and Edgar M. Johnson

"DTjc*l L I".FEEO': 2 -0

HUMAN FACTORS TECHNICAL AREA

/

Re h s t U. S. Army

Research Institute for the Behavioral and Social Sciences

January 1981

Approved for public release; distribution unlimited.

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHA\VIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

L. NEALE COSBY

JOSEPH ZEIDNER Colonel, INTechnical Director Commander

Ill

NOTICES IDISTRIBUTION: Primary distribution of this report has been made by ARI.Please address correspondence concerning distribution of reports to: U.S.Army Research Institute for the Behavioral and Social Sciences, ATTN:PERI-TST, 5001 Eisenhower Avenue, Alexandria, Virginia 22333.

FINAL DISPOSITION: This report may be destroyed when It Is no longerneeded. Please do not return It to the U.S. Army Research Institute forthe Behavioral and Social Sciences.

NOTE: The findings in this rePort are not to be construed as an officialDOpartment of the Army position, unless so designated by other authorizeddocuments. •

UnclassifiedSECURITY CLASSIFICATION OF THIS PAGE MWhent Dat& Bnterei) '_REPOT DOU~tETAT• PAEREAD INSTRUCTIONS

REPORT DOCUMENTATION PAGE BEFORE COMPLETING FOR!I

1. REPORT NUMBER 2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

Technical Report 504 , '

4. TITLE (and Subtiotle S. TYPE Of REPORT A *F.MOD COVEAEr

A Decision Support Framework for Decision 1 FinalAid Designers

6. PC,_SMING ORG. REPORT NUMBER

7. AUTHOR(e) S. CONTRACT OR GRANT NUMBER(*)

Ruth H. PhelpsStanley M. Halpin

gEdar M. Johnson9. PERFORMING ORGANIZATION NAM. AND ADDRESS 1. PROGRAM ELEMENT. PROJECT, TASK

US Army Research Institute for the Behavio" .i AREA k WORK UNIT NUMBERS

and Social SciencesPERI-OS 5001 Eisenhower Ave, Alex., VA 22333 2Q162717A790

12. REPORT DATEI,. CONTROLLING OFFICE NAME AND ADDRESS US Army Research ER

Institute for the Behavioral and Social Sciences January 1981

(PERI-0S) 5001 Eisenhower Ave., Alexandria, VA 22333'S. NUMMER OF PAGES18

14. MONITORING AGENCY NAME & ADDRESS(It dffeent from Controlling Office) IS. SECURITY CLASS. (of thidse reot)

Unclassified

IS5. DECL ASSI FCATION/DOWNGRADINGSCHEDULE

1I. DISTRIBUTION STATEMENT (of thil F"Port)

Approved for public release; distribution vnllmited

17. DISTRIBUTION STATEMENT (of the abstract enterd in Block 20, It dilffrent from Report)

Same

It. SUPPLEMENTARY NOTES

19. KEY WORDS (Continue on r.vinee 4,ds if neceo•,,a and Identlf7 by block number)

Decision MakingDecision AidingDecision Making Structure

Z& ANSTRACT' (ta sift reverse 011' nIt eg• O ldE #~fl by block nmbe)A Decision Support Framework is presented (Mp1V-•) which serves two purposes:first, to organize and integrate various decision aids according to theirfunction, and secondly to provide the decision aid designer with a- systematiccontext in which to develop decision aids as well as examine which aspects ofthe decision problem would mont benefit from decision aiding. The main compo-neats of the framework are discussed in detail with Army intelligence decisionmaking examples: (1) analysis of the decision requirements; (2) development ofdecision aids to provide the decision maker with information as well as tools .........

DD _ i i03 mio. OF. MV 4515 OUS.ETM iM '- W3 UnclassifiedSSECURMTY CLASSIFICATION OF THIS PAGE (Ph.m Date Entered)

Z/(/ rr

uclasaifiedStCUnITY CLASIWFICATION Of TI4S PAG••(llan Dati EXMIlo

Block 20.

for evaluating, weighting and integrating t.:he information to make a decision;and (3) evaluation of the success of the decision aids in leading to a logical,rational decision.

I!

11

I

iismC;~~Ty Unclassified

SEUIYCLASSIFICATION OF THIS PAOS(W1Ief Data Nntered)I

IS!

Technical Report 504

A DECISION SUPPORT FRAMEWORK

FOR DECISION AID DESIGNERSAccession For'NTIS -GA

DITIC TAB

Ruth H. Phelps, Stanley M. Ha!pin, and Edgar M. Johnson Uum : ... _

, .Dizt

Submitted by:Stanley M. Halpin, Acting Chief

HUMAN FACTORS TECHNICAL AREA

Approved by:

Edgar M. Johnson, DirectorORGANIZAT;ONS AND SYSTEMSRESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for PersonnelDepartment of the Army

January 1981

Army Project Number Intelligence Systemc2Q162717A790

AODrOved for Publ•c raem: distribution unlimitd.iii

|4

ARI Research Reports and Technical Reports are ititended for sponsors ofR&D tasks and for other research and military agencies. Any findings readyfor i~mplementation at th~a time of publication are presented in the last part

i•hs Brief. Upo: "ompketion of a major phase of the task, formal recom-.nendaitions for o. ..#.,dl action normally are conveyed to appropriate militaryagencies by briefing or Disposition Form.

ivi

I1

ii!H

ivi

* . ,

FOREWORD

The Human Factors Technical Area is concerned with the demands of the

iuture battlefields for increased human-machine complexity to acquire, transmit,process, disseminate, and utilize information. Research is focused on theinterface problems and interactions within command, control and intelligencecenters and is concerned with such areas as topographic products and procedures,tactical symbology, user-oriented systems, information management, staffoperations and proceduxres, and sensor systems integration and utilization.

One area of special interest is the development of procedures to support

and enhance the decision making process within command, control and intelligencecenters. The current effort summarizes a framework which both identifiesthe requirements for developing and evaluating decision aids and organizes de-cision aids according to the functions they serve for the decision maker. Thisframework thus provides a concise way of categorizing already existing de-cision aids as well as provides guidelines for developing and evaluating newaids. Examples drawn from the areas of Army command and control and intelligenceS~are used to demonstrate the usefuiness of the framework.

Research irn decision aiding is conducted as an in-house effort with addi-tional support from contracting organizations which are selected for theirunique contributions to this area. This effort is responsive to the require-ments of Army Project 2Q162717A790,

XISEP ID ERe'lhnic 16D rector

Iv

7P

• , .. .. . _ .. .... . .. .. . . .

A DECISION SUPPORT FR".MFWORK FOR DElISION AID DESIGNERS,

BRIEF

Requirement:

To develop and demonstrate a framework for the developers of decisiunaids for organizing and categorizing decision aiding procedures.

Approach:

The development of the framework was based on a study of existing de-cision aids, identifying differences and similarities in the purpose, de-cision, execution, and evaluation of the aids. The largest differences werein the service provided to the decisionmaker, e.g., provide information orlogical reasoning support.

Product:

The framework developed (a) lists the steps a decision aid designershould ideally complete in the full development and implementation of theaid or aiding system, and (b) distinguishes two categories of aids, thosethat provide information and those that provide support for logically andrationally evaluating amd integrating information in making a decision.See Figure I for a schematic of the framework. Amny command and control ex-amples of the two types of decision aids are summarized to help explain theuse of the fraim;work as well as distinguish the types of decision aids.

Utilization:

This framework should provide decision aid designers with an outline ofsteps to be followed in the cycle of development, from initial conceptuali-zation and implementation to evaluation and revision of the aid. Specialattention is devoted to the definition of requirements prior to aid develop-ment and an evaluation of the completed aid, two areas that rarely are givenadequate weight in aid development. Use of the framework by system develop-ers, both of large aut6mated systems and small manual procedures, could en-hance the useability of their systems and procedares since applying theframework requires a thorough study of the decisionmak-r's requirements andconstraints as wall as identification and categorization of the decisionaids to be developed. Since the framework is strongly based on psychologi-cal principles, it also provides an alternative to the more popular systemsand engineering perspectives on decision aiding.

viiO

A DECISION SUPPORT FRAMEWORK FOR DECISION AID DESIGNERS

CONTENTS Page

Page

INTRODUCTION ........................................ ....... . 1

DECISION SUPPORT FRAMEWORK .... ..................... ............ 2

DECISION AIDS ............. ............................ 4

Information Aids ................. ........................... 4Integration Aids ................... ......................... 5

EVALUATION ................. .............................. 8

Validity and Reliability 9Flexibility............................. 9Improved Decisions .... ....................... ............. 9

CONCLUSIONS ...................... ............................ .. 10

REFERENCES ................... .............................. .. 1]

DISTRIBUrrION ................... ............................ ... 13

LIST OF FIGURES

Figure 1. The Decision Support Framework ...... ............... 3

- -.-, w-- N

A DECISION SUPPORT FRAMEWORK FOR DECISION AID DESIGNERS

INTRODUCTION

The need for decision-making support is being increasingly recognizedin a variety of disciplines: military (Kibler, Watson, Kelly, & !'helps,1978; Levit, Alden, Erikson, & Heaton, 1977), public policy making (Hammond,Roirbaugh, Mumpower, & Adelman, 1977), land management (Gardiner & Edwirds,1975), medicine (Fryback & Thornbury, 1978), and oil exploration (von Winter-feldt, 1976). Decision aids are currently under widespread development, butlittla effort has been devoted to examination of the various functions de-cision aids fulfill or to development of a system or framework for coordi-nating related decision aids.

The purpose of this paper is to present a preliminary Decision SupportFramework which is intended to be used by the designers of decision supportsystems to organize and integrate different types of decision aids into aunifying system. The framework is both a descriptive tool to clarify the Irelationship among decision aids and a type of aid itself. It will providethe decision aid designer with a systematic context in which to develop aidsas well as to examine which aspects of the decision problem would most bene-fit from decision aiding.

Before presenting the decision support structure, however, a few gen-eral comments concerning the nature and function of the type of decisionsupport being discussed will help to define the scope and limitations ofthis paper. The role of decision support is to increase the range of a de-cisionmaker's capabilities to make a rational decision. Such a function isaccomplished by providing a decisionmaker with an informational base, aswell as organizational, computation, and psychological tools for making alogical decision based on that information. Implicit in this role are twoassumptions: (a) Decision support is used when human judgment is a criti-cal element, and (b) decision support in no way replaces the decisionmakeras a problem solver. By definition, these are tools to support the humanjudgment and decision-making process.

The decision support framework presented in this paper encompassesthe various functions decision aids may serve, the relationship among dif-ferent aids, and an evaluation of those aids. While the framework is dis-cussed in generic terns, clearly the -tructure and configuration of specificdecision aids will depend on the nature of the decision problem, the typeof information required, and the consequences of the decision. Corversely,while the examples of different types of decision aids will be drawn fromArmy tactical intelligence decisionmaking, the classes of aids are intendedto be general and apply to a variety cf JIcision-making contexts.

The remainder of the paper is organized into three main sections:S(a' an overview of the Decision Support Framework; (b) a more detailed

discussion of functions of and relationships among specific classes ofdecision aids; and (c) factors to be included in an evaluation of the de-cision aids. .1. '

S. ~rT

DECISION SUPPORT FRAMEWORK

In Figure 1 is a diagram of the Decision Support Framework. This frame-work is designed to depict the decision support components, not the decisionprocess itself. The pivotal point of the framework is of course the actualDECISION to be made. Based on an analysis of that decision the REQUIREMENTS,or the information and tools needed, to make the decision are defined. Suchrequirements may include type of information, timeliness of information, datacomputations needed, and how the information should be organized, integrated,and evaluated. Lists and taxonomies such as those generated by Brown andUlvila (1976) may be useful in determining the REQUIREMENTS for the specificdecision problem. The importance of this analysis should not be underempha-sized since it is the basis for determining the content and nature of thedecision aids. In addition, clear and systematic analysis of the require-ments is critical for insuring the validity of the EVALUATION of the decisionaids.

Depending on the requirements established, the types of decision aidsneeded are determined. At the most basic level, there are two types of IN-FORMATION AIDS. Data-based aids make available to the decisionmaker the dataon which the decision is based; these aids can bent be viewed as automateddata banks which make selected raw data, or perhaps summarized data, avail-able to the decisionmaker. Data aids may select the data for a decisionmakerbased on either predetermined or user-specified criteria. Calculation aidsprovide the decisionmaker with the results of statistical computations orother mathematical computations such as numerical changes in the data bases,distances, velocities, objective probabilities. Regardless of its form, the gIoutput of both aids is information or selected data.

In many, perhaps most, situations the decisioiamaker may feel overwhelmedwith the sheer volume of information available and have great difficulty inselectinq the most relevant information and/or making a logical evaluation ofthat information. INTEGRATION AIDS are sets of procedures designed to help adecisionmaker logically evaluate and integrate the information provided byINFORMATION aids. The composition and organization of specific aids dependson the characteristics of the decision problem as well as the psychologicaldifficulties known to influence this class of decisions. Thus, INTEGRATIONaids can serve a single or a combination of functions, such as those listedin Figure 1: organizing and structuring the information, helping to ovc:.omejudgmental and cognitive limitations and biases such as faulty memory, andsimplifying the evaluation and weighting of the information. The goal ofevery INTEGRATTON aid is to help the decisionmaker arrive at a logical, ra-tional DECISION, not to replace the decisionmaker.

Once the DECISION has been made, an EVALUATION of the various aidswithin the decision support framework is necessary for the decision aid de-signer to determine if in fact they did *ielp the decisionmaker reach a ra-tional decision. As shown in Figure 1, such an evaluation should considerat least the validity and reliability of the aid as well as its flexibilityand the degree to which it led to an improved decision.

In summary, the proposed Decision Support Framework is deiigned to helplogically define and organize the decision problem, describe the classes ofnecessary information, and provide a manageable structure for evaluating

2

I1

:c .1 P-

- '.4

u '-

me

various decision aids. The following sections of the paper will focus on amore detailed discussion of two types of decision aids, INFORMATION and IN-TEGRATION, and their EVALUATION.

DECISION AIDS

The purpose of decision aids within the Decision Support Framework isto provide the decisionmaker with data on which to base a decision or withhelp in evaluating the data for a specific oroblem. While the two classesof aids, INFORMATION and INTEGRATION, are indeed based upon different pur-poses, such a distinction is not always cle-r in practice. The presentframework defines the class of aid with respect to its function for the de-cisionmaker. Thus, although an aid may integrate, summarize, or performsome algorithm on the data, if the output of the aid is a piece of informa-tion or a restructuring of information it i.- considered an INFORMATION aid.Only when an aid helps the decisionmaker perform the summarization or evalu-ation of the information is it classified as an INTEGRATION aid. Such adistinction is iustified by the focus of the Decision Support Framework onthe actual decisionmiker as the reference or pivotal point, rather than the

data or other environmental demands.

Informational Aids

Clearly no rational decision can be made without data or information.However, in many situations decisionmakers feel they do not have the type,accuracy, or timeliness of information used to make a rational decision.As shown in Figure 1 two of the many possible functions INFORMATION aidscould serve are to provide data and calculations on those data to the de-cisionmaker. However, since the needs for INFORMATION aids will totallydepend on the specifics of the decision problem, only a few general com-ments and an example will be presented. DATA aids are designed to fulfillthis function, usually by automating a data base such that large amounts ofdata can be stored and readily retrieved by a decisionmaker. Data may bestored as individual items of information or summarized over some specifiedtime period or level of detail. CALCULATION aids provide additional infor-mation by making computations on the raw data such as velocity, percentages,or even moie sophisticated algorithms involving matching templates or statis-tically optimal solutions. However, for both DATA and CALCULATION aids theoutput of the aid is information which then must be evaluated and integratedby the decisionmaker.

Example: Information Aid. The U.S. Army Research Institute has devel-oped a Graphic Movement Analysis Aid (GRAMA) to provide an Army tacticalcommander's staff with decision support. The specific decision problem isto determine which avenue of approach the enemy is most likely to select.The purpose of GRAMA is to identify those routes which minimize travel timeand therefore are most likely. Specifically, GRAMA will help an intelli-gence analyst answer questions such as: "How long will it take an enemyunit to move from location A to location B, and what route will they follow?"A similar question may be asked concerning the optimal routes for as many as10 units simultaneously moving from 10 locations to 10 destinations. Toperform the calculations necessary for these questions, the computer

41

algorithms require data on: (a) the road network--what is connected to whatby what type of road; (b) the conditions of the move--day/night, wet/dry,vehicular/foot travel; (c) the types of units involved--how fast can theymove given the conditions and the roads. GRAMA has a preloaded network ofup to 600 points and their interconnections, and a "Speed" table which de-fines how fast an average unit can move under a combination of road and en-vironmental conditions. The main interactive features of the program allowthe user to work within this structure. Nodes may be added or deleted fromthe network, or specific subnetworks of interest within the larger networkmay be defined.

Thus, the output of GRAMA is a rank ordering of various avenues of ap-proach based on the available data concerning environmental conditions,enemy units, etc. The commander's staff must now incorporate this withother information in determining the most likely enemy avenue of approach.

Integration Aids

The general purpose of INTEGRATION aids is to help a decisionmakerevaluate information from data aids, calculation aids as well as othersources, and then to integrate that information to make a rational decision.INTEGRATION aids are designed to help a decisionmaker ultimately relate allthe relevant information to the various decision alternatives under consid-eration. Therefore, definition of decision alternatives is a critical pre-liminary step to the use of INTEGRATION aids. While a decisionmaker mayrequire help and thus one aspect of aiding may be devoted to alternativedefinition, it is assumed for the present framework that the alternativeshave been defined either in the decision REQUIREMENTS or as a result of theINFORMATION aids.

There is ample evidence from psychological literature that severalfactors degrade the quality of decisions which are based on even moderateamounts of information from single or multiple sources. The primary func-tion of INTEGRATION aids is to help overcome or circumvent these factors.For any one decision problem the degrading factors must be identified andthe aids developed to take into account those factors. While identificationof the degrading factors requires considerable analysis and research on thespecific decision problem, knowledge of the factors already known to inter-fere will help focus such efforts. Since an exhaustive list is beyond thescope of this paper, reference should be made to fairly complete descriptionsavailable elsewhere (Tversky & Kahneman, 1974; Hogarth, 1975; Lichtenstein,Fischhoff, & Phillips, 1977; Slovic, Fischhoff, & Lichtenstein, 1977; Fisch-hoff & Slovic, in press). Three categories of factors which influence adecisionmaker's integration of information are listed in Figure 1 and brieflysummarized heren T

Structure of Information. The output of INFORMATION aids, in additionto other sources, will make available to the decisionmaker a large volumeof data, summaries of data, or calculations on which to base a decision.Perhaps the most basic function an INTEGRATION aid can perform is to organizeand structure this potentially overwhelming mass of information. Such struc-turing serves at least two functions: It allows all information to be cate-gorized such that none is unintentionally overlooked and it also encourages

5 1

a morf' logical evaluation of information which varies'in type as well assource. The most popular structuring techniques are based on decomposingthe decision problem into a series of distinct categories of factors. Thesefactors may be exclusive of each other, may be hierarchical in nature, i.e.,some are subsets of others, or some combination of the two. Once such astructure is formed, the information can be sorted into the defined cate-gories. Alternatively the structure can be used to determine which infor-mation needs to be gathered to arrive at a logical solution.

Weighting of Information. One of the most difficult tasks after theinformation has been organized into categories or factors is to assess whichinformation is relevant for making a particular decision. The basis for de-termining relevance is diagnosticity; that is, the information that is mostrelevant is that which differentiates or discriminates among the optionsbeing considered. For example, a decisionmaker is selecting a new car andthe information obtained about the potential cars is categorized into twofactors: gas mileage and cost. If the three cars A, B, and C all get 30miles per gallon, then gas mileage is not diagnostic because it does notdifferentiate the three cars under consideration. However, if the pricesare A = $3,000, B = $4,000, C = $5,000, then price is a highly diagnosticfactor since it does differentiate the options. The psychological concept ]of diagnosticity is often very difficult for decisionmakers to consistentlyapply. In the above example, it may be hard for a decisionmaker who is en-ergy conscious to accept that gas mileage is not relevant when deciding amongthe cars A, B, and C. The difficulty seems to be in focusing on the factorswhich discriminate the specific options, and ignoring factors which in gen-eral are important but not discriminating in this decision problem.

If, as in many cases, the factors are all diagnostic, then the decision-maker is faced with assessing the relative diagnosticity or relative weightof the factors. Procedures for assigning relative weights, based on thediagnosticity of the factors, can vary from a simple rank ordering to moresophisticated assignments of normalized subjective probabilities. In mostcases, however, numerical values are assigned which correspond to the per-ceived relative weights. While there is no technique that can guarantee toplace the factors in optimal order, psychological INTEGRATION aids are de-signed to help a decisionmaker determine which factors are diagnostic, thenassess the relative impcrtance of those factors.

Judgmental and Cognitive Biases. Psychological research has documentedseveral judgmental biases which can severely degrade the quality of decisions.It is very important to recognize that these judgmental biases are not inten-tional and are so pervasive and compelling that training and instruction de-signed to help decisionmakers overcome specific biases have largely been in-effective. Because training has failed, it is even more critical that Ldecision aids be designed to compensate or help minimize the impact of thesejudgmentally degrading biases. While it is beyond the scope of the presentpaper to discuss them all at length, a few will be briefly outlined to demon-

strate the need to address such factors when designing decision aids.

One biasing factor is the illusory correlations which are inferredabout the relationship of two events A and B. Because A and B are observedto co-occur a number of times, they are believed to be highly correlatedw'th each other. However, simple co-occurrence is not sufficient evidence

6

S. . .... .. ..

to infer correlation. It is necessary to consider how often B occurs and Adoes not. For 'xample, if high gas mileage cars are also high in price, apositive correl4tion might be assumed. What has not been considered is thenumber of high gas mileage cars with low price and the low gas mileage carswith high price. Decisionmakers apparently can readily search for positiveinstances of the relationship between two or more variables, but have dif-ficulty in either searching for or assimilating negative instances.

Decisionmakers also have miscon':eptions concerning probabilistic infor-mation. The probability of rolling an even number using a fair die is .5;the gambler's fallacy is the misperception that if a number of rolls occurswithout an even number that the likelihood of one occurring soon is increased.It is commonly said that the event in question "is due." The decisionmakerfails to consider that the probabilities are based on thousands of rolls anda series of 10 rolls will not necessarily exhibit the same proportion of oddand even rolls. It is illogical to expect that an observed low or high fre-quency will be compensatory within a small sample.

Decisionmakers also have a t adency to inaccurately recall the confi-dence with which they made a decision after the decision outcome is known.In such hindsight biases, it appears that once the outcome is learned, de-cisionmakers unintentionally distort their perception of their predecisionprocesses. A related problem is the feeling that when presented with newinformation, the decisionmaker sees the information as obvious, incorporatesit, then fails to recognize that the information was not known previously.Both of these tendencies have been dubbed the "i knew it all along" effect.The danger in such a bias is that what a decisionmaker can learn from thedecision outcome or the new information is severely inhibited.

When aids have been constructed to help structure and weigh informa-tion, as well as minimize judgmental biases, cognitive limitations willalso be largely overcome. Simply by providing an explicit structure andmaking weighted information and necessary calculations readily retrievableby the decisionmaker, limitations such as memory capacity can easily beavoided. Thus, a decisionmaker is relieved of the burden of rememberingall the information and evaluations of that information. In addition memorybiases such as the disproportionate ease in recalling first and last infor-mation obtained will also be avoided.

Example: Integration Aid. A very critical decision problem for anArmy tactical commander is to select the most advantageous course of actionto pursue in attacking enemy forces, terrain, or cities. That is, whatroute will most quickly allow the commander's forces to reach their desti-nation and accomplish their mission with the fewest forces, least equipment,and fewest losses. Such a decision has obviously grave consequences formany people as well as the comnander. The problem is complicated by notonly these consequences but by the extreme time pressure under which theinformation is gathered, evaluated, integrated, and a decision made. Acomputerized (IBM 5100) INTEGRATION aid has been developed by the U.S. ArmyResearch Institute (Kibler et al., 1978) to help a commander both focus at-tention on and critically evaluate relevant information. The aid has beennamed TACVAL since its function is to aid in tactical evaluation of alter-native courses of action.

7

The structure of TACVAL is based on multiattribute utility theory,

which requires that the decision problem be broken down into its contribut-ing component parts or factors. Thus, the first step in designing the aidwas to identify and define an exclusive, but not exhaustive, set of factorswhich would encompass the most relevant information for selecting coursesof action under most circumstances. There are a total of 24 factors groupedinto 5 categories organized into a two-level hierarchical structure.

To use TACVAL, a commander, or more realistically a member of the com-mander's staff, first defines the courses of action to be evaluated. Then,each course of action is independently evaluated on each of the 24 factors.Such evaluations involve assigning a numerical score to both the value ofeach alternative and the relative weight or diagnosticity of each factorfor making thib particular decision. After all factors are evaluated vari-ous calculations are then performed based on the user's inputs and prepro-grammed algorithms. Two of these calculations are a normalized weightedaverage score for each course of action indicating the user's relative pref-erences among the alternatives and a sensitivity analysis which identifiesthe factors which are most sensitive to changes in the user's evaluation.In addition, there are options for editing the evaluations based on new in-formation or reevaluation of old information and listing the relative impor-tance of the 24 factors.

In terms of the functions of INTEGRATION aids listed in Figure 1,TACVAL performs at least the following: (a) The user searches, requests,and organizes the available information based on the 2-level hierarchical,24-factor structure. Thus all the relevant information is systematicallyarranged and attention to irrelevant information can be minimized. (b) Thereare explicit instructions and scoring conventions built into the aid whichhelp the user in a step--by-step fashion to evaluate the factors and assignnumerical weights. Problems in determining factor diagnosticity are reducedby explicit elicitation procedures. (c) Judgmental biases are also minimizedby providing the user with a hard copy of all the factors and evaluationsof the alternatives on those factors. Hindsight biases are eliminated sincethe user has a complete record of all evaluations, updating, reevaluations,and resulting calculations. (d) Coqnitive limitations such as memory over-oad are minimized since all evaluations and factors are explicit and infor-.ation is organized within the framewor,,. In addition since each factor is

evaluated independently, the user need only be concerned with informationrelevant to that factor and need not be concerned with recalling other fac-tors or information. Finally, psychological problems in combining the evalu-ations and weight of the factors are eliminated since they are arithmeticallycomputed.

iA

"EVALUATION

The purpose of the EVALUATION component of the Decision Suppoyt Frame-work is to provide critical feedback for the decision aid designer concerningthe success of the aids. In this context it is not the correctness of theDECISION that is assessed, but the value of the decision aids in making arational decision. As shown in Figure 1, the EVALUATION step serves atleast three fLuctions, each of which will be briefly discussed.

: ~8 I

Validity and Reliability

If the factors selected do not allow critical information to be in-cluded in assessments or the input judgments di not accurately reflect thefeelings of the decisionmaker, then the output of the aid will be misleae-ing. Validity assessments should include extensive consultation with ex-perts to assure the factors, their labels, and e.efinitions do in fact accu-rately capture the scope of the decision problem. In addition the judgmentson those factors must reflect the true opinions of the user. While evalu-ation of the validity of input judgments can be elusive, checks such asinternal consistency of judgments and questioning of the user should be at-tempted. Measures of the reliability of the user's judgments are also neces-sary, not so much to assess the user, but to be sure the procedure unam-biguously elicits the user's responses. Experimental validation of bothfactors and judgments could include comparisons across decision problemsand decisionmakers as well as variations in the factors and information.

A related validation issue is the credibility of the decision aidstructure, process, and output to the decisionmaker. While the decision-maker may not be the best expert on psychological biasing factors or infor-mation organization, the ultimate fidelity of the aid dependýs on the decision-maker's acceptance. Thus, efforts 'should be made to assess the user'sbeliefs and confidence in the aid as well as abilities to make valid judg-ments of the information.

Flexibility

Since most decision situations are not completely static, a decisionaid can never include all factors which may be of conceivable importance inunforeseen circumstances. Therefore, it is necessary to assess the flexi-bility of the decision aid to incorporate unique information, factors, orevents. One possible approach to-evaluate flexibility is to present a userwith unexpected but obviously important information, then assessments ofhow and to what degree the information is incorporated into the decisionaid structure can then be made.

Improved Decisions

While there is agreement from many disciplines that aids are necessaryto support complex and consequential decisionmaking, there is very littlereported evidence that decision aids do in fact lead to improved decision.Perhaps the most basic reason for this failure to evaluate decision aids isthe lack of a criterion for assessing the quality of dicisions. In somecases the actual consequences are too far removed krom the decisions, whilein others there is no realistic context in which to test the decision aids,e.g., a tactical commander at war. However, in the bulk of the cases, anobjective criterion could certainly be developed even if only in hypotheti-cal or contrived contexts. The crux of t!ie evaluation issue lies in opera-tionally defining the DECISION and REQUIREMENTS before the aids are evendeveloped. Without clear objective definitions of the purposes, goals,and the decision itself, meaningful measures of the contribution of thedecision aids to a rational decision cannot be made. Assuming operational

9

measures have been developed, experimental comparisons of decisions anwewith aids, without aids, with various modifications, etc., should be madetc assess the contribution of the aids to making a rational decision.

As a final point on evaluation, it should be noted that in some situ-ations the evaluation should include m-. lures other than the rational de-cision. Based on the analysis of the decision REQUIREMENTS, perhaps mea-sures such as the type and amount of information used, or the degree towhich different decisionmakers resolve conflict about the decision, etc.,may also be valid evaluation criteria. At times a decisionmaker is justas concerned with how the decision is made as with the decision itself.

CONCLUSIONS

The Decision Support Fre".tework categorizes and relates different typesof decision aids based on the functions they perform for a decisionmaker.However, beyond its use as an organizational tool, it requires the decisionaid designer to carefully analyze the decision to determine the requirementsfor aiding the various functions which must be performed by the decision-maker, as well as operationally define the decision criteria for validevaluation of the decision aid. In short, using a complete framework suchas presented here serves as a guide to help decision aiders develop logicaland useable decision aids.

0k

10

REFERENCES

Brown, R. V., & Ulvila, J. W. Selecting analytic approaches for decisionsituations, A matching of taxonomies. (DDI Technical Report 76-10).McLean, Va.: Decision and Designs, Inc., October 1976.

Fischhoff, B., & Slovic, P. A little learning: Confidence in multi-cuejudgment tasks. Attention and Performance, VIII, in press.

Fryback, D., & Thornbury, J. Informal use of decision theory to improveradiological patient management. Radiology, 1978, 129, 385-388.

Gardiner, P. C., & Edwards, W. Public values: Multiattribute utility mea-surement for social decision making. In M. F. Kaplan F S. Schwartz(Eds.), Human judgment and decision processes. New York: AcademicPress, 1975.

Ham•wond, K., Rohrbaugh, J., Mumpower, J., & Adelman, L. Social judgmenttheory: Applications in policy formulation. In M. F. Kaplan & S.Schwartz (Eds.), Human judcnment and decision prccesses in appliedsettýis. New York: Academic Press, !977.

Hogarth, R. M. Cognitive processes and the assessment of sukjective prob-ability distributions. Journal of the American Statis' ial Associ-ation, 1975, 70, 271-289.

Kibler, A., Watson, S., Kelly, C., & Phelps, R. A prototype aid for evalu-ating alternative courses of action for tactical engagement. (ARITechnical Report TR-78-A38). Alexandria, Va.: U.S. Army ResearchInstitute for the Behavioral and Social Sciences, November 1978.

Levit, R., Alden, D., Erikson, J., & Heaton, B. DevelopAment and applica-tion of a decision support aid for control of battlefield operations:A preliminary evaluation of a decision support complex in SIMTOS.(ARI Technical Report TR-77-A3.) Alexandria, Va.: U.S. Army ResearchInstitute for the Behavioral and Social Sciences, March 1977.

Lichtenstein, S., Fischhoff, B., & Phillips, L. D. Calibration of proba-bilities: The state of the art. In H. Jungermann & G. de Zeeuw (Eds.),Decision making and change in human affairs. Amsterdam: D. Reidel,1977.

Slovic, P., Fischhoff, B., & Lichtenstein, S. Behavioral decision theory.Annual Review of Psychology, 1977, 28, 1-39.

Tversky, A., & Kahneman, D. Judgment under uncertainty: Heuristics andbiases. Sciences, 1974, 184, 1124-1131.

von Winterfeldt, D. Personal communication, Seventeenth Annual BayesianConference, Los Angeles, Ca.. Feb. 15-16, 1976.

£If;11

DISTRIBUTION

I US AmMY WESTERN COMMAND ATTN$ APPLI DEPANTMENT OF THE NAVY TRAININ3 ANALYSIS AND EVALUATION GROUPI HOUA ATTNI OAAG-EDI HOt ICATA ATTN. ATCAT-OP-W2 HOUA RESEARCH AND STUUIES OFCI MILITARY OCCUPATIONAL DEVELOPMENT UIV DAPC-MSP-Os RM 8S2C HOFFMAN BLDG 14 OASO (NRA ANC LlI HODA ATTNi DANO-RORI HO TcATA TECHNICAL LIURAMYI HODA OOCSPERI USAAuCOI STCI HQOA ATTNI DANI-ISII USA AVIATION SYSTEMS COND ATTN. DRSAV-ZDAI USA cORAOCOM ATT4: AMSEL-PA-RHI USA aRRADCOM ATTN: ATFE-LO-AC1 HEADwUARTERS9 US 4ARINE CORPS ATTN$ CODE MP1-aO2 US AwMY EUROPE AND SEVENTH ARMYI IST INFANTRY DIVISION AND FT. RILEY ArTNI AFZN-UPT-TI USA iNTELLIGENCE AND SECURITY COMMAND ATTN$ IAOPS&TNG-T2 No Ta4ADOC TECHNICAL LINRARY1 NAVAL TRAINING EQUIPMENT LEN ATTN: TECHNICAL LIWRARYI MILIIARY OCCUPATIONAL DEVELOPMENT UIRECTORATE ATTNI ATZI-NCR-MS-M, RH 3N33 HOFFMAN BLDG 1II DATA ANALYSIS DIVISION ATTNt ATZI-NCR-MDo HOFFMAN SLOG 1II USA @ILPERCEN ATTNS DAPC-POO-TI USAFACFA CHIEF, ORGANIZATLONAL .FFECTIVENESS *RANGHI 8TH INFANTRY DIVISIONI HODA ARMY FORCE 400ERNIZATION COORD14ATION OFPICEI NAVAL AIR SYSTEM COMNANU /I OCSOWS (DIST 41) ATTN; UANO-RQI1 1230 USARCOM RESERVE CENTER1 US AmNY SOLDIER SJPPORT CENTER1 DIRELTORATE OF AR4OR AVIATION ATTN: ATZK-AADI USAAWMC * FT. KNDX AVIATION UIVISIONI USA tORCES COMMAND AFIN - DEPUIT C OF S FOR INTELLIGENCEI USA fORCES COKMAND AFOP - DEPUIV CHIEF OF STAPF FOR OPERATIONSI US AmMY AIR DEFENSE SCHOOL ATTN$ ATSA-DruI DIRECTORATE OF TRAINING ATTNI ATZQ-T1 DIRELTORATE OF C04BAT DEVELOPMENTS ATTN. ATZQ&OI HQOA4COM MARINE CORPS LIAISON OFCI OEPAwTMENT OF THE ARMY US ARMY INTELLIGENCE # SECURITY COMMANDI' US A"MY SAFETY CENTER ATINI LIbHAHIAq4 ALOG 4905I USA MISSILE COMMA40 ATTN: ORSMI-NTNI US AwMY CECOM ATTNi DRSEL-ATOD1 USA oORCES COMMAND1 PN TwADE ;I US MILITARY DISTRICT OF WASHINGTON OFC OF EQUAL OPPORTUNITYI NAVAL CIVILIAN PERSONNEL COMO SOUTHERN FLO DIV

22 ARI LIAISON OFFICEI ?TA ^RMY TRAINING COMMANDI HQ UbAREUR ATTN: DCSOPSI HQOA. OCS FTUDY OFFICEI U.S. NAVY TRAIN146 ANALYSIS EVALUATION GROUPI USACuEC ATTN% ATEC-EX-E HUMAN FACTORSI USAF*GOS/TAC SENIOR ARMY ADVISORI INTEw-UNIV SEMINAR ON ARMED FORCkS * SOC1 USA tLECTRONIC PROVING GROUND ATTN: STEEP-MI-ES1 OASA (RUA) DEPUTY FOR SCIENCE AND TECHNOLOGYI OFC uF NAVAL RESEARCH /1 AFHRL/LRTI AFHRL/LRLGI AIR tORCE HUMAN RESOURCES LAN ATTNI AFMRL/TSR •tIRJxi • ULa-4O ]rL )

,- . - . .. .-

I FEDEggAL AVIATION AOMINISTRAT16N CENTRAL REGION LIBJRApy, ACE-661 AFANa'L/bd1 AFAMkL/HEI NAVAL PERSONNEL R ANU U CtNTEI4 COMMA~40 AND SUPPORT SVSTEMSI NAVY PERSONNEL R AND 0 CENTER /*

1 NAVY PERSONNEL R AND D CENTER UIRLCTOR OF PROORAMS

1 NAVY PERSONNEL R AND D CENTERI US AR4MY AVN ENUiINEERING FLIGH'r ACTIVITY ATTNI DAVTE-Tfl2 OFC uF NAVAL WLSEANCH FLHSONNFL AND TRAININGi RESEARCH PR06RAMSI NAVAL PEPSONNEL R # UI CENTLR/1 OFC uF NAVAL RLSLARCm PRUJECT UtFICEi4, ENVIRONMENTAL PHYSIOLOGYI DEPT, OF NATIONAL DEFENCE DEFENCL ANJ) CIVIL INSTITUTE OF ENVIR MED1 NAVAL AEROSPACE MEUICAL NSCH LAU AEROSPACE PSVCHOLOGY DEPARTMENT

I USA INAUOC SYSTEMS ANALYSIS ACTIVITY ATTNt ATAA-TCA

I HEADw~UARTERS9 CUAST LiUARU CHIEl'9 PSY(GHOLOG1C#-ý RSCM O~RI USA kESEARCH ANU TECHNOLOO~Y LAS /I USA tNG1NEER TUP03HAPH'C LAOS ATTN: E7L-GSL1 USA tNGINLER TOPOGRAPHIC LAU$ ATTN: STINFO CENTERI USA tNCINEER TUPOGRAlFHIC LAOS ATTN: ETL-TO-SI USA mOBILITY EWiU1DMENT R AND U '-OMU ATTNI RORME-TQ (SCHOOL)I NIGHI VISION LAO ATTN; UHSEL-NV-SUUI USA IRAINING BOARD AITNI ATTG.ATd-TA1 USA mUMAN ENGiINELRIN5i LAUI USAHkL LIAISON WEPs USAAVNC1 USA mATERIEL SYSTEMS ANALYSIS ACIlVITY ATTN: CRXSYýCI USA KESEARCH OFC /II NAFEL HUMAN ENGIJEENIN6i OHANCtiI b3AITeLLE-COLUMbijS LAbOWATURILS TACTICAL TECHNICAL OFCI USA .aRCTIC TEST CEN ATTN$ AMbTL-PL-TSI USA LOLD REGIONS TEST CEN ATTN$ SIECA-OPI USA LONCEPTS ANALYSIb AGCY ATTNI CSCA-ROPI USA LONCEPTS ANALYSIS AULY A7TNI LSCA-JFI HQJ WaiAIR DIV OF 4EUkOPSY(,nIArRYI USACmaCDA ATTNI ATZL-CAC-IL1 USACACDA ATTN: ATIL-CAL-IMI USAC.mC ATTN: ATLL-CAC-IAI USAC.ACDA ATTN; ATZL-CAC-AI USA tLECTRONIC wA4FARE LAU CHIL}, INTLLLIGENCE MA7ER IDEVEL *SUPP OFFI USA .iSCH (lEVEL # 5TANUARUIZA Goo.' U.K.I USA wESEARCH AND) DEVLLOPM...NT LAbS CHIEF. BEHAV SCIENCES DIVY FOOD SCI LAOI TRAJANA ATTN: SAilS.ORI NAVAL AIR SYSTt.mS COMMANDI ATTN: AIN-5313I ECC'M ATTN: AMbEL-CT-OI USACUEC IECHNICAL INFORMATION Ct.NTERI USAAI4L L189ARY1 HUMAN RES~URCEU HSCH OiRi (1UMRRUII SEVILLE RESEARCH CORPOWATIu41 U5A lRAUOC SYSILMS ANALYSIS ACTIVIYý': ATTN: ATA~wSL (TECH LIPRARY)I UNIFURMED SERVIcES UNIT 0# TrEE HALhTH SCI OEPARTMENT OF P'SYCHIATRYI USA gLOMPUTER SYSrEMS COMMAND ATTNI COMMAND TEQHNICAL LIBRAF%'.' H-9

I HUMAN RESOURCES RSCv-. QWH, (hUMARRWI HUMRI4O LIBRARYI EUSTLS I)IRECTOHATE9 USMAMRUL TLCHNICAL LIBRARVI RAND CORPORA ION/1 RAND CORPORATION ATTN: LIbRAw.y UI FEDEWAL AVIATION ADMLNISTNATION AIIN. CAM! LIBRARY ACC-4401I NAFEL LiBRARY, ANA-641 QRONtNGER LISRARY ATTN:I ATZF-s',-L B~LDG 1313I CENTtR FOR NAVAL ANALYSISI NAVAL HEALTH RSCH CEN LlbHARYI NAVAL ELECTRONICS LAL ATTNI WE~kARCm LIBRARY1 NAVAL PERSONNEL W4 ANI) U CLN LlbkARY ATTN: CODE P106

14

JN

I AIR o.ORCE HUMAN RESOURCES LAU AITN: AFHRLbOTSI HQ,# IT# PJAChUCA ATTNA Tk.(A REF DIV1 uSA ACAUEMY OF HEALTH SCILN,ýLS STIMSON LIBRARV (DOCUMENTS)I SCHOuL OF SYSTEmh AND LOGISTICSii uSAmtRDC TEZHNICAL LIbRANYI flEPA.4TMENT OF THL NAVY THAIN1Nb ANALYSIS AND EVALUATION GPi NATluNAL CENTER FOR HEALTH STAT1ITICS /I USMA DEPT OF bEHAVIORAL SCI ANU LEAUEI4Sm.IpI OLU uOMINION UN~IVERSITY I'LAFOWMINCL ASSESMENI LAOORATORYI USA 60.MMAND AMU 4ENERAL STAFF CULLk.GL ATTN% LIBRARY1 USA iRANSPORTATIUO4 SCHOOL USA TNANSP TECH INFO AND RSCH CENI NASAHNO/I NMRDL~ PROGRAM MA4AUI~ER FOR HUMAN Pk.RF)HMANCEI NAVAL MEDICAL H A'40 U COMMAND (44)I USA ADMINCEN IEC4NICAL RLSEARCM dRANCH LIBRARV2? HODA USA MEL) RSCA4 AND ')EVt.L COMMAN~DI USA t1ELL) ARTY BU)I NAT LLEARINGHOUSE FOR MENIAL ,ICALTM INFO PARKLAWN BLOG1 U OF~ TEXAS CEN FINR COMMUNICATIUN HSCtII INSTITUTE FOR UEFENSE ANALYSESI USA IRAINING SUPPORT Ck.NTtK ATINI ATIC-OST-PAI AFr$Ri TECHNOLO6Y 3FC (h)I PUR~uE UNIV OLPT Of PSYCHOLObIL-AL SCILNCESI USA #MOBILITY EQUIPMENT R AND Ui COMMAND ATTNt DRUME-2GI NO* uSA MOW AITNI ANPL-OtI DA Uz ARMY RLTR'AI'4INO~ bUE RESEARCH + EVALUATION DIVISIONI CALSWAN HUMAN FACTORS ANW THAININb CENTFRI USA ^EROMEDI-AL RESEARCH LAO SLLENTIF~Ic INFORMATION CENTLRI USAF SCHOOL OF At.RUSPACk. PttlUcINL AEI4OMEO)ICAL LIBRARY £TSK-4)I US MILITARY ACAUE~4Y Uk.PT* OF HISTURY, dLOG 601I USA iNTELLIGEN(.E CEN ANU ')Ch AlINI SCHOOL LIONA14YI USA INTELLIGENCE CEN ANU bCR AtTNI ATSI-UPI MARIA~ CORPS INST1TUTE1 NAVAL SAFLTY Ct~ENrEI USAAvNC ANI) FT. RJCKLW AtINI ATLU-ES1 US ANMY AVN TNia LIHRRAY ATTN% CHILF LIBRARIANI USAAvNC ATTNI ATIZUm-I US MILITARY ACAUE4Y UIRELI0R Of INSTITUTIONAL RSCMI USA AIR DEFENSE SCHOOL ATTN% ArSA-CO-MSI 0SAAuS-LIbRARYUOCUMQ~4TSI lISA sIR DEFENSE bOANU ATTNII VILLS RLP'DSITORYI USA INFANTRY BOARD ATINI ATLO-16-ALI USA iNTELLIGLNLE CEN ANU "NR AIINI ATSI-L)T-SFLI uSe% uRONANCE CL14 ANI. SCH ATINt ATtL-ID.TAC

IUSA aRMON SCHOUL ATTNI AIZK-TBJI USA aRMOR CENJLR DIRECTORATE of CUMdAI OEvELUPMENTSINAVAL POSTGRADUATE SCH AIIN: DOULLY KNO)X LIbRARY (CODE 1424)

1 USA IRANSPORIATIOU4 SCHUOL OLg.u1Y ASSt. COMMANDANT EOUCA. TECHNOLObYI USA bIGNAL SCHUOL ANU VT. 6URUON A774: A!ZHATI USA ARMOR CENTLH + FT. KNOX OFFICL OFiARMOR FORCE M6T + STANDARDIZATIONI CHIEs. OF NAVAL EUJCATI1QN AND TN42 /II USA ziIGNAL SCHOOL 4 FT. bORLION LUUCArIONAL TECHNOLOGY DIVISION1 HQ AIC,#'XPTO TRAItNdN(i bYSTEMS oL.VELOP"4kNT13 USA INTELLIGENCE CEN ANU bCM A~tNI ATbI-LRMI US AmMY ARMOR CENTER ATTN% ATiIK-IU-P4UI USA wUARTERMASrER SCHOOL UIRLCIURATE OF TRAININO PEVELOPMENTSI US CvAST GUARD ACADEMY/1 USA iRANSFORTAlIU4 SCHOOL DIRECTORATE OF TRAINING *DOCTRINEI UiA INFANTRY SCHOOL LIbHANY / I1 USA INFANTRY SCri OL ATTN: ATSH-IVI US ANMY INFANTkY SCHOOL AITNI ATSH-CDI USA INFANTRY SCHOOL ATTN& ATSH-IOI-LRU

I USA INFANTRY SCHOOL ATTNI ATSH-EVI USA AP # CHEM SCH/TNG CEN * FT. MCCLELLAN ATTIN ATZN.PTSI USA NP + CHEM SCHITNG CEN * FT, MCCLELLAN DIRS COMBAT DEVELOPMENTI USA. NP # CHEM SCHMTNG CEN • Fl. MCCLELLAN DIR: TRAINING DEVELOPMENTI USA wP * CvEM SCHITNG CEN • FT, MCCLELLAN ATTIN ATZN-14P-ACLI USA iNSTITUTE OF ADMINiSTRATION ATTNI RESIDENT TRAINING MANAGEMENTI USA fIELD ARTILLERY SCHOOL MORRIS SWETT LIBRAPY1 USA iNSTITUlE OF ADMINISTRATION ACADEMIC LIBRARYI USA wAR COLLEGE ATTN$ L1WRARyI USA tNGINEER SCt4OOL LIBRARY ANU LEARNING RESOORCES CENTERI USA aRMOR SCAOGL (USARMS) ATTN& LIBRARYI ORGAeWIZATWOýýL EFFECTIVENESS CEN # SCH ATTNI LIBRARIANgI US ARMY INTELLIGE4CE CENTER # SCHOOL ATTNI ATSI-TPI US ARMY INTELLIGENCE CENTER # SCHOOL ATTN$ ATSI-RM-MI US ARMY INTELLIGE4CE CENTER # SCHOOL ATTN: AT$I-TO-PMI US ARMY INTELLIGE4CE CENTER + SCHOOL ATTNt AT$l-C9-CSI US ANMY INTELLIGEMCE CENTER * SCHOOL ATTNt ATS$-E$I DEPARTMENT OF ThE AIR FORCE AIR UNIVERSITY LIBRARY (ATC)I HQ THADOC TRAIN14G DEVELUPMENT INSTITUTE2 BRITiSH EMBASSY BRITISH ULFENCE STAFF,2 CANAUIAN JOINT STAFFI COLR (W) LIBRARYI FRENCLH ARMY ATTACEI AUSTRIAN EMBASSY DEFENSE, MILITARY A.4U AIR ATfACmE3 CANAuIAN DEFENCE LIAISON STAFF ATTNI COUNSELLOR, DEFENCE R AND DI ROYAL NETHERLANDS EMBASSY MILITARY ATTACHEI CANAuIAN FORCES BASE CORNwALLIS ATITNi PERSONNEL SELECTION. CANAOIAN'FORCES PERSONNEL APPL HSCH U41TI ARMY PERSONNEL RESEARCH ESTABLISHMENTI NETHERLANDS EMBASSY OFFICE OF TME AIR ATTACHE6 LIBRARY OF CONGRESS EXCHANGE ANU GIFT DlVI OEFEOSE TECHNICAL INFORMATION CEN ATTNt OTIC-DOA-2

140 LI|RARY OF CONGRESS UNIT DOLUMENTS EXPEDITING PROJECTI US GoVERNMENT PR14TING OF(. L'IRARYs PUBLIC DOCUMENTS DEPARTMENT1 US GuVERNMENT PQ1ITING OFC LIBRARY AND STATUTORY, LIe DIV (SLL)1 THE ARMY LIBRARY ATTNX ARMY STUDIES SEC3/ /

NUMBER uF ADDRESSEES 222

TOTAL NUMBER OF CUPIES 406

16

--

1-