Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
AD-Ai33 264 CLASSIFYING BUGS IS A TRICKY BUSINESSIU) YALE UNIV NEW 1/HAVEN CT DEPT OF COMPUTER SCIENCE W LJOHNSON ET ALAUG 83 YALEU/CSD/BB-284 NOUG 4-82 IT 0714
UNLSIFIDD/G9/ N
E1EhEEEEI.7
ILE' 1&2.2k ~ILO
iiiii 11 IL1.8
J1L25 .411 111 _6___
MICROCOPY RESOLUTION TEST CHART
NATIONAL DURBAU OF S1ANDtO&R)II-1113-A
' i
W~)-a-
TVERIA
CLASSIFYING BUGS IS A TRICKY BUSINESS
W. Lewis Johnson. Stephen Drap rand Elliot Soloway
YaleU/CSD/RR #284
August 1983
r' OCT n
YALE UNIVERSITYDEPARTMENT OF COMPUTER SCIENCE
ri.s d ,,. .i t4- 83 100gi 092-"- "- 4 'o- .. ':83 10 04 J
i ... .~ ~ ~~Li .... • , I I , , , I
CLASSIFYING BUGS IS A TRICKY BUSINESS
W. Lewis Johnson. Stephen Draperand Elliot Soloway
YaleU/CSD/RR #284
August 1083
SECURIT' CLA SSifCAtO T P OFI RAGE fU'h.. Data Entered)
REPORT DOCUMENTATION PAGE REDISRCIN
I EOTUMBER VTA~s60 noT5CTZG%69f
#284 A-0iNA33 oya. TITLI (and S.biaaie) S. YPE OF REPORT II PERIOD COVERED
Classifying Bugs is a Tricky Business Technical6. PERFORMING *"a. REPORT "UNDER
7- AUTHON(a S. CONTRACT on GRANT MuNDER()
W. Lewis Johnson, Stephen Draper and N00014-82-K-0714Elliot Soloway
9PERFORMING ORGANIZATION NAME AND ADDRESS A0 PORAM ELEOMNT PROMECTTAS
Department of Computer Science AE I OKUI UBR
Yale UniversityNew Haven, CT 06520 NR 154-492
I* CONTOIIO,.ING OFFICE NAME ANC ADDRESS 12. REPORT DAT2
Personnel and Training Research Programs August 1983Office of Naval Research (Code 458) 13. NUMDER OfPAGES
Arlington, VA 22217 1714 Meop.1ORING AGENCY NAME &ADRESS(it differentif horn Controlling Office) IS. SECURITY CLASS. (of this eport)
unclassified154. OECL £55I 1 ICATION, DOWN GRADING
SCH EDULIE
16 DISTRiouTION STATEMENT (of this Report)
Approved for public release; distribution unlimited.
17 OISRUFtiglON STATEMENT tat the abstrt entered in Block 20. II aff.,it how Repent)
14 SUPPLEMENrARY NOTES
7th Annual NASA/Goddard Workshop on Software Engineering,Baltimore, 1982.
IS KEY WORDS (Contjnu. anfavSor&* eid# If flc@Owy and Idenftt by Weetk anm ,)
Software EngineeringProgram BugsProgram UnderstandingProgramming Plans
20 AGSTRACT (CaflttmOn -01- ide if II ncs.. Old 16E*lttP 67 Wleek umbeM)
see attached page
DD, 1473 EDITION or I NOV, SB IS OSSOLETES N 0102- LF- 01- 6601 SECURITY CLASIVICAIWOP orTull PASE M%4 bC~oo I
RuITV CLASaIuCrATION OF TNIS PASS t'i Doeh s XISMOl
In order to build a computer-based programming tutor for novice programmers, 0'-wa-iAeded to first classify the bugs we found in their programs on the basisof type and frequency. However, the enterprise of class.. ication turns outto be a complicated process. While one may want to be able to simply usefeatures in the program itself as the basis for the classification, it turnsout that such a scheme will result in classifications that seem to miss themark, i.e., the classifications will not tell you what misconception the
1,prorammer was operating under which caused the bug. To remedy this situation-iie arguae that the programming plans that the programmer intended to useshould be the basis for a classification scheme. Thus, a bug c!ossificationmust take the programmer directly into acc.ount. In this paper, te compareseveral different methods of bug classification currently being v.sed insoftware engineering projects, and show their weaknesses; while 0ir'methodof using intended programming plans is not without problems, -e-ergiie thatit presents a better alternative than the other methods currently beingemployed.
Accession For
-!T
P17TC T '
iN 0102. F. L014.6601
92CURIy CLAIIATIONCor TNi PAIDSGrU1 ll. *I ibewo
To Appear: 7th Annual NASA/Goddard Workshop on Software Engineering, Baltimore, 1982.
Classifying Bugs is a Tricky Business
W. Lewis Johnson'
Stephen Draper*Elliot Soloway*
Department of Computer ScienceYale UniversityP.O. Box 2158
New Haven, Connecticut 06520
• Institute for Cognitive ScienceUniversity of California, San Diego Mail Code COS
La Jolla, California
This work was co-sponsored by the Personnel and Training Research Groups, PsychologicalSciences Division, Office of Naval Research and the Army Research Institute for the Behavioraland Social Sciences, Contract No. N00014-82-K-0714, Contract Authority Identification Number.Nr 154-492. Approved for public release; distribution unlimited. Reproduction in whole or part ispermitted for any purpose of the United States Government.
Johnson, Draper, Soloway Page 2
1. Context: Motivation and GoalsAbout 2 years ago we decided to build a computer-based programming tutor to help students
learn to program in Pascal; we wanted the system to identify the nen-.pntactic bugs in a
student's program and tutor the student with respect to the misconceptions that might have
given rise to the bugs. The emphasis was on the system understanding what the student did and
did not understand; we felt that simply telling the student that there was a bug in line 14 was
not sufficient -- since oftentimes the bug in line 14 was really caused by a whole series of
conceptual errors that could not be localized to a specific line in the program. However, in order
to design the system we needed to know what bugs students did make in their programs and
what misconceptions they typically labored under. On the basis of bug types found in a numberof pencil-and- paper studies with student programmers (novices, intermediates, and advanced)
[9, 10), we built and classroom tested a first version of such a programming tutor [11]. In the
process of testing that system we instrumented the operating system on a CYBER 175 to
automatically collect a copy of each syntactically correct program the student programmers
attempted to execute while sitting at the terminal; we call this form of data "on-line protocols".
We collected such protocols on 204 students for an entire semester (7 programming assignments).
We have systematically analyzed only a small portion of these data: the basis for this paper is
the hand analysis of the first syntactically correct program that students generated for their first
looping assignment,' i.e., 204 programs.
The story we tell in this paper deals with our experiences in analyzing these 204 on-line
protocols. In particular, we will describe the observations we made in trying to build a bug
classification scheme; the actual details of what bugs we found, their frequency, etc. can be found
in [51. The key observation is the following: while one might think that building a classification
scheme for the bugs would be straightforward, it turns out not to be so simple; in fact, we will
argue that:
Bugs cannot be uniquely described on the basi. of features of the buggy program alone; one
must also take the programmera* intentions and know.ledge st ate into accunt.
2. A Slmplifed ExampleConsider the problem statement in Figure 1, which is a simplified version of the first looping4 problem that the students in our study had to solve in Pascal. From a novice's perspective the
difficult part of this problem is making sure that the negative inputs are filtered out before theyare processed. There are two common approaches to solving this type of problem in an Algol-like
language such as Pascal. In Figure 2 we depict a solution in which a negative input causes
1This problem is given in Figure 8, which will be discuused in action 4.
Johnson, Draper, Soloway Page 3
execution of one branch of a conditional, while a non-negative input caues execution of the
major computation of the loop. We call this type of structure a Sksp-uad P1611:2 a
conditional statement is used to guard the main computation from illegal valuse. Notice that one
pass through the loop will be made for each input value. The second approach is given in Figure
3; here an embedded loop filters out the illegal values. Notice that one pms through the outside
loop will be made for each - and only each - legal value. We call the nestedl loop structure an
Embedded Flter Loop Plan.
Write a program that reads in integers, that represent the daily rainfall in the New Haven area.and computes the average dafly rainfall for the input values. If the input is a negative number, donot count this value in the average, and prompt tb. use to input another, legal value. Stopreading when 9990 is input; this is a sentinel value and should mot be used is the averagecalculation.
Figure 1: Simplified Looping Problem
READ(RAINFALL)WHILE RAINFALL 0 99M 00
BEGINIF RAINFALL < 0
THENWRITELN('SA INPUT. TRY AGAIN')
ELSEBEGIN
TOTAL :s TOTAL + RAINFALL;DAYS :s DAYS + 1;
REAO(RAINFALL);
END;
Figure 3: Using a Skip-Gu~ard Plan
Now consider the buggy program in Figure 4. The problem with this program is that if the
user first types a negative input, and then types the sentinel value 99M9, this value will4 - incorrectly - be processed as a legitimate value. A number of questions come to mind:1. How should we classify this bug'
2. What piece of code is to blame'
3. What mental error on the student's part might have caused this bug'
2%* [11. 3, 9lfor a more complete discussion of programming plan&.
Johnson, Draper, Soloway Page 4
READ(RAINFALL)WHILE RAINFALL 0, 0" DO
BEGINWHILE RAINFALL < 0 DO
BEGINWRITELN('BAD INPUT. TRY AGAIN');READ(RAINFALL)
END;IF RAINFALL 4) 99999 THEN
BEGINTOTAL TOTAL + RAINFALL;DAYS DAYS + 1;READ(RAINFALL)
END;END;
Figure 3: Using an Embedded FMte Loop Plan
4. Vhat piece of code should we change to make the program correct?
In order to answer these questions, however, we need to answer another one first:
What programming approach was the user trying to implement? That is, did the student intendto implement the skip-guard plan or did he try to implement the embdded Jfter loopplan?
Answers to the first 4 questions will be different depending on how we answer this last question.
READ(RAINFALL)
WHILE RAINFALL 0, 99999 DOBEGIN
WHILE RAINFALL < 0 DOBEGIN
VRITELN('BAD INPUT. TRY AGAIN');READ(RAINFALL)
END;TOTAL : TOTAL * RAINFALL;DAYS : DAYS * 1;READ(RAINFALL)
'END;
Figure 4: Sample Bugg Program
We will continue this example by presenting first an argument that supports the choice of the
skip-guard plan, and then an argument that supports the choice of the embedded filter
loop plan; we will then describe a basis for making a choice between the two competing
Johnson, Draper, Soloway Page 5
positions. Consider, then, Figure 5 in which we depict the buggy program again, plus a
generalized, template version of the asp-gvord plan. We can describe the buggy program in
terms of a difference description between the it and the generalized plan. As shown in Figure 5.there are 3 differences:
1. need an IF instead of a WHILE inside the loop,
2. have an extra read inside the loop,
3. will always execute the processing steps since there is no way to skip around theprocessing.
The first difference is a plausible bug for a novice to make; in our examination of novice
programs we have seen novices confuse IF and WHILE: students sometimes construct a loop with
simply an IF, and sometimes they use just the test part of the WHILE statement3 12. 61.Similarly, the second difference is also plausible for novices; again, we have found that novices
often add bits of spurious code, oftentimes attempting to mimic the redundancy they often use in
formulating plans and actions in the real world. Finally, if we assume that the programmer
really meant to simply test RAINFALL, then all that is missing is an ELSE to cause the skip
around the computation; novices notoriously have trouble with the ELSE parts of conditionals.
Thus, the buggy code in Figure 5 is not that different from the skip-guard plsa; w.hen
considering differences from only this plan it is entirely conceivable that the novice
programmer was trying to implement this plan in his code.
Now consider Figure 6 in which we again depict the buggy program. This time, however, we
show differences between it and a generalized, template version of an embedded filter loop
plan. Notice that the code matches the plan well; the only bug is a missing guard before the
code that processes the input: the running total update and the counter update must be
protected from including a sentinel value in the computation.
The analysis in Figures 5 and 5 would lead to different answers to the first 4 questions above.
For example, if we believe that the analysis in Figure 5 is correct, we might say the following to
the student: 4
It seems that you are haying some trouble with conditional statements. For example, did yourealize that there exists a statement called IF that allows you to test ..
To correct your program, you might wast to add an ELSE clause...
2While this may moem strange to us u expert programmer. if we take a moment to reflect, we can uee that usingWKILE rot a conditional and a loop, and IF for only the conditional part is somewhat arbitrary, given their meaningsin English.
4We do not want to argue about the beet pedagogial strasw for interacting with the student; that in itself me avery difficult question. The particular response shown in simply meant to illustrate one type of response to thissituation.
Johnson, Draper, Soloway Page 6
READ(RAINFALL)WHILE RAINFALL ( 99"9 DO Skip-Guerd Ran
BEGINWHILE RAINFALL < 0 DO IF x < fin
BEGIN THENWRITELN(*BAD INPUT. TRY AGAIN'); BEGINREAD(RAINFALL) print error message
END; ENDTOTAL S TOTAL * RAINFALL; ELSEDAYS DAYS * 1; BEGINREAD(RAINFALL) process input
END; END
BUG DESCRIPTION:
1. need an IF instead of a WHILE2. have an extra READ in inner loop3. missing ELSE; processing of Input
will never be skipped
Figure 6: Bug Description Assuming Skip-Guard Plan
Moreover, we would classify the bugs as an (1) incorrect statement type, (2) spurious read, (3)
missing ELSE. On the other hand, if we believe that the analysis in Figure 6 is correct, then we
might say something like the following to the student:
You should notice if the sentinel value follows the input of a negative value that your programwill compute an incorrect average.
The bug type then might be a missing guard (conditional) plan.
By this time the reader's intuition is surely saying that the correct analysis of the buggy
program in Figure 4 is that the programmer intended to implement an embedded filter loop
plan. The bug counts (3 for the skip-guard plan and 1 for the embedded filter loopplan) provide quantitative eupport for this decision. However, we feel that the key in the
decision process -- and the basis for our intuition - is our underetanding of the student's
program provided by the plan analysis in Figure 5: thus, the bug categorization and bug count
follow from our understanding of the program - and not the other way around. We purposely
choose an example over which there would be little controversy. However, the point was (1) to
show how much reasoning we often do about programs implicitly, and (2) to show how different
bug categorization and bug counts could be u a function of choice of intended underlying plan.
While the above decision was relatively clear, let us perturb the buggy code a bit further and
Johnson, Draper, Soloway Page 7
READ(RAINFALL) Embedded alter Loop PlanWHILE RAINFALL <> 99999 DO
BEGIN WHIZE < in DOWHILE RAINFALL < 0 DO BEGIN
BEGIN print error messageWRITELN('BAD INPUT, TRY AGAIN'); READ xREAD(RAINFALL) END
END; sentinel guard planTOTAL : TOTAL * RAINFALL; process inputDAYS DAYS * 1;READ (RAINFALL)
END;
BUG DESCRIPTION:
1. missing conditional (guard) onprocessing the input
Figure 6S Bug Description Assuming Embedded Ailter Loop Plan
see how murky these type of decisions can - and do - become. In Figure 7 we show three
buggy program fragments; let us compare the bug categorization and bug counts using the two
-rnative plans for each of the programs.
e Figure 7 a
Using the embedded filter loop plan we get the following bug differences:
1. the WHILE and IF keywords have been interchanged
2. there is a missing read for a new value
3. there is a missing guard on the subsequent input processing
s, Using the ekip-guard plan we get the following bug differences:
S1. missing ELSE on the internal IF
0 Figure 7b
o Using the embedded filter loop plas we get the following bug differences:
1. the WHILE ad IF keywords have been interchanged
2. there is a mising guard on the subsequent input prcesig
P Using the skip-guard plan we get the following bug differences:
1. spurious READ
2. missing ELSE on the internal IF
Johnson, Draper, Soloway Page 8
*Figure 7c
Using the embedded filter loop plan we get the following bug differences:
1. missing read for a new value
2. there is a missing guard on the subsequent input processing
Using the ship-guard planswe get the following bug differences:
1. the WHILE and IF key-words have been interchanged
2. missing ELSE on the internal IF
We would argue that the programmer of the code in Figure 7a intended to encode a
skip-guard plan: again, the bug counts (3 for the embedded filter loop plan and 1 for the
skip-guard plan) support the intuition that it is more plausible that the programmer simply
left out an ELSE, as opposed to swapping keywords, etc. However, the code in Figures 7b and c
are not so easily analyzed: the bug counts are the same and the plausibility of the bug types are
reasonably similar. In order to make a reasoned decision we need to bring other evidence from
the program to bear. For example, in Figure 7b the programmer used a WHILE loop to correctly
implement the outer loop; this is some evidence that he understancdi. how and when to use thisconstruct. Thus, we might be confident that the programmer really meant IF in the program in
Figure 7b. On the other hand, the inclusion of the spurious READ is unsettling. However, theprogram in Figure 7c is certainly the most problematic: the bug counts are the same, the
plausibility of the bugs are similar, and the additional outside information is equivocal. The
moral of this program is that it can be exceedingly difficult to make decisions about plans -- and
bugs --by simply looking at the code.
The point of these latter examples is to illustrate how quickly the decision about what the
programmer intended gets murky, and how additional information outside the buggy area needs
to be brought to bear. We see again that for the programs in Figure 7 the bug categorization
and bug frequencies change depending on what decision is made about the programmer's
intention.
Finally, the fact that the programs we have shown are notices' programs is really irrelevant to
the point in question: the problem is that the intention of the programmer effects the hug
categorization and the bug count. Quite reasonably, we would not expect a professiona
programmer to mistake an IF for a WHILE. The observation that we would not expect this
particular confusion would in fact aid us in inferring the intention - it would not, we believe,
simply make the problem go away. In fact, we might well see buggy code such as Figure 4,
Figure 7 from a professional programmer.
Johnson, Draper, Soloway Page 9
a b
REAC(RAINFAL.) IEAO (RAINALL)WHILE RAINFALL -o 999B DO WHILE RAINFALL 9o W"1 DO
BEGIN BGINIF RAINFALL < 0 TWEN IF RAINFALL C 0 THEN
WRITELNPCAD INPJT TRY AGAIN') BEGINTOTAL * TOTAL * RAINFALL IITELN('0 INPUT TI GAIN')DAYS -OATS • I EAD(AINFALL).
(EAOAINALL) OWEND TOTAL - TOTAL * RAINFALL.
OAYS -OAYS * IREAD(RAINFALL)
END
READ(RAINFALL)WILE RAINFALL "I DO9t 0
BEGINWHILE RAINFALL ' 0 00
vITEL('IAD IOW TRY AGAIN).TOTAL - TOTAL - RAINFALL.DAYS * DAYS • 1.REAG(RAINFALL)
END
Figure 7: Clouding the Waters: Additional Buggy Programs
* 3. Methods for Specifying the Intention of a Program
In the above section, the basis for describing bugs was the difference between a program and
the programming plans that specified a correct program. There are other methods of specifying
the intention of a program:
* 1/0 Behavior
* Programming Plans
* Corrected Version of the Buggy Program
* Program Description Language (PDL)
In what follows we will examine each of these in turn, and explore their good points and the bad
points with respect to using a method as a basis for developing bug difference descriptions.
1/O BEHAVIOR
An I/0 specification for the problem in Figure 1 would be quite close to the problem statement
itself. The obvious problem with this method is its vagueness with respect to the code: many
different code fragments can misbehave in the same manner (e.g., there we many, many ways to
generating an infinite loop -- but the 1/O result is the same in all cases). One needs to be able
to make finer-grain distinctions than are facilitated by a comparison of the code to simply 1/0
Johnson, Draper, Soloway Page 10
specifications.
PROGRAMMING PLANS
The major problem with this method is the need to guess what plan the programmer intended
to implement. However, once the decision is made, then describing the bug as a difference
between the plan and the code is relatively easy. One method of coping with the plan decision
problem is interviews with the original programmers; this technique has been used to "validate"
change report data in several software monitoring projects (e.g., 1121). Unfortunately, in a classof 200 students writing code at different terminals, interviews with subjects is a bit more
difficult.
The major benefit derived from building a bug description using this method is an accurate
reporting of the cause of the bug. That is, clearly the goal of a bug taxonomy in which one
captures bug type and bug frequency is the ability to pinpoint the sources of the bugs: one
would like to know which bugs came from misunderstandings of the specifications document and
which bugs arose from coding errors, etc. For example, in the previous section if we assumed
that the programmer intended to implement a skip-guard plarn then we would say that there
were a number of coding level bugs (e.g., WHILE instead of IF, missing ELSE, spurious READ).
However, if we assume that the programmer intended to implement an embedded filter loop
pln then the source of the bug may be a problem of specification interprtation: the
programmer may not have thought that someone would ever input the sentinel value after
inputing an illegal (negative) value. Thus he felt no need to guard subsequent computation. (An
interview with the programmer would be particularly useful in this specific case.) Thus, bug
categorization and bug origin is directly influenced by the choice of underlying plan structure in
the buggy program.
CORRECTED VERSION OF THE BUGGY PROGRAM
The typical method of describing a bug is to compare the original buggy program with the
corrected version of that program (e.g., [12, 7, 11). While there is no guessing as to the intention
of the original programmer, we see 2 basic problems with this approach:
e 7Te choice of the particular corrected program used as the measure ie relativelyarbitrary. That is, there are few hard guidelines for making changes to code. Thus,different program mersa could well take the same buggy program and correct it indifferent ways. This would result in two different bug descriptions - an intuitivelyunsatisfactory situation. Moreover, different bug descriptions could lead to differentconclusions as to the origins of the bugs, which, afterall, is the the point of doing thebug categorization in the first place. For example, if the buggy program in Figure4 were corrected by implementing a Akip-guard plan, then the difference betweenthe buggy program and the corrected program would result in a bug descriptioncontaining 3 coding level bugs. On the other hand, if the program is corrected byputting in aguard around the subsequent computation to protect against a sentinelvalue, then the bug description would only contain I bug, a missing conditional
Johnson, Draper, Soloway Page 11
(guard plan) - which may or may not be a coding level bug (as discussed above).While we might prefer the programmer to make the latter change, there is no way toguarentee this situation.
Interviewing the original programmer might shed some light on his intentions -- andguide the subsequent bug analysis or even bug correction. However, this additional,programmer-supplied, information goes beyond the corrected program - andapproaches a bug description based on the programmere orginal plani While we havesome methodological reservations about using interviews collected after the fact, 5 themain issue is that information gotten from the interview is of a different sort than theinformation gotten from the corrected program - where the former information ismuch more akin to the programming plans described above.
a 7sat is actually counted can be quite problematic. For example, if we correct thebuggy program in Figure 7c by adding the missing ELSE, we also need to add aBEGIN-END block around the running total update and the counter update. Shouldwe count this as 1 bug or 2 bugs? It seems unfair to count the BEGIN-END blockagainst the programmer, since this change is required by the 'real* change. On theother hand, however, in the next section we will show programs in which the 'real"bug ie a missing BEGIN-END block. Thus, it is not inconceivable that a programmercould add the ELSE in Figure 7c, but forget to put in the now necessary BEGIN-ENDblock. What one counts is a tricky issue.
The upshot of these two problems with categorizing and counting bugs based on a corrected
version of the program was suggested above: one is less confident of the origins of the bugs, and
thus is less confident about percentages of bugs with those origins. Depending on the particular
corrected solution and the particular choice of counting scheme, one could paint a picture of a
program that contained many more coding level errors, say, than specification-based errors. The
worst part of this situation is that we would not have a good way of knowing how right or wrong
this analysis was - since we don't know how the bug categories and counts would have turned
out if a different corrected version were used as the basis for difference descriptions.
PROGRAM DESCRIPTION LANGUAGE (PDL)
PDL's come in all flavors; some are very close to the code, while others are more high level,
and closer to the plan level description. The former PDL would suffer from the same problems as
using a corrected version as the standard. The latter type of PDL would suffer from the problems
associated with using the programming plans as the standard.
1
'The problems with using interview data has received significant attention in psychology. For example, Eriessonand Simon (41 have argued that one can reliably only we verbal information given by the subject as the en hset isdoinp ta 9"kh. They argue that sch a concurrent verbal report is effectively an on-line dump troi short-termmemory. In contrast, a report after the fact could be a story about what the subject thought be was thinking, andthat significant distortions can occur in this type of situation. While one might arguably feel that the Eriesson andSimon pooition is a bit extreme, nonetheleu, it seems only prudent to exercise eare in interpreting interview data.
Johnson, Draper, Soloway Page 12
4. An Extended ExampleLet us now consider an actual example from the on-line protocol data. In Figure 8 we depict
the problem the students were trying to solve; in Figure 9 the program on the left is a buggy
program generated by a student in our study. If we take a "local view" of the bugs in this
program, we can generate a corrected version as shown in Figure 9 (right hand side). Notice that
if we do a difference description between the corrected and the buggy versions we can come up
with 8 changes:
9 The rainyday counter, COUNTI, will be always be updated; in order to correct forthe times when a negative rainfall is input, we need to decrement COUNTI. Thus, [1]added a begin-end block after (NUM < 0) test, and [21 added a decrement of therainyday counter.
9 COUNT2 must be made to contain the number of rainy (not just valid) days.COUNT2 keeps track of the non-rainy valid days in the loop. Thus, we need tosubtract the non-rainy days (COUNT2) from the total valid days (COUNTI) in orderto get the number of rainy days: (3J changed addition of COUNTI and COUNT2 tosubtraction of COUNAT from COUNTI.
* The guard on the average calculation is incorrect. Thus, [41 changed guard on averagecalculation to COUNTI.
* The divisor in the average calculation should be the valid day counter, COUNTI, notthe valid, but non-rainy day counter, COUNT2. Thus, [5] changed COUNT2 toCOUNTZ in the divisor of the average calculation.
a If there is no valid input the program should neither calculate the average, nor shouldthe program print it out -- as well as not printing out the maximum. Thus, [S] addeda begin-end block after division guard around average calculation and outputstatements.
* The WRITELNs give a message about what should be output; in order to make themessage agree with the actual output, the variables need to be changed: [7] the validday counter needs to be COUNTI, while the [8] rainy day counter needs to COU?%T72.
Given the number of changes that need to be made to the counters (COUNTI and COUNT2), it
would appear that the student has some confusion over the roles of the two counters.
The Noah Problem: Noah needs to keep track of the rainfall is the New Haves are to determinewhen to launch his ark. Write a program which he can use to do this. Your program should readthe rainfall for each day, stopping when Noah types "9999, which is not a data value, but asentinel indicating the end of input. If the user types in a negatp-e value the program shouldreject it, since negative rainfall is not possible. Your program should print out the number ofvalid days typed in. the number of rainy days, the average rainfall per day over the period, andthe maximum amount of rainfall that fell on any one day.
Figure 6: The Noah Problem: A First Looping Problem
However, consider now a different corrected version of this buggy program as depicted in
Figure 10. A difference description between the buggy version and the corrected version yields the
following set of bugs:
9 We can make COUNTI only keep track of the rainy days; this is consistent with code
Johnson, Draper, Soloway Page 13
DUCCGY EXAMIPM cosauRanS VWSoNKGPI EGIN
WAITILN ('PLEASE~ I IU MOM? OF RAINFALL') WITELM ('PLASI INPU 111111? OF RAINALL)11R ADLIN 111 101111READ(ALIP) OMAll(111A.UCOUN?1 - 0 MI '0COUNT2 - 0 CNW2 s0SIR *O so '0"I~es - 0 NI~. 0.WHIL[ (VIA < SENTINAL1) 00 NILE IRAS 0- SENIRAL) 00
EGIN RGINIF (FMie 0) IF (No ' 0)
ATi4N TNN
C2PSTI COUT'. CMXTI 'COAT II F (NIR H IGOWP IF (11110 WIWM?WE%~ THEN
SIF (MR 0)
MAY.'* C"AT2 CDLN?? - COANT? - I:F X" 01 If (AMM 0)
T1HE.4 TiNEWRITELN ('.ILLEGAL !IPUT INPUT NEW VALUE*)11111t (do thee list )
QEAD% cmi uemalE* ; (owt ikeh~i.)REN ul IIIWTELN ('ILLEGAL INPUT IPUT NEW VALUE')
CONT CONT2 - COU1TI READL11IF ("N a 0) ftAS(tap)
TwfN Eo*OTAL - SUP/COUNT7 mut =mm - audit (ehpd ue@ (hie 0)WRITELN ('AVERAGE RAINFALL WAS TOTAL *IKCHE PER DAY-) IF Cttl 3 0 )w (6ut w 0 )HRITELN CH"IOIIST RAINFALL WAS NIG~ * INCHES') THEN
WRITELN (,^"jT2 VALID DAYS WERE ENTERED) be- (*sod this thu. )VPIIEL% ( COUNT? RAINY DAYS IN THIS PERIOD ITOTAL StP/inmtZ. to how~d &UP fine 0)
WRITELN ('AVERAGE RAINFALL WAS TOTR INCHES PER DAT1VRITELN (HNIGHEST RAINFALL WAS HIGWTIM INPCHES*
VRITEL(Gatt VALID DAYS WERE ENTERED') (at e ahs how~ 0)k*ITELNI(Imt. RAINY DAYS IN THIS PERIOD (dimpd thee howi
* (11 added a beg5aCed block after (111A 4 0) test. Rod 12 added a decremnt of the ratoyday counter
a 131 c.4angeo add tioR of COUNITi sod COUNT2 to subtraction of COUNT? from COATI
is 141 Clanged guard oH average CalcH latioH to COANTI
*e i changeo COUN'? to COUNTI a the divisor of the aversge CsIcelatlee
0 ill added a begHf-end block after ,HiStoo guard around swerage calculation sod output statemeetsis t(lJ Te va' day C-)vnter needs to be COUNIT) wklse 1 1 r ainy day counter needs to COUNT?
Figure 0: A Buggy Pogram and a Corrected Veision
Johnson, Draper, Soloway Page 14
already in the program: the line that adds COUNT2 and COUNTI now makes sense-- COUNT2 now keeps track of the valid days, and the divisor in the averagecalculation suggests that COUNT2 should be the valid day counter. In order to makeCOUNTI perform in this manner, we need to [1) add a begin-end pair around allcomputation after NUM. > 0 test, up to the NUM " 0 test.
9 If there is no valid input the program should neither calculate the average, nor shouldthe program print it out - as well as not printing out the maximum. Thus, we needto [2] add a begin-end block after division guard around average calculation andoutput statements.
e The guard on the average calculation is incorrect. Thus, [33 changed guard on averagecalculation to COUNTI.
Which description should we choose? And why! Notice that neither of the corrected versions
were that unreasonable. However, it would seem to us that one should choose the second bug
description over the first. The basis for that decision is the hypothesized plan structure
underlying the buggy version: it appears to us that the student was trying to structure the
actions in the main loop in terms of case. For example, the program explicitly tested for NUM> 0, NUM - 0, and NUM < 0 and took the appropriate actions -- almost. In order to makethe case structure work, the code following the NUM > 0 up to the NUM - 0 test should be
grouped together. While one cannot put too much faith in the indentation of a novice's
program, 6 it appears that the indentation supports this analysis. Thu, what is missing from themain loop is a begin-end pair surrounding the code between the NUM > 0 test and the NUM-
o test. On this analysis, the student does not have a misunderstanding surrounding the two
counters, but rather has a coding level misunderstanding about how to block code together.
Moreover, this same misunderstanding can explain the lack of a begin-end pair surrounding the
average calculation in the next two write statements. The reduced bug count in the second
description follows directly from this analysis: in effect there awe only 3 bugs in this program, 2
of which have the same underlying origin.
This example illustrates a point made earlier. the bug categorization and bug count follow.
from an understanding of the program that is provided by tke hypothesized plan structure of
the program. That is. to understand a buggy program, one must make inferences about what
plan structure the programmer intended to implement; the program only *makes sense" in terms
of these plan descriptions.
$We have observ~ed in the on-line protocols that the physical layout of a student's program suffers U the studentmakes changes to his program in the procss of debugging it.
Johnson, Draper. Soloway Page 15
BEGIN BGYZA P BEGIN N T =CI W DVZ m
WRKTELN ('PLEASE' KMPUIT AURT OF RAINFALL') WITEIM ('PLEASEI IIDPJT MOT OF MVRAKWML*)READLN PEAK",RAALP1READ(MM ALIN)COi4TI 0 cUT1 a 0COUAIT2 0 COIWdI - 0,
WIKLE (WIN a, SENTKNAL) DO WHILE (010443 SONIKAL) DO
IF (AA 0) IF (m 2 0TWEN TNEe
c~jNTI - cowl I RX K mm *i. WARIF (FILM ' IGURM) CA6TI =CmOTI * I
THEN KF (NURHM WMHKONP - WIN TUEN
.r (11101 - 0) WIr~A * MTHENk ASI. (ad" #A" be. e)CMAT2) -COWM2. -I If (KIN xO)
KF (41.1M 0) THENlTWE N CMLRT2 w COLIET2*I
WRITELN ('ILLEGAL XNFLJT INPUT NEW WALLS') KF (111011 c 0)
AMINOM) ITELIN K TILMAL INPUT KmpJt NEW WALL)
COJET? * CMT2 * Coil~l REAIAII)KF (AIl 0) ENDTWE'E COL2 - C~itW2 - CG.*T)'OTAL S 9J/CCIA3T2 IF (M W ). 0) fO mkp tso .. )
WRITELAI (AVERAGE RAINFALL WAS TOTAL -IES PEN DAY-) THEM
WRI11TELNS (HK4GMEST RINFtALL VAS PIWRALI I 1NESI 64.. aud tame $.)WRI'EI.N (CKIJET2 VALKD DAYS WERE ENTERED-) ?OTAL - SUVCOMT2IKITEJE (COUNTI RAINY DAYS KN TmKS PERIOD PWITELMN ( AVERAGE RAINFALL. WAS TOSAL. INCHES PER DAY2
EPIC VNITELIN (W101OEST RAKNFALL VAS *I~MAM INCHES0" (Is ae kne )gITELOO (OM2 VALKD DAYS WKR ENTERED)W~AKI (OAffI NAdKM DAYS KM THIS PERIOD
0 111 add a beg."-end pair afrrovd all Compgtatioa after Mis b 0 oSt vp to Lb.SM 0 tent
* 121 add a beg-ft-esd bloCk after diviSie. 5gard &roved average CalColation sod ovtpvL StatementS
* jai Cheaned gosard oH average caIcuIALIOR to COW?)J
Figure 10: A Bugggy Progam an an Alternative Cormeted Version
Johnson, Draper, Soloway Page 16
S. Concluding RemarksWe have argued that a bug description is a difference description between the realization and
the intention specification. We have presented a number of techniques for specifying the intention
and have pointed out the problems associated with each type of specification in developing anaccurate picture of bug types and bug frequency. While no technique is without its problems, we
have argued that the understanding provided by a plan analysis of the buggy program stands a
better chance, as compared to the other techniques, of providing a more accurate categorization
and count of the bugs - and thus a more accurate reflection of the origins of the bugs.
I
Johnson, Draper, Soloway Page 17
Reference.
I. Basili, V., Perricone, B. Software Errors and Complexity: An Empirical Investigation. Tech.Rept. TR-1 196, University of Maryland, Dept. of Computer Science, 1962.
3. Boua, J. Understanding the Novice Programmer. Dissertation, in preparation.
3. Ehrlich, K., Soloway, E. An Empirical Investigation of the Tacit Plan Knowledge inProgramming. in Human Factors in Computer Systems , J. Thomas and M.L. Schneider (Eds.).Ablex Inc., in press.
4. Ericsson, A. and Simon, H. "Verbal reports as data." Pschoogcal Review 87(1980),215-251.
S. Johnson, L., Draper, S., Soloway, E. The Nature of Bugs in Novices' Pascal Programs. inpreparation
6. Miller, L. A. "Natural Language Programming: Styles, Strategies, and Contrasts." IBMfSysems Journal .00(1981), 184-215.
7. Ostrand, T., Weyuker, E. Col'.,cting and Categorizing Software Error Data in an IndustrialEnvironment. Tech. Rept. 47, New York University, Dept. of Coimputer Science, 1982.
6. Rich, C. Inspection Methods in Programming. Tech. Rept. AI-TR-804, MIT Al Lab, 1981.
9. Soloway. E., Ehrlich, K., Bonar, J., Greenspan, J. What Do Novices Know AboutProgramming? In A. Badre, B. Shaciderman, Ed., Direction#e in Humn-~aComputer Int eractiosoe,Ablex, Inc., 1982.
10. Soloway, E., Bonar, J., Ehrlich, K. . Cognitive Strategies and Looping Constructs: AnEmpirical Study. Communications of the ACM, in press.
11. Soloway, E., Rubin, E., Woolf, B., Bonar, J., Johnson, L. MENO-11: An IntelligentProgramming Tutor. Journal of Computer-Based Instruction, to appear.
12. Weiss, D. Evaluating Software Development By Analysis of Change Data. Tech. Rept.TR- 120, University of Maryland, Dept. of Computer Science, 1981.
- OFFICIAL DISTIRUBTION LIST -
Aray Privat Sector
Technical Dect.or I copy Or Michael G..eseretb 1 copy
U S Army Research Institate for the Department of Cooputer ScienceBehavioral &ad Social Sciqecs Sutefordi University5001 Eisenhower Avenue Stanford. California 34305Aleziadre1 Virginia 22333
Dr. Dadra Geataer I copy"r James Baker I copy bolt Beriesn I %iheseArmy Resetrch InStitute 10 Moulton Street5001 Eisenhoe, Avienue Cambridge, Plssachilsett 02136A;IeIaoria. Vigtii'a 22333
Or Robert Glbser I copyDr B. trice J Parr I copy Lsrning Research & Development Ceter
U S Army Research Institute Unsversity of Pittsbrgk5001 Eisenhower Avenie 3939 O'Mara StreetAlonledri Virginia 22333 Pottstibrgh. P lesylvtIeia 15260
D- Mt, Ite S Nat: 1 copy Or Joseph Gogeen I copy
1l as Tockbica| Arel SRI InternationalU S Army Restore,. lnstitate 333 Rvensiwood Avenue$001 Eseloer Aveiue MeNlo Path. California 34025Alexandria. Virginia 22333
Dr Bert Gres I copy.e Ni'Shall Norwo I copy Jos$ Nopkins University
S Army AeSvarc insti tto for the Oepartment of PsychologyGflavioral A Social Sciences Charles A 34th Street!101 Eisenhower Ave.iue laltimore. Maryland 21218
Ailiandrin. Virginia 22333
C~ Ote4 ' Quei. Sr I copy Oi' Jane 0 tteio I copyD;rector T-a inng Research Lab LROCArst Researep Institute University of Pittsbargh
50.i E-senhoner Avenue 3939 O'Mara StreetAilein*.oa Vrgts 2-2333 Pittsborgh. PePsylvania 15213
Comemader. US Army Research Instittte 1 copy
fel the Behavor$l A Social Sciences Dr Brbarn Hayes-Roth 1 CopyAttn PER!-BR (Dr .udith Oresln,) Department of Computer Science
iCVl Eisenhower Ave-,e Stanford Uiversity
Alzandris Virginia 22333 Stanford. California 65305
Joseph Pso.ka. Ph D I copy Dr Frederick Noyes-Roth I copyAttfn PEIN-IC Telouledge
Army Research Instit'te 525 University Avenue
$001 Eisenhower Avenve Polo Alto, California 94301Alelnandrin. Virginia 22333
Glann Grteeld. EdDr Robert Sasmor I copy Mhenn Intelligence Newsletter I copyU S Army Research, InSi ttt for the P 0 Goo 1163
4 Behavior; and Social Sciences Birmingham. Michigan 460125001 Esenhower Avenne
Alsandria. Virginia 22333 Or Earl HNt. I copyI Dpartmset of PsychologyOr Robert wisher I copy Uiversity of VashligtooArmy R*seares Isitate Seattle. Wasnlogtol 99103
5001 iseshower AveteAlesnadria. Virginia 22333 Or Marcel Jost I copy
Department of PsyckologyCiregoe-velloe UniversityPittsbersk. Peasylvanie 15213
Air Force
U S At; Force Office of Scientific I copyResearca Or David Kieras I copyLife Sciences Directorate. III Department of Psychologyselilng air Force last University of Arizonawalshagton. K 2033 T ame.. Aizon 5721
Or (art A Allaest 1 cool or alter Kistsci coolNQ AFNIL (AFSC) Department, of PaycoologyBrooks APB. loans 7M35 University of Colorado
loude*r. Colorado 80302Bryan DaolIaIw1 copyAFIRL/LRT Df Stephen Kosslyn I copyLowry ASS, Colorado 30230 Department of Psychology
The Jobs Nopkins UsiversityOr Gmnevieve Noadded Icopy Baltimore. Maryland 21213Program ManagerLife Scieaces Directorate Dr Pat Langley copyAFOSA The Robotics InstitutetBoiling AFB. K 20332 Cilraigis-Mullen University
Pittsburgh. Pennsylvania 1S213Or join Tangmey I cpAFOSIIARL Or Jill Larkin I copySo ing APB. K 20332 Department of Psycooogy
Carteee-M lion UiversityOr J s Yaisaee I copy ittsalurga. Pennsylvania 15213
Lowry AS, Colorado 30230Or Alast Lesgold 1copy
Mlarins, Corps Lesaieg MIO CenterUm~versity of Pittsbartlh
4 Willit Sm roenp I copy 333 0 tNara StreetEduacation Advisor (E031) Pittsbergi. Peensylvania 15213Education Center. MCDECvaeties. virginia 22134 Dr Jim Loeit 1 copy
University of CaliforniaSptc-al Assistant for Marine I copy It Sea DiegoCorps 11tters Laboratory for Comparative,Code loom Nae Cognition - 0003AOffice of Naval Research La Jails. California 9211380C I Qeincy StreetArl~agton. Virginia 22217 Or Michael Levine 1 copy
Department of Edocatiolial PsychologyOr A L Slafkoshy I copy 210 Eduocation BldgScientific Advisor (Code RD-I) Usiversity of 1llinoisMo. U S Marine Corps Coampeege. Illinois 61361Wash ingtos OC 20380
O r Marcia Lime 1 copyDepartment of Defente University of Califora
Director. Adolesent Reasoning ProjectDefense, Technical Information Center 12 coptiesl berheley, California 34720Caimeron Station. lodgAletandris. Virgios 22314 Or Jay MClelland4 I copyAttn TC Detnton of PsychologyMIMilitary Almistast for Training and I e"py Cambridge. Maamseelsetts 62133Permssnl TechnologyOffice of to* Under Secretary of Defense or JaM R Killer I copyfor Rfesarch a 9161neting Camipsea Thoegot Corporationfosm. 30123, The Postagoa 1721 West Piano RiepealWsingSt". K 20301 Plan.. teea" 75075
Major Jack Thorpe 1 espy Df Math Miller copyDARPA Cemptet Theoagt Corporation1400 W. las, blvd 2722 vast Pias ighueyArlington. Virginia 22200 Plnes. le" 7&073
Navy Or Teeflirte I eopyIlaeo PM
Notern boers I copy 833 Coyote HIll baedCode 17 11 Palo lso. Calafqreas 94804Nome factors Laboratory
I~v~RE~l~cuOr Atllenebare I copyAnusaft. Florida 23 biehawoeat Toehoogy Laboratriote
2IOU fleas Avenue. Furth FlooCode 3711 I cpy ledoodo both. Catiforms 6617Atta Arther S WasveNoelt Iraiciag fqoipeust Center Or heald Normn I eopyOrtasdo. Florida 32013 Cogmitiv "SWaes. C-013
Ussw of Califors. See SpeoeLasose Scientist I copy La Jolla. California M313Office of novat ResearchBrache Off ice. LooeoIca 36 Or JeIs Orlesky coepyFPO low York. Neu York 03510 lastieteu for Defease Analyses
1301 1 basrgard StreetOr Richard Cantone I copy Alexandria. Virginia 22311Navy Researcht LaboratoryCode 7510 Professor Seymour Papert I copyWash ingte. KC 20376 2Q- 109
NITChief of laval Edocefios &ad Tressiag I copy Cuorisso. feumaceseitte 3213Lassom OfficeAir Force Massa Resource Laboratory Sr Nsey Psssaos 1IspOperations Tram..5g Division University of ChieapWILLIMS API. Ar izona 65224 Gradueate School 011 lesaem
1101 E N6th StreetChicago. 11t16048 N063
Or Stanley Cot yen I copy Dr Rickard A Pollak I espyOffice of levei Tiehuoiogy Director. Special Projects900 1 91iscy Street NECCArlington. Virginia 22217 21154 Militie.. Valley Loe
SteI luatar. Nieseetae 5506CDt mile Cora c1opyOffice of lovel R"eserc Or Peter Pota"ge copy300 1 9aiscy Street Deparment of PsychoologyCode 270 University of ColoradoArligtoo. Virgirnia 22217 Iselder. Colorado 66309
Or Je Ford I espy Or fred Reif I espylevy Permoae RI Ceur Ploysim esSparteetSa Do*$o. Cat ifersa 12152 Univer'sity of Califoraia
Garteley. Californae 34728Or Judle Fraski,. I espyCode 7510 Sr Larm es oakc 3 epy,Navy Research Laboratory LADCWshaagtoa. K 20175 Vaiversity of Pittelburgli
am3 Otllera StreetDr Nsi*e Gaynor I eopy Psttuburgb. Poessylease 15223Nacy Research LaboratoryCede 7510 Nory S oilf 19 1 pWaba OaC 21175 Pregra. is Cogatswe ete
Coaer for Romse Zeforntie toSr Jie Notion I "opy Usivesrvit of Colifefe. 2607 "49Code 24 La Jala. California I"6Deny Peresel NIo Castersee Deep. California 3152 Or Mileespy
Anes Ientioe"e for lseeseegr I Noese 2my INS Tlhins Jef forge Steet. WAbay Pesase M I Coeutr PosIIGO. KC low
e Miep. Cliferes 3223
Dr Ernst Z Rotbopf I copy
Dr lorose J K.'r I copy Bell LaboratoriesChief oP Naval Tecanical Traieiag Nurray Mill. gow JerSey 07174
Naval Air Staion PleclupS (75)Millington. Teoscssee 3054
Dr Vill$im I Names I copyDr James Lester I copy Georgia leostatee of Tocksology
O Detaclment School of Industreil a System495 Summer Street Eagiseorigbat.O. ess& cSatLS 02210 Atlsts. Georgia 30332
Or William L Naoy (02) 1 copy Dr David R11amltrt I copyChief of levll EsecatcOs and Treimniag Ceter for Neone Iformation Processinglvat Ar Statoa Univfsity of California, See DiegoPeasacola Florida 32506 Ls Jolls. Clitforais M2913
Dr Jot WcLac. ac I copy Dr Michael J SSer I copyNavy Personnel RID Ceiler Perceptronics. Ise
Sao D-ego Cal,forca 921!2 6271 Veriel AveaceWoodliad Hills. Califotici 11364
Dr Roger Scall I COpy
Dr William Montag e I Copy Yale UmeovrsltyNFRDC Code 13 Departmaat of Cospetef Science
San Diego. Calfornia 2152 P 0 low 2156low Iove. Cossecticat 06520
Libtrary Ci . 'O.L I copyNavy Ptnrscrtiu I&Ca ce ter Dr Walter Sciaoder I copy
Sea Diego Calfor-. 92152 Psychology D epsruNta603 E Dec il
*ecss,€ai Director I copy Chospi0lg. Illoaoots 6120
lvy Personnel iAo Cet.or
Sac Diego California 92152 Dr Also Sctooefold I copyMathematiCS ais Edecatio
Comsdcg I ff cer 6 cops Tie University of Rochester
*ova; Research Laborstory Rochester. lwe York 14627
Code 62627waSonivgon. DC 203;0 ir Colls Sheppard 1 copy
Applied Psychology Unio
Office of Navel Research I cooy Admiralty orose Tectsology EstCode 433 Teldiagtoo. Middlesex
OC N guinty Street United 1i16o4Arfhgtoa Virginia 22217
Dr N Wllace Siaciho 1 copy
Persomp l I Trining Research Group 6 copies Program Director
Code 442PT Iapoler Resarch sad Advisory Service
Office of lval Research Smithsonian ZastitatioaArlvgtoao Virgii 22217 601 north Pitt Street
Alelladria. dorgicie 22314
Office of the Che of lval Operations 1 copyRealrcs Dovelopumet A Stedies Iravch Or Edeard E Soith 1 copy
OP 115 Del?. ftrcae A NowecaWvaigtoc DC 20350 50 %ttO a.rett
¢lCamridge. MessactasattaI 021331
LT Frank C Potio. Rx. USI1 (Ph ) 1 copy
ClT (1-432) Or Ihaord Sam 1 copy"A USSchool of ooidacatas
Pfesaaole. Floride 32509 Steford Uciversi?,Steeford. California 34305
Dr Gary Pec I copyOperavties ftesecb DevelopmectCoda UIPs Or Lattryc I Spoti eopyNavel PostgrIdweto School Psychl ogy DoporIstoa
Noaterty. allfrIlc 1340 Ioa laiversityProvw deat. Rhode Island M2812
I,
Dr Gl Recard I copyCode slii Dr Roert Sternberg 1 copyITEC . Departaest of Psychology
Orlaeno, Florida 32613 Yale UiersityBox IA. Yale Station
Dr Worth Sceasold I copy Now Noves. Connecticut 06520CUE7 (1-5)AS. Pescot. Florida 32506 Dr Albert Stevens I copy
Sait Branch A lenan10 "otilto Street
Or Robert G Smith I copy Cambridge. assoelsetts 02238Office of Chief of avel Operations
OP-987h David E Stome. Ph D I copywNasinon. DC 20350 Nozeltiae Corporaton
7680 Old Spriaghouse RoadOr Alfred F SoOde, Director I copy McLean. Virginia 22102Trinns Analysis A Evalvation Group
Department cf the Navy Dr Patrick Snppes I copy
Orissoo. Florioda 3213 Institute for Mathematical Studies inthe Social Sciences
Dr Richard Sorensen I copy Stanford University
Navy Perscnre, RID Center Stanford, California 94305
San Diego Ca! fornia 9212L Kihemi Tatsuoao I copy
Or rrederick Stenheiv ser I copy Coeputer Based Education Research Lab
CN0 - OPI:5 252 Engineering Research LaboratoryNavy A-net Urbona. Illiois 61601
Arl!ngton Virg:nia 20370
Dr Miaurice Ttsuok 1 copy
Roger weissilr-Saylc 1 copy 220 EducatioO Bldg
Department cf AdPinistratine Sciences 1310 S Sixth Street
laval Postgraduate School Champaign. Illinois 61820Ponterey California 93940
Dr Perry I Thoradyhe 1 copy
"r John I i cfe I COpy Perceptroics. IoncNavy Persoptei AD Center 54S Middlefield Road. Suite 140
San oiego. California 92152 Menlo Park. California 94025
Or WaliaC bulloCk III copy Dr Do glas Tovne I copyNavy Persornel RID Center University of So California
Son Diteg. Calfcali 921!2 Oevauvoral Ttchnology Labs1645 S Elena Avenue
Prnvate Sector Redondo leach. Californ3 i 90277
Pr John R Ar.oson 1 copy or %art Van Leon 1 copyDepartment of Psychology XeroI PARC
Carnegie-01eilo, University 3333 Coyote Hill RoadPittsburgh Fennsylvania 1S213 Palo Alto. California 04304
Dr Joh- Anett 1 copy Dr Keith T Wescourt 1 copy
Department of Psychology Percoptrosics. Tnc
University of waraick 545 Middlefield Road. Suito 140
Coventry CV4 7AJ Raeio Park. California 94025' ENIUAND
Or Michael Ateood I copy V1llin S 9i bttes I copy
ITT - Programming Sell Laboratories1000 Oronoqee Lane 2D-610Stratford. Coasectacat 05497 Noledel. le Jersey 07733
Dr Almo laddeley I copy Dr Mike Williams copy
Medical Research Council xeror PARC
Applied Psyciology Uit$ 3333 Coyote Hill N oa36 Chaucer Road Polo Alto. California 94304
Cambridge C82 2fFE[aLA¥O
Covi rai Asgeces
Dr Patricia A heler 1 copyOr Pistne a gg I copy N IE-P% lid. Stop #IDepartment of Psyciology 1200 lthb Street IWUniversity of Colorado Washington. C 20OOBoulder. Colorado 60309
Dr Swsis Chipma I copyfs Carolb A Bagley I cOpy Learmgii d evelopoestNGmmeS t& Educatioil Caopetg s Nional lJstiete of EllcatiowComsort'im 1200 loth Street 4W2354 Hiden Villey Limo Wishingto. 20206Stilliiter N. iesota 55062
Edwars Esty I copyOr Joatha Sharon I copy Department of Edecitiog. 0111e! 3t00m Avenge MS 40Berwyn, PtnanSylnvii 12312 1200 lth Street. IV
Wish igton. K 20200Mr Avran Srr 1 copy
tevartP nt of Coapvt.er Scieice Eduard J Fuete 1 copyStarfold Umlve'sty Department of EducationSt~ifcra Iiforvia 94305 1200 loth Street, NW
Wiasington, C 20208Cr John Slack I copyYale Unverst y TAlE. TUA I copySot 1!A Yale Statuci Nitleoal Institite of EdocatloiNew avev Coeiect&cvt 05.10 1200 loth Street. iw
Washeigto. DC 20206Or Jobs S Brown I copyXEPX Pitao Alto Research Center Or Jobe Keys I COpy3.33 Coyote Road Naioni losti-tie of EdecatomPIIo-Alto. Californmi 94304 1200 loth Street. IW
Washngton KC 20208Or .'lce lvchain I copy
eczl-ment of Cosu.Vtr Sc:emce Or Arthur eImed I copyStialord University 724 BrowsStarford. Cai fornia 94305 U S Dept of Educatom
Viii iigod. DC 20206
Or otvo Cartorvlt I cepy
aetaptient of Psychology Dr Andrev 0 tioner I copyCarveg-e-feloc Uiversity Office of Scientific sad EotmeegriPttsbvrgh Pemssyllamsi 15213 Personnel ad Edecatlo
Istlovil Sclemce FolaiteloDr Pat Carpenter 1 copy Wasimgtl K 20550 -
Oe;artpvnt of PsychologyCirnttog 1e-el lo Us~uersltVPtts vrgh Pessylvesis 15213 Everett PIler I copy
Researci ScientistOr Willam Cho*e * I copy Nilb StOt 239-3Department of Psychology NASA Anes Research CeterCarvegle-eliom University Moffett Fleld Callforalm 94035Pittsvrgh Plemsylianma 15213
Or Mary Stod ard I copyOr Nichollse Cho I copy C 10. 018i Stop C96Leiromeg 0 I Center LOS Aleo Utiosal LaboratoriesUniversity of Pittsberg Los AIimos 11 Necico 67545399 O'ara Street
Pttsburgh. Ptleylillii 15213 Chief. Psycholo ic¢l esesirc BrIoc I copyU 8 Coast Gefr (-P1/21TP42)Wash ington C 20593
Dr Willias Cleucey I copyDepartmest of Computer Scitace Dr Freak Withrou I copyStalford univrsity U S Office of EdecatioeStssfori. Califormis 94306 400 Marylead Avewm Ski
Wasiogtca. DC 20202or hilas " douseas IcopyBait Bereech A *@w3. INC Dr Josepb L Tovag. Director I copySo moattos Street Memory A Cognitive ProcessesCambridg. Massachosetts 02138 Ma1tsoa Sciesce Fomagatmee
Wash iagtoe. DC 20550
ERIC Fie-lity-Acquisitofts 1copy4M3 Rugby Ave***SBethestsa Mary lad 20014
Mir Wallace Fearzeig I copyDepartmvot of Ediocatiomal TechaologyBolt leramek and Nesome10 o [t. Street
Cameridge Niassacftesetts 02238
or Deuter P'etc~tr IcopyWICAT Resellrel Zaittate1675 S State StreetOres Utala 22333
Dr Joe. 4 Fredtfisis IcopyBolt Bersek A Uemanlo oNo. 'to StreetCambridge Plassacbesetts 02138