Cognitive layouts of windows and multiple screens for user interfaces

Int. J. Man-Machine Studies (1986) 25, 229-248

Cognitive layouts of windows and multiple screens for user interfaces

KENT L. NORMANt, LINDA J. WELDON§ AND BEN SHNEIDERMAN~

Departments of t Psychology and $ Computer Science and § Center for Automation Research, University of Maryland, U.S.A.

(Received 25 June 1985 and in revised form 20 May 1986)

In order to make computers easier to use and more versatile many system designers are exploring the use of multiple windows on a single screen and multiple coordinated screens in a single work station displaying linked or related information. The designers of such systems attempt to take into account the characteristics of the human user and the structure of the tasks to be performed. Central to this design issue is the way in which the user views and cognitively processes information presented in the windows or in multiple screens. This paper develops a theory of the "cognitive layout" of information presented in multiple windows or screens. It is assumed that users adopt a cognitive representation or layout of the type of information to be presented and the relationships among the windows or screens and the information they contain. A number of cognitive layouts are derived from theories in cognitive psychology and are discussed in terms of the intent of the software driving the system and congruence with the cognitive processing of the information. It is hypothesized that the particular layout adopted by a user will drastically affect the user's understanding and expectation of events at the human-computer interface and could either greatly facilitate or frustrate the interaction. Ways of ensuring the former and avoiding the latter are discussed in terms of implementations on existing multiple-window and multiple-screen systems.

Introduction

Designers of new computer terminals, personal computers, and workstations are exploring innovative tools for restructuring the presentation of information and control of processing. These new tools include the use of windows that display menus, files, or other objects on the screen (Card, Pavel & Farrell, 1984; Miller & McMillan, 1984; Myers, 1984; Weiser, 1985). Windows are used to partition the screen into discrete sections each containing different types of information. For example, a word processor may display the working text in one window and the set of available editor commands in a window at the bottom of the screen. Windows can also be arranged to overlap one another. Exposed areas of partially obscured windows serve to remind the user of their contents. In some cases windows can be thought of as "pop-up" windows that temporarily obscure other windows. These windows contain menus that pertain to actions to be applied to the underlying window. In other cases windows may overlap to show the structure of the file whose elements are within the windows. Window management may be under user control or system control (Bly & Rosenberg, 1986; Burey, Davies & Darnell, 1985; Cohen, Smith & Iverson, 1986). The top panel of Fig. 1 displays two types of windows, overlapping and non-overlapping.

Send correspondence to Dr Kent L. Norman, Department of Psychology, University of Maryland, College Park, MD 20742, U.S.A.

229

0020-7373/86/080229+20503.00/0 © 1986 Academic Press Inc. (London) Limited

230 K . L . N O R M A N E T AL.

(o)

Non-overlapping Overlapping

Cb)

Screen I Screen 2 Screen 3

FIG. 1. Configurations of multi-window (a) and multi-screen (b) displays.

A closely related innovation is that of multiple-coordinated screens (e.g. Shneider- man, Grantham, Norman, Rogers & Roussopoulos, 1984; Shneiderman, Shafer, Simon & Weldon, 1986). The workstation, for example, may have three screens side by side as shown in the bottom panel of Fig. 1. Information displayed on the screens is linked in some way depending on the task at hand. The screens may be "fused" together so that in effect one is looking at a single display with three times as many lines but split into three discrete pages. In other applications one screen may display a menu of commands that can be selected for actions applied to information on a second screen. Alternatively, screens may contain linked information. For program files, one screen may display variable declarations, another the program statements, and a third comments.

Visual scope

The variety of possible layouts is endless, but whatever the layout of the screen, there is an attempt to represent information in a meaningful visual form and to increase the visual scope of the user. "Visual scope" is defined here as the degree to which the user is able to integrate information across a display of multiple windows or screens and to grasp the whole of whatever is being displayed. Although multiple windows and screens increase the perceived viewing space, they will not necessarily increase the visual scope if the user sees no relationship or pattern that spans the display. Indeed, they may decrease the visual scope, by adding irrelevant and distracting information.

c O G N I T I V E LAYOUTS 231

When there is a mismatch between the user's visual expectations about the pattern of the display and the actual operation of the system, it is hypothesized that performance will be impaired. On the other hand, if multiple windows and coordinated screens are used in a manner that supports the user's expectations, they will become powerful tools for creating new work environments and increasing the visual scope of the user.

A related measure has been proposed by Woods (1984). He defines "visual momentum" as the ability to extract and integrate information across successive displays. Visual momentum is created by providing a smooth transition between one display and the next. When it is broken, it results in (a) cognitive tunnel vision due to a low field of attention, (b) an inability to locate "important" data, (c) a tendency to get lost in display networks, (d) memory bottlenecks due to an increase in mental workload, and (e) a decrease in problem-solving performance.

In a similar manner, when the visual scope of a display is broken, a number of problems will result, such as (a) information overload due to additional distracting information, (b) confusion and disorientation, (c) difficulty in locating needed information on the display, and (d) loss of position and location of markers such as the cursor.

Modes of representation

In order to evaluate the utility of layouts for multiple windows and coordinated screens, it is necessary to consider how the user views and interprets the layout. To facilitate

Cognitive layout User's mental model of the surface layout

i Surface layout Physical arrangement of I information on the screen I

J

I Machine layout jlnternal representation of | the surface layout \

FIG. 2. Three modes of representing the user interface.

thinking about the operation of windows and multiple screens, a distinction will be made between three modes of representation. The modes of representing the human- computer interface will be called "layouts". Figure 2 defines and schematically displays the surface layout, the machine layout, and the cognitive layout.

232 K . L . N O R M A N ET AL.

SURFACE LAYOUT

The most obvious and apparent layout is the physical layout of the display and its functional properties. Here the user sees windows, screens, and information types displayed thereon. The functional properties are the apparent relationships between windows (spatial location) and the linking of information from one window to another (semantic properties). But it must be emphasized that the surface layout is concerned only with what is seen on the screen both statically and dynamically.

Design issues at this level include (a) the use of color and gray levels for highlighting, (b) the amount of information that can be presented in the windows or screens, (c) the size and resolution of the display, (d) the time to fill windows or screens with information, and (e) the physical location of the windows. The designer plans the surface layout not only for efficient communication of bits of information but also to convey a sense of how the whole surface is to be interpreted. For example, overlapping windows take advantage of the perceptual tendency of the user to infer depth from interposition cues. Consequently, an overlapped window is not seen as a window that has become " L " shaped, but rather a rectangle behind another rectangle.

MACHINE LAYOUT

Internal to the computer in terms of hardware and in terms of software is the computer representation of the surface layout. The machine layout drives and supports the surface layout. It includes all of the hardware display devices and interface circuitry as well as the software drivers that create windows and coordinate displays on multiple screens. The machine representation is extremely important both from a practical point of view in that it must support the surface layout and from a conceptual point of view in that it embodies a representation of not only the surface layout but also inferences about how the user views and responds to the information on the screen. The machine representation determines the speed of operation, screen fill, display resolution, and other factors in the surface layout.

COGNITIVE LAYOUT

Finally, the user interprets and responds to the surface layout. Generally, the user derives from the system a "mental model" of what is going on. Alternatively, the user may impose (correctly or incorrectly) a mental model onto the surface layout. The term "mental model" is used to convey the idea that the user has a structure with elements and relationships that map to the elements and relationships at the interface level. For example, the user may conceive of a data base as a set of metal file cabinets with drawers for different areas and alphabetized folders in each drawer containing pieces of paper with text on them. The model may prove useful, not because it is a true representation, but because it is concrete to the user and because there is a formal similarity between the mental model and the actual computer representation of the data structure and its physical properties.

The idea of a cognitive layout is a special case of a mental model. It is restricted to a mental model of the surface layout, that is, it is a representation of the elements and relationships among elements that appear at the user interface. In addition to the cognitive layout of the interface, there will probably be a more extensive mental model of the computer operations having to do with information not currently appearing on

COGNITIVE LAYOUTS 233

the screen and with functions on information as it passes from one apparent location to another.

As an example of how these layouts are used, we take the concept of windows. A window from the machine layout may be represented as a device driver to which specified files may be written. From the machine point of view, we often think of the surface layout as a window into which something is projected. At the surface layout what appears is not a window. Rather it is a display of text with boundaries separating it from other text. At the cognitive level, the layout may be represented as overlapping sheets of paper. Although the designer may conceive of the surface layout in terms of windows, it is probable that the user would never think of these as "windows" were it not for the terminology generated by system designers.

The remainder of this paper will be concerned with the variety of surface layouts that may be generated and the types of cognitive layouts that the user brings to the system. The next section will explore surface layouts using multiple windows and screens. In the last section cognitive layouts will be discussed in terms of how they are suggested by tasks, induced by user expectations, or evoked by the surface layouts. Each cognitive layout entails different modes of processing information and will serve to predict which surface layouts will be superior in terms of user performance.

Surface layouts

Windows and multiple screens are being used to expand the amount of information presented to the user. With overlapping windows on a single screen, there is no increase in the real amount of information displayed at any one time. However, there is an apparent increase in that users infer that the information is there although it is covered up. A real increase in the amount of information is achieved only with larger/higher resolution screens or with multiple screens.

While multiple windows actually divide the display area into small chunks, multiple screens truly multiply the actual area. Surface layout with multiple screens is analogous to that of non-overlapping windows. However, to expand the utility of the display, a number of methods of coordinating the screens have been proposed and developed by Shneiderman, Shafer, Simon & Weldon (1986). The surface layouts presented below employ these methods. In each case we will assume that there are two or three screens.

FUSION

Three screens may be configured as if they are really one long screen with three times as many lines but chopped into three pages. The top panel of Fig. 3 illustrates the bot tom/top fusion layout for three screens. When screens are initially filled, the first screen displays lines 1 to n, the second screen displays lines n + 1 to 2n, and the third screen displays lines 2 n + l to 3n. If the lines are scrolled up one line, the screens display lines 2 to n + 1, n + 2 to 2n + 1, and 2n +2 to 3n + 1 respectively, As simple as fusion may appear, there are a number of complex layout issues. If one is editing a file, where is the current line displayed? Should it be anywhere among the 3n lines or should it be constrained to the middle screen? The problem is that the first and third screen are in a sense peripheral to the middle screen. We will see later that there may be a pervasive cognitive tendency to think of the middle screen as the "working" screen.

234 K. L. NORMAN ET AL.

H , eo

I 26 • 8

~ o , ~1

Bottom-top fusion

15 I~c:::: .......................... ............... ; ................. ................. ~ ~

lib ~t~ ................................. ->,~

[i ..... ' .... ' ""°H

" 25 i" ..... 61

RiohHeft fusion

i ~ 1,6 ̧~̧ (~ i,: ~:.~ :i~z:~i~,~:,,~:i~:~i~ ~:~j~'~ll

FIG. 3. Two configurations for the fusion of coordinated multi-screen displays.

Fusion may also be done in a right/left manner to generate a screen with lines three times as long. The bot tom panel of Fig. 3 shows three screens configured so that lines on one screen continue on the next.

SYNCHRONIZED AND INDEPENDENT SCROLLING

Scrolling describes the movements of lines on the screen up or down. In synchronized scrolling, the screens are linked, so that as lines are scrolled on one screen, linked lines on the other screens are simultaneously displayed. Synchronized scrolling is illustrated in the top panel of Fig. 4. This layout provides the capability for additional information relevant to what is on one screen to be available on an adjacent screen. There are a number of applications of synchronized scrolling. A program listing of code on one screen could have linked comment statements on a second screen. Assembled code, nesting structure, error messages, etc. could also be shown using synchronized scrolling.

The surface layout of synchronized scrolling may be thought of as a linking of the lines of two or more different files. I f scrolling is line by line, the cognitive layout may interpret the display as showing only 1 to n lines of a file having lines that are three times as long as would have appeared on one screen as in right/left fusion. I f scrolling is linked by groups of lines, then the cognitive layout will probably interpret the screens as accessing different files chunked into related groups of lines.

In independent scrolling, the user can browse through one or more files and indepen- dently access different points (see the bot tom panel of Fig. 4). Independent scrolling is useful in programming and writing when one needs to refer back or forward to other parts of the file for comparison. An interesting issue in independent scrolling has to

cOGNITIVE LAYOUTS 235

t f IPrOgr°m . . . . . . . . . . . ~1 t I" I SynchrOnized`` ~ I / statements I I I / lines of r l

screen I!/ on the B I:/ other screen H

2~ I:!~ z~ ~t

Synchronized scrolling

tlj- i One part of a file - i { ~ i i ' '~":: ........................... ~ ............................ ii ~J'52 Another part o f i i ! .................................................................... ~]!;i~ l~J the same or [!I I ~ can be v,ewed ~1 ~ 'I

i N can be viewed H :iii I!/76 on the °ther" ~i

Independent scrolling

FIG. 4. Two configurations of scrolling on coordinated multi-screen displays.

do with how the user refers to a particular screen and which screen is currently active. Should there be only one cursor or should there be a cursor on each screen?

TRIGGERED CHANGES

A surface layout may be structured so that if the user makes changes to the contents of information on one screen, it produces changes in the information displayed on a linked screen. This layout is illustrated in Fig. 5. A change, for example, in a numerical value of some variable on one screen would trigger a change in a graph displaying the variable on another screen. In another example, a program listing could be shown on the middle screen. The leftmost screen could show a file of input cases, and the rightmost screen could show a file of output cases. Either the program listing or the input cases could be changed to allow for the debugging of the program or editing of input cases.

SELECTION

A special case of the idea of triggered changes is that of selection. One screen is defined as the working screen. An item is selected on the working screen. Instances of the item and/or detailed information about the item could be displayed on another screen without changing the contents of the working screen. This layout is illustrated in the top panel of Fig. 6. A particular file may be selected from a list on the left screen which is then displayed on the fight screen. Selection could also be used to find an instance or the declaration of a variable and then position the file on the right screen. For example, in debugging a program, the cursor could be used to select a program variable. On the adjacent screen the data declaration statement and other occurrences of the variable could be displayed (Shneiderman et al., 1986). Similarly, one could

236 K.L. NORMAN ET AL.

f~ Data File: Prices ~ li| I02332 12 IO II H 172525 II 2245 I~! I~ 22 23 42 532231

25 43 23 22 31 48

Raw data file Summery statistics Projections

Data reduction and projection

Il l appears on this ~ 8 while (s<>eof) ~|

Lower case Procedure capitalize

Input-Program-Output

I~ APPEARS ON B H THIS SCREEN

Upper case

FIG. 5. Some configurations for triggered changes on coordinated multi-screen displays.

select a citation in an article shown on one screen, display its occurrence in the list of references on the next screen, and finally, display other pertinent bibliographic information on the last screen.

COPY

A copy o f the working screen could be made and displayed on another screen for reference. The copy layout is shown as in the middle panel o f Fig. 6. For example, a directory listing of documents on a disk shown on the working screen could be copied to another screen for future reference rather than having to generate a printout. The copy screen could also be thought of as a clipboard for transferring text or graphics between applications. Although there is no actual necessity in having the clipboard displayed, there is much utility to the user in having a constant visual reminder of its contents.

CLONE

The copy, as described above, gives only a static snapshot of what happened on the working screen at the point of copy. In some cases we may want the copy to be a working screen as well. A clone is a copy that creates a concurrent job; and, in a sense has a life o f its own. The bottom panel o f Fig. 6 illustrates the clone layout. One may be running a program and entering data up to a point at which you want to create two divergent runstreams. In programming, one may write a procedure on one screen using the editor, then produce a clone o f this work on a second screen. The second screen can then be used to produce a second version o f the procedure for comparison.


~T Ongoing Projects: H Annual Report I~J |The Experiment I f l B°°kChapter ~

The Experiment: I I N~I We are conducting H ' ~ research on the

Select , / f use of multiple I~ I rE screen terminals. ~iil

,~ i i~ i i i~ l ~ . . . . . . . ~"i'i l i'~i ~ i~ i ~.

Selection

~directory e: I Bprog I.bas !~! ~l myprog, pas I II{] parapr°g" c°m " I~!

,gopoprog W

Copy

3 direct°ry a: 1~ I ~ prog I.bos |9 " ) myprog, pas /~.

paraprog, com /~ E igOpOprOg" sys "*' J!l

Copying

~,:prog ml , .~ll Enter Nume: Fred nter Time: 2:30 I Start Clone d~Hello Fred. What's nter Name: Ted M

Cloning

FIG. 6. Illustrations of selection, copying, and cloning on coordinated multi-screen displays.

The user can then compile and execute both versions on the screens to be run concurrently. The problem with clones will be how to refer input to them. A toggle is needed to switch keyboard input from one concurrent process to the other.

Cognitive layouts Each of the cognitive layouts that follows is based in part on theoretical work in cognitive psychology. These theories will help to delineate the implications of evoking a particular cognitive layout and the types of surface layouts and tasks that help to induce that cognitive layout in the mind of the user.

LINEAR ARRAY LAYOUT

The simplest cognitive layout is a close match to the physical layout of three screens in a linear spatial array as illustrated in Fig. 7. Cognitively, an order is inferred, whether left, center, right or top, middle, bottom. The order allows for spatial relationships: "to the left of", "to the right of", "on the bottom of". Furthermore, the layout provides conceptual anchors: "leftmost", "rightmost", "top", "bottom". These are important points of reference.


Fusion:

Pagel

Text like this can just go on and on from one page to another because people ore so wordy and terribly

Panorama :

Page 2

verbose, but what does it matter when we hove so many more screens to on which to view oil of this wonder-

Po~e 3

full text. through which we love to scroll and search for words not unlike"ook" and "birch"

Inferred O r d e r :

16 June 84 Section I

I t s y b i t s y

17 June 84 Section 2

Modsywodsy

18 June 84 Section 5

Bigsywigsy

FIG. 7. Linear array layout for (a) bottom-top fusion. (b) panoramic right-left fusion and (c) inferred order from information.

The linear array layout engages what has been called "spatial paralogic" by Desoto, London & Handel (1965). In their work on problem solving involving comparisons, it was assumed that objects were spatially represented by subjects solving problems. Verbal information about order was spatially represented and comparisons were assessed by inspection of the spatial image. In the same way, when a user must make comparisons between information displayed in windows, a part of that comparison will be in terms of its spatial location.

The linear array layout is persuasive because it matches the physical layout of the screens. It may be further supported by the surface layout. The idea of fusion suggests an ordering of the screens that correspond to the sequence of lines in the file. Other types of surface layouts may not involve an ordering of the screens. Nevertheless, users may infer an ordering as illustrated in the bottom panel of Fig. 7.

One very obvious conclusion about the relationship-between information and the location of the screen, is that information whose order is congruent with location will


be less confusing than information that is not congruent. If pages of text go from previous to current to next, then they should be arrayed on the left, center, and right screens, respectively. The important point is that interface designers should determine first the inferred order on the part of the user and follow this order in their design.

INFORMATION INTEGRATION LAYOUT

Many tasks require the user to extract information from different windows and in some way integrate the information in order to make a decision or judgment. This entails the cognitive operation of information integration described by Brunswik's (1946) lens model. The spatial layout illustrated in Fig. 8 includes the user since he or she is an

Source A Source 8 Source C

Information Informotion Infor motion

@ FIG. 8. Information integration layout for multiple sources of information.

integral part of the picture. The direction of information is from the source (the screen) to the integrator (the user). In general, there is no implied order of the windows as is the case in the linear layout; however, the user may infer an order due to the surface layout or some inherent property of the information displayed on the screens, such as the utility of the information.

An example of this task would be trying to interpret what a computer program is meant to do. Using synchronized scrolling, one screen may display the code, another comments, and a third a flowchart. The user must extract and integrate information from the three screens in order to assess what the function of the program is.

Another example can be taken from the task of an air traffic controller. One screen may display the location of airplanes. A second screen displays aircraft descriptions, altitudes and headings, and flight plans. A third screen may display weather data and the text of communications between the ground control and the pilots. Again the user must extract information from the sources and combine it in some manner to make judgments about flight instruction to the pilots.

240 K . L . NORMAN ET AL.

The screens are in a sense windows into different data bases, networks, or application programs. Although each is an independent source of information, the displays may be coordinated to present information relevant to the particular task at hand. Research in cognitive psychology has shown that it is important to display multiple sources simultaneously so that the user can compare and combine information. Biases due to order effects and memory limitations are introduced when the user has only one screen and must switch from source to source to get the relevant information.

SELECTIVE ATTENTION LAYOUT

In a similar manner to the information integration layout, the user may view the screens as information channels, but instead of integrating the information, the task requires that selective attention be paid to one of the screens. One theory in psychology has been that attention is much like the filter used in selecting a broadcasting channel (Broadbent, 1958). This layout is shown in Fig. 9. The user attends to only one screen

Channel A Channel B Channel C

Information currently being ignored F/RE I/.I

Information currently being attended to

Information currently being ignored FOOD !

f \ z

FIG. 9. Selective attention layout.

at a time, but may switch screens at will. A simple example of this might be the user who signs on to three different terminals at once, starts working on one, but switches to another when the computer exhibits a long response time.

Alternatively, others have theorized that attention is not like channel switching. Rather the mechanism of attention is more like an active filter that attenuates the signals from all but one source. The attended source is fully processed; however, some information from other sources may make its way into attention depending on its salience and level of processing (Deutsch & Deutsch, 1963).

An example of the attention layout would be the task of reading a manuscript for errors on one screen while system messages and news items are appearing on one or several other screens. The user's attention may be shifted if a salient item appears in


the news (particularly if it mentions the user's name). Perhaps the most striking example of the attention layout is that of multiple screen displays in control rooms and cockpits. A person may voluntarily shift attention from one screen to another searching for information, or attention may be drawn involuntarily by highly potent information such as flashing inverse video.

LEVELS OF PROCESSING LAYOUT

Generally, as humans attend to a source of information, they process the information so that features are extracted, inferences are made, and links with items in long-term memory are made (Craik & Lockhart, 1972). Information at the early stages of processing is generally not remembered; but as information is subjected to further processing, it becomes more meaningful and easier to remember. A number of situations suggest the idea that information on one screen is processed and appears on another screen. Each screen displays the information at successive levels of analysis. Figure 10 illustrates this layout. The top panel shows how levels may successively extract features from the formal syntactic properties of the input to the semantic information that it contains. The first screen shows the letter "A" outlined and in a stylized font; the second screen shows only the font and the letter, and the third only the letter.

Levels of processing may also be applied to levels of abstraction from detail to generalization. In the bottom panel of Fig. 10, for example, program code" appears on the first screen, line-by-line comments on the code appear on the second screen, and narrative comments on the function of the code appears on the third screen. Another example that suggests the levels layout is shown in the top panel of Fig. 5. One screen

-el Feature extraction from formal syntactic 1o semantic levels

Extraction from detail to generalization

FIG. 10. Levels of processing layout.

242 K.L. NORMAN ET AL.

displays the file of the raw data; the next screen displays the file of descriptive statistics; and the third displays the results of statistical inference. Although the user may scan the first two screens of raw data and descriptive statistics for errors, little if anything will be remembered since it is not as meaningful as the deeper levels of processing. It should be pointed out that the levels layout is related to the linear array layout since it implies an ordering of the information on the screens.

M E M O R Y STORAGE LAYOUT

The model of different memory stores in psychology is a pervasive one. Three memory stores have been proposed. The first is a short-term sensory store (Sperling, 1960). It is extremely rich in amount of information, but it decays rapidly with time. It can be scanned for information in much the same way that one would visually scan a photograph. The second is short-term memory. It is thought of as a limited memory buffer (Miller, 1956). New items bump out old. Furthermore, they must be maintained in memory by repetition. Related to the short-term store is the idea of a working memory as discussed by Feigenbaum (1970). The third memory store is long-term memory. Long-term memory is thought of as unlimited in capacity, but items may nevertheless be lost due to interference or lack of appropriate retrieval cues. Long-term memory is largely semantic and organized in some meaningful way (Bower, Lesgold & Tieman, 1969). The "three-box" memory storage layout is shown in the top panel of Fig. 11.

Users may adopt a memory stores layout for the same reasons as the levels of processing layout. Information may be scanned and extracted from a central screen

Three-box model of memory:

On,,m,ted ] detail of input

Short-term sensory store

,#

Limited working space

Short-term memory

Unlimited data base of information

Long-term memory

Clipboard

Temporary storage

Application program

Working memory

Files: disks folders

File storage

FIG. 11. Memory storage layout.


representing the not ion of a working memory as shown in the bottom panel of Fig. 11. It may be temporarily displayed via a clipboard on a screen to the left. Finally, the information may be more or less permanently catalogued or referenced by a file management system shown on a screen to the right. Just as one has constant access to all memory stores in the brain, the user should have constant (simultaneous) access to different types of memory stores at the human-computer interface.

ZOOM IN-ZOOM OUT LAYOUT

One cognitive representation that is visually powerful is the idea that one screen is a blow-up of a section o f another screen. The inverse is that one is a bird's eye view or global picture o f another. This layout becomes particularly powerful when it is induced by an expanding image that zooms into focus on a particular part o f the image. Figure 12 shows instances o f this layout.

Magnification end enhancement: <H A Menu selection:

Select One:

Copy ~ Edit

File

Edit: A

Change ~ . Search

Format

Change

Insert

Delete

Over Print

Hierarchical Data Bases:

Operational A Productivity Performance Functional Objectives: Objectives:

A Productivity A, Task Time • B Reliabihty 8 Quality D Secority \ l DNombero, D. Flow of Into Actions E Ease of Use

A Task Time Performance Indicators:

A, Meetings 8. Documents C. Distribution

q D. Communicate

FIG. 12. Zoom in-zoom out layout.

Even though this layout is predominantly visual, one can imagine it occurring with verbal and conceptual material as well. For example, one screen may display the table of contents o f a document with one of the sections highlighted. The next screen displays an outline o f the chapter with the title o f one section highlighted. The third screen

244 K . L . NORMAN ET AL.

shows the text in that section. The multiple screen surface layout is particularly useful because the user can refer to the contents and location of the text by glancing at different screens without losing the point of reference. In the visual process of zooming in the picture is lost. The user must hold it in short-term memory. Similarly, while zooming out the detail is lost and the user must remember it. In the three screen layout, three visual slices are retained. The user can then flip from one to the other while holding each in place. It remains to be seen if for specific tasks the three-screen layout is superior to the one screen which visually zooms in or out.

The blow-up layout may also be induced by hierarchical menus and by search in a database. For example, one screen may display a representation of the menu tree with the current location marked, the second screen may list the current choice and the alternatives that may be selected. The third screen may give detailed explanations of the consequences of selecting each of the alternatives. In fact, the user may indicate a particular alternative and the detailed description of that alternative may be presented on the last screen.

The zoom in-zoom out layout is related to spatial and network models of long-term memory. For example, Collins & Quillian (1969) proposed a network model for the time that it takes to answer questions such as "Does a bird have wings?" Response latency was a function of the number of assumed links that must be traversed to associate objects with properties. When the zoom in-zoom out layout is suggested by the human-computer interface, it may help to facilitate mental processing and reduce time in the use of hierarchical databases by visually pulling the user across the relationships.

PERSPECTIVE LAYOUT

The ability to view a scene from one perspective and infer what it looks like from another is a part of the cognitive development of the mind. In thinking and problem solving, it is often necessary to rotate objects mentally or to assume different perspectives. Mental rotation is a difficult and predictably time consuming act (Shepard & Metzler, 1971) yet it is essential in many forms of information processing. In order to help the user, information may be displayed on different screens to show a situation or an object in different orientations or from different perspectives. Each perspective may show a facet or a different attribute of the same object. Multiple views of programs, e.g. flowcharts and text, were shown in the PECAN system (Reiss, 1984). A visual representation of a perspective layout is shown in Fig. 13.

Plonor perspectives:

FIG. 13. Perspective layout.

coGNITIVE LAYOUTS 245

Probably the most powerful induction of the perspective layout is generated by viewing a solid figure from three different angles. The views may be orthogonal, in the sense that the three facets are at right angles; or they may be oblique. If they are orthogonal, as shown in Fig. 13, one screen may be viewed as showing points on the X, Y axes; another on the X, Z axes; and the third on the Y, Z axes. Single-screen presentations may show different facets by rotating the solid and projecting its two- dimensional representation or linear perspective drawing; but in doing so only one facet can be shown at a time and the user must remember what the image looked like from the other perspectives. The three-screen layout allows the simultaneous presentation of multiple facets. The user can rapidly glance from one perspective to another without having to hold in memory previous views for comparison.

Computer simulations for training and guidance systems may effectively use the perspective layout by generating graphic images of objects from different perspectives. In maneuvering a space craft, computer-generated views from different locations may be necessary in order to position objects in space when they are obscured by other objects.

The perspective layout may be effectively used when working with a relational database. For example, one screen may display the information from the perspective of author's name, the second from the perspective of book title, and the third from the perspective of subject. Search may be directed from each of these facets. Facets may also be combinations of attributes in the database such as (a) author's name and book title, (b) book title and subject, and (c) subject and author's name. The difficulty of formulating and comprehending Boolean sets may be reduced by creating facets out of combinations of attributes.

The perspective layout may take the position of either viewing from the outside looking in or viewing from the inside looking out. Information on multiple screens may be thought of as converging in on a single entity or from a single vantage point diverging out to multiple surrounding objects. In the first case, the visual scope of the user is increased by artificially adding monitors focused on a single object. In the second case, the visual scope is increased by providing a panorama around the user as in the linear array layout and the information integration layout. A well-designed human-computer interface must convey a clear, unambiguous orientation of perspective or there will be a loss of orientation and the visual scope will totally collapse.

Techniques for inducing a cognitive layout

No matter how reasonable or elegant a surface layout may be in the eyes of the designer, if the appropriate cognitive layout is not created in the mind of the user, it will be a constant source of confusion and frustration. Although manuals may help to suggest a cognitive layout and create expectations about how things work, it will be the perceptual aspects of the surface layout that ultimately convince the user. Early work in the psychology of perception testifies to the strong perceptual forces that lead a person to view a stimulus in one way rather than another. The principles of perceptual organization are now well known and have been demonstrated over the years (i.e. Wertheimer, 1923). These principles as well as a number of other techniques from art, film, and psychology can be used to create the appropriate cognitive layout. The


following ideas are offered as a starting point and are by no means exhaustive of the techniques that can be used.

SPATIAL GROUPING

The perceptual system tends to group objects that are (a) in close proximity or (b) similar in size, shape, color, or orientation. Information on the same screen is automati- cally grouped by physical proximity and by separation from information on other screens. This principle works against surface layouts such as fusion and synchronized scrolling. It is difficult for the user to group lines of text or program code that straddle two screens. On the other hand, surface layouts such as the information integration layout and the selective attention layout are facilitated by physical grouping of information from different sources. When information must be grouped across screens such as lines on a spreadsheet, then strong forces must be used to cause the user to see the lines continuing on the next screen. Type font, size, color and texture may be effectively used with discretion.

TEMPORAL GROUPING

The timing at which information appears on multiple coordinated screens can also have a powerful effect on perceptual grouping of information and on the inferred order of information. Although an instantaneous screen fill may be achieved at the machine layout, it will probably be more effective to trigger the fill of multiple screens and of windows on different screens at different times. When the linear array layout is to be suggested, screens should fill in their inferred order rather than fill simultaneously. In the case of triggered changes, the cognitive layout of levels can be induced by delaying changes on other screens slightly so that a cause-effect schema is suggested. For bot tom-top fusion in which multiple screens display one long continuous file, fill should start on the first screen and then proceed to the next. For left-right fusion in which multiple screens display long lines that extend across screens, fill should start simultaneously at the top of each screen and proceed downwards. In each case, the designer needs to know the inferred relationships among information and the order of events in the appropriate cognitive layout and then create the surface layout to support that view.

ANIMATION

Animation illustrates things that words and still pictures cannot. Few if any instructions are needed for most video games when designers show an animation of the course of play before the game is initiated. Similarly, confirmatory feedback about how the system operates may be provided by animation. When a system opens a file for text editing, it may show an expanding rectangle to illustrate the operation. When it closes a file, it shows a collapsing rectangle. Movement provides a powerful cue to the perceptual system which must interpret what happened and where it went. Animation will prove particularly useful in inducing the levels of processing, memory storage, zoom in-zoom out and perspective layouts. The levels of processing layout is meant to show the metamorphosis of information from one screen to another. Rather than merely displaying the products at each discrete level, animation may be used to show successive frames stripping away some features and developing others. The memory storage layout shows the origin and destination of information at the human-computer

coGNITIVE LAYOUTS 247

interface. Animation may further help to induce this layout by showing the transfer of information from one store to another. The zoom in-zoom out layout can be animated by showing expanding and collapsing figures from their point of origin. Finally, the perspective layout may use animation techniques to rotate graphic objects on the screen. Loss of orientation can be avoided by starting at a known viewpoint and gliding the perspective to the new viewpoint in a continuous manner.

Conclusions

The modes of representing the interface between the user and the display in a multiple window or multiple-screen workstation are categorized as either surface layouts, machine layouts, or cognitive layouts. Surface layouts represent the physical formats in which the information is displayed. Machine layouts are the internal, machine-level representations of the surface layouts. Cognitive layouts are the representations of the surface layouts !n the mind of the user. Designers are creating many new and innovative surface layouts for workstations of the future. However, it is hypothesized that unless designers take into consideration the cognitive layout of the work on the part of the user, the most elegant and rational surface layouts in the eye of the designer may only lead to confusion and frustration in the mind of the user. Instead, the surface layout must suggest a cognitive layout that increases the visual scope of the user. The cognitive layouts discussed in this paper are by no means exhaustive, but represent a point of departure in system design meant to stimulate further thinking about how the user interprets what is happening at the human-computer interface.

This work was supported by a grant from IBM Federal Systems Division.

References

BLY, S. A. & ROSENBERG, J. K. (1986). A comparison of filed and overlapping windows. ACM Chr '86 Conference Proceedings, Human Factors in Computing Systems, 101-106.

BOWER, G. n., LESGOLD, A. M. & TIEMAN, D. (1969). Grouping operations in free recall. Journal of Verbal Learning and Verbal Behavior, 8, 481-493.

BROADBENT, D. E. (1958). Perception and Communication. London: Pergamon Press. BRUNSWIK, E. (1946). Organismic achievement and environmental probability. Psychological

Review, 50, 255-272. BURY, K. F., DAVIES, S. E. & DARNELL, M. J. (1985). Window Management: a Review of

Issues and Some Results from User Testing (HFC-53). San Jose, CA: IBM Human Factors Center.

CARD, S. K., PAVEL, M. & FARRELL, J. E. (1984). Window-based computer dialogues. INTER- ACT '84, First IFIP Conference on Human-Computer Interaction, pp. 355-359 (September).

COHEN, E. S., SMITH, E. T. & IVERSON, L. A. (1986). Constraint-based tiled windows. IEEE Computer Graphics and Applications. 6, 35-45.

COLLINS, A. M. & QUILLIAN, M. R. (1969). Retrieval time from semantic memory. Journal of Verbal Learning and Verbal Behavior, 8, 407-428.

CRAIK, F. I. M. & LOCKHART, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671-684.

DESOTO, C. B., LONDON, M. & HANDEL, S. (1965). Social reasoning and spatial paralogic. Journal of Personality and Social Psychology, 4, 513-521.

DEUTSCH, J. A. & DEUTSCH, A. (1963). Attention: some theoretical considerations. Psychological Review, 70, 80-90.

248 K.L. NORMAN ET AI_,

FEIGENBAUM, E. A. (1970). Information processing and memory. In NORMAN, D. A., Ed., Models of Human Memory. New York: Academic Press.

MILLER, G. A. (1956). The magic number seven, plus or minus two: some limits on our capacity to process information. Psychological Review, 63, 81-97.

MILLER, i . J. & MCMILLAN, T. (1984). Windows while you work. Popular Computing, May, 96-110.

MYERS, B. A. (1984). The user interface for Sapphire. IEEE Computer Graphics and Applications, 4, 13-23.

REISS, S. P. (1984). PECAN: program development systemsthat support multiple views. Proceedings of the Seventh International Conference on Software Engineering, pp. 324-333.

SHEPARD, R. N. & METZLER, J. (1971). Mental rotation of three-dimensional objects. Science, 171, 701-703.

SHNEIDERMAN, B., GRANTHAM, C., NORMAN, K. L., ROGERS, J. & ROUSSOPOULOS, N. (1983). Evaluating multiple coordinated windows for programming workstations. Paper presented at the Eighth Annual Software Engineering Workshop, NASA Goddard Space Flight Center, October 1983.

SHNEIDERMAN, B., SHAFER, P., SIMON, R. & WELDON, L. J. (1986). Display strategies for program browsing: concepts and experiment. IEEE Software, 3, 7-15.

SPERLING, G. (1960). The information available in brief visual presentations. Psychological Monographs, 74, 1-29.

WEISER, M. (1985). CWSH: the windowing shell of the Maryland window system. Software: Practice and Experience, May, 515-519.

WERTHEIMER, M. (1923). Untersuchungen zar Lehre von Gestalt: II. Psychologische Forshung, 4, 301-350. Abridged translation by M. Wertheimer (1958). Principles of perceptual organization. In BEARDSLEE, D. C. & WERTHEIMER, M., Eds, Readings in Perception. Princeton: Van Nostrand.

WOODS, D. D. (1984). Visual momentum: a concept to improve the cognitive coupling of person and computer. International Journal of Man-Machine Studies, 21, 229-244.