Journal of Computer Assisted Learning (1996) 12, 89-102

Measuring users' performance with interactive information systems R. Oliver Edith Cowan University, Western Australia

Abstract The purpose of this study was to investigate the discrete forms of knowledge required for successful use of interactive information systems (11s). Using an existing framework, an instrument was created to measure the achievement of three discrete forms of knowledge among novice users of an CD-ROM encyclopedia. The instrument included tasks designed to measure users' knowledge and skills in managing and controlling the system, navigating between information nodes and selecting appropriate search strategies given information related problems. The study examined hierarchical knowledge and compared the development of these forms of knowledge and skills among novices using two different 11s. The results found significant differences in the user performances between two systems in tasks that measured navigation skills and an understanding of the information organisation. It is suggested that these differences were due to differences in the interface design of the two systems and suggestions are made for selecting and implementing 11s systems in school and educational settings.

Keywords: CD-ROM; Interactive systems; Performance; Schools

Introduction

There are many interactive information systems (11s) available for personal use in schools and home environments. The most accessible form appears to be based on CD-ROM but many more systems are available to remote users through telecommunications networks. The growth and spread of these systems is rapid and it is likely that they will soon be a familiar part of the educational landscape in both CD-ROM and network forms.

Interactive information materials and environments appear to offer significant advantage over conventional materials when used for instruction and information seeking purposes. A number of studies have demonstrated significant achievement gains attributable to the learning and information opportunities afforded by the implementation of 11s (e.g. Saga, 1992; Riding

Accepted 15 June 1995

Correspondence: Dr. Ron Oliver, Edith Cowan University, Bradford St, Mt Lawle 6050, Western Australia. r.oliverdcowan.eciu.au

89

90 R.Oliver

& Chambers, 1992). While there may be many intuitive reasons to suggest the advantages of the increased use of 11s as educational tools, studies have demonstrated that there are many factors that can influence the quality and quantity of learning outcomes from these computer applications in schools.

High among the factors that may impede successful interaction are the human-computer interface. Most computer users frequently struggle with the unique characteristics of computer interfaces. The higher the cognitive demand associated with controlling the interface, the less the user is able to attend to the key purposes of the interaction, that is, the content of the package (e.g. Heller, 1990; Trumbull et al., 1992). This problem is compounded with 11s where individual interface designs of the multimedia and hypermedia applications have many unique and novel characteristics.

In schools today, there is growing use of encyclopedias and electronic books on CD-ROM. Depending on the way in which they are used and the skill level of the user, these systems have the capacity to either reveal or conceal much of the contained information. It is clear that the skills and knowledge required for success in the domain of 11s extends beyond those required for conventional systems. The purpose of this study was to investigate the discrete knowledge forms which impinge on successful use of 11s and to establish a framework that could be used to investigate skills and knowledge development among naive users.

Interactive information systems

Previous research by the author with naive users of CD-ROM has revealed large differences in their abilities to access and retrieve information in response to information-related tasks and activities (Perzylo & Oliver, 1992; Oliver & Perzylo, 1994). Observations of the actions and reactions of naive users suggested several forms of knowledge and skill as being influential in describing a students' success with the electronic information system.

Computer skills It was apparent that previous computer experience played a big part in students' confidence and, as a consequence, their capacity and willingness to explore the software. Although students' confidence and skill in computer use increased quickly as they developed familiarity with the system, it was evident that the syntactical skills required to drive the 11s were an integral component of the knowledge required for successful 11s use. Another impediment to successful information retrieval appeared to stem from the students' poor conceptions of the size and extent of the information base and a lack of familiarity with the available retrieval mechanisms. Observations of the actions and reactions of the naive users of the 11s revealed patterns and trends for which mental model theory appears to offer some explanations.

Mental models Most mental models, defined simply as conceptual cognitive structures used to explain and confront novel problems (Jib & Reeves, 1992), formed by

Measuring users' performance 91

people in response to devices and systems are naive, imprecisely specified, and inconsistent. Nonetheless, these models function to guide much of the user's behaviour (Norman, 1983). Mental models guide the naive user of electronic information systems by providing a framework and schema linking the important elements of the system. When a novel or unknown condition is encountered, the user's mental model acts to guide the decision making that follows.

People construct mental models for most systems with which they interact and many divergences occur in the models constructed by different users of a single system. The construction of mental models is influenced by personal traits as well as by previous experience (Ackerman et al., 1989). Successful use of IIS like other knowledge domains is very much dependent on the development of adequate models of the information base and its retrieval mechanisms. Mental models are frequently aided in interactions with 11s through the use of navigation metaphors and analogies in interface design. There is always the prospect, however, that the user's mental model may not match that planned by the software developer. Previous research, for example, has shown that the use of metaphors and analogies as controlling and design mechanisms can limit rather than strengthen naive users' models of electronic systems (e.g. Allwood & Kalen, 1993; Gay & Mazur, 1989).

The ease with which novices can create an adequate model of the interface of an IIS is an important consideration in judging the overall quality of that program. The quality of interactions with any 11s is very much dependent on the functionality and accuracy of the mental model which the user constructs of the system (Reeves et al., 1993).

The possible impact of users' mental models on successful use of 11s was identified in our previous classroom studies which had revealed a distinct preference among users for one 11s over another. This was despite the fact that students had similar amounts of instruction and activity with each. It was suspected that users' mental models of the preferred 11s were more successful and accurate than those of the alternative system.

Knowledge forms Within any learning environment, there are forms of knowledge specific to the learning domain that describe achievement and performance. Alexander and Judy (1988) describe these forms of domain-specific knowledge as declarative, procedural and conditional. These forms of knowledge are considered discrete and tend to develop hierarchically. Declarative knowledge is factual and represents knowledge of the discrete components at the lowest level. Procedural knowledge is defined as the compilation of declarative knowledge into functional units. This level of knowledge is achieved through linking and associating the declarative knowledge and represents a form of comprehension and understanding. Conditional knowledge is described as the understanding of when and where to employ procedures and is achieved when a learner makes strong links and connections between the discrete components of knowledge at the lower

92 R.Oliver

levels. Our experience in previous studies had identified various forms of 11s knowledge that could readily be described as pertaining to elements of these three forms. The following section demonstrates one way in which the domain specific skills associated with successful 11s usage can be identified and described using the knowledge hierarchy described by Alexander and Judy (1988).

Management: With IIS declarative knowledge represents knowledge of the computer interface and is demonstrated by users' skills in program management and control. The basic knowledge elements that govern success with operating and controlling an ITS are influenced by the users' capacity to control and manage the interface itself. The declarative knowledge represents a user's ability to control and guide tasks involving interactions with the software to seek and display information. This includes a knowledge of such processes as choosing menu options, inputting data in response to dialogue box displays and moving from one section of the program to another. This level of knowledge is very much oriented to computer skills and computer literacy and is likely to develop quickly through practical activity and experience.

Navigation: Procedural knowledge is demonstrated by 11s users who are able to use efficient procedures to access specific information nodes and to move freely between associated nodes. In essence, this is navigation skills. Procedural knowledge is demonstrated by the user who is able to complete tasks involving a sequence or series of steps. These steps usually combine to form an action that can be applied to complete a specific information retrieval task. With 11s it is possible to identify a range of actions of this form that users need are able to employ in response to differing information requirements. For example, users demonstrate procedural knowledge when they browse or use an index lookup to initiate an inquiry. An incapacity to use the index search strategy or to move between media elements would indicate a user with limited procedural knowledge who has not yet developed an understanding of the connections between the discrete steps and is unable to anticipate precisely what will be the outcomes of actions.

Organisation: While the previous two levels of knowledge have described knowledge that is directly measurable in terms of users' familiarity and competence with elements of the information system, conditional knowledge involves an understanding of knowing when and where to apply the procedural steps. It encompasses information skills as well as computing and interface skills. For example, conditional knowledge is demonstrated by a user who is able to choose the most appropriate search strategy in response to an information need. This user understands the nature of the information organisation and is able to choose from among the available procedures to directly access the required information.

The nature of the these three different knowledge forms, management, navigation and organisation suggests a hierarchy. The descriptions of the

Measuring users' performance 93

two higher knowledge forms assume some knowledge from the form(s) that precede them. The initial forms describe knowledge of a syntactical nature while conditional knowledge is a form of semantic knowledge involving an understanding of the interactions of the different 11s elements. Measurements of conditional knowledge are likely to provide the best assessments of users' mental models of 11s systems.

Research aims

Having identified possible forms of knowledge by which 11s users' performance could be measured, the subsequent steps in this research project became to:

create an instrument to measure the attainment these discrete knowledge forms by novice users of an 11s; to implement the test among novice users to investigate the influence of instruction and activity on the development of 11s skills and knowledge; to implement the test among novice users to determine whether the knowledge forms represent a learning hierarchy, and to use the instrument to investigate differences in usage patterns and user preferences between two discrete 11s systems.

Research method

An instrument was planned that was capable of providing information on users' knowledge acquisition across each of the forms described. The instrument required users to complete information seeking tasks at the keyboard with the researcher observing the performance and measuring the level of success. It was intended that the tasks be of a generic form and able to implemented across different 11s systems. In the actual choice of tasks it was necessary to consider the two 11s which the students would be using and to ensure that there was compatibility and uniformity in the knowledge acquisition tasks when completed on alternative 11s systems.

Choice of 11s systems The two systems that were chosen had been previously used by the researcher, one of which was found by users to be distinctly preferable to the other (Oliver & Perzylo, 1994). The preferred 11s employed retrieval mechanisms that appeared to more closely match those with which the students were familiar with conventional paper-based systems. The significant differences between the interface design and information organisation strategies suggested that successful use would require some contrasting mental models on the part of users. The following descriptions of the 11s indicate the differences in the information organisation and retrieval patterns of each.

System A: All information nodes were linked and accessible through associated information and hyperlinking. The index system was consistent

94 R.Oliver

across all media and the retrieval of discrete media forms used a series of refinements through a uniform dialogue box. There was little separation or distinction between media forms and strategies available for information retrieval. A series of icons indicated the media form of the required information in the uniform dialogue box. The indexing and organisation system used pictorial elements to facilitate the selection of media sources.

System B: (preferred system) The hypermedia nodes were all stored as discrete entities able to be accessed individually as well as through associated nodes and hyperlinking. For example, each of the pictorial, sound and animation items within the system was able to be accessed through a distinct index system as well as through reference to specific associated text nodes. The media nodes of similar forms were accessed through a discrete index system as a separate component of the main menu. Both systems employed a graphical user interface.

lnstrument development Having chosen the 11s for use in the study, the next phase involved the development of an instrument capable of identifying users' knowledge acquisition and development. The instrument was comprised of a series of tasks that students would be required to carry out using both systems. The use of a common sample of students was designed to enable any differences in performance and achievement to be attributed to the software rather than to factors associated with the choice of subjects.

Management: This scale used tasks that required users' to control and guide the 11s. The tasks involved interacting with the software to display specific information and required very low level information skills. Activities included entering given text into a dialogue box and choosing appropriate options to display and manage the retrieved information. To successfully complete the items, users were required to select from available options using buttons and on-screen icons, open and close windows as well as using scrolling bars to view information. The speed and accuracy in completing these tasks was assumed to be indicative of users' familiarity and comfort with control aspects of the system interface. Successful completion of the tasks was judged to be independent of information skills.

Navigation: This scale provided a measure of users' success in completing tasks that required specific sequences and combinations of steps to move between nodes. Tasks were selected that required users to choose search parameters and to apply these parameters to locate specific information. Users were required to move between information nodes to locate precise information files. This knowledge form required elements of declarative knowledge as well as knowledge of sequences and processes required to complete tasks. In measuring this knowledge form, a record was to be maintained of the level of success achieved across tasks as well as a measure

Measuring users' performance 95

of the ease with which the success was achieved. Both measures provided necessary data to fully describe task performance and achievement.

Orgunisation: This scale provided a measure of users' understanding of the manner in which the information was organised within each system. Tasks required users to locate information that could not be sourced by a single word description. Success in these tasks could be achieved by correctly identifymg the key words and applying appropriate search mechanisms. Performance in this scale was likely to be dependent on the information skills of the users as well as the presence of an accurate mental model describing information organisation, navigation and management.

Table 1. Knowledge forms and tasks by which each was measured

Knowledge form Description Tasks

Management Knowledge of the Performing controlling processes and controls governing use of the program.

functions to effect a desired outcome.

Navigation Comprehension of the processes associated with linking and steering between nodes and sources.

Seekingspecific information nodes through links and associations with existing nodes.

Organisation Understanding of manner in which information is stored and organised.

Seeking information given broad topics r uiring refinement 3 searching.

Table 1 provides a summary of the different scales against which the development of the knowledge forms were to be measured as well as descriptions of the tasks and the nature of cognition and understanding represented by each. The content validity of the tasks was aided by the contributions of subject experts and the reliability of each of the scales within the instrument was measured by the calculation of the correlation coefficient between users' scores in each scale across each 11s.

Method With the software chosen and data collection instruments organised, it remained to implement the two 11s. Two multimedia computers were placed in two upper primary level classrooms of 16 and 13 students respect- ively. Over a two-week period, instruction was provided in the use of each of the CD-ROM based electronic encyclopedia. The instruction consisted of several whole-class demonstrations of the available search strategies and small group practical sessions. The instruction was directed towards use of the software, with demonstrations of the available features and navigation with the interface. The topics covered included; locating information through title searches, using keywords within articles, hyperlinking and

96 R. Oliver

using indices to recover non-text media. Students were shown how to copy, paste and print text and pictures. No attempt was made to discuss or develop information seeking skills that were specific to either system.

Following the instructional phase, students undertook a research based project and used both systems as information sources. The students were rostered to ensure equal usage of both systems. Each student was allocated at least four thirty minute sessions with each system over a 6 week span, for information seeking activities. During this period, students were observed while using the system to gather knowledge on the forms of information retrieval that students were inclined to use and the difficulties faced by novices in using an electronic information system for the first time. At end of the 6 weeks, all students completed the test instrument.

Results

Measures of performance across the three scales were gained through an observation schedule. The first outcome measured was the level of success achieved. For each of the tasks, a numerical score of 0 - 3 was recorded against each student based on the level of success achieved. If the required information was located a 3 was recorded, if the action was completely unsuccessful, a score of 0 was recorded. Scores of 1 and 2 were recorded for minor and major levels of success that could not be rated as 0 or 3. The second scale was ease of use and it too was quantified by a numerical score indicative of the path taken to find the information. Students who employed the most efficient path in solving a task were scored with a 3. Students who chose paths that led away from the required information scored 0. Other students were scored with 1 and 2 depending on the direction and efficiency of the path chosen in retrieving the information. There were three tasks associated with each knowledge level and with the maximum score for ease of use and success level on each task being 3, students were given scores out of 9 in each section of each level.

It may be argued that the measures of success and ease of use are related in some fashion and as a result should not be considered as separate entities. The reason why the scales were separated is that it is possible for a student to receive a high score on the success scale while achieving a very low score on ease of use. Measures of correlation were determined between each of the success and ease of use scales across the three mental models and the resulting values are included in the results for each scale.

Hierarchical knowledge levels Table 2 shows the mean scores achieved by the students in each of the knowledge forms. A comparison of the achievement between the forms was intended to investigate whether the knowledge forms represented a hierarchy. It became evident in observations of the students that although the development of navigation skills and an understanding of information organisation required management knowledge, there was not the distinct

Measuring users' performance 97

need for knowledge of one form for students to be able to demonstrate knowledge of higher forms. There appeared to be several reasons for this.

In the first instance, the test instrument did not provide a measure of the full extent of students' knowledge of each form. For example, there were elements in the test of management that students had not yet learned and results reflected this. Since these elements were not required in the navigation tasks, scores for these tasks were frequently higher than those achieved by the same student in the lower level. Table 2 reveals that among both IIS, students obtained higher mean scores for the items measuring navigation than for the items measuring management skills and knowledge.

Secondly, students' organisational knowledge appeared to include knowledge that extended beyond the domain of these IIS to information systems in general. Some students in the class identified by their teacher as being more able than others, were seen to make successful decisions in tasks requiring conditional knowledge as a consequence of their general information skills. In some instances, these students demonstrated evidence of organisational knowledge before they had developed a knowledge of the management and navigation procedures of the IIS which they were using. These factors tended to suggest that the tasks contained in the instrument, could reasonably identify and measure discrete knowledge forms but these knowledge forms could not be considered hierarchical. Success in higher levels was found not necessarily to be dependent on success in lower levels.

Table 2. Students' scores in scales measuring achievement in knowledge forms.

System A System B mean sd mean sd

Management level of success 7.69 1.83 6.79 1.89 level of ease 6.89 1.82 6.60 1 .so

Navigation

Organisation

level of success 4.07' 1.90 8.27* 1.45 level of ease 4.65 1.84 8.07* 1.41

level of success 4.41 2.21 6.89 1.9s level of ease 4.55 1.76 5.75 1.62

Note: * p < 0.05

Comparisons of students' performances between the two IIS systems provided some interesting results. The following discussion reviews students performances in each of the knowledge forms.

Management: Most students completed the items from the this scale with relative ease and efficiency across both systems. The mean score across all items was quite high with low standard deviations (Table 3). Analysis of variance, using the Scheffe F-test, revealed no significant difference between both groups on either the success (F(1,55) = 3.35, ns) or ease of use (F(1,55) = 0.43, ns). The results reveal that all students achieved a similar level of competence with the graphical user interface of both systems. There were

98 R. Oliver

relatively few differences in the interfaces and most controls operated in a consistent fashion across both systems. From an alternative perspective, it could be concluded that the learners' mental models of the systems design were reasonably accurate and successful across both systems.

Table 3. A comparison of learners' management knowledge and skills

Level of success mean sd mean sd

Ease of use

System A 7.69 1.83 6.89 1.82 System B 6.79 1.89 6.60 1.50

There was a high correlation between level of success and ease of use across this scale (t = 0.86). This was caused by the consistently high scores achieved by students in both components of the scale. There were few students whose scores on both scales differed by more than one unit.

Navigation: Table 4 shows the performance of the students on the tasks measuring their capacity to employ procedural routines within both Systems A and B. There were significant differences in both level of success and ease of use measured across both systems. Students demonstrated significantly higher levels of understanding of the navigation and linking facilities of System B than they did with System A (F(1,55) = 17.11, p < 0.001), these being the principal tasks by which procedural knowledge was assessed.

It was clear across most interactions that System B provided a control structure with which students were more at ease and able to implement in different ways with little difficulty as the task specification changed. On the other hand System A involved steps and procedures that required more thought and decision making. In most instances students tended not to pause and reflect but to experiment with likely options. The end results saw less success with the information seeking tasks.

The measure of ease of use showed a similar significant difference across this scale (F(1,55) = 8.10, p c 0.01). In carrying out the required tasks with System A, students were often able to judge the lack of success that was being achieved. To overcome this, they sought to implement a number of alternative steps, the majority of which provided little extra advantage. The outcome of these actions saw activities with System A taking longer than with System B with generally lower levels of success.

Table 4. A comparison of learners' navigation knowledge and skills

Level of success Easeofuse mean sd mean sd

System A 4.07 1.09 4.65, 1.84 System B 8.27 1.45 8.07. 1.41

Note: * p < 0.001

Measuring users' performance 99

The correlation coefficient calculated to establish whether there was a relationship between the success and ease used to measure this knowledge level, was quite strong (T = 0.77). As with the previous scale this result demonstrated that it was likely that the level of success achieved and the ease of use exhibited were not completely independent.

Organisation: This scale provided a measure of students' understanding of the manner in which information was organised within each system. The tasks that were posed required students to demonstrate an ability to choose and implement the appropriate search strategy for series of information seeking tasks. Since this represented a higher level of knowledge than the previous scale, the significant difference in users' knowledge displayed previously was expected to be evident again in this scale. The results demonstrated clear differences in the students' organisational knowledge between the two systems being used (Table 5).

Table 5. A comparison of learners' organisational knowledge and understanding

Level of success mean sd mean sd

Ease of use

System A 4.41 2.21 4.55 ** 1.76 System B 6.89 * 1.95 5.75 ** 1.62

Note: * p < 0.001 ** p c 0.01

Students achieved significantly higher levels of success in these tasks with System B, (F(1,55) = 12.54, p < 0.001). Observing students, it was clear that they were generally far less successful in accessing the information sources within System A. There were numerous occasions where the required information could not be sourced by a student using System A while the same student had little trouble completing the same task with System B. In terms of ease of use and the efficiency with which information was located, the students took considerably longer to complete these tasks than those described by the previous two scales and among the systems, students took significantly longer and employed more stages when working with System A (F(1,55) = 7.10, p < 0.01).

The correlation between the scores for level of success and ease of use (T = 0.69) was less than that observed for the previous scale. The strength of the correlation was high enough however to conclude that the measures of level of success and ease of use bore some relationship to each other and were once again, not entirely independent.

Discussion

The results from this study provide some interesting insights into the application of IIS systems among naive users in schools. The students demonstrated differing levels of success in information retrieval tasks

100 R. Oliver

between the systems. The identification of discrete knowledge forms among the task selection provided the means to identify possible causes for the varying performances. It was evident through observation of student activity that certain aspects of the system interfaces differed and this had contributed to the development of different levels of knowledge and understanding within the domain of each system.

In measures of user performance in use of the graphical interfaces of both systems, few differences were found to exist. When students were confronted by information problems that required the use of standard inter- actions, they were able to perform the tasks as successfully and as efficiently with both systems. It could be judged from this that the majority of students had developed a range of generic skills able to be transferred between systems and that both systems used the graphical user interface in a consistent fashion. In tasks measuring students' capacities to successfully navigate between nodes within each system, significant differences emerged. A possible explanation for this might be that the success and accuracy of students' mental models of the navigation controls and functions of the information seeking tasks differed significantly between the two systems. A more accurate model was evident in students' use of the preferred system.

There were significant differences evident in students' organisational knowledge of both systems. These tasks were associated with the organisation and storage structures of the information contained within the different systems. This factor was clearly evident in observations of student usage of the systems. Students appeared to have developed more accurate mental models describing the organisation of information in System B than they had done so for System A. It appeared that they were aided by the proximity of the organisation structure within System B to the linearly indexed procedures of conventional materials. The mental models used for information access in conventional materials served them well in their interactions with the hypermedia System B. On the other hand, System A while using a neater and more efficient organisation structure proved more difficult to use as students frequently incorrectly interpreted the actions of search and retrieval strategies.

It must be remembered that these results were achieved with novice users given equal exposure to the two systems. It is interesting to speculate if and when the students would become as comfortable with System A as they obviously were with System B. It was clear that in the short time frame of this study, this had not occurred. This issue would appear to have important ramifications in practical settings where students were using the systems under their own control and at their own volition. It is likely that the problems associated with developing accurate mental models describing the information organisation and structure and storage with System A would lead many students to avoid use of this system in preference to another judged more easy to use. Interviews with the subjects indicated that this would likely be the case. On the other hand, with experience and practice, it seemed likely that performance and achievement measures with System A could increase significantly.

Measuring users' performance 101

Summary and conclusions

The purpose of this study was to investigate whether it was possible to identify the discrete forms of knowledge required for successful IIS use. Using a framework described by Alexander and Judy (1988), an instrument was created to measure the achievement of three discrete forms of knowledge among novice users of an 11s. It was expected that the implementation of this instrument would show the knowledge forms to be hierarchical in nature, the success in higher levels directly dependent on success in lower levels. The results of the study did not bear this out. Although the knowledge forms could be considered discrete, it was found that students could demonstrate achievement in higher levels through knowledge gained beyond the 11s domain.

Students performances made it evident that many are now developing sound mental models describing the control and functions of systems that use a graphical user interface. Few students in this study had any problem recognising scrolling windows when they appeared or the means by which windows were closed, moved or manipulated. Furthermore, the majority of students seemed well able to choose from among the range of menu items, to choose options represented within dialogue boxes and to return to the main menu. On the other hand, their levels of understanding and their mental models of the content organisation differed significantly and saw large differences in the levels of success achieved with the two IIS used in this study. Students showed a preference for the IIS with which more accurate and successful mental models appear to have been formed. The preferred system showed itself to divulge its information more easily and students recognised this factor in their interactions.

The results seem able to be explained by previous studies that have demonstrated problems in users' interpretations of interface metaphors. Although many publishers build design metaphors into their interface and navigational structures, naive users are often unaware of this encompassing and potentially useful structure (Gay & Mazur, 1989). In many cases, the design metaphors can hinder more than aid the users and this appeared to be the case in this instance. It would appear that in choosing a non- conventional design metaphor, publishers can limit the utility of their systems particularly in settings where users are not given explicit instruction in the use of the system.

As we move more and more to electronic forms of information storage, there appears a need for publishers to develop some consistency in interface system design particularly in areas of information organisation and structure. This study demonstrated that the system that provided an organisation that closely followed known structures was used more successfully by students. It appeared to enable students to develop more accurate and useful mental models to guide usage. Novice users quickly became aware of the accuracy and utility of their understanding of the system and this factor had an obvious impact on their preferences in choosing between the two.

102 R.Oliver

From an instructional perspective, teachers need to be made aware of the need to provide instruction that goes beyond providing syntactical descriptions of the operation of 11s. This level of knowledge can develop quickly with appropriate computer experience and activity. To enable students to become efficient and effective information seekers from 11s requires a knowledge and understanding of the information organisation within a system. This study has found that this knowledge does not always develop easily through experience and activity. It also found that when systems differ in their information organisation strategies, without appropriate instruction, the effective use of a system can be limited. Teachers and information specialists need to be able to recognise the difficulties that different 11s can pose to novice users and to select for use those products that pose the least difficulty in this regard.

References

Alexander, P. & Judy, J. (1988) The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58, 4, 375- 404.

Ackerman, P., Sternberg, R. & Glaser, R. (eds) (1989) Learning and individual differences: advances in theory and research. W.H. Freeman, New York.

Allwood, C. & Kalen, T. (1993) Using a patient administrative system: a performance evaluation after end-user training. Computers in Human Behaviour, 9,2/3,137-156.

Gay, G. & Mazur, J. (1989) Conceptualising a hypermedia design for language learning. Journal of Research on Computing in Education, 21,2,119-126.

Heller, R. (1990) The role of hypermedia in education: a look at the research issues. Journal of Research on Computing in Education, 22,4,431-441.

Jih, H. & Reeves, T. (1992) Mental models: A research focus for interactive learning systems. Journal of Education Training Research and Development, 40,3,39-53.

Norman, D.A. (1983) Some Observations on Mental Models. In Mental Models (eds. D. Gentner & A.L. Stevens) pp. 176-199. Lawrence Erlbaum Associates, NJ.

Oliver, R. & Perzylo, L. (1994) Children's' information skills: making effective use of multimedia sources. Educational and Training Technology International, 31, 3, 219- 230.

Perzylo, L. & Oliver, R. (1992) An investigation of Children's' use of a multimedia CD-ROM product for information retrieval. Microcomputers for Information

Reeves, T., Ellis, R., Ring, G., Ring, J. & Oliver, R. (1993) Mental models: a research focus for human-computer interface design. In Proceedings of Australian Association for Research in Education National Conference. AARE, Fremantle.

Riding, R. & Chambers, P. (1992) CD-ROM versus textbook: A comparison of the use of two learning media by higher education students. Educational and Technology Training International, 29,4,342-349.

Saga, H. (1992) Are we ready enough to learn from interactive multimedia? Educational Media International, 29,3,181-188.

Trumbull, D., Gay, G. & Mazur, J. (1992) Students' actual and perceived use of navigational and guidance tools in a hypermedia program. Journal of Research on Computing in Education, 24,3,315-328.

Management, 9,4,225-240.