Learning through virtual reality: a preliminary investigation

Learning through virtual reality: a preliminaryinvestigation

Stella Millsa,* , Maria Madalena T de Arau´job

aDepartment of IT, Cheltenham & Gloucester CHE, The Park, PO Box 220, Cheltenham GL50 2QF, UKbDepartment de Sistemas e Producc¸ao, Universidade do Minho, Azure´m, 4800 Guimara˜es, Portugal

Received 1 December 1997; received in revised form 8 June 1998; accepted 8 June 1998

Abstract

Our understanding of learning through the use of Virtual Reality (VR) is still in its infancy but asmall core of work is emerging that is of growing importance. The literature is utilised to derive threedesign principles that are pertinent to VR systems used for learning. These principles form the basisfor the design of a small VR world which was used for teaching a managment technique to studentsin Higher Education (HE). Thus, this project naturally divided into two stages: first, software wasdeveloped for Portuguese HE students to learn the basic concept of apportioning resources subject toconstraints, while Stage 2 comprised a formative experiment to test for differences in the learning ofthe technique. The conclusion was that overall the traditionally taught group faired better, but notstatistically significantly better, than the software based group. Issues of enjoyment and learningwere also raised. More studies are needed before any generalities can be drawn.q 1999 ElsevierScience B.V. All rights reserved.

Keywords:Virtual Reality; VR Systems Design; Learning through VR

1. Introduction

Virtual Reality (VR) has become a popular medium for a number of application areassuch as modelling safety critical systems and simulating engineering systems such asautomotive engineering. Another area of application is that of education and VR hasbeen used successfully in simulating situations for training such as on the flightdeck ofaeroplanes [1]. However, there were few, if any, projects which have attempted to evaluatethe learning achieved through using VR, particularly comparing this with traditional classlearning, perhaps because of the difficulties in designing a tight experiment for such acomparison. Confounding variables can easily become significant, but even so, compara-tive methods can be used to expose issues arising from using software for learning. This

Interacting with Computers 11 (1999) 453–462

0953-5438/99/$ - see front matterq 1999 Elsevier Science B.V. All rights reserved.PII: S0953-5438(98)00061-7

* Corresponding author.

paper describes a pilot study that used a formative method to compare the learning gainedby using VR with that of traditional teaching by a lecturer; assessment was by a shortwritten test.

2. VR in Education

The terms Virtual Reality and Virtual Environments have come to have a number ofdifferent meanings [2], from what is more exactly described astelepresenceto totallyimmersive systems in which the user is isolated from the real world. The sense in whichVR is used here is somewhere between these two extremes in that it is non-immersive butallows the user to move around the computer generated world in the same way as the userwould move around a real world — a true example of non-immersive VR or desktop VR[3]. Such systems have been used in education for a while; for example, [4] related a fire-training scenario used by the military, but less safety-critical situations have also beenexplored within a learning environment such as laboratories involving scientific experi-ments and wind-tunnels [5].

All these applications (whether immersive or non-immersive) exploit the visual strengthof VR which is known to be important in gaining conceptual insights [6]. Thus, educa-tionally, VR is important, and possibly at its best, when visualising situations which arenot generally available to the student; this is why it has yielded apparent success in trainingin safety critical situations but it is just as valid in other situations such as planetaryexploration [7] or fashion design [8].

Another area of application is in children’s education. Indeed, it is admitted that in theUS, children do not experience enough learning of skills and concepts which are related tosociety’s needs [9] and perhaps VR is a way of addressing this lack, at least in part. Itcertainly seems that, theoretically at least, VR systems should be used to simulate aconcept which is difficult for the user to visualise. This could be through lack of experienceor abstraction which can be supplemented for the user by exploiting VR’s strength ofvisualisation.

Moreover, it is equally important that the system develops the student’s independentthinking as well as increasing the extent to which a student learns and retains specificknowledge [9]. It is also desirable that the system increases the student’s motivation forlearning. Indeed, [10] emphasised the apparent experiential quality of learning throughVR, asserting the the virtual world should be a place in which the student can experience asense of presence and interact with the other characters (whatever they may be) within thatplace (virtual world). Thus the students are encouraged to use the VR system to help buildtheir own knowledge worlds which become vehicles of learning [9,10]. In essence, this fitsthe psychological theory of constructivism [11] which takes an exploratory approach tolearning. Constructivism has become more popular since the late 1960s [9] and computerlearning generally, and VR in particular, lend themselves to such an approach.

As with any learning, motivation is important and constructivists claim that an explora-tory method of learning can, at least, maintain motivation and may increase it. However,more studies are needed in order to learn more about the causes of motivation in studentsnotwithstanding that motivated students learn more easily than non-motivated students.

S. Mills et al. / Interacting with Computers 11 (1999) 453–462454

Thus, if the software can increase motivation then this will enhance the learning of thestudents.

Summarising these points, it is important the strengths of VR are used in learning and,consequently, the system:

1. should exploit the visualisation of the world it portrays;2. encourage the student to explore and to learn through constructing their own knowledge

patterns;3. should increase the student’s motivation for learning by allowing the student to feel a

sense of presence within the world.

Of course, these criteria are ideal and it may be that technological advances are neededbefore they will become a norm, but they should be the intended goal of all designers ofVR systems for use in education.

The study described here endeavoured to produce a VR world which satisfied thecriteria discussed earlier and this was used to teach Portuguese undergraduates a manage-ment technique. The experience of the students was compared with a similar group, usedas a control group, and taught traditionally by a lecturer. Thus the project naturally dividedinto two stages: that of developing the software and that of conducting the experiment.

3. Stage 1 — software development

The area of application for the software was chosen to be the problem of assigningresources subject to constraints — a problem very common in operational research. Theparticular example was taken from a case-study [12] although only the first part of thestudy was implemented. The scenario consisted of planning for the expansion of nursing-home provision subject to building and travel cost constraints in any one of four geogra-phical areas. Town A, for example, could accommodate one nursing home of 20 patientsfrom the four towns after which further homes had to be built elsewhere. Some towns hadsufficient resources for two nursing-homes but travel and time costs for the patients alsohad to be minimised, thus exhibiting a problem of where to build a nursing home thatsatisfied building restrictions and also limited travel for the patients. Thus the problem wasone of non-linear programming with variable constraints.

While this scenario could be adequately represented in multimedia, a more interactivesituation requiring movement in a three-dimensional world was deemed better as thiswould exhibit more obviously the movement of the patients and the restriction of buildingin certain towns. However, the drawback to this is the time taken to build the VR world inthe software, which could be two years or more for a fairly simple world such as theevacuation of a house on fire [4]. Therefore, the chosen scenario was implemented insoftware which could be developed in a few months and so within the timescale of theproject. However, while the textures of the software and some of its design could benefitfrom better development, it was deemed natural enough by the users at the prototypingstage for sound understanding by the students. Thus, the software was designed toexploit the visualisation aspects of VR while encouraging the students to explore andlearn by using buildings as the nursing-homes and three-dimensional movement for the

S. Mills et al. / Interacting with Computers 11 (1999) 453–462 455

transportation of the patients. It was intended that the patients would be represented aspeople and roads used as links between the towns but unfortunately, software limitationsdid not allow this.

The design followed the established method of storyboarding [13] and prototyping. Theinitial screen designs had to be modified in order to accommodate the technical limitationsof the software thus illustrating the development still needed in VR development shells inorder to allow the easiest learning environment for the end-user [14]. The major problem


Fig. 1. A typical screen.

Fig. 2. A typical screen showing mathematical explanation.

here was ensuring that the user could see the method of calculation used to arrive at theresult; this was important as the underlying theory embedded in the calculation wasessential for a complete understanding of the management method and without this theoryfurther application of the method would be impossible. Eventually it was decided to givethe mathematical explanation of the technique in a dialogue box at the end of each shortsection of the implementation (Fig. 2). However, reading and understanding these couldeasily be ignored by weaker students, particularly those who completed the sectionscorrectly. Consistency was maintained between screens by using similar designs foreach screen and each overall section. Figs. 1 and 2 show examples of screens used inthe system.

On completion, the software was evaluated as a prototype by a small group of users(students and two members of the teaching staff) at the Portuguese University and conse-quently, some small changes were made to the language (Portuguese) and to the third(final) section of the software in order to achieve better consistency. It should be noted (seeFig. 2) that the software did not support accents etc. being added to the text. These users


Table 1raw results data from test

User Group 1— (Lecture taught) Group 2 — (VR taught)

1 0.25 02 0.75 03 0.375 0.6254 0.5 0.3755 0.375 0.256 0.875 0.6257 1 0.58 0.375 0.8759 0.625 0.62510 1.125 011 0Average (Mean) 0.525 0.352

Fig. 3. SUMI scores.

also made a (subjective) evaluation of the software using the criteria discussed earlier asheuristics and all the users declared the software satisfied them on all accounts.

In addition, the software was evaluated after use by asking each user to complete aSUMI questionnaire [15] which yielded ratings (see Table 1) for learnability, efficiency,affect, helpfulness and control as well as giving an overall usability rating. Full discussionof these ratings can be found elsewhere [14]; suffice to say that a rating of 50 indicatessoftware which is ‘state of the art’ [15] and the software here rated overall at 47. From Fig.3, it can be seen that the software had high ratings of learnability (62) and helpfulness (54)while control and efficiency were low at 41 and 42, respectively. Affect was adequate at49. However, these divergent results were probably caused by the fact that the instrumentcaters for different types of software such as spreadsheets and databases and thus includesquestions about data-entry and data-processing. In a VR World, these types of input do notusually occur, so the scores for efficiency and control which relate to ease of (data)manipulation and interaction through user input are skewed accordingly. This raises theneed for a more VR specific type of instrument to be developed.

4. Stage 2 — evaluation of learning

The evaluation of learning has evolved as a global term with a variety of differentmethods of assessment. However, in this case, it was decided that since the project wasusing real undergraduate material, the learning should be assessed in the traditional way ofan examination. While it is appreciated that this may not be the best method of assessment,it is the method normally used within the Portuguese university, and therefore was realisticwithin the experimental environment. Consequently, a short written test, heavily focusingon the material learnt, was given. At this preliminary stage of experimentation, no allow-ance was made for confounding variables such as different performances under examina-tion conditions. It is worth repeating that designing a summative experiment in order toevaluate learning is fraught with difficulties caused by confounding variables and, conse-quently, a formative approach has been used in order to identify issues for further work.

4.1. Method

In order to evaluate the learning experience through VR, the experimenters selected acase-study [12] which was used as the instrument for the software and was also taught tostudents by a lecturer, who had a large amount of experience in teaching this work and hadalso received good teaching testimonials from past students. An after-only experimentalresearch design of independent subjects [16] was chosen, with a group of students beingtaught traditionally by a lecturer as the control group and a second equally sized group ofsimilar students using the software as the experimental group. From the prototypingevaluation, the general learning time seemed to be around 10 minutes, so the experimentalgroup was allowed to use the software for a maximum of 30 minutes in order to adequatelylearn the resourcing technique while the control group was taught in a class in the usualway. All the students then were asked to complete a single question test which utilised thetechnique learnt, and the results were compared. This was taken to be synonymous with


the ability to apply the knowledge learnt to a similar scenario to that used in the learningexperience.

The question for the test was based very closely on the technique learnt and requiredonly a change of scenario; in place of a healthcare environment an organ pipe manufac-turer was chosen. Corresponding data were changed as well but the question intentionallyrelated very closely to that of the original case-study.

4.2. Participants

Twenty-one students from a Department of Engineering Production from a Portugueseuniversity took part in the experiment. They were all final year undergraduate students ofEngineering and had volunteered to take part. The total group was representative of thefinal year cohort in terms of academic excellence, age and gender. Although none of thestudents had studied material similar to that used in this project, they all had somemathematical knowledge as well as computer systems engineering.


Fig. 4. Results of Group 1 (traditionally taught students).

Fig. 5. Results of Group 2 (VR taught students).

4.3. Procedure

The students were divided voluntarily into two groups of 10 and 11 students each, thesmaller group (Group 1) being taught by the lecturer and the larger group (Group 2) usingthe software. After being briefed that they would be given a maximum of 30 minutes to usethe software, the larger group was allowed to study the technique through the softwarerepeating as many times as they wanted. When each student indicated that they had learntthe management technique from the software (which was well within the scheduled 30minutes, they were asked to complete a copy of the SUMI questionnaire [15]. At the sametime but in an adjacent room, the smaller group of students were taught the technique bythe lecturer. After each learning session had finished, the students were given a 10 minutesbreak. On return, they were all shown into the examination room and given the test to becompleted under normal examination conditions.

4.4. Results

The raw results of the experiment are given in Table 1. In general, the results for thesoftware students (Group 2) were disappointing on two accounts: first, only nine of the 11students returned from the break to take the test and secondly, two of these failed to scoreso that only seven students gained positive marks (Fig. 4). The 10 students who learnt withthe lecturer traditionally (Group 1) all scored, one student (Number 7) actually gaining fullmarks (Fig. 5). The number of participants is too small to be subject to a full statisticalanalysis, but the Mann–Whitney Test revealed that while the traditionally taught group(Group 1) performed better than the software taught group (Group 2), the results are notstatistically significantly better for 0.005 error tolerance. Consequently, the null hypoth-esis is proved.

However, this does not address the issue of why students did not return from the breakand why, of those who did return, two did not score at all. However, since anonynmityprevailed and no attempt was made to record other factors of the students such as indivi-dual learning preferences, no further conclusions about this can be drawn from this dataand the total number of scores of Group 2 must be used for data analysis.

4.5. Discussion

The most important issue arising from the results is that the difference between themeans of the two groups suggests that the traditionally taught students learnt more thor-oughly and were more able to apply their knowledge than the computer taught students.However, the samples were far too small to generalise this [17,18] and, in any case,explanations should first be sought from the data themselves. We should note that theMann–Whitney Test revealed there was no significant difference between the two sets ofresults for an error tolerance of 0.005 which was supported by similar results from t-tests.Unfortunately, a dispersion analysis was not possible using the Mann–Whitney Test sincethe means of the two groups are different. Even though there is no statistical evidence tosuggest any quantative difference between the two groups, behavioural evidence indicatesthat more qualitative data may have revealed some differences.

In the first place, students were given the choice of learning traditionally or by software


but the experimental design did not allow for investigating the learning preferences of thestudents or their psychological types. Thus it may be that the types of students who aremore easily suited to certain learning styles were not amicably matched with the learningenvironments. The software was designed to exploit the constructivist approach to learn-ing and it could be that some students in Group 2 preferred to learn in a more instructionalway. This could explain the negative reaction of the two students who failed to take the testas well as those who scored less ably using the software. Further work is now needed tocorrelate the learning preferences, the psychological types and the SUMI results of thestudents.

Another problem was that the case-study used an instructional approach by the nature ofthe problem in that all mathematical exercises are sequential. VR emphasises the visualaspect of learning particularly in an exploratory way which may not be the best method forlearning sequential tasks. In addition, the software prevented the interface from having anaturally textured landscape that added cognitive load for the user [19]. The structure ofthe system intentionally did not emphasise the mathematical method, thus requiring thestudent to think ahead in terms of future applications of the method. The task of calculatingthe distances in the problem was assigned to the computer since this is a mundane task, butthis meant that a part of the method was removed from the student’s sequential learning toa dialogue box only visible at the end of the period. Consequently, a student in Group 2 didnot see all aspects of the complete method receiving the same emphasis — as, in fact, thetraditionally taught students did since the lecturer taught the actual calculation as part ofthe method. Again, the visual and constructivist approach which seems so excellent for VRapplications may have caused the student to ‘gloss over’ the very details that s/he wouldneed for the test.

Turning now to the more general concept of the learning experience using VR, [10]described a case-study which involved students at summer school using VR to create theirown worlds based on their experiences of life. The school lasted a week and the studentswere asked if they had enjoyed their time and whether they wanted to experience VRagain. ‘Enjoyment’ scored 6.5 and ‘repeating the experience’ scored 6.8 on a scale whichran from 1 to 7, but when asked if they felt that VR was a good learning environment, thestudents rated this at 5.7. The authors do not suggest reasons for the difference in thesescores but it suggests there is need for further investigation. While these results do notexplain the failure of the two students to return after the break, these experimental findingsdo echo those of [10] in that both sets of results seem to indicate a difference in concept forthe users between enjoying using a VR system and actually learning from it. Clearly,further work is required to investigate possible reasons for this.

5. Conclusion

The project has shown that there may be a difference in learning in higher educationthrough virtual reality when compared with learning traditionally. However, if this is sothen further work needs to be done to investigate the causes; in particular, the design of thesoftware and student learning preferences, including learning styles and psychologicaltypes, need to be analysed more closely. Indeed, more studies of performance and learning


preferences are needed in order to ascertain whether there is correlation between these twoin the case of computer supported learning. Of course, this requires the sound design ofstudies that has been shown here to be fraught with confounding variables. However,summative studies could give useful information about the value of learning with VR.Further trends concerning enjoyment and learning need to be investigated too. Finally,there is a need for an instrument that can be used to evaluate the usability of virtualenvironments so that the software’s worth per se can be ascertained. These advanceswould help greatly in the evaluation of using virtual environments for learning.

Acknowledgements

The authors wish to thank The British Council for their financial support in this project.They also appreciate the kindness ofSuperscape plc. in allowing the free use of a fullevaluation copy ofvrt 4.00 for the duration of the project.

References

[1] L. MacDonald, J. Vince, Interacting with Virtual Environments, Wiley, Chichester, 1994.[2] S.R. Ellis, Pictorial Communication in Virtual and Real Environments, 2nd ed., Taylor and Francis,

London, 1993.[3] S. Mills, M.M.T. de Araujo, Distance Learning by Virtual Reality — Two Preliminary Issues, Video and

Computer Conferencing in Education and Training Conference, 23–24th September 1996, University ofAberdeen, Inverness.

[4] C. Mason, Desktop Virtual Reality in the RAF, Paper given at Multimedia in Business Conference, June1996, Cheltenham and Gloucester College of Higher Education.

[5] R.S.Kalawsky,TheScienceofVirtualRealityandVirtualEnvironments,Addison–Wesley,Wokingham,1994.[6] M. Kaiser, Knowing, in: S.R. Ellis, Pictorial Communication in Virtual and Real Environments, 2nd ed.,

Taylor and Francis, London, 1993.[7] M.W. McGreevy, Virtual Reality and Planetary Exploration, in: A. Wexelblat (Ed.), Virtual Reality Appli-

cations and Explorations, Academic Press, London, 1993.[8] N.M. Thalman, Tailoring Clothes for Virtual Actors, Chapter 13 in MacDonald and Vince, 1994.[9] N.I. Durlach, A.S. Mavor (Eds), Virtual Reality Scientific and Technological Challenges, National

Academic Press, Washington, DC, 1995.[10] M. Bricken, C.M. Byrne, Summer Students in Virtual Reality, in: A. Wexelblat (Ed.), Virtual Reality

Applications and Explorations, Academic Press, London, 1993.[11] P. Fensham, R. Gunstone, R. White, The Content of Science — A Constructivist Approach to its Teaching

and Learning, The Falmer Press, London, 1994.[12] S.R. Heinmann, E.J. Lusk, Health facility planning: an example of a decision flexibility approach, Opera-

tional Quarterly Review, 27 (2, ii) (1979) 449–457.[13] M.J. Bunzel, S.K. Morris, Multimedia Applications Development using IndeoVideo and DVITechnology,

McGraw-Hill, New York, 1994.[14] S. Mills, M.M.T. de Araujo, Learning from virtual reality — a lesson for a designer, HCI Letters, 1 (1998)

28–31.[15] SUMI, Software Usability Measurement Inventory, University of Cork, Ireland, 1996.[16] L.B. Christensen, Experimental Methodology, 4th ed., Alleyn and Bacon, Boston, MA, 1988.[17] P. Johnson, Human Computer Interaction Psychology Task Analysis Software Engineering, McGraw-Hill,

Maidenhead, 1992.[18] G. Lingaard, Usability Testing and Evaluation, Chapman and Hall, London, 1994.[19] K. Carr, R. England, Simulated and Virtual Realities Elements of Perception, Taylor and Francis, London,

1995.


Documents

Learning through virtual reality: a preliminary investigation