Tools for Teaching Mining Students in Virtual Realitybased on 360°Video Experiences

Denis Kalkofen*1, Shohei Mori1, Tobias Ladinig2, Lea Daling3, Anas Abdelrazeq3, Markus Ebner1, Manuel Ortega2,Susanne Feiel2, Sebastian Gabl1, Taras Shepel6, James Tibbett4, Teemu H. Laine7, Michael Hitch5,

Carsten Drebenstedt6, Peter Moser2

1Graz University of Technology, 2Montanuniversitat Leoben, 3RWTH Aachen University, 4SeePilot,5Tallinn University of Technology, 6TU Bergakademie Freiberg, 7Lulea University of Technology

(a) (b)

Figure 1: Our system enables teaching mining students in virtual reality with 360°videos. (a) After exploring the environment, theteacher presents a specific procedure to all students by controlling the video playback and by consecutively enabling and disablingseveral prepared annotations (shown in red in this example). (b) Students follow the presentation using a smartphone based VRheadset, which enables for orientation updates.

ABSTRACT

In recent years, Virtual Reality (VR) technology has found theirway into higher education. Its power lays in its ability to provideimmersive three-dimensional (3D) experiences that help conveyingeducational content whilst providing rich interaction possibilities.Especially in mining engineering education, VR has high potentialto reshape the provided learning content. Field trips, i.e. mine visits,are an integral part of the education and necessary to transfer knowl-edge to students. However, field trips are time and cost intensiveand mines often have tight entry regulations. As a result, the numberof field trips is limited. VR-based field trips offer a considerablealternative presupposed they replicate the complex mining environ-ment realistically. In addition, VR mines have the advantage oftaking students close to events (e.g. explosions) that are impossibleto demonstrate in a real mine. However, generating realistic 3Dcontent for VR still involves complex, and thus time consumingtasks. Therefore, we present the design of a VR Framework forteaching mining students based on 360° video data, its evaluation inthree different lectures, and its extension based on the feedback wereceived from students and teachers from four different universities.

1 INTRODUCTION

Higher mining education has always faced a number of challenges.Traditionally, complex concepts have been conveyed using two-dimensional (2D) material only, which is challenging for both teach-

*e-mail: kalkofen@icg.tugraz.at

ers and students. Therefore, more realistic three-dimensional (3D)experiences have been recognised to be an indispensable methodin mining education to augment the theory. Field trips have beenan important part of this. However, mines are high-consequenceenvironments and mining sites are commonly outside universities,which is why field trips to mining sites are often difficult to organize.In addition, it is highly challenging for educators to lead 20–40students through a mine. As a result, many students hardly obtainaccess to real operations throughout their studies.

Virtual Reality (VR) offers a solution to overcome these barriersby taking students safely into such a high-consequence environmentwithout leaving the classroom. Moreover, VR enables experiencingan even wider variety of scenes, compared to a single field trip. Thus,VR can help students to get in contact with the complex world of rawmaterials production and help them develop a better understandingof mining.

However, properly designing VR experiences for higher educationis still an active topic of research [6, 12, 14]. While VR has beendemonstrated to provide a powerful tool for improving learningoutcomes [1], the VR application requires careful design choices [4].For example, a common problem in VR education is to keep studentsfocused [17]. VR systems have introduced eye tracking for detectingattention drift. The knowledge about the students eye directionsenables presenting visual cues, such as arrows, to refocus theirattention towards the region of interest [16]. However, while thisdirection of research helps staying focused, it adds complexity tothe VR system development.

A large body of work also proposed using gamification in VR-based education since it is expected to increase the motivation ofstudents [10]. For example, Feng et al. [9] discuss the design of VRgames for training building evacuation, and Stelian et al. [11] use

455

2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)

978-1-7281-6532-5/20/$31.00 ©2020 IEEEDOI 10.1109/VRW50115.2020.00093

(a) (b) (c) (d)

Figure 2: Initial tools. (a) Students can explore the scene by teleporting to different locations. Selecting a camera icon loads the correspondingvideo data. (b) The teacher augments 360° videos with drawings and notes. (c) Only the teacher sees the icons to trigger the preparedaugmentations. (d) The teacher can interactively enable or disable each augmentation by selecting respective icons.

gamification for learning and training of medical procedures. Mostof these approaches are designed for a single user and do not fitwell into lecture halls. For educating groups of students, researchersproposed using Second Life [7] and other, similar multi-user virtualenvironments [5]. Multi-user VR environments support the feelingof learning in a group and support controlled user guidance [2]. Inguided tours, the teacher is commonly represented by a 3D avatarand live performance capturing is used to control its animations [15].While this supports controlled field trips of groups of students, itrequires an interactive multi-user 3D environment. This, however, isoften difficult and thus time consuming to generate, especially forcomplex surface and underground mines. Furthermore, as recentstudies indicate [13] full 3D stereoscopic experiences require sixdegrees of freedom (6DOF) head tracking. However, mobile head-mounted displays that support 6DOF tracking are still expensive andthus, cannot easily be offered to a large group of students.

Since we are specifically aiming at an integration of the VR expe-riences into the teachers existing workspaces, we aim at supportingthe teaching of many students. To provide universities with a prac-tical solution, we focus on VR systems based on smartphones (e.g.using a cheap cardboard head mount), which we assume most of thestudents own already. Since the current generation of smartphonesoffer only rotation tracking, we do not rely on stereoscopic render-ings of a 3D environment. Instead, we are aiming at a teachingenvironment that mainly consists of 360° video experiences.

Therefore, in this paper, we present an interactive framework forteaching mining students in VR, mainly using 360° video experi-ences. We present the design of the framework and its refinementbased on feedback, which we collected from students and teachersduring and after testing the system in three different lectures. Base-line requirements of the design elements are inspired by the workof Radianti et al. [12]. Based on the feedback, the design elementshave been extended by tools that support interactive teaching in360° video experiences. In particular, we developed tools to sup-port attention control, notes and bookmarks, object selection duringcommunication, and class overview visualization.

2 SYSTEM OVERVIEW

We derive our initial system from the design element frameworkproposed by Radianti et al. [12]. Note that the work of Radianti etal. [12] is based on a survey that excludes 360° video experiences.However, we believe that basic design elements apply to both, 3Dand 360° video based VR experiences.

Realistic surroundings. Our main objective is to provide an alter-native for real field trips in mining education. Thus, we aim at avirtual environment that is as close as possible to its real counterpart.The students need to learn about the structure of a mine and manycomplex processes, which are necessary to understand the produc-tion process of raw materials. Since this can hardly be modeled with3D modelling tools with reasonable efforts, we support the playbackof 360° videos. Filming real mining environments enables to presentlocations and processes with high fidelity.

Passive observation. Since we focus at large lecture halls, wedesigned the VR framework to support passive consumption of pre-pared material. The students can freely rotate their heads, however,their position in space and time is predefined and cannot be alteredduring the teacher’s presentation. This design element allows us toensure that students are located at the intended point in space andtime.

Role management. We ensure that all students are located at thesame place at the same time by synchronizing the teacher’s videoplayback with that of the students. Therefore, we distinguish be-tween the roles of teacher and students, and provide the teacher withadditional tools for controlling the presentation. See Figure 2(c)for a screenshot of the teacher’s view. The teacher can control theplayback of the video for all students, jump to a certain frame, andcontrol the visibility of any additional information.

Prepared annotations. In addition to the design elements proposedby Radianti et al. [12], we added support for controlled display ofvideo annotations (see Figure 2(b)). Therefore, we allow augmenting360° videos with sketches, which the teacher prepares using thetouchscreen of a smartphone. During the presentation to students,the teacher can interactively toggle any annotation on and off to showor hide the prepared sketches for the students. See Figure 2(c) foran example of the teacher’s interface for toggling prepared sketcheson and off, and Figure 2(d) for a example after two annotations havebeen enabled.

Moving around. Before the teacher starts the presentation, allstudents are allowed to move around, which helps them form anoverview of the environment. Therefore, we allow students to tele-port between several viewpoints of a single scene. Available loca-tions are indicated by icons which appear as video annotations (seeFigure 2(a)). To support building a mental map of the environment,we spatially register the viewpoint icons, i.e. we place the icons in avideo frame where the corresponding viewpoint is located.

456

(a)

Lookingdown

(b)

Figure 3: Region of interest and classroom overview. (a) Similar to the work of Liu et al. [8] we control the playback with the user’s gaze. However,instead of looping the video, we pause it and we provide an arrow so that they the student can reorient towards the region of interest. The red andgreen outlines indicate the region of interest that needs to be watched in this example. When the student is not facing the region of interest, theoutline becomes red and the arrow appears. The outline appears in green when the student’s head gaze is pointing towards the region of interest.(b) After receiving a notification about unfocused students, the teacher looks down to see an overview of all connected devices. The icons ofdistracted students are highlighted and pointing to an icon reveals its detailed state.

3 EVALUATION

In order to evaluate the VR framework, we developed three differentlectures on mining using 360° video material. Specifically, we de-veloped lectures on Open Pit Bench Blasting, Continuous SurfaceMining, and on Loading and Hauling in Mining. The lectures wherepresented at four different universities in Austria, Germany, and Swe-den. The test lectures began with a brief introduction of the projectand the lecturer (3-5 min). Then a technical assistant explained howto use the VR headsets (approx. 5 min.). Afterwards, the lecturebegan. The lecture was held in form of an oral presentation withinterleaving VR scene demonstrations. In addition, PowerPoint andPDF presentations were delivered on a pull-down screen and on aninteractive whiteboard. Whenever scheduled during the lecture, thestudents were asked to use the VR headsets. In total, 125 studentsattended the test lectures.

Students and teachers were asked to fill in feedback forms. Inaddition to a usability evaluation, the focus of the forms was onassessing the usefulness of the VR technology for learning progress.It became clear that students saw high potential in the use of VR,especially if there is no practical experience through field trips orinternships. Navigation within the system by the teacher was consid-ered essential by the students. In addition, the students expressedthe wish for different modalities of interaction with the teachers, andthey emphasized the need for taking notes during the lecture.

The teachers also stated that they would like to further use VR forteaching purposes. Nevertheless, they all depended on a technicalassistant who could prepare the technology for them and solve minorproblems during the lecture. At this point, there is still an urgentneed to sensitize teachers to new technologies and to support themaccordingly in the use and adaptation of the tools. In addition,they commented on the diminished view of the classroom and theyexpressed their wish to observe the students’ behaviour and attention.

The results of the evaluation were directly transferred to thefurther development of the VR framework, which is described in thenext section.

4 VR FRAMEWORK EXTENSION

We extended the VR framework in order to integrate the feedbackfrom students and teachers. In particular, we added tools for con-trolled video playback based on regions of interests, for classroom

overview for teachers, for interactive pointing, for taking notes dur-ing VR lectures, and for recording and reviewing lectures.

4.1 Playback Control by Regions of InterestTo ensure that students see what they are supposed to see, we controlthe playback of 360° videos similar to the method proposed by Liuet al. [8]. Therefore, we introduce regions of interest and we playthe video forward only when the student’s viewing direction fallswithin the region of interest. However, unlike Liu et al. [8] we donot loop the video. Instead, we pause the video playback and wedisplay an arrow so that the student can reorient towards the regionof interest (notice the white arrow pointing to the right hand side inFigure3(a) left).

Pausing the video playback includes pausing the sound, whichnotifies the student immediately about the status of the playback. Inaddition, we highlight the region of interest by colouring its outline.We draw a green outline when the student is facing towards theregion of interest and we color it in red otherwise (see Figure3(a)).

This is a powerful tool to ensure that students receive the informa-tion that is required to follow the teacher’s explanations. However,since each student explores the video material at its own speed, theway the content within the region of interest is explained needs to beadapted. If the video presentation is synchronized, we can assumethat each student is seeing the same video frame at the same point intime, allowing the teacher to explain the demonstrated process whilethe video is playing. However, if students pause and play their videopresentation asynchronously, they consume the material at differentspeeds. Therefore, the teacher needs to explain the process beforestarting a video sequence, which requires breaking the video mate-rial into short sequences. We support such stop and go presentationsby introducing playback barriers, i.e. selected frames at which allstudents wait for the teacher to continue explaining the scene.

4.2 Classroom OverviewThe regions of interest feature enables us to identify students that arenot paying attention. We detect unfocused students by computingthe amount of time they do not look towards the region of interest.In order not to clutter the teacher’s view, we first inform him or herabout unfocused students, by using a blinking icon in the corner ofthe interface. If the teacher wants to identify the unfocused students,he or she needs to look down (Figure 3(b) left), where a classroom

457

Pointer

(a)

Smart pen

Current position

Bookmarks

(b)

Figure 4: Virtual pointer and notes. (a) The teacher can put a point of interest to temporally draw attention during the lecture (right). Then, anarrow appears to guide the students to the point (left). The circle pointer blinks to get further attention (right, bottom). (b) Students can take notesusing smart pens. Digitized strokes appear on a pad in VR (left). Students can review the recorded VR lecture and bookmarks (right, top) enablethem to jump to preferred time points (right, bottom).

overview appears. The overview visualization presents details aboutall connected devices, including student identification numbers andwhether they are focused or not (Figure 3(b) right).

4.3 Interactive Points of Interest

Video playback based on regions of interest requires the teacherto prepare barriers and to define the regions of interest in videosequences. Since interesting objects may move around, the regionsof interest need to be updated in each frame. Therefore, we haveimplemented region tracking using the well known Lucas-Kanadetemplate tracker [3].

However, even if we automatically update regions of interest, theteacher has to generate the barriers and he or she needs to outline theregions at keyframes. Since this might become time consuming, wealso support directing the students attention with interactive pointsof interest. Points of interest can be generated at any point in timeby pointing with a handheld controller or the user’s head gaze. Oncea student receives a point of interest, the system presents him or heran arrow (Figure 4(a) left)) that indicates the direction towards thepoint of interest. In addition, we allow to emphasize the point ofinterest with an animation, such as a blinking circle as illustrated inFigure 4(a) right, bottom).

We support operating points of interest in two different modes.In order to ensure that students look at the object that the teacher isexplaining, we constantly distribute the teacher’s viewing directionas point of interest to all students. However, our point of interest toolcan also be used during a questions and answers sessions. Therefore,we also enable students to point. However, to control pointing,students need to request permission from the teacher before theirpoint of interest is distributed to all other students and the teacher.

Note that points of interest have no impact on the playback ofvideo. We use points of interest while playing the video at the samespeed on all devices in the lecture hall. Therefore, care must betaken when pointing during video playback.

4.4 Taking Notes

We realized a system for taking notes in VR environments by con-necting a smart pen to the VR device. The student can use the pento take notes at any time during the VR lecture. As the smart pendigitizes the writings and drawings, we can immediately transmitthem to the VR device and for visualization in the VR environment(Figure 4(b) left). Therefore, the smart pen enables to write ona physical sheet of paper, which is placed in front of the student,whereas the written note appears within the VR environment.

4.5 Recording and Reviewing Lectures

We implemented a feature that allows teachers to record VR lectures,including their voices. The purpose of this feature is to providestudents with the possibility to review the lecture by revisiting theVR experience at a later point in time. Similarly, the student can setbookmarks at any point in time during the lecture. These bookmarksare available to the student when he or she is reviewing the recordedlecture. Figure 4(b) right shows the visualization of inserted book-marks during reviewing of a recorded lecture.

5 CONCLUSION AND FUTURE WORK

Virtual Reality offers a powerful platform for higher education. Espe-cially in higher mining education, virtual field trips have the potentialto bring the experience of visiting a real mine into the classroom.Due to the complexity of a real mine, virtual field trips need to be asrealistic as possible. However, generating realistic 3D content forVR still involves complex and thus, time-consuming and expensivetasks. Therefore, we designed a VR framework for teaching miningstudents based on 360° video data which is cheap to produce. Wederived our initial design from the work of Radianti et al. [12] andwe extended it based on the feedback from students and teachersfrom several universities.

The most important feedback referred to lack of tools for control-ling the students’ attention, for taking notes, and for providing anoverview of the students in a classroom. We believe that the tools wehave derived from the feedback build the basis for effective teachingin VR with 360° videos. However, future evaluations are required toidentify their strength and weaknesses, and the learning effect in theproposed VR teaching environment.

Since the time the students spend in a classroom is limited, wefurthermore aim at extending our system to support other learningenvironments. The notes and recording components enable review-ing a lecture at home. However, the recording only allows followingthe teacher’s presentation. To furthermore support an independentlearning approach, we will combine our video based framework withtraditional printed books. Using an Augmented Reality interfacewill allow linking 360° videos to written text, so that traditionallecture handouts and books can be combined with VR experiences.

ACKNOWLEDGMENTS

This work was enabled by the European Institute of Innovation andTechnology (EIT) Raw Materials in the project MiReBooks: MixedReality Handbooks for Mining Education (18060).

458

REFERENCES

[1] D. Allcoat and A. von Muhlenen. Learning in virtual reality: Effects

on performance, emotion and engagement. Research in LearningTechnology, 26, 2018.

[2] C. W. Borst, N. G. Lipari, and J. W. Woodworth. Teacher-guided

educational VR: Assessment of live and prerecorded teachers guiding

virtual field trips. IEEE Conference on Virtual Reality and 3D UserInterfaces, pp. 467–474, 2018.

[3] J.-Y. Bouguet. Pyramidal implementation of the Lucas Kanade feature

tracker description of the algorithm. Technical report, Intel Corp.,

2001.

[4] B. Chavez and S. Bayona. Virtual reality in the learning process. In

A. Rocha, H. Adeli, L. P. Reis, and S. Costanzo, eds., Trends andAdvances in Information Systems and Technologies, pp. 1345–1356,2018.

[5] E. Dieterle and J. Clarke-Midura. Encyclopedia of Multimedia Tech-nology and Networking, chap. Multi-User Virtual Environments for

Teaching and Learning, pp. 1033–1041. 2008.

[6] Z. Feng, V. A. Gonzalez, R. Amor, R. Lovreglio, and G. Cabrera-

Guerrero. Immersive virtual reality serious games for evacuation

training and research: A systematic literature review. Computers& Education, 127:252 – 266, 2018.

[7] P. Hollins and S. Robbins. Living Virtually Researching New Worlds,chap. 13 The Educational Affordances of Multi User Virtual Environ-

ments, pp. 257–271. 2009.

[8] S. J. Liu, M. Agrawala, S. DiVerdi, and A. Hertzmann. View-dependent

video textures for 360◦ video. In ACM Symposium on User InterfaceSoftware and Technology, p. 249–262, 2019.

[9] R. Lovreglio, V. Gonzalez, Z. Feng, R. Amor, M. Spearpoint,

J. Thomas, M. Trotter, and R. Sacks. Prototyping virtual reality seri-

ous games for building earthquake preparedness: The Auckland City

Hospital case study. Advanced Eng. Informatics, 38:670 – 682, 2018.

[10] M. Morales, H. R. Amado-Salvatierra, R. Hernandez, J. Pirker, and

C. Gutl. A practical experience on the use of gamification in mooc

courses as a strategy to increase motivation. In L. Uden, D. Liberona,

and B. Feldmann, eds., Learning Technology for Education in Cloud –The Changing Face of Education, pp. 139–149. Springer InternationalPublishing, Cham, 2016.

[11] S. Nicola, I. Virag, and L. Stoicu-Tivadar. Vr medical gamification for

training and education. Studies in Health Technology and Informatics,236:97–103, 2017.

[12] J. Radianti, T. A. Majchrzak, J. Fromm, and I. Wohlgenannt. A sys-

tematic review of immersive virtual reality applications for higher

education: Design elements, lessons learned, and research agenda.

Computers & Education, 147, 2020.[13] J. Thatte and B. Girod. Towards perceptual evaluation of six degrees

of freedom virtual reality rendering from stacked omnistereo represen-

tation. Electronic Imaging, 2018(5):352–1–352–6, 2018.[14] P. Wang, P. Wu, J. Wang, H.-L. Chi, and X. Wang. A critical review

of the use of virtual reality in construction engineering education and

training. International Journal of Environmental Research and PublicHealth, 15(6), 2018.

[15] J. W. Woodworth, N. G. Lipari, and C. W. Borst. Evaluating teacher

avatar appearances in educational VR. In IEEE Conference on VirtualReality and 3D User Interfaces, pp. 1235–1236, 2019.

[16] A. Yoshimura, A. Khokhar, and C. Borst. Eye-gaze-triggered visual

cues to restore attention in educational VR. In IEEE Conference onVirtual Reality and 3D User Interfaces, pp. 1255–1256, 2019.

[17] A. Yoshimura, A. Khokhar, and C. W. Borst. Visual cues to restore

student attention based on eye gaze drift, and application to an offshore

training system. In Symposium on Spatial User Interaction, pp. 1–3,2019.

459