Paper ID #24588

Design and Development of a Virtual Reality Educational Game for Archi-tectural and Construction Reviews

Dr. Fadi Castronovo, California State University, East Bay

My name is Fadi Castronovo, I am an Assistant Professor of Engineering at Cal State East Bay. I receivedmy doctorate in Architectural Engineering at Penn State with a minor in Educational Psychology. I amstrongly focused on my teaching and research. In my teaching, I strive to provide an engaging and activelearning experience to my students, by applying innovative technology and researched pedagogical inter-ventions. I translate this passion for pedagogy in my research by evaluating the intersection of innovativetechnology and learning. In this website, you can learn about my experiences and work.

Dr. Dragana Nikolic, University of ReadingSilvia Mastrolembo Ventura, Politecnico di Milano & University of BresciaMs. Vrinda Shroff, California State University, East BayAnh NguyenMs. Nguyen H.P. Dinh, Mission College

Hello, I am Nguyen Dinh, a Computer Science student at Mission College, Santa Clara, California.

Dr. Semih Yilmaz, California State University, East BayDr. Reza Akhavian, California State University East Bay

Dr. Reza Akhavian is an Assistant Professor of Construction Management at the School of Engineering,California State University East Bay (CSUEB) and the director of the Data-informed Construction Engi-neering (DiCE) Group. He received his Ph.D. in Civil Engineering (with Construction Engineering andManagement concentration) from the University of Central Florida (UCF). He also holds an M.S. (UCF,2012) and a B.S. (University of Tehran, 2010) in Civil Engineering. He has more than 30 articles publishedin peer-reviewed journals and conference proceedings and serves as a member of the editorial board ofthe American Society of Civil Engineers (ASCE) Journal of Construction Engineering and Management(JCEM). His industry experience includes working with Bentley Systems and DPR Construction.

Dr. Cristian Gaedicke, California State University East Bay

Dr. Cristian Gaedicke earned the Ph.D. in civil engineering from the University of Illinois at Urbana-Champaign in 2009 and is a licensed professional engineer (Civil). His research interests include con-necting STEM education to engineering practice, sustainable construction materials, infrastructure, andconstruction engineering. Dr. Gaedicke has participated in engineering education projects sponsored bythe NSF and Chevron and has served as co-PI on projects with MESA and Project Lead the Way.

c©American Society for Engineering Education, 2019

Design and Development of a Virtual Reality Educational Game for Architectural and Construction Reviews

ABSTRACT The growing adoption of building information modeling platforms in the construction

industry requires instructors to design and leverage innovative pedagogical interventions. Furthermore, construction instructors must support their students in gaining essential problem-solving skills such as the analysis and evaluation of proposed construction and design solutions. In this study, the research team aimed at designing and developing an educational virtual reality game to support students in evaluating and reviewing BIM-based design of residential buildings. Research has illustrated that reviewing designs using traditional methods, such as drawings, can tax a students’ cognitive process. To tackle this issue, the team has developed the Design Review Simulator (DRS), a virtual reality educational simulation game. The learning objectives of the game are to support students in developing evaluation and reviewing skills of mistakes in construction projects through the use of virtual reality. We used a five-phase instructional design framework – A.D.D.I.E. – to analyze, design, develop, implement and evaluate the DRS game. Currently, the authors have performed the first three phases of the A.D.D.I.E. process. In the (1) analysis phase, the team evaluated literature, the future users of the games, and defined learning objectives. We then (2) designed and (3) developed the DRS in the Unity 3D game engine. The design proposal used as a test bed for this project was a residential townhouse in the heart of the San Francisco Bay Area and the BIM model was developed in Autodesk Revit. The model was provided by a local architectural firm specialized in renovations and retrofits. The team and the architects selected this model due to the high presence of modeling mistakes, making it effective example for educational purposes. The paper reports the analysis, design, and development efforts of the DRS in order to support other instructors and designers in developing future virtual reality games for construction education. Therefore, this experience’s contribution provides a direction to future game developers in designing similar educational virtual reality games.

1. INTRODUCTION The growing adoption of virtual reality (VR) in architecture, engineering, construction (AEC) industry has illustrated its potential for improving communication and evaluation of design proposals, and coordinating building systems [1,2]. With the growing interest in VR by the industry, academia has also started to explore the educational potential of VR in the AEC disciplines [3]. For example, VR has been explored for its effects on engagement, enjoyment or memory [4] or as a means of interactive storytelling [5] as a way to support dynamic and more active learning. At the same time, VR as inherently three-dimensional and interactive environment can support the students in design and engineering disciplines who are required to build their spatial reasoning skills. In this paper, the authors detail the process in designing and developing an educational virtual reality game, the Design Review Simulator (DRS) with the aim to help students build design review skills. The game was designed to challenge the students in finding and evaluating various types of design mistakes. In this paper, we discuss the development of the game and frame it in light of the existing literature to further understand the value of VR in the design process. In addition to the development experience, we offer a plan for the implementation and evaluation of the game with an assessment instrument designed for the game. 2. LITERATURE REVIEW 2.1. Design Review and Visualization

With the growing adoption of building information modeling (BIM) approaches, several innovative technology applications for various use cases have started reshaping the process with which facilities are planned, designed, built, and operated. One such application is the use of high-end visualization tools for reviewing building information performance over its lifetime, from the planning to the operations phase. Research on visualizations tools has been particularly focusing on evaluation and review of design as a way to reduce problems before the facility reaches the construction phase. Design review is a process of analyzing and evaluating the proposed design solution against program requirements, specifications, and user and owner requirements [6]. This process may use different methods and representations, such as physical mock-ups and paper drawings to digital and virtual prototypes [7–12]. Three-dimensional models notably supported the design review process by reducing the cognitive load demand by allowing the user to visualize information more easily [13–15]. Growing implementation in the las two decades of BIM and interactive workspaces [16–22] has also lead to the renewed interest in immersive virtual reality for its potential to improve the design review process [1,23–31]. For example, Castronovo et al. [32], Dawood [33], and Liu et al. [2] have found that VR can support integrated design teams in the delivery process of facilities, through the analysis of the proposed design before it reaches the construction phase. Similarly, others have found the application of VR to support the collaborative design development and verification building and spaces, such as patient rooms and hospitals [7,8,34] nuclear power plants [35], and courtrooms [36]. Lastly, immersive VR has been used to support AEC students to enhance their design generation process and evaluate their proposed solutions [3,37,38] based on the enhanced level of understanding that a virtual facility prototype provides if compared to traditional representations, as demonstrated in previous research works [39].

2.2. Educational Gaming

Educational research has focused attention to the use of multimedia tools to enhance the student learning in a classroom environment [40,41]. This can be attributed to the influence of constructivist learning theory, which advises instructors to develop learning environments where students learn by actively engaging with the material presented [42]. In particular, Mayer’s cognitive theory of multimedia learning emphasizes that instructors should develop interventions were students can learn by developing “mental representations” from words and pictures [41]. One such approach learners can benefit from is educational simulation games. Such games enable the learner to interact with a repeatable and individualized learning environment [43]. Research has found that educational games enable the students to engage with the learning material by providing them with a risk-free environment [44–46]. In particular, educational games allow learners to visualize, explore, and interact through instant feedback with the desired concepts [47–49]. Such interventions have shown that students can develop real-world knowledge and meeting learning objectives [50–52]. The application of simulation games in the AEC education has illustrated that students can gain problem-solving skills transferrable to real-world scenarios [52–54], while being engaged and motivated [55,56]. However, little research, within the domain of AEC education, examines the value of educational games for teaching students architectural design review skills. 2.3. Virtual Reality and Education

Virtual reality has been recognized as a type of media that can engage students in experiential educational experiences. In particular, increasing research in immersive VR is focusing on its ability to promote spatial presence and immersion of the user inside a virtual environment [57]. Through the use of immersive VR, students’ engagement and motivation in learning are enhanced [4,5]. For example, at the University of Iceland, a program called “Innovation Education” provided VR experiences to their engineering students, allowing them to understand the concepts in their textbooks [58]. The benefits they found from this study were that students not only gained basic textbook knowledge, but experiential knowledge as well. In this virtual environment, the students were able to see and experiment with the software and use trial and error to solve problems. They were able to test their products in the simulated world, as well as to gain knowledge and express their creativity with their projects. Based on these benefits, VR can be seen as a suitable learning environment that stimulates and educates students. VR, in fact, offers an opportunity for students to emotionally connect with how their actions and consequences affect the environment [59]. According to Riva et al. [59], VR can provide a strong mental connection with the presented content because of its immersive capability and a sense of presence within the reality virtually represented. Controlling the environment and problems presented through a virtual medium could allow for a distinct user content experience that is unparalleled in conventional audiovisual implementations. Once students are able to make an emotional and cognitive connection with the consequences of their actions, they would be more likely to solve complex and real-world problems. While some reach has started evaluating the impact of VR to teach design review skills in undergraduate students [3], further investigation in the interaction that VR and design review skills have on pedagogy. Based on the value VR possesses and the existing research gap, the research team, has set the objective to evaluate the potential effect that a VR educational game has on learning.

3. RESEARCH GOAL With the growing implementation of BIM and visualization technologies in AEC education and industry, instructors are tasked to develop and deliver educational experiences that sufficiently challenge students’ thinking and engage them in an active and dynamic learning process. As discussed previously, educational gaming has shown the potential for engaging students in developing problem-solving skills. Building on the constructivist principles embodied by simulation games on one hand, and the salient features of VR for spatial and exploratory learning, the authors of this paper set forward the goal to develop an educational game – Design Review Simulator (DRS) – to engage students in developing design review skills. The development team was composed of five faculty members, one doctoral student, and two undergraduate students. The role of the faculty was to guide the development of the educational game, while the doctoral students supported the development of the literature review and learning objectives of the game. The undergraduate students focused on the development of the game. We used a five-phase instructional design framework – A.D.D.I.E. – to analyze, design, develop, implement and evaluate the DRS game. Here, we document the first three phases of the virtual reality educational game and we also discuss the development of the educational assessment material for the classroom implementation. 4. INSTRUCTIONAL DESIGN OF THE DESIGN REVIEW SIMULATOR To understand better the value of VR in how it may support undergraduate students in reviewing and evaluating the facility design, the team developed the virtual reality educational game DRS. The main learning objective for the game is for the students to learn to identify and evaluate the types of design-related and construction-related mistakes that can be found in the BIM model of a townhouse. The development platform that was chosen for the game was the Unity 3D game engine, and for the VR headsets, the team chose the Oculus Rift. During the gameplay, the students are immersed inside the 3D model of a townhouse in the heart of San Francisco and are asked to walk around and identify design mistakes such as layout, material and coordination issues or missing components. To guide the development of the game, the team selected the A.D.D.I.E. framework for instructional design, stands for analysis, design, development, implementation, and evaluation [60] (see Figure 1). The framework developed by the Center for Educational Technology at Florida State University is commonly leveraged for the design of instructional experiences [61]. 4.1. Analysis 4.1.1. Audience and Environment

In the analysis phase, the first two steps are to identify the target audience and environment for the DRS. The chosen audience for this game is undergraduate students pursuing degrees in the AEC field and who are currently in their second or third year of studies when they are typically introduced to design problems. However, at other institutions, the game can be implemented at the instructors’ discretion and preference. The environment for the game is a classroom equipped with VR-ready desktops and VR headsets. The initial implementation of the game took place in the Visualization and Immersion Classroom (VIC) at the first author’s University. The VIC was

designed to deliver VR experiences for large groups, therefore it is equipped with 48 high-end computers and 15 Oculus Rift headsets (see Figure 2). (The project has already received IRB approval and all of the students in Figure 2 have signed the IRB consent forms for the researchers to use their data and pictures taken during the implementation).

Figure 1. A.D.D.I.E. Process for the Design Review Simulator Game

4.1.2. Game Learning Objectives

The last step in the analysis phase is the definition of the learning objectives, which educational gaming research has identified as an essential step in the development of games [53,62]. Additionally, the definition of the learning objectives enables the development of the assessment instruments for the implementation and evaluation phase of the game and measure the extent to which the students learn by playing the game. The learning objectives of the game were developed by using the revised Bloom’s Taxonomy [63,64], particularly focusing on the cognitive domain. Within this domain, the team selected the remembering and understanding cognitive thinking skills levels. For each of these levels, the team selected the following action verbs: define, explain, match, and identify. These action verbs are aligned with a cognitive level of thinking skills set by the taxonomy. Through the selection of these levels and verbs, the four resulting learning objectives for the DRS were defined (see Table 1). These game learning objectives are not aligned to any particular course learning objectives. The researchers decided not to align the game’s learning objectives to a course learning objectives to maximize implementation across multiple courses.

Table 1. Game Learning Objectives for the Design Review Simulator

Cognitive Domain Action Verb Game Learning Objective

At the end of the pedagogical intervention the students will be able to:

Remembering Identify Define Match

1. Identify the different types of design mistakes 2. Define the provide design mistake 3. Match the provide mistake to the correct design mistake description

Understanding Explain Summarize 4. Explain/Summarize the identified design mistake

Figure 2. Cal State East Bay Students in the Visualization and Immersion Classroom

4.2 Design 4.2.1. Game Story and Mechanics

According to Schell [65], when developing a game, it is essential to create four components: mechanics, story, technology, and aesthetics. The story of the game was set at the beginning of the game when the learner is introduced to the challenge. The narrative at the start of the game introduces the students to their task: “You have been hired to review the proposed design model for the renovation of a townhouse in the heart of San Francisco. You will need to perform a virtual walkthrough to find design mistakes. Mistakes include poor spatial layout (20 pts), coordination issues (15 pts), missing elements (10 pts), and improper material choices (5 pts). Try to find as many mistakes as possible! Good luck and have fun!”.

By developing the story, the team then set the game mechanics outlining the game flow (Figure 3). Following the introduction, the learners are transported to the virtual model of a townhouse in the heart of San Francisco. To identify and flag a design mistake, they point the crosshair to a design mistake and press “enter”. If the mistake is correctly identified, the game displays a message with the type and description of the mistake. The game was designed to present to the students four types of design mistakes, each with a specific point associated with it (see Table 2). The point system was based on the severity of the type of mistake. The team selected for the game the following mistakes: (1) spatial layout, (2) coordination issues, (3) missing elements, and (4) improper material choices.

Table 2. Types of Mistakes

Mistake Type Associated Points Spatial Layout 20

Coordination Issues 15 Missing Elements 10 Improper Material 5

Figure 3. Game Mechanics

The game was designed for learners to work as a group, therefore, after five mistakes, the

learner that is the “driver” is asked to switch with another leaner. Once the learners have found all of the mistakes, the game then prompts them to close the game. Based on the game mechanics the team then developed a game scene structure (see Table 3). The technology and aesthetics components of the game are discussed in the following section.

Table 3. Structure of the Design Review Simulator Game Scene Game Mechanics Purpose of the Scene Start Screen Start Game Start the game

Introduction Scene Introduction Narrative Introduce the game and learning objectives.

Main Game Scene Walk around the townhouse Point and find mistakes Load mistake message Prompt switch after 5 mistakes

Main scene where the user has to walk around the townhouse and find the design mistakes

End Screen End Game End the game 4.2.2. User Interface and Interaction The next game design phase focused on developing user interface with considerations for the aesthetics and a specific look and feel of the game [65]. The aesthetics include the graphical user interface of the game (GUI) with which the learner will interact with the virtual environment. The development of the aesthetics of the game was guided by educational design research and theory, such as Mayer’s cognitive theory of multimedia learning [41]. Therefore, the interface included elements that provided verbal and pictorial information aligned to the learning objectives, and instructions that directed the learners to read specific words and images. The design of GUI started with the drawing of storyboards that mapped the user interaction and the aesthetics of the game. In these storyboards, the design and research team translated the game mechanics into graphical elements and interaction processes. For example, in the start screen, the user is introduced to the game, the instructions for the game, and buttons to start the game or find information about the development team (see Figure 4). Another example is the main screen

storyboard, where learners can walk around the house, click on mistakes and see their score and numbers of identified mistakes (see Figure 4). Once the entire game was storyboarded, the design team started the development of the game in the Unity 3D game engine.

Figure 4. Start Screen and Main Screen Storyboard

4.3. Development 4.3.1 Game Development For the development of the game, the research team chose the Unity 3D game engine because it allowed the team to develop virtual reality interfaces and a relatively easy transfer of BIM models. Additionally, the game can be published for multiple platforms, such as Windows and Macs [66]. In a first development step, the team collaborated with an architecture firm in San Francisco that specializes in the renovation of townhouses. Together with the architecture firm, the team selected a townhouse model, which was generated directly on-site with Revit and which had not yet been reviewed by the firm. Upon reviewing the model, we identified several design mistakes, which made the model suitable to use for the game. The model was then exported in an FBX format from Revit and imported into Unity. The game engine was used in conjunction with the Oculus Rift Virtual Development Kit. This kit allowed the teams to generate a custom GUI that would work with the Oculus Rift headset (see Figure 5 and 6). The team then translated the storyboards and the game mechanics into GUI elements. The coding language that was used for the game was C#.

Figure 5. Start Screen

Figure 6. Mistake Message

Table 4. Assessment Instrument

Learning Objective Question Type Question

1. Identify the different types of design mistakes. 3. Match the identified mistake to the correct design mistake description.

Multiple Choice

1. Which of the following is not a potential type of design mistake?

2. The following is what type of mistake in the picture to the right? (Coordination)

3. The following is what type of mistake in the picture to the right? (Spatial Layout)

4. The following is what type of mistake in the picture to the right? (Missing Element)

2. Define the identified design mistake. True or False

5. A material mistake can be defined as building elements have the incorrect material assigned. (T/F)

6. A coordination design mistake can be defined as building elements that are not positioned correctly in the design. (T/F)

7. A spatial layout design mistake can be defined as building elements that don’t align to each other or interfere with each other. (T/F)

8. An architectural can be defined as building elements that are not included or present in the design. (T/F)

4. Explain/Summarize the identified design mistake.

True or False

Multiple Choice

9. Design review is an essential process to be performed during construction. (T/F)

10. Evaluating design with paper drawings is an _____ process.

4.3.2 Educational Material In addition to the game development, the research team designed a set of educational material and assessments to support the adoption of the game in a classroom environment. The material was developed to assist future instructors in implementing the game in their courses. The material includes an Instructor’s Guide, User’s Guide, and a Knowledge Test. The Instructor’s Guide contains instructions of how to implement the game, a presentation on the game, learning

objectives of the lecture and activity, technological requirements, duration of the activity and testing, and a solutions key to the Knowledge Test. The Knowledge Test includes a set of ten questions, ranging from multiple choice to true or false, to test the students’ achievement of the game’s learning objectives (see Table 4). The questions in the test are aligned with the game learning objectives set in Table 1. This material will be hosted on the research team’s website STEM Educational Gaming Research Group. Together with the game, this material will be open-source and free for download for any instructor to implement in their courses. 5. CONCLUSION AND FUTURE WORK Virtual reality and educational games have been largely recognizing for their ability to enhance learning in the classroom. While both VR and educational games have been explored for their effects on the levels of engagement, knowledge retention or motivation, their potential to support the students in design and construction-related disciplines still remains largely untapped. The purpose of this paper was to document the development of an educational simulation virtual reality game to teach about the design review process and support students in the development of evaluation skills. Based on this potential, our goal was to develop a VR game that would support the students in learning how to review and evaluate facility design and in the process, for us to understand the value of VR as an education tool. VR as a platform for the DRS game was chosen due to its capability to immerse the user an environment bringing close to realistic experience, which is conducive to learning and performing design reviews. The research team used the A.D.D.I.E. framework to guide the development of this game. In particular, in this paper, the team illustrated their process in the analysis, design, and development of the DRS game. The main contribution of this paper is to provide an experience in developing an educational game for gaining of design review skills alongside the framework for evaluating such skills through the assessment instrument in Table 4.

The next phases in the research and the A.D.D.I.E. framework are the implementation and evaluation of the designed intervention or game. As the next step, the research team will gather evidence illustrating the role that an educational virtual reality game has in supporting the development of design review skills. The team has already received IRB approval for the implementation of the game in their construction management courses. Additionally, the research team has designed an experiment to evaluate the impact that the game has on the students’ learning. In particular, the team will utilize an ABBA experimental design, where the student groups performing design review through the DSR game will be compared to those who will use 2D drawings of the townhouse (game vs. 2D drawings) in order to quantitatively compare the user performance adopting the two media for the same purpose. Additionally, demographic and educational background, as well as familiarity/experience with VR/computer-based games, will be collected as control variables. This experimental design will evaluate the between and within subjects’ effects on meeting the learning objectives. By assessing the impact of the game in the classroom, the team can provide further evidence that VR and educational game have educational value in the classroom. To conclude, future publications will document the implementation and evaluation process, and the team hopes to contribute to the growing body of knowledge that is evaluating the impact that VR has in the classroom.

BIBLIOGRAPHY [1] Whyte J. Industrial applications of virtual reality in architecture and construction. Journal of

Information Technology in Construction (ITcon) 2003;8:43–50. [2] Liu Y, Lather J, Messner J. Virtual Reality to Support the Integrated Design Process: A

Retrofit Case Study. Computing in Civil and Building Engineering (2014), American Society of Civil Engineers; 2014, p. 801–808.

[3] Castronovo F, Danielle O, Liu Y, Messner J. Application of immersive virtual reality systems in an interdisciplinary design studio course. Proceedings of the Joint Conference on Computing in Construction (JC3), vol. 1, 2017, p. 337–45.

[4] Lin J-W, Duh HB-L, Parker DE, Abi-Rached H, Furness TA. Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. Virtual Reality, 2002. Proceedings. IEEE, IEEE; 2002, p. 164–71.

[5] Cavazza M, Charles F, Mead SJ. Developing re-usable interactive storytelling technologies. Building the Information Society, Springer; 2004, p. 39–44.

[6] Computer Integrated Construction Research Program. BIM Project Execution Planning Guide - Version 2.0. University Park, PA, USA: The Pennsylvania State University; 2010.

[7] Kumar S. Procedures to incorporate interactivity in virtual prototypes using a game engine environment. International Conference on Construction Applications of Virtual Reality, 2011, Weimar, Germany: 2011.

[8] Kumar S. Experienced-based design review of healthcare facilities using interactive virtual prototypes. Doctoral Dissertation. The Pennsylvania State University, 2013.

[9] Staub-French S, Atul K. 3D and 4D modeling for design and construction coordination: issues and lessons learned. ITcon Vol 12, Pg 381-407 2007.

[10] Akanmu A, Anumba C, Messner J. Integration of Virtual Models and Physical Construction. Proceedings of the 6th International Conference on Innovation in Architecture, Engineering and Construction (AEC), Penn State University, University Park, PA, USA: 2010.

[11] Akanmu A, Anumba C, Messner J. Critical review of approaches to integrating virtual models and the physical construction. International Journal of Construction Management 2014. doi:10.1080/15623599.2014.972021.

[12] Kalyan TS, Zadeh PA, Staub-French S, Froese TM. Construction Quality Assessment Using 3D as-built Models Generated with Project Tango. Procedia Engineering 2016;145:1416–23. doi:10.1016/j.proeng.2016.04.178.

[13] Eastman CM. Cognitive processes and ill-defined problems: a case study from design. Proceedings of the 1st international joint conference on Artificial intelligence, 1969.

[14] Rieber LP. Visualization as an aid to problem-solving: examples from history. Proceedings of the National Convention of the Association for Educational Communications and Technology, Nashville, TN: 1994.

[15] Rice A. Exploring the Impact of Emerging Landscape Visualization Tools on Spatial Perception and Design Education. In: Buhmann E, Ervin S, editors. Trends in Landscape Modeling, Heidelberg: Wichmann; 2003, p. 173–82.

[16] Dalton B, Parfitt M. Immersive visualization of building information models. Design Innovation Research Center Working Paper 2013;6.

[17] Johansson M, Roupé M, Viklund Tallgren M. From BIM to VR-Integrating immersive visualizations in the current design process. Fusion-Proceedings of the 32nd eCAADe Conference-Volume 2 (eCAADe 2014), 2014, p. 261–269.

[18] Roupé M, Johansson M, Viklund Tallgren M, Jörnebrant F, Tomsa PA. Immersive visualization of Building Information Models. Living Systems and Micro-Utopias: Towards Continuous Designing, Proceedings of the 21st International Conference of the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA 2016), 2016, p. 673–682.

[19] Liston K, Fischer M, Kunz J. Requirements and benefits of interactive information workspaces in construction. 8th International Conference on Computing in Civil and Building Engineering, August, 2000, p. 14–17.

[20] Balakrishnan B, Kalisperis LN, Muramoto K, Otto G. Multimodal virtual reality environment for architectural design (re) presentation. Proceedings of the 11th International Conference on Computer Aided Architectural Design Research in Asia (CAADRIA), 2006, p. 513–519.

[21] Leicht RM, Messner JI, Poerschke U. INVOLVE: Developing interactive workspaces that impact communication and task performance when using virtual prototypes. Journal of Computing in Civil Engineering 2012;28:191–201.

[22] Leicht RM. A framework for planning effective collaboration using interactive workspaces. Doctoral Dissertation. The Pennsylvania State University, 2009.

[23] Bouchlaghem NM, Liyanage IG. Virtual reality applications in the UK’s construction industry. CIB REPORT 1996:89–94.

[24] Whyte J. Virtual reality and the built environment. Oxford, U.K.: Architectural Press; 2002. [25] Gopinath R. Immersive virtual facility prototyping for design and construction process

visualization. Doctoral Dissertation. The Pennsylvania State University, 2004. [26] Shiratuddin MF, Thabet W, Bowman D. Evaluating the effectiveness of virtual environment

displays for reviewing construction 3D models. Proceedings of the 4th International Conference on Construction Applications of Virtual Reality, Lisbon, Portugal: 2004, p. 87–98.

[27] Dawood N. VR-roadmap: a vision for 2030 in the built environment. ITcon Vol 14, Special Issue Next Generation Construction IT: Technology Foresight, Future Studies, Roadmapping, and Scenario Planning Pg 489-506 2009.

[28] Bassanino M, Wu K-C, Yao J, Khosrowshahi F, Fernando T, Skjærbæk J. The impact of immersive virtual reality on visualisation for a design review in construction. Information Visualisation (IV), 2010 14th International Conference, IEEE; 2010, p. 585–589.

[29] Rohani M, Fan M, Yu C. Advanced visualization and simulation techniques for modern construction management. Indoor and Built Environment 2013:1420326X13498400. doi:10.1177/1420326X13498400.

[30] Fernando TP, Wu KC, Bassanino MN. Designing a novel virtual collaborative environment to support collaboration in design review meetings. Journal of Information Technology in Construction 2013;18:372–96.

[31] Berg M. Exploring the impact of virtual reality in design review processes. Master Thesis. University of Twente, 2014.

[32] Castronovo F, Nikolic D, Liu Y, Messner JI. An evaluation of immersive virtual reality systems for design reviews 2013.

[33] Hobbs B, Dawood N. Harnessing the power of virtual reality–the potential for VR as a virtual integrated environment for project development in construction. Proceedings of the Berkeley-Stanford CE&M Workshop: Defining a Research Agenda for AEC Process/Product Development in, 2000, p. 26–28.

[34] Dunston PS, Arns LL, McGlothin JD. An Immersive Virtual Reality Mock-Up for Design Review of Hospital Patient Rooms. Proceedings of the 7th International Conference on Construction Applications of Virtual Reality, Penn State University, University Park, PA: 2007.

[35] Whisker VE, Baratta AJ, Yerrapathruni S, Messner JI, Shaw TS, Warren ME, et al. Using immersive virtual environments to develop and visualize construction schedules for advanced nuclear power plants. 2003 International Congress on Advances in Nuclear Power Plants (ICAPP), Córdoba, Spain: 2003.

[36] Maldovan KD, Messner JI, Faddoul M. Framework for reviewing mockups in an immersive environment. Proceedings of the 6th International Conference on Construction Applications of Virtual Reality, vol. 2006, 2006.

[37] Rahimian FP, Arciszewski T, Goulding JS. Successful education for AEC professionals: case study of applying immersive game-like virtual reality interfaces. Visualization in Engineering 2014;2:4. doi:10.1186/2213-7459-2-4.

[38] Vassigh S, Davis D, Behzadan AH, Mostafavi A, Rashid K, Alhaffar H, et al. Teaching Building Sciences in Immersive Environments: A Prototype Design, Implementation, and Assessment. International Journal of Construction Education and Research 2018;0:1–17. doi:10.1080/15578771.2018.1525445.

[39] Paes D, Arantes E, Irizarry J. Immersive environment for improving the understanding of architectural 3D models: Comparing user spatial perception between immersive and traditional virtual reality systems. Automation in Construction 2017;84:292–303.

[40] Rieber LP. Multimedia learning in games, simulations, and microworlds. The Cambridge Handbook of Multimedia Learning 2005:549–67.

[41] Mayer RE. The Cambridge Handbook of Multimedia Learning. Cambridge University Press; 2005.

[42] Schunk DH. Learning Theories: An Educational Perspective. 6 edition. Boston: Pearson; 2011.

[43] Aldrich C. Simulations and the Future of Learning: An Innovative (and Perhaps Revolutionary) Approach to e-Learning. Jossey-Bass Inc., Publishers; 2003.

[44] Squire KD. From content to context: Video games as designed experiences. Educational Researcher 2006;35:19–29.

[45] Squire K. Changing the game: What happens when video games enter the classroom? Innovate 2005;1.

[46] Gee JP. What video games have to teach us about learning and literacy, New York : Palgrave Macmillan, 2007; 2007.

[47] Katsaliaki K, Mustafee N. A survey of serious games on sustainable development. Simulation Conference (WSC), Proceedings of the 2012 Winter, IEEE; 2012, p. 1–13.

[48] Blackman S. Serious games... and less! ACM Siggraph Computer Graphics 2005;39:12–6. [49] Michael DR, Chen S. Serious Games: Games That Educate, Train, And Inform. Thomson

Course Technology; 2006. [50] Hirose Y, Sugiura J, Shimomoto K. Industrial waste management simulation game and its

educational effect. Journal of Material Cycles and Waste Management 2004;6:58–63. [51] Philpot TA, Hall RH, Hubing N, Flori RE. Using games to teach statics calculation

procedures: Application and assessment. Computer Applications in Engineering Education 2005;13:222–32.

[52] Castronovo F. Assessing Problem-Solving Skills in Construction Education with the Virtual Construction Simulator 2016.

[53] Castronovo F, Leicht RM, Messner JI, Zappe SE, Nikolic D. Enhancement of Learning Objectives in Construction Engineering Education: A Step toward Simulation Assessment. Construction Research Congress 2014, American Society of Civil Engineers; 2014, p. 339–48.

[54] Castronovo F, Meter PNV, Messner JI. Leveraging metacognitive prompts in construction educational games for higher educational gains. International Journal of Construction Management 2018;0:1–12. doi:10.1080/15623599.2018.1492760.

[55] Lee S, Nikolic D, Messner JI, Anumba CJ. The Development of the Virtual Construction Simulator 3: An Interactive Simulation Environment for Construction Management Education, American Society of Civil Engineers; 2011, p. 454–61. doi:10.1061/41182(416)56.

[56] Nikolic D. Evaluating a simulation game in construction engineering education: the virtual construction simulator 3. Pennsylvania State University, 2011.

[57] Bowman DA, McMahan RP. Virtual Reality: How Much Immersion is Enough? Computer 2007;40:36–42.

[58] Thorsteinsson G. Developing an Understanding of the Pedagogy of Using a Virtual Reality Learning Environment (VRLE) to Support Innovation Education. The Routledge International Handbook of Innovation Education Edited by LV Shavinina Oxford: Routledge ISBN-10 2013;415682215:456–70.

[59] Riva G, Mantovani F, Capideville CS, Preziosa A, Morganti F, Villani D, et al. Affective interactions using virtual reality: the link between presence and emotions. CyberPsychology & Behavior 2007;10:45–56.

[60] Branson RK, Rayner GT, Cox JL, Furman JP, King FJ, TECHNOLOGY FSUTCFE. Interservice Procedures for Instructional Systems Development. Defense Technical Information Center; 1975.

[61] Morrison GR. Designing effective instruction. Hoboken, NJ: Wiley; 2011. [62] Cannon HM, Burns AC. A Framework For Assessing The Competencies Reflected In

Simulation Performance. Developments in Business Simulation & Experiential Exercises 1999;26:40–4.

[63] Bloom BS. Taxonomy of educational objectives: The classification of educational goals. United Kingdom: Longman Group United Kingdom; 1969.

[64] Anderson LW, Krathwohl DR. A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. New York: Longman; 2001.

[65] Schell J. The art of game design a book of lenses. - Description based on print version record. Boston: Elsevier/Morgan Kaufmann; 2008.

[66] Unity Technologies. The future of visualization in architecture n.d.