A N I M M E R S I V E V R E D U C A T I O N A L E X P E R I E N C E

Mu

ted

Blu

e

Interactive Virtual Reality Museum experience that teaches users about the Blue Whale and the

world’s oceans.

Huy Ngo

Thesis submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of

Master of Fine Artsin

Creative Technologies

Thomas Tucker, ChairWallace Santos Lages

Sterling Nesbitt

May 2, 2019Blacksburg, Virginia

Keywords: Virtual Reality, Museum, Education, 3D Modeling, Ocean, Blue Whale.

Muted BlueH U Y N G O

A B S T R A C T

Muted Blue (Whale Level) is a Virtual Reality educational

experience that explores the possibility of creating an

alternative learning method in an informal learning setting

such as a museum. This project seeks to merge art, technology

and education to create an immersive, interactive learning

experience that is geared toward Gen Z and Millennial but can

be enjoyed by every age group. Unreal 4 game engine along

with other 3D packages made it possible to develop a Virtual

Reality experience that can be used as a complimentary piece

that can possibly exist along side a museum exhibit.

G E N E R A L A U D I E N C E A B S T R A C T

Muted Blue (Whale Level) is an interactive museum experience

that aims to give audience a more informative and interactive

digital experience. The main focus of Muted Blue is allowing

users to explore and learn at their own pace. This Virtual

Reality Museum idea in particular, is geared toward the

potential museum goers that are in the Gen Z and Millennial

generations because technology has been an integral part of

their lives. Muted Blue (Whale Level) is capable of creating an

impossible space that cannot be recreated inside of a real

museum while providing meaningful, educational information

along with beautiful visuals that will assist with the learning

process and inspire the users to learn and care more about our

ocean.

-iv

A C K N O W L E D G E M E N T S

I would like to thank my committee members, Thomas Tucker, Wallace Santos Lages and Sterling Nesbitt for all of your help, guidance, expertise, patience and support throughout the creation of this project as well as my graduate school experience at Virginia Tech.

I’d also like to thank all of my friends and colleagues for their invaluable moral support and in helping me troubleshooting technical issues. Without you, this project would not have come to fruition.

-v

C O N T E N T

Abstract .................................................................................................. 3

General Audience Abstract ........................................................................ 4

Acknowledgements .................................................................................. 5

List of Figures .......................................................................................... 7

Introduction ............................................................................................ 8

Preliminary Works .................................................................................. 11

Original Concepts & Deviation .................................................................. 12

Architectural Explorations ....................................................................... 13

Subject Matter (Blue Whale) .................................................................... 14

Production (3D Modeling) ........................................................................ 15

Production (Animation) ........................................................................... 16

Production (Sound) ................................................................................. 17

Production Challenges ............................................................................. 19

User Evaluation ...................................................................................... 20

Conclusion .............................................................................................. 21

Bibliography .......................................................................................... 22

-vi

L I S T O F F I G U R E S

1.1 Museum Journey Map .........................................................................

2.1 Top: Cave Mesh in Unity ..................................................................... 2.2 Middle: Test of physical cave ............................................................... 2.3 Bottom: Maya view of Cave mesh .........................................................

3.1 Render of the bat .............................................................................. 3.2 Pointcloud of the head ......................................................................3.3 Pointcloud of the body ...................................................................... 3.4 Retoped bat’s wings .........................................................................

4.1 Leopard skeleton ............................................................................... 4.2 Assembled dinosaur leg ..................................................................... 4.3 Horned owl skeleton .......................................................................... 4.4 3D model of leopard ...........................................................................

5.1 Iteration 1 ........................................................................................ 5.2 Iteration 2 ........................................................................................ 5.3 Iteration 3 ........................................................................................ 5.4 Iteration 4 .......................................................................................

6.1 Dead Whale in Indonesia ................................................................... 6.2 Dead Whale in Indonesia ................................................................... 6.3 Dead Whale in Thailand ....................................................................

7.1 Shark Model ..................................................................................... 7.2 Diffuse Map ..................................................................................... 7.3 Normal Map ..................................................................................... 7.4 UE4 packed Map ..............................................................................

8.1 Shark with bone joints placed ............................................................ 8.2 Whale with bone joints placed ...........................................................

9.1 Testplaying of Muted Blue .................................................................

9

101010

11111111

12121212

13131313

141414

15151515

1616

20

-01

Muted Blue is a Virtual Reality educational

experience that aims to place users in a beautiful

underwater environment populated by animals and

plant life while educating museum goers about the

blue whale and the ocean.

While creating a VR Museum experience,

there are 2 challenges that needed to be addressed.

The first is how to create an enjoyable VR experience

regardless of the application, whether it be gaming

or education, the second is the museum visiting

experience. VR gaming involves the use of a VR

headset that places the player in the world. The VR

controllers allow the player to interact and modify

the environment in ways that traditional gaming

experience could not allow. Through engaging

the senses more directly and bringing a haptic

experience with more sophisticated hardware, VR

gaming is slowly gaining and redefining the value

of being a protagonist in videogame. Educational VR

experiences are becoming more popular as they have

a different approach to the learning experience by

forgoing the traditional structure of a classroom and

allow the users to teach themselves about the subject

matter in a more informal setting. VR educational

apps such as Apollo 11 VR and Anatomy 4D were

studied as they provide clear information about the

subject matter along with ease of navigation and

good visuals that in combination, create an excellent

educational VR experience.

Apollo 11 VR is a cinematic VR experience

that teaches the user about the moon landing. The

app has two options with one being completely

cinematic and the other gives the player a few

interactive options. Regardless of the way the

user choose to play, a majority of the information

is relayed through JFK’s voice as well as a visual

recreation of the moon landing. This allows the

players to learn through the action that is happening

in front of them instead of extensive texts and

scripts.

Anatomy 4D is another app that was

studied. This app allows user to explore human’s

anatomy through augmented reality. This app lets

you explore several human organs such as the

human heart completed with pulsing animation and

a description text that descirbes the function of each

part of the heart. The inclusion of visual aid helps

the students gain a deeper understanding of human

anatomy.

The second challenge that need

to be explored is the museum visiting experience.

The museum journey map usually follows a pattern

I N T R O D U C T I O N

-02

of Awareness - Interest - Plan - Visit - Return.

Museums are doing this in several different ways.

Often times, Awareness and Interest are raised

through Social Media sites while the Visit itself

consists of up-to-date content as well as new

methods of presentation such as digital aid that

appeals to the average goers as well as an experts

in the subject matter.

In order to achieve the immersiveness of

a VR experience and an educational experience of

a museum exhibit, a semi limited but self guided

VR experience was created that adds Awareness/

Interest into the VR world itself with visual cues to

spark interest in the user. The visit would follow

the same structure with a beautiful environment

and visual cues to spark interest in the user to

explore the VR space. A motion tracking system is

utilized to give the physicality of moving through

an actual space while the visuals let the user know

that they are existing in an impossible space with

that space eliciting real feelings and emotions.

I N T R O D U C T I O N

1.1 Museum Journey Maphttps://www.bdcnetwork.com/blog/future-museums-ultimate-vis-itor-experience

The effort to merge art and technologies

to visualize history and produce immersive and

visually interesting educational experiences have

been a continuous theme in my early works.

From visualizing history, digital recreation of bat

and reconstruction of dinosaur bones, my works

aim to merge the gap between art, technology,

entertainment and education.

The Vauquois World War I project is an educational

Virtual Reality experience that allows the user to

walk through the underground tunnel of Vauquois,

a World War I battlefield in Northern France. This

experience includes a 12ft tracking space using the

VIVE Pro headset in which the user will walk through

the digital tunnel recreated from 3D scanned data.

Accompanying this are two physical walls created to

match the physical width, height and texture of the

actual tunnel. The user can then walk through the

virtual tunnel which has educational informations

in the forms of sound, images, and text that the

user can explore at their own pace. The physical

tunnel then guides the user physically through it

by engaging the sense of touch which the textures

and the locations of various artifacts placed in the

physical tunnel match the locations of their digital

counterpart.

P R E L I M I N A R Y W O R K S

2.1 Top: cave mesh in Unity

2.2 Middle: Test of the physical cave

2.3 Bottom: Maya view of the cave mesh

-03

-04

I worked with the School of Physics at

Shandong University, China and the Department of

Mechanical Engineer at Virginia Tech to recreate an

anatomically accurate 3D model of a Hipposideros

Armiger and simulate its flight and noseleaf

movements. Over a period of 2 months, I worked

with engineer students to study the behavior

and movements of the Hipposideros Armiger and

motion capture its flight and 360 degree turns in

our flight tunnel. The end result of accurate wing

flaps and noseleaf movements were achieved with

a combined technique of point matching and hand

animation clean up in Maya.

P R E V I O U S W O R K S

3.1 Render of the bat

3.2 Pointcloud of the head

3.3 Pointcloud of the body

3.4 Retoped bat’s wings

-05

Initially, the idea was to recreate a

prehistoric VR museum that educates the user

about the creatures that lived during that time.

The skeletons would then become fully animated

with muscle system and skin as well as animations

when the user interact with them. This includes

some of the oldest bone specimens of dinosaur

ever found as well as the flora of that time. The

Virginia Tech geology department would provide

the fossil samples of these dinosaurs as well as the

3D scanners that could be use to digitally recreate

these fossils. It quickly became apparent that the

idea was extremely difficult to execute due to lack of

complete skeletons in the Virginia Tech collection

as well as limited information about these ancient

creatures.

The idea then deviated to a museum space that

includes the skeletons of many modern day

animals such as a leopard, brown bear, and

american bisons. This idea came from the readily

available information about these animals as well

as an online archive of already 3D scanned and

processed skeletons of these animals, courtesy of

the Idaho University Museum of Natural History.

The diversity of the type of animals presented an

intriguing challenge in Modelling and animation

techniques. It quickly became clear that the

challenge of animating several different animals

both quadruped and biped would be a huge

undertaking for the time frame given.

O R I G I N A L C O N C E P T S & D E V I A T I O N

4.2 Assembled dinosaur leg 4.3 Horned owl skeleton

4.4 3D Model of leopard

4.1 Leopard skeleton

-06

The initial space that the player would load

into would serve as a hub for player to explore and

choose different animals to learn about. In order to

build the space in VR, the interior of several famous

museums were researched, such as the interior of

the Louvre, The Smithsonian Museum of Natural

History and New York’s Museum of Natural History.

Classical structures were also explored such as that

Pantheon and the Roman water gardens. More

modern architecture space and philosophy were also

explored in the process of designing these spaces.

Tadao Ando’s and Peter Zumthor’s architectures

with opposing ideas of centrifugal and centripetal

architectures respectively. Centrifugal architecture

tends to make the user aware of the natural world

around them by bringing the world into the space

whereas centripetal architecture tends to make

the user aware of their action and how such action

effects their surrounding. I wanted to bring this

philosophy into my VR environment and have the

environments tell their part of the stories. Each

iteration is an exploration into different aspect

of these two architectural philosophies discussed

above.

A R C H I T E C T U R A L E X P L O R A T I O N S

5.1. Iteration 1. Exploring the idea of nature taking over man made structure.

5.2. Iteration 2. Exploring the structure of the Smithsonian Africa hall and recreating it in Unreal 4 engine.

5.3. Iteration 3. Exploring the architectural philosophy of Ando with use of concrete material and bringing in sunlight to connect with the outside world.

5.4. Iteration 4. Final environ-ment which explored the form of classical Greek and Roman architecture.

-07

The Blue Whale level is inherently a small part in

a much bigger, more comprehensive VR experience

that explores several different animals both land

and sea and sky. One of the most appealing aspects

of VR is that it allows the user to occupy a space

that otherwise would not be possible or very

hard to experience in real life. The sheer size of

the blue whale as well as its habitat presented a

unique opportunity for VR development. From

being the largest and iconic animal on earth, blue

whale is also one the most prominent examples of

humanity’s negative environmental effects on the

ocean. Dead whales caused by plastic swallowing

or ship collisions as well as lost whales beaching

themselves due to ocean sound pollution are some

of the most sobering reminders of the negative

environmental effects humanity are leaving on

the planet. For these reasons, the Blue Whale was

chosen to be the first “room” of this VR nature

museum.

S U B J E C T M A T T E R ( B L U E W H A L E )

6.1 Dead whale in Indonesiahttps://www.cnn.com/2018/11/20/asia/indone-sia-whale-plastic-scli-intl/index.html

6.2 Dead whale in Indonesiahttps://www.nerdiest.org/dead-whale-with-1000-plas-tic-pieces-in-stomach-found-in-indonesia-video/

6.3 Dead whale in Thailandhttps://www.nbcnews.com/news/world/dead-whale-found-thailand-17-pounds-plastic-its-stom-ach-n879581

-08

The development of VR museum started with

asset creations. To create VR ready 3D models, a

multitude of softwares were used. Softwares like

Zbrush, Maya, and Substance Painter were used

extensively for during the modelling process with

some shapes being created at a lower polycount

first in Maya and then brought into Zbrush for

extra detailings. The models are then put back in

Maya for retopologizing, a process used to reduce

the number of polygons on a 3D model to optimize

its performance within a game engine. The asset

is then UV unwrapped which is a process used to

flatten out the 3D mesh and convert them into a 2D

layout which is then used to apply textures on top.

The asset is then imported into Substance Painter

for the texturing process. The textures are then

packed with optimizations for Unreal Engine 4 and

exported out as maps; this includes Albedo, Normal,

Roughness, Metallic and Ambient Occlusion. The

model and the maps are then imported into Unreal

Engine 4, the developing platform used to develop

VR Museum (Whale Level). The models are used to

populate the scene and light were added as well as

materials to create the desired visuals of the scene.

P R O D U C T I O N ( 3 D M O D E L I N G )

7.2 Great white diffuse map

7.1 Shark model

7.3 Great white normal map

7.4 Great white UE4 packed map

-09

After creating the 3D meshes and generating

textures for them, the next step is to animate them.

In order to animate the whale as well as the

various fish species in the scene, they first need

to be rigged. Rigging is a process where “bones”

need to be placed inside the mesh and the mesh

is then skinned on top. After bone placements are

finished, a process of weight painting needs to be

done. Weight painting is used to dictate how much

influence each bone/joint has in a certain part

of the mesh. To achieve a fish like movement in

the animation, an Inverse Kinematic(IK) handle

was attached from the root bone to the tip bone

with a spline attached along the bones. A sine

wave deformation technique was applied to the

curve which in turn drove the IK handle to move

the bones in a fish-like animation. The last step

was to export this animation in an FBX file format

to ensure that the animation is carried over into

Unreal Engine 4. For the Blue whale mesh, there

were 29 bone joints placed, the shark meshes have

17 bones in them.

P R O D U C T I O N ( A N I M A T I O N )

8.1 Shark with bone joints placed

8.2 Whale with bone joints placed

-010

Blue whale songs , ocean and underwater sounds

were used to create ambient sounds in the level.

Audacity sound software was used to digitally

remove excess clicks and as well as adding reverbs

and bass boosted the sounds to create sounds that

would fit the visual of each level.

Blue whale sound scripts details.

1 HeadHollogramTriggerBox:

Whale Communication Information sound

script: Blue whales have evolved to use various

acoustic sounds that allow them to talk to each

other across vast oceans. Blue Whales have

exceptionally deep voices and can produce sound

at as low as 14Hz, with volume louder than 180

decibels, which makes the Blue Whale the loudest

animal on the planet. These whale “songs” are

used for a multitude of purposes, to navigate the

world's oceans, to find food sources, and to

communicate.

2 TriggerBox_Body: Whale size and facts

about whale body script: This giant is the largest

animal ever known to have lived on the planet.

General length ranges from 80-100 feet while

weighing in at around 200 tonnes, or

441,000 pounds.

3 TriggerBox_Krill: Blue Whale Diet script:

Despite their massive size, blue whales eat tiny

crustaceans called krill. During its peak

consumption period, a blue whale can eat up to

8000 lbs of krills.

4 TriggerBox_Tail: Blue Whale swim speed

script: The Blue Whale’s massive tail fluke

allows it to travel the world’s ocean at speed of

up to 30 MPH in short bursts while their

travelling speed is around 12 MPH. They slow

down to 3 MPH when feeding. They typically

swim at a depth of about 43 feet with an average

dive depth of 330 feet but is capable of diving up

to 1640 feet.

5 TriggerBox_Baleen: Blue Whale Baleen

information: Blue whales is a species of “baleen”

whale. Instead of teeth, they have plates of baleen,

these plates are used to filter their food. When

eating, the whale opens its mouth and lets both

water and krill into it. It then uses its tongue to

push the water through its baleen plates, which

trap the krills inside to be swallowed.

P R O D U C T I O N ( S O U N D D E S I G N )

-011

6 TriggerBox_WorldMap: Blue Whale

locations and migration behavior script: Blue

whales can be found in almost every ocean on the

planet ranging from the equator to the ice edges in

the North and South poles. Whales migrate often

and like to spend the winters in warmer, tropical

water, migrating to the polar regions in the spring

and summer.

Effects of plastic and noise pollution

on the world’s oceans details

Unfortunately for these gentle giants.

Humans’ activities have left severe damage on their

habitat. Plastic. The non-biodegradable substance

that plays such a pivotal role in human day to day

life is also choking up our ocean and the marine

lives that live in it. Every year, 8 million metric

ton of plastic is dumped into our ocean that then

get dispersed by the currents through out every

corner of the world’s ocean.

At least 267 species worldwide are affected by

plastic pollution. Sea birds that feed on the surface

of the ocean can ingest floating plastic which later

transfer to the chicks during feeding. Sea

turtles, sharks, dugongs and other marine

animals are caught in abandoned fishnets.

Cetaceans such as the whales and dolphins are

found dead with stomachs full of plastic. Not

only do plastic affects whales directly, it also

affects their food source. As plastic spends time in

the ocean. It starts to break down into smaller and

smaller pieces due to photodegradation from

sunlight , the broken down plastic then get mixed

with fish and whales’ food sources like plankton

and other small organisms.

Sound pollution such as ship horns and other man

made noises significantly interfere and

obscure marine animals’ ability to hear

natural sound and communicate with each

other. Blue whales communication have been

reduced by as much as 90% due to man made

Ocean Noise. In many cases this lead to ship

collisions as well as blue whale getting lost in

unfamiliar water due to man made ocean noises

blocking their communication system.

P R O D U C T I O N ( S O U N D D E S I G N )

-012

Creative Challenges. How to differentiate this

experience from other VR experiences. The idea

of this was to create an educational experience

that educate the user about blue whales and ocean

pollution. It was a challenge to find a balance

between taking creative liberties of the visuals

to create an interesting VR environment that

will immediately captivate the audience for long

enough with interesting visual cues that encourage

exploration of the scene.

Presentation Challenges. Find an effective way

to present the research information in a concise

and direct manner without boring out the user. A

balance of educational information coupled with

hyper-stylized environment was created in order

to both captivate the audience while providing just

enough educational materials that will not bore out

the user.

Collision Challenges: The most challenging part

of the development process was to develop an

interesting and engaging VR experience. The user’s

collider accurately triggering the content of the

content “orbs”. This problem was solved by making

the collider follow only the position of the headset

and not the rotation. The second most difficult

challenge was to get the Qualisys tracking system

to work within the cube and to track the VIVE

headset. Due to a latency issue and a difference in

the up axis between Qualisys and UE4, this problem

was not solved in time and the technology was not

used in the final demonstration of this project.

Audio Challenge: Implementing the directional

audio system in perform and make it accurately

emmulate the spatial sounds in the VR scene proved

to be a challenge as there are no overhead speakers

in the space to represent the overhead sound of

the whale swimming by in the scene, which is an

integral audio/visual cue for the user to look up.

This issue was solved by making the whale sound

travel from the speakers on the left side of the room

to those on the right. While this did not entirely

compensate for the lack of overhead speakers, it

was useful in providing a sense of direction in the

way the sound was traveling.

P R O D U C T I O N C H A L L E N G E S

-013

Evaluations from 15 different testers have

been that visually, the whale level environment is

beautiful with a lot of sea creatures moving and

colors. User evaluations also stated that the first

level provides a nice aesthetic break from the other

2 levels but maintained a general theme. The last

level does a good job of visually separating itself

from the whale level while retaining the ocean

environment. Audiowise, the environmental

sounds and music have generally received positive

feedbacks with suggestions on changing the blue

whale sounds to make it not so repetitive. The

content orbs that allows the user to get information

about blue whale as well as loading the levels

have receieved generall positive reviews. The

experience was described by Executive Director,

Applied Research in Immersive Environments

and Simulations Todd Ogle as “intuitive and easy

to follow” with Assistant Professor Phat Nguyen

saying “I can see that this experience will be a great

hit in science museums and with class rooms”.

Professor Todd Ogle also stated “it was engaging

but needs more methods of interaction with the

user and more contents”.

U S E R E V A L U A T I O N

19.1 Tesplaying of Muted Blue

-014

C O N C L U S I O N

I set out to create a beatiful world in Virtual Reality that aims to aid the learning process as well as

improve the entertainment value in the museum going experience. The topic of the ocean allowed me to take

the user to fantastical environments that cannot exist in real life while trying to educate the users about a

real issue in today’s world. 3D packages such as Maya, Zbrush, and Unreal Engine 4 allowed me to create a

fantasy virtual reality world. Technical challenges such as collision, play space, and sending OSC messages

were solved with a combination of research, repeating trials and seeking help from other people. Feedbacks

have suggested to include more ways to interact with the environment to further enhance the experience as

well as including more learning through visual component rather than just audio. User evaluations of the

experience also stated that the experience was user friendly and the visual was beautiful with many stating

that they see real values in using this to help educate people in an informal learning setting.

-015

B I B L I O G R A P H Y

1. How Museums are using Virtual Reality? (2019, April 15). Retrieved fromhttps://www.museumnext.com/2019/01/how-museums-are-using-virtual-re-ality/

2. 10154013767872392. (2018, June 15). Virtual reality Rembrandt, Degas andthe Temple of Dendur. Retrieved May 2, 2019, from https://www.gearbrain.com/virtual-reality-museum-art-summer-2577767924.html

3. Virtual reality museums take art to the next level. (2019, March 25).Retrieved May 2, 2019, from https://provr.io/blog-en/virtual-reality-muse-ums-take-art-to-the-next-level/

4. Paqua, M., & Paqua, M. (2018, May 15). Visitor Journey Mapping In Mu-seums. Retrieved May 2, 2019, from https://museumsdigitalculture.prattsi.org/visitor-journey-mapping-in-museums-f18442ee1d99

5. Natural History Museums. (n.d.). Retrieved May 2, 2019, from http://framestorevr.com/natural-history-museums

6. Pivovarov, K. (2018, March 21). The Future of Museums: The UltimateVisitor Experience. Retrieved May 6, 2019, from https://www.callisonrtkl.com/you-are-here/the-future-of-museums-the-ultimate-visitor-experience/

7. Blue Whale. (2019, June 14). Retrieved June 19, 2019, from https://kids.nationalgeographic.com/animals/mammals/blue-whale/

8. Kids Do Ecology. (n.d.). Retrieved June 19, 2019, from http://kids.nceas.ucsb.edu/

9. More Ideas, Less Waste. (n.d.). Retrieved June 19, 2019, from https://earth911.com/

10. National Geographic. (n.d.). Retrieved June 19, 2019, from https://www.nationalgeographic.com.au/

11. The Problem of Marine Plastic Pollution. (2017, December 20). RetrievedJune 19, 2019, from https://www.cleanwater.org/problem-marine-plastic-pollution