5

Click here to load reader

i- CAVE: Intelligent Agent-Augmented CAVE System CAVE.pdf · i- CAVE: Intelligent Agent-Augmented CAVE ... intelligent agent-augmented CAVE system, namely i-CAVE, ... Mu l t i -a

  • Upload
    docong

  • View
    213

  • Download
    0

Embed Size (px)

Citation preview

Page 1: i- CAVE: Intelligent Agent-Augmented CAVE System CAVE.pdf · i- CAVE: Intelligent Agent-Augmented CAVE ... intelligent agent-augmented CAVE system, namely i-CAVE, ... Mu l t i -a

i- CAVE: Intelligent Agent-Augmented CAVE

System

Yundong Cai

School of Computer Engineering

Nanyang Technological University

Singapore

[email protected]

Zhiqi Shen

School of Computer Engineering

Nanyang Technological University

Singapore

[email protected]

Bai Xiaoguang

School of Computer Engineering

Nanyang Technological University

Singapore

[email protected]

Abstract— Cave Automatic Virtual Environment (CAVE) is a

virtual reality environment with immersive 3D experience and

engaging interactions, which would be a very good medium of

entertainment, training or exercise for elderly. However, the raw

CAVE system doesn’t provide semantic understanding of players’

behaviors and emotions in real-time, which leads to a challenge for

application developers to create a personalized virtual environment

for various players. In this paper we proposed an intelligent agent-

augmented CAVE system, namely i-CAVE, to bridge the gap

between application developers and end players, especially the

elderly which has the special requirements of interactions. We have

implemented the system with EON Studio in a Live CAVE

environment and a case study called “Walking with Dinosaurs”.

The interviewed players give very positive feedback on the

immersion and engaging interactions.

Keywords- CAVE, semantic information, agent

I. INTRODUCTION

Cave Automatic Virtual Environment (CAVE) is a virtual

reality environment with immersive 3D experience and

engaging interactions, which would be a very good place of

entertainment, training or exercise for elderly. However,

CAVE system doesn’t provide semantic understanding of

players’ behaviors and emotions in real-time, which leads to a

challenge for application developers to create a personalized

virtual environment for various kinds of players. In this paper

we proposed an intelligent agent-augmented CAVE system,

namely i-CAVE, to bridge the gap between application

developers and end players. With the augmentation of

intelligent agents, we are able to achieve good immersion and

engaging interactions for the players.

II. BACKGROUND

Interactive digital media (IDM) attracts a lot of research and

industry today to education and entertainment. Cave

Automatic Virtual Environment (CAVE) is one of the world’s

leading 3D immersive virtual environments, which is used by

a lot of researchers to develop interactive and immersive

virtual worlds. Institute of Media Innovation (NTU) has setup

one CAVE system and launched immersive virtual

applications successfully. Currently, the research efforts on the

CAVE system mainly focuses on human computer interaction

(HCI), i.e. virtual environment projection on the walls, gesture

and motion recognition etc., and the virtual world application

developments [1].

However, it is a great challenge for novice developers to

create a CAVE application with EON studio by EON Reality.

Even more, how to naturally immerse players into the

interactive environment is one of the most significant

challenges for human computer interaction. Though the

current CAVE improves immersion compared to traditional

virtual worlds in computers, it still lacks the capabilities to

understand the players’ behaviors in real-time in order to

provide a personalized player experience. By learning the

player’s behaviors or emotions in CAVE, we will be able to

provide a personalized virtual world in order to cater to the

player’s preference.

Nowadays, artificial intelligence technologies such as personal

mobile agents and other soft computing techniques have

become more and more mature, which makes it possible to

achieve the virtual world personalization in the CAVE system.

Currently CAVE only provides the low-level data collections

of motions and gestures, but there is a lack of semantic

learning of the recorded data and adapting of the virtual world

and virtual characters inside the virtual world. Agents are

goal-oriented, autonomous, reactive and proactive objects with

social communications, which can be used to handle the

semantic analysis in order to advance the immersion in the

CAVE.

The CAVE system was built in the immersive room at

Institute for Media Innovation (IMI) in 2011. It’s an

immersive virtual reality environment with rich interactions

Page 2: i- CAVE: Intelligent Agent-Augmented CAVE System CAVE.pdf · i- CAVE: Intelligent Agent-Augmented CAVE ... intelligent agent-augmented CAVE system, namely i-CAVE, ... Mu l t i -a

with users. Besides the traditional input devices like keyboard,

mouse, gamepad and microphone, it makes use of latest

hardware including infrared emitters, position trackers, gesture

gloves, kinect and so on. Electromagnetic 6DOF tracking is

implemented to monitor the viewers' position and orientation.

Scenes data and users’ action are processed by graphics

workstations linked together in a local area network [7]. The

3D dynamic images are sent by 5 high-end projectors mounted

on the ceiling, reflected by mirrors and projected to the curved

screen which covers 330 degrees of view horizontally.

Stereoscopic audio sound system is also integrated into the

system. With the stereoscopic active lightweight glasses, users

can be fully immersed in the virtual world.

III. PROPOSED I-CAVE

The objective of our research project is to develop an

intelligent agent-augmented CAVE system, namely i-CAVE,

which overlays a semantic agent layer on top of the current

CAVE system to achieve high level semantic analysis and

virtual world adaptation. Triple I system means “Interaction,

Immersive and Innovation”, we aims to add an extra “I” which

is “Intelligence” to it. Other CAVE researchers will be able to

benefit from the i-CAVE system for they own research and

development.

The intelligent agents in i-CAVE should have the following

capabilities:

Semantically analyze the players’ statuses (mood,

emotion, energy etc.) through their behaviors,

gestures or facial expressions inside CAVE

The player’ statuses (i.e. mood, emotion, energy etc.)

are tracked in the CAVE for adaptive human

computer interaction. By learning from the collected

data with data mining techniques, we will be able to

deduce whether the player is happy or sad, energetic

or lethargic, which would be a basis to provide

him/her a personalized virtual environment.

Adapt virtual characters in the system to the players’

preferences in real-time so that the players can enjoy

a more personalized immersive experience

Based on the player’ statuses (mood, emotion, energy

etc.), i-CAVE system could adjust the virtual

characters’ behaviors and emotions inside the

environment seamlessly in order to serve the players’

needs. For example, if a player is very active, the

virtual characters will come over to interactive with

the player interactively; otherwise, the virtual

characters might act by themselves with fewer

interactions with the player.

Adapt virtual environment in the system to players’

preferences in real-time by guiding the players in the

virtual environment with visual clues or tuning the

virtual environments

A big challenge to play inside the virtual

environment is that, the players might get lost more

easily than inside the conventional media. Therefore,

the player might need some guidance in the CAVE

depending on the system scenario. Thus, i-CAVE

needs to provide virtual clues to guide the players

after learning the players’ requirements semantically

in real-time. Moreover, based on the players’

requirements, the virtual environment should adapt

accordingly. For example, if a player is bored with

the current complex virtual world, a simplified virtual

world should be presented to attract the player.

A. Architeture

Cave Automatic Virtual Environment (CAVE) is a virtual

reality environment with immersive 3D experience and

engaging interactions, which would be a very good place of

entertainment, training or exercise for elderly. However,

CAVE system doesn’t provide semantic understanding of

players’ behaviors and emotions in real-time, which leads to a

challenge for application developers to create a personalized

virtual environment for various kinds of players. In this paper

we proposed an intelligent agent-augmented CAVE system,

namely i-CAVE, to bridge the gap between application

developers and end players.

Agents are goal-oriented, autonomous, reactive, proactive

objects with social communications, which are used widely in

intelligent systems. i-CAVE means intelligent agent-

augmented CAVE system. As shown in Figure 1, i-CAVE

overlays a semantic multi-agent system layer above the

current CAVE system with low level data collection agent

layer. The agents are able to handle the high-level semantic

analysis of user interactions and adapt the virtual characters

and virtual environments in real-time, which are: 1) Agent

Manager: it is the main coordinator to control all the agents in

i-CAVE, which controls the running process of all other

agents with scheduling and synchronization; 2) Semantic

Learning Agent: the agent gathers data from the low level data

collection agents, learns from the data to deduce the players’

real-time semantic statuses, and then guides the environment

agent and character adaptive agents to make the adaptations;

3) Environment Agent: the agent gathers the player

information inside the CAVE virtual environment, and

provides useful guiding information to the player. Moreover, it

adapts the immersive virtual environment to the players’

preferences; and 4) Character Adaptive Agent: the agent is

used to model the non-player characters (NPCs) inside the

virtual environment that will interact with the players. The

agents will adapt their behaviors and emotions inside the

virtual environment according to the analysis of the semantic

analysis agent.

Page 3: i- CAVE: Intelligent Agent-Augmented CAVE System CAVE.pdf · i- CAVE: Intelligent Agent-Augmented CAVE ... intelligent agent-augmented CAVE system, namely i-CAVE, ... Mu l t i -a

Semantic Multi-agent System

CAVE

Application Layer

i-CA

VE

Low Level Data Collection Agent Layer

Agent Manager

Semantic Learning

Agent

Action Detection

Agent

Emotion Detection

Agent

Speech

Recognition Agent

Environment

Agent

Semantic Multi-agent

system

Character

Adaptive Agent

Figure 1 Semantic Multi-Agent System in i-CAVE

In order to interact with the virtual environment more

smoothly, we adopt the gesture detection technology of Kinect

by Microsoft. For example, the player can walk in the virtual

world with “step forward”, “step backward”, and “rotate”

freely with normal gestures. Currently, there are different

gesture detection protocols for Kinect available for

researchers. Based on our prior R&D experience of Kinect-

based products, we have developed a standard protocol for our

case study that could result with natural interactions, which

might be used as a guideline for other similar game

development in CAVE system.

B. Design Considerations for Elderly

CAVE provides a very useful environment for elderly’s

entertainment, training, rehabilitation etc. Different from

normal adult players, there are several typical features for

elderly to be considered in the research and development:

1. Motor limitation

Elderly don’t preserve fast speed, strong power etc.

This prevents them from hard exercises or complex

tasks.

2. Mental limitation

They might not have good memory and might be

weak mentally. They are not able to memorize and

induce very quickly.

Therefore, it would be helpful to make the virtual reality more

immersive and interactions more engaging.

In our approach, elderly can be better served in the 3D virtual

environment to cope with the listed limitations by using the

personalized agent.

IV. IMPLEMENTATION

The application is developed with EON Studio on PC, and

then converted to work on CAVE system.

EON Studio provides commonly used functions as nodes and

prototypes, and allows developers to create new nodes or

prototypes. The world is populated by importing assets or

dragging nodes to the simulation tree as a node object. Each

node object contains some fields or events. For general

developers, they can implement application logic by linking

input and output events between objects to change their

behavior. For advanced users, they can program in VBScripts

or JScripts. We used both of them to take the advantages of

easy implementation and sophisticated control. For example,

keyboard events are implemented with routes system, while

the animation and movement of Meilong (dinosaur) is

controlled by JScripts, as shown in Figure 2.

Figure 2. Visual Programming on Dinosaur Behaviors at

Different User Interactions

“Walking with dinosaurs” is a very famous documentary

television mini-series produced by BBC. However, the

audiences are not able to interact with the dinosaurs and the

world. In order to verify the proposed system, we have

implemented “walking with dinosaurs” in the real CAVE

system. Figure 3 shows the screenshot of the dinosaur world in

Page 4: i- CAVE: Intelligent Agent-Augmented CAVE System CAVE.pdf · i- CAVE: Intelligent Agent-Augmented CAVE ... intelligent agent-augmented CAVE system, namely i-CAVE, ... Mu l t i -a

EON studio. Three types of dinosaurs are involved, which are

meilong, jinzhousaurus and T-Rex, which have different kinds

of behaviours. At clicking a dinosaur, the information will be

shown at the left side of the screen, as shown in Figure 4.

Within the game, the players are able to move around

freely in the virtual world and interact with dinosaurs face-to-

face through behaviors (e.g. clapping hands, threatening). The

interviewed players gave very positive comments on the

immersive environment and engaging interactions.

Figure 3. A screenshot of multiple dinosaurs modelled with

intelligent agents.

Figure 4. A screenshot of Showing Dinosaur Information

Figure 5 shows the game in the real CAVE system. The

detailed demonstration in EON CAVE system can be found at

here (http://www.youtube.com/watch?v=HOeAaXpiy_o).

Figure 5. Walking with Dinosaurs in the Virtual Environment

(Real gameplay in IMI immersive room)

V. EVALUATION

As the current prototype is still on the beta stage, we have

conducted the interviews on 5 selected users. The evaluation is

based on two experiences of user: immersion and engagement.

Immersion reflects the level that the user feels presence in the

virtual reality with minimum of disbelief, while engagement

reflects the evaluation of user interaction in terms of

complexity, freedom and fun. Each evaluation metric is

measures from 1(least) to 5(most).

Figure 6. Evaluation based on Immersion and Engagement

From the feedback, we can see the current i-CAVE reaches

high on the “immersion” and “engaging interactions” with

limited “complexity”, which would be helpful for the elderly

in the play.

VI. CONCLUSION

In this paper, we have proposed an intelligent agent-

augmented CAVE system, namely i-CAVE, to bridge the gap

between application developers and end players, especially the

elderly which has the special requirements on interactions. We

have implemented the system with EON Studio and delivered

Page 5: i- CAVE: Intelligent Agent-Augmented CAVE System CAVE.pdf · i- CAVE: Intelligent Agent-Augmented CAVE ... intelligent agent-augmented CAVE system, namely i-CAVE, ... Mu l t i -a

into a Live CAVE environment with the “walking with

dinosaurs” case study. Moreover, we have developed a

template of interaction protocols in CAVE to facilitate the

interactions for elderly. The interviewed users gave very

positive comments on the user experiences of immersion and

engagements.

ACKNOWLEDGMENT

We would like to acknowledge the funding support by

Institute of Media Innovation, Nanyang Technological

University, Singapore.

REFERENCES

[1] Carolina Cruz-Neira , Daniel J. Sandin , Thomas A. DeFanti, Surround-screen projection-based virtual reality: the design and implementation of the CAVE, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.135-142, September 1993

[2] Zhiqi Shen , Chunyan Miao , Robert Gay , Dongtao Li, Goal-Oriented Methodology for Agent System Development, IEICE - Transactions on Information and Systems, v.E89-D n.4, p.1413-1420, April 2006

[3] Yuan Miao, ChunYan Miao;, XueHong Tao, ZhiQi Shen, ZhiQiang Liu, “Transformation of Cognitive Maps”, IEEE Transactions on Fuzzy Systems (SCI impact factor:3.624), Volume: 18 , Issue: 1, Page(s): 114 – 124, 2010.

[4] Yundong Cai, Chunyan Miao, Ah‐ Hwee Tan and Zhiqi Shen, “Creating an Immersive Game World with Evolutionary Fuzzy Cognitive Maps", IEEE Computer Graphics & Applications (CG & A), vol. 29, Issue 6, 2009.

[5] Song Hengjie, Miao Chunyan, Shen Zhiqi “Implementation of Fuzzy Cognitive Maps based on Fuzzy Neural Network and Application in Prediction of Time Series,” IEEE Trans. Fuzzy Systems (SCI impact factor:3.624), vol. 18, no. 02, pp. 233‐ 250, 2010.

[6] Michael Wooldridge and Nicholas R. Jennings. Intelligent agents: Theory and practice. Knowledge Engineering Review, 10(2):115-152, 1995.

[7] Ma, M. T. A., Nanyang Technological University. (2011, Dec 15). Infrastructure. Retrieved 28 March 2012, from: http://imi.ntu.edu.sg/IMIResearch/Pages/Infrastructure.aspx