Click here to load reader
Upload
docong
View
213
Download
0
Embed Size (px)
Citation preview
i- CAVE: Intelligent Agent-Augmented CAVE
System
Yundong Cai
School of Computer Engineering
Nanyang Technological University
Singapore
Zhiqi Shen
School of Computer Engineering
Nanyang Technological University
Singapore
Bai Xiaoguang
School of Computer Engineering
Nanyang Technological University
Singapore
Abstract— Cave Automatic Virtual Environment (CAVE) is a
virtual reality environment with immersive 3D experience and
engaging interactions, which would be a very good medium of
entertainment, training or exercise for elderly. However, the raw
CAVE system doesn’t provide semantic understanding of players’
behaviors and emotions in real-time, which leads to a challenge for
application developers to create a personalized virtual environment
for various players. In this paper we proposed an intelligent agent-
augmented CAVE system, namely i-CAVE, to bridge the gap
between application developers and end players, especially the
elderly which has the special requirements of interactions. We have
implemented the system with EON Studio in a Live CAVE
environment and a case study called “Walking with Dinosaurs”.
The interviewed players give very positive feedback on the
immersion and engaging interactions.
Keywords- CAVE, semantic information, agent
I. INTRODUCTION
Cave Automatic Virtual Environment (CAVE) is a virtual
reality environment with immersive 3D experience and
engaging interactions, which would be a very good place of
entertainment, training or exercise for elderly. However,
CAVE system doesn’t provide semantic understanding of
players’ behaviors and emotions in real-time, which leads to a
challenge for application developers to create a personalized
virtual environment for various kinds of players. In this paper
we proposed an intelligent agent-augmented CAVE system,
namely i-CAVE, to bridge the gap between application
developers and end players. With the augmentation of
intelligent agents, we are able to achieve good immersion and
engaging interactions for the players.
II. BACKGROUND
Interactive digital media (IDM) attracts a lot of research and
industry today to education and entertainment. Cave
Automatic Virtual Environment (CAVE) is one of the world’s
leading 3D immersive virtual environments, which is used by
a lot of researchers to develop interactive and immersive
virtual worlds. Institute of Media Innovation (NTU) has setup
one CAVE system and launched immersive virtual
applications successfully. Currently, the research efforts on the
CAVE system mainly focuses on human computer interaction
(HCI), i.e. virtual environment projection on the walls, gesture
and motion recognition etc., and the virtual world application
developments [1].
However, it is a great challenge for novice developers to
create a CAVE application with EON studio by EON Reality.
Even more, how to naturally immerse players into the
interactive environment is one of the most significant
challenges for human computer interaction. Though the
current CAVE improves immersion compared to traditional
virtual worlds in computers, it still lacks the capabilities to
understand the players’ behaviors in real-time in order to
provide a personalized player experience. By learning the
player’s behaviors or emotions in CAVE, we will be able to
provide a personalized virtual world in order to cater to the
player’s preference.
Nowadays, artificial intelligence technologies such as personal
mobile agents and other soft computing techniques have
become more and more mature, which makes it possible to
achieve the virtual world personalization in the CAVE system.
Currently CAVE only provides the low-level data collections
of motions and gestures, but there is a lack of semantic
learning of the recorded data and adapting of the virtual world
and virtual characters inside the virtual world. Agents are
goal-oriented, autonomous, reactive and proactive objects with
social communications, which can be used to handle the
semantic analysis in order to advance the immersion in the
CAVE.
The CAVE system was built in the immersive room at
Institute for Media Innovation (IMI) in 2011. It’s an
immersive virtual reality environment with rich interactions
with users. Besides the traditional input devices like keyboard,
mouse, gamepad and microphone, it makes use of latest
hardware including infrared emitters, position trackers, gesture
gloves, kinect and so on. Electromagnetic 6DOF tracking is
implemented to monitor the viewers' position and orientation.
Scenes data and users’ action are processed by graphics
workstations linked together in a local area network [7]. The
3D dynamic images are sent by 5 high-end projectors mounted
on the ceiling, reflected by mirrors and projected to the curved
screen which covers 330 degrees of view horizontally.
Stereoscopic audio sound system is also integrated into the
system. With the stereoscopic active lightweight glasses, users
can be fully immersed in the virtual world.
III. PROPOSED I-CAVE
The objective of our research project is to develop an
intelligent agent-augmented CAVE system, namely i-CAVE,
which overlays a semantic agent layer on top of the current
CAVE system to achieve high level semantic analysis and
virtual world adaptation. Triple I system means “Interaction,
Immersive and Innovation”, we aims to add an extra “I” which
is “Intelligence” to it. Other CAVE researchers will be able to
benefit from the i-CAVE system for they own research and
development.
The intelligent agents in i-CAVE should have the following
capabilities:
Semantically analyze the players’ statuses (mood,
emotion, energy etc.) through their behaviors,
gestures or facial expressions inside CAVE
The player’ statuses (i.e. mood, emotion, energy etc.)
are tracked in the CAVE for adaptive human
computer interaction. By learning from the collected
data with data mining techniques, we will be able to
deduce whether the player is happy or sad, energetic
or lethargic, which would be a basis to provide
him/her a personalized virtual environment.
Adapt virtual characters in the system to the players’
preferences in real-time so that the players can enjoy
a more personalized immersive experience
Based on the player’ statuses (mood, emotion, energy
etc.), i-CAVE system could adjust the virtual
characters’ behaviors and emotions inside the
environment seamlessly in order to serve the players’
needs. For example, if a player is very active, the
virtual characters will come over to interactive with
the player interactively; otherwise, the virtual
characters might act by themselves with fewer
interactions with the player.
Adapt virtual environment in the system to players’
preferences in real-time by guiding the players in the
virtual environment with visual clues or tuning the
virtual environments
A big challenge to play inside the virtual
environment is that, the players might get lost more
easily than inside the conventional media. Therefore,
the player might need some guidance in the CAVE
depending on the system scenario. Thus, i-CAVE
needs to provide virtual clues to guide the players
after learning the players’ requirements semantically
in real-time. Moreover, based on the players’
requirements, the virtual environment should adapt
accordingly. For example, if a player is bored with
the current complex virtual world, a simplified virtual
world should be presented to attract the player.
A. Architeture
Cave Automatic Virtual Environment (CAVE) is a virtual
reality environment with immersive 3D experience and
engaging interactions, which would be a very good place of
entertainment, training or exercise for elderly. However,
CAVE system doesn’t provide semantic understanding of
players’ behaviors and emotions in real-time, which leads to a
challenge for application developers to create a personalized
virtual environment for various kinds of players. In this paper
we proposed an intelligent agent-augmented CAVE system,
namely i-CAVE, to bridge the gap between application
developers and end players.
Agents are goal-oriented, autonomous, reactive, proactive
objects with social communications, which are used widely in
intelligent systems. i-CAVE means intelligent agent-
augmented CAVE system. As shown in Figure 1, i-CAVE
overlays a semantic multi-agent system layer above the
current CAVE system with low level data collection agent
layer. The agents are able to handle the high-level semantic
analysis of user interactions and adapt the virtual characters
and virtual environments in real-time, which are: 1) Agent
Manager: it is the main coordinator to control all the agents in
i-CAVE, which controls the running process of all other
agents with scheduling and synchronization; 2) Semantic
Learning Agent: the agent gathers data from the low level data
collection agents, learns from the data to deduce the players’
real-time semantic statuses, and then guides the environment
agent and character adaptive agents to make the adaptations;
3) Environment Agent: the agent gathers the player
information inside the CAVE virtual environment, and
provides useful guiding information to the player. Moreover, it
adapts the immersive virtual environment to the players’
preferences; and 4) Character Adaptive Agent: the agent is
used to model the non-player characters (NPCs) inside the
virtual environment that will interact with the players. The
agents will adapt their behaviors and emotions inside the
virtual environment according to the analysis of the semantic
analysis agent.
Semantic Multi-agent System
CAVE
Application Layer
i-CA
VE
Low Level Data Collection Agent Layer
Agent Manager
Semantic Learning
Agent
Action Detection
Agent
Emotion Detection
Agent
Speech
Recognition Agent
Environment
Agent
Semantic Multi-agent
system
Character
Adaptive Agent
Figure 1 Semantic Multi-Agent System in i-CAVE
In order to interact with the virtual environment more
smoothly, we adopt the gesture detection technology of Kinect
by Microsoft. For example, the player can walk in the virtual
world with “step forward”, “step backward”, and “rotate”
freely with normal gestures. Currently, there are different
gesture detection protocols for Kinect available for
researchers. Based on our prior R&D experience of Kinect-
based products, we have developed a standard protocol for our
case study that could result with natural interactions, which
might be used as a guideline for other similar game
development in CAVE system.
B. Design Considerations for Elderly
CAVE provides a very useful environment for elderly’s
entertainment, training, rehabilitation etc. Different from
normal adult players, there are several typical features for
elderly to be considered in the research and development:
1. Motor limitation
Elderly don’t preserve fast speed, strong power etc.
This prevents them from hard exercises or complex
tasks.
2. Mental limitation
They might not have good memory and might be
weak mentally. They are not able to memorize and
induce very quickly.
Therefore, it would be helpful to make the virtual reality more
immersive and interactions more engaging.
In our approach, elderly can be better served in the 3D virtual
environment to cope with the listed limitations by using the
personalized agent.
IV. IMPLEMENTATION
The application is developed with EON Studio on PC, and
then converted to work on CAVE system.
EON Studio provides commonly used functions as nodes and
prototypes, and allows developers to create new nodes or
prototypes. The world is populated by importing assets or
dragging nodes to the simulation tree as a node object. Each
node object contains some fields or events. For general
developers, they can implement application logic by linking
input and output events between objects to change their
behavior. For advanced users, they can program in VBScripts
or JScripts. We used both of them to take the advantages of
easy implementation and sophisticated control. For example,
keyboard events are implemented with routes system, while
the animation and movement of Meilong (dinosaur) is
controlled by JScripts, as shown in Figure 2.
Figure 2. Visual Programming on Dinosaur Behaviors at
Different User Interactions
“Walking with dinosaurs” is a very famous documentary
television mini-series produced by BBC. However, the
audiences are not able to interact with the dinosaurs and the
world. In order to verify the proposed system, we have
implemented “walking with dinosaurs” in the real CAVE
system. Figure 3 shows the screenshot of the dinosaur world in
EON studio. Three types of dinosaurs are involved, which are
meilong, jinzhousaurus and T-Rex, which have different kinds
of behaviours. At clicking a dinosaur, the information will be
shown at the left side of the screen, as shown in Figure 4.
Within the game, the players are able to move around
freely in the virtual world and interact with dinosaurs face-to-
face through behaviors (e.g. clapping hands, threatening). The
interviewed players gave very positive comments on the
immersive environment and engaging interactions.
Figure 3. A screenshot of multiple dinosaurs modelled with
intelligent agents.
Figure 4. A screenshot of Showing Dinosaur Information
Figure 5 shows the game in the real CAVE system. The
detailed demonstration in EON CAVE system can be found at
here (http://www.youtube.com/watch?v=HOeAaXpiy_o).
Figure 5. Walking with Dinosaurs in the Virtual Environment
(Real gameplay in IMI immersive room)
V. EVALUATION
As the current prototype is still on the beta stage, we have
conducted the interviews on 5 selected users. The evaluation is
based on two experiences of user: immersion and engagement.
Immersion reflects the level that the user feels presence in the
virtual reality with minimum of disbelief, while engagement
reflects the evaluation of user interaction in terms of
complexity, freedom and fun. Each evaluation metric is
measures from 1(least) to 5(most).
Figure 6. Evaluation based on Immersion and Engagement
From the feedback, we can see the current i-CAVE reaches
high on the “immersion” and “engaging interactions” with
limited “complexity”, which would be helpful for the elderly
in the play.
VI. CONCLUSION
In this paper, we have proposed an intelligent agent-
augmented CAVE system, namely i-CAVE, to bridge the gap
between application developers and end players, especially the
elderly which has the special requirements on interactions. We
have implemented the system with EON Studio and delivered
into a Live CAVE environment with the “walking with
dinosaurs” case study. Moreover, we have developed a
template of interaction protocols in CAVE to facilitate the
interactions for elderly. The interviewed users gave very
positive comments on the user experiences of immersion and
engagements.
ACKNOWLEDGMENT
We would like to acknowledge the funding support by
Institute of Media Innovation, Nanyang Technological
University, Singapore.
REFERENCES
[1] Carolina Cruz-Neira , Daniel J. Sandin , Thomas A. DeFanti, Surround-screen projection-based virtual reality: the design and implementation of the CAVE, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.135-142, September 1993
[2] Zhiqi Shen , Chunyan Miao , Robert Gay , Dongtao Li, Goal-Oriented Methodology for Agent System Development, IEICE - Transactions on Information and Systems, v.E89-D n.4, p.1413-1420, April 2006
[3] Yuan Miao, ChunYan Miao;, XueHong Tao, ZhiQi Shen, ZhiQiang Liu, “Transformation of Cognitive Maps”, IEEE Transactions on Fuzzy Systems (SCI impact factor:3.624), Volume: 18 , Issue: 1, Page(s): 114 – 124, 2010.
[4] Yundong Cai, Chunyan Miao, Ah‐ Hwee Tan and Zhiqi Shen, “Creating an Immersive Game World with Evolutionary Fuzzy Cognitive Maps", IEEE Computer Graphics & Applications (CG & A), vol. 29, Issue 6, 2009.
[5] Song Hengjie, Miao Chunyan, Shen Zhiqi “Implementation of Fuzzy Cognitive Maps based on Fuzzy Neural Network and Application in Prediction of Time Series,” IEEE Trans. Fuzzy Systems (SCI impact factor:3.624), vol. 18, no. 02, pp. 233‐ 250, 2010.
[6] Michael Wooldridge and Nicholas R. Jennings. Intelligent agents: Theory and practice. Knowledge Engineering Review, 10(2):115-152, 1995.
[7] Ma, M. T. A., Nanyang Technological University. (2011, Dec 15). Infrastructure. Retrieved 28 March 2012, from: http://imi.ntu.edu.sg/IMIResearch/Pages/Infrastructure.aspx