0018-9162/05/$20.00 © 2005 IEEE September 2005 25

P E R S P E C T I V E S

P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y

From VisualSimulation toVirtual Realityto Games

During the past two decades, the virtual reality community hasbased its development on a synthesis of earlier work in inter-active 3D graphics, user interfaces, and visual simulation.1

Doing so let developers create a more open technology thanthe visual simulation community could, increased the number

of people working in 3D, and developed a science, technology, and lan-guage considerably beyond that of the earlier field.

Beginning in 1997 with the publication of an NRC report titled “Modelingand Simulation—Linking Entertainment and Defense,”2 the video game com-munity has pushed into spaces previously the domain of the VR community.Clearly, the VR field is transitioning into work influenced by video gamesand thus now influences that industry as well. Because much of the researchand development being conducted in the games community parallels the VRcommunity’s efforts, it has the potential to affect a greater audience.

Given these trends, VR researchers who want their work to remain rele-vant must realign to focus on game research and development. Research inthe games arena affects not just the entertainment industry but also the gov-ernment and corporate organizations that could benefit from the training,simulation, and education opportunities that serious games provide.

DEFINING SERIOUS GAMESPeople respond differently to the emotionally charged term game depend-

ing on whether they played or did not play video games while growing up.This is basically a generation-gap issue because children who have grownup since the 1980s have been exposed to video games their entire lives.

Before we can seriously tackle the issue of what a games research agendamight be, we must define what the term means. Dictionaries tend to definea game as a physical or mental contest, played according to specific rules,with the goal of amusing or rewarding the participants. When seeking a def-inition of the more specific term video game, we are likely to encounter adescription such as “a game played against a computer,” which would moreaccurately be worded as “a game played with a computer.” To fully fleshout this definition, we might propose the following: “Video game: a mentalcontest, played with a computer according to certain rules for amusement,recreation, or winning a stake.”

Leveraging technologyfrom the visual simulation and virtualreality communities,serious games provide a delivery system fororganizational videogame instruction andtraining.

Michael ZydaUSC Information Sciences Institute

26 Computer

Developing a science of games opens up a hugepotential for the wider application of games in gov-ernmental and corporate arenas. The formal defi-nition might read as follows: “Serious game: amental contest, played with a computer in accor-dance with specific rules, that uses entertainmentto further government or corporate training, edu-cation, health, public policy, and strategic commu-nication objectives.”

Bing Gordon, chief creative officer of video andcomputer games developer Electronic Arts, oncetold me that he defines video games as “story, art,and software.”

When designing a game, the development teamblends these elements into a finished product. Thedesign team crafts the story, which provides thegame’s entertainment component. The art teamprovides the game’s look and feel. The program-ming team develops the code that implements storyrequirements, interface features, networking, Webconnectivity, scoring systems, AI scripting, gameengine changes, and just about anything technicalor programmatic that the entire development effortrequires.3

Serious games have more than just story, art, andsoftware, however. As Figure 1 shows, they involvepedagogy: activities that educate or instruct,thereby imparting knowledge or skill. This addi-tion makes games serious. Pedagogy must, how-ever, be subordinate to story—the entertainmentcomponent comes first. Once it’s worked out, thepedagogy follows.

A human performance engineering team worksclosely with the design team to oversee this peda-gogy insertion. The team’s lead is part instructional

scientist and part subject-matter expert for thedomain around which the teams are building theserious game.

Building serious games takes more than simplyhanding their development to a traditional gameteam, however. The team must interact with theinstructional scientists and subject matter expertsthat comprise the human performance engineeringteam. To thrive, this new organization must have auniversity’s facilities and support or similar researchorganization. A research agenda that supports seri-ous games also benefits the entertainment industry,one of the US’s largest economic sectors.

CREATING A SCIENCE OF GAMESThe development and wide release of the

America’s Army game began a revolution in think-ing about the potential role of video games fornonentertainment domains. It also sparked a dis-cussion about how to advance game technology’sstate of the art to support future entertainment andserious games.4 This, in turn, promoted interest increating a science of games and an allied educa-tional program. Experiences with digital-gamenatives—those who have grown up playinggames—indicated that a game-centered researchand educational program could offer many posi-tive benefits. While much speculation regardingthese benefits is anecdotal, substantive evidenceshows that game experiences affect digital-gamenatives positively. If researchers construct and per-form their studies carefully, they may be able to har-ness these positive effects for societal gain.

The announcement of America’s Army, shownin Figure 2, at the 2002 Electronics Entertainment

Figure 1. From gameto serious game.Unlike their enter-tainment-only coun-terparts, seriousgames use pedagogyto infuse instructioninto the game playexperience.

Pedagogysubordinate

to story

Closeworking

relationship

Pedagogy

Human performanceengineering team

Designteam

Art

Game

Artteam

Software

Programmingteam

GameSeriousgame

Story

Expo prompted the US Army to commission astudy of the game to see if it could be used for train-ing. In the summer of 2002, a US Army captainfrom Fort Benning, who was tasked with spendingabout a week playing the game, reported that hewas unimpressed. Although he deemed the game agood recruiting tool, he indicated that there “wasinsufficient fidelity in the game for it to be of anyuse in training.”

Cyberpunk author William Gibson once notedthat the street finds its own uses for things.5 We sawthis effect in action in October 2002, when a staffsergeant from Fort Benning approached theAmerica’s Army booth at the Army’s annual AUSAConference in Washington, D.C. An enthusiasticgamer who had played America’s Army from dayone, he told the staff, “We love this game atBenning. We use it for training.”

The sergeant had bypassed the Army’s require-ments documents and formal studies and deployedthe game on his own initiative. He went on toexplain that when new recruits had trouble withthe rifle range or the obstacle course, his team hadthose recruits play America’s Army and requiredthem to complete those levels in the game, as shownin Figure 3. When the recruits did so, they thenwent back to the range and usually passed the rangetests. This made us wonder what other applicationswould benefit from game-based instruction—espe-cially after we developed a revised version ofAmerica’s Army with new levels for a variety ofDepartment of Defense, Army, and Secret Servicetraining needs.

Six to nine months after its release, motherswould meet me and complain that “my son is play-ing America’s Army five to six hours a day, sevendays a week. What is going to become of him?” Iwould usually answer that these children would betwice as likely to consider a career in the US Armyas those who didn’t play the game, something theArmy counts on with respect to the game’s recruit-ing mission. When I asked the mothers if their chil-dren knew a lot about the US Army, the mothersusually responded that “they know everythingabout the Army, having learned it from the game.Wouldn’t it be nice if playing games could teachthem something more useful?”

These comments led us to wonder how much ofK-12 science and math education could be taughtvia games and how we might exploit students’capability for collateral learning—the learning thathappens by some mechanism other than formalteaching. Ultimately, we wondered if we couldincorporate all K-12 science and math education

in a highly immersive, highly addictive game—wecalled this our “first-person education” grand chal-lenge, a play on the phrase “first-person shooter.”6

Given the many anecdotes about digital-gamenatives being better surgeons and displaying excep-tional business skills, it became clear that creatinga science of games—a scientific and engineeringmethod for building them and understanding andanalyzing game play—had become essential.

September 2005 27

Figure 3. Training simulator. Despite the initial evaluator’s skepticism, America’sArmy proved to be an effective military training simulator. Soldiers who playedthe rifle range segment of the game, for example, earned improved scores on the real-life rifle range.

Figure 2. America’s Army. The most widely used and successful serious game todate, this title initially served as a recruiting tool.

28 Computer

GAME PRODUCTION CHALLENGESToday, game production poses some

incredible challenges. Teams come togetherto build a single game, then dissolve. Trainingtimes on game engine toolsets increasesteadily. Code modules, crafted for specificgames, offer less than 30 percent reuse in sub-sequent efforts. First-person shooters are cur-rently the only games that have reusableengines. These engines are proprietary, how-ever, used for a few games only, licensed atexorbitant expense, cost more than $6 mil-lion, and take more than a year to develop.Game production times increase steadily asgraphics cards provide more visual capabil-

ities and the game-playing public demands moreinteractivity and verisimilitude.

The demand for better computer characters andstory increases with the complexity of visual dis-plays and with the release of each new, more com-plex-than-ever game. Game play innovation isbecoming a competitive necessity. The big hits havebeen sports, first-person shooters, adventure, andSims-type games. For the game industry to continueto grow, additional genres must become moresophisticated, with better backstories and thor-oughly researched, developed, and deployed foun-dational game-play technologies.

The game-playing public is becoming ever moreenamored of portable entertainment platforms, andcomputing speeds are becoming ever faster. Thesetrends pose the critical challenge of providing well-designed interfaces, ever-increasing storage capac-ity, and expanded wireless network bandwidth.Efficient streaming of content to wireless portabledevices and for game downloads from the Web willalso become priorities.

GAMES RESEARCH AGENDATo influence the future of both serious and enter-

tainment games, developers must create a researchand development agenda that transforms the gameproduction process from a handcrafted, labor-intensive effort to one with shorter, more pre-dictable production timelines that still manages toprovide innovations and increased complexity. Thisresearch agenda has three components: the infra-structure, cognitive game design, and immersion.

InfrastructureThe underlying software and hardware necessary

for developing interactive games include massivelymultiplayer online game architectures, gameengines and tools, streaming media, next-genera-

tion consoles, and wireless and mobile devices.Many application domains use massively multi-

player online game architectures, including the mil-itary, security and homeland defense, and onlineeducation. MMOGs pose the fundamental researchquestion of how to develop dynamically extensibleand semantically interoperable software architec-tures. This functionality requires building game orsimulation clients that can connect to a runningMMOG, download the appropriate code for dis-play and interaction, then operate with other onlineplayers. This work, of interest to gaming in gen-eral, has special relevance for the large govern-mental game-based simulation sector.

Currently, only static solutions that dramaticallydrive up the cost of large-scale simulation and gam-ing have been developed—no dynamic solutionsare available. Developers must solve the MMOGarchitecture problem not just for game clients butalso for large-scale computational architecturessuch as grid computing. Game engines and toolswill be vital to researchers bent on attacking thelack-of-reuse problem in gaming. They can alsoprovide the technology for moving games fromcrafted systems built by the game industry gnomesto engineered systems used widely in the govern-ment and corporate sectors.

The America’s Army project has made someattempts to broaden first-person-shooter gameengine reuse to other domains, but all such useshave suffered from major limitations.3 For example,many game engines lack support for large terrainboxes and can only handle a 1 km × 1 km area eventhough most real-world applications require muchlarger spaces. Other limitations include onerousand expensive game-engine licenses and the gen-eral unavailability of these engines for R&D andto the serious games community at large.

Thus, developers need an open source gameengine that includes a development toolset aswidely available and utilized as Linux. With anopen source engine, developers could explore addi-tional capabilities, including a larger terrain box,dynamic terrain, physical modeling, and otherrequirements that the entertainment world ignores.In addition, an open source engine would make fea-sible exploring many other directions, including themodeling and simulation of computer characters,story, and human emotion.

Streaming media will play a prominent role inthe delivery of dynamic content to PC-based gamesand mobile devices. This will gain more importancein games as computing becomes smaller, faster, andmore capable. In turn, these developments will

America’s Army hasmade some attempts

to broaden first-person-shooter

game engine reuseto other domains,but all such uses

have suffered frommajor limitations.

encourage research into the proper interfaces forsuch devices and in determining how to best deploythem for serious and entertainment purposes.

Cognitive game designTaking a cognitive approach to game develop-

ment will give developers the tools to create theo-ries and methods for

• modeling and simulating computer characters,story, and human emotion;

• analyzing large-scale game play;• innovating new game genres and play styles;

and• integrating pedagogy with story in the inter-

active game medium.

Computer-generated autonomy involves model-ing human and organizational behavior in net-worked games, for example, deploying thetechnology from a game like The Sims for a seri-ous purpose, such as a training aid for nursing.Developers can use this approach to model andsimulate hospital operations in game form, pro-viding an immersive experience for the nursetrainee.

Creating a compelling computer-generated storyhas long been a game development challenge. Toovercome this challenge, developers must compu-tationally model story by deploying engines andtool suites that dramatically simplify the construc-tion of networked game storylines.

The modeling and simulation of human emotionlies at the frontier of networked games and simu-lations. For the entertainment world, the future ofgaming includes developing an experience soimmersive that it engages the players’ emotions ona visceral level. The military, homeland security,defense, and hospital trauma sectors need a similargame-based simulation capability that spans thespectrum of entertainment and serious game devel-opers. This capability must be thoroughlyresearched to determine in advance its potentialhuman impact.

Understanding and analysis will play a key rolein the games research agenda. When researchersplace humans in large-scale MMOGs or single-player modules, they must determine what happensduring game play and how the experience affectsthe players. Current serious game usage and large-scale simulation require human monitors to watchnetworked play. When the game ends, the humanmonitor tells the researchers which team won andwhy.

To refine this process, developers mustacquire an automated understanding andanalysis capability that can generate a high-level report of what happened during gameplay over a specified period, from a particu-lar viewpoint, with the option to query thesystem for additional detailed information.Several defense, homeland security, and edu-cational applications require such automatedanalyses if gaming is to make meaningful con-tributions to the serious game domain. Inaddition, the entertainment industry mightfind such analyses useful as a marketing andgame-refinement tool.

Pedagogy and story integration involvedetermining theories and developing prac-tices for inserting learning opportunities into story,such that participants find the story immersive andentertaining because the embedded instructionremains subordinate to it. The game industry hasalready witnessed the failure of edutainment, anawkward combination of educational softwarelightly sprinkled with game-like interfaces and cutedialog. This failure shows that story must comefirst and that research must focus on combininginstruction with story creation and the game devel-opment process.

ImmersionCreating technologies that engage the game

player’s mind via sensory stimulation and provid-ing methods for increasing the sense of presencecontribute to building a feeling of immersion. Thiswork includes

• computer graphics, sound, and haptics;• affective computing—sensing human state and

emotion; and• advanced user interfaces.

Sensory channel research plays a fundamentalrole in games technology. As they develop morecapable graphics engines, researchers need to knowhow to appropriately use that new capability forserious games as well as how to generate new tech-nology that industry can put into the next-genera-tion graphics chipsets it provides.

Spatial and immersive sound are key componentsfor whatever training and educational systemsresearchers build with gaming. Developers mustimplement future engineering requirements andhuman-performance engineering to ensure that theycan employ sound appropriately and effectivelywhile minimizing cross-modal sensory conflicts.

September 2005 29

Defense, homelandsecurity, and educational

applications requireautomated analysesfor gaming to make

meaningful contributions to the

serious gamedomain.

30 Computer

Haptics, another key research area, takes gamesinto the tactile realm. Considering that the R&Dwork they are doing now might be implemented intechnology the game industry deploys in the next10 to 20 years, developers will need to work dili-gently to continually improve sensory stimulation.

Affective computing measures a person’s physi-cal and emotional state. In the next two years, low-cost sensors will become available that measure aplayer’s emotional state and provide this informa-tion as input when running a game. Software devel-opment kits that read those devices and transmitthe player’s emotional state to the game will needto be developed. In turn, the game must be able touse that state along with many other inputs andrespond appropriately. Although this capability willhave a major impact, at this point developers donot really know how this process will work, partlybecause they lack good models of human emotionsand of how computer characters should react tothem.

Game designers must understand these things ifthey are to engineer and implement these capabil-ities so that these features behave predictably andreliably. This type of research effort could broadenthe scope of both entertainment and serious games.Ultimately, a video game may not only make us cry,

it may be aware that we are crying and respondappropriately.

Developers use presence to measure the immer-sive experience of a game player or virtual realityexplorer. Whether building a virtual reality envi-ronment or a game, developers attempt to providethe illusion that they have entered a virtual world.Ultimately, developers must be able to engineerpresence so reliably and convincingly that gamemakers can author such worlds a priori for theimmersive experience rather than just hoping thedeveloped world will be convincingly engaging.

Advanced user interfaces will become key ascomputing moves from the standard desktop PCto mobile platforms. Much can be gained bystudying how the game industry has developedalmost universal interfaces that let gamers transi-tion seamlessly from, for example, playing Quaketo playing Unreal Tournament. To make signifi-cant progress in deploying serious games, devel-opers must understand interfaces from the gameperspective.

SERIOUS GAMESApplying games and simulations technology to

nonentertainment domains results in serious games.As Figure 4 shows, this work includes

Figure 4. Broadapplicationspectrum. Serious-game methodologycan be applied todomains as diverseas healthcare,defense, human per-formance engineer-ing, and game evalu-ation.

Seriousgames

VirtualFallujah

Nationbuilding

Spacewar

Network centricwarfare

Sensors, information sharing, agents, and

interoperability

Training andsimulation

Missionrehearsal

Combat modelingand analysis

Health

Public policy

Gameevaluation

Education

Strategiccommunication

Human performanceengineering

Serious games and simulations:developing a theory for the deployment of games and simulations forpurposes of education and training; human performance engineering;applications of games to health, public policy, and strategiccommunication; game evaluation; serious game development.

• development across all application domains:healthcare, public policy, strategic communi-cation, defense, training, and education;

• human performance engineering; and• game evaluation.

A new phenomenon in the video game world,serious-game development could potentially eclipsethe entertainment world in size if developers per-form the proper research when building this impor-tant area. Serious games use entertainmentprinciples, creativity, and technology to build gamesthat carry out a government or corporate objective.Moving toward making serious games requireshuman performance engineering—developing prin-ciples, processes, and procedures for their deploy-

ment. Researchers must have training and educa-tion objectives for their serious game, and theymust understand how to use the entertainmentworld’s creativity with appropriate human perfor-mance engineering principles.

GAMEPIPE LABORATORYThe first serious games must be constructed in

carefully controlled university environments. The“Educational Programs” sidebar describes the cur-ricula developed at the University of SouthernCalifornia to advance work in serious and enter-tainment games. The USC Viterbi School ofEngineering has formed the interdisciplinaryGamePipe Laboratory to research, develop, andteach the technologies necessary for creating future

September 2005 31

The USC Viterbi School of Engineering has several degreeprograms that work with the GamePipe Laboratory, includinga BS and MS in game development and a game developmentminors program. These degree programs have been developedjointly with the Department of Computer Science, the Schoolof Cinema’s Interactive Media Program, and the School ofEngineering’s Information Technology Program. Courses forthe MS program have begun as of Fall 2005. The BS programis planned to come online in Fall 2006.

BS and MS programsThe BS program seeks to educate students so that they can

engineer next-generation games immediately upon graduation.Students receive a solid grounding in computer science in addi-tion to the art and design expertise required for functioning inthe game industry. This program offers the School of Cinema’score game design sequence: programming for interactivity, 3Dgame design workshop, and intermediate game design work-shop. This sequence of design courses also functions as a seriesof deficiency courses for the MS in Game Development, ensur-ing that our BS and MS programs match up and that our MScandidates entering from other universities receive the properremediation.

The interdisciplinary MS in Game Development programstrives to graduate professionally educated students who canengineer next-generation games and their required technolo-gies. The program has a computer science MS core—analysisof algorithms, game AI, and computer graphics and rendering.This is followed by a game development core of game mechan-ics design and development, game engine programming, andgame hardware architectures.

Once they complete the computer science and game devel-opment cores, students select a concentration area and take

two courses in it. The concentration areas are infrastructure,cognition and games, immersion, and serious games. Thecourse offerings supporting each area allow considerable vari-ability in the educational programs.

Students complete the MS by taking a three-credit advanced-game-project course in each of the last two semesters. Thesecourses have students integrate and apply gained knowledgeon a significant game development effort that they then demon-strate the week before graduation. In addition to this advancedgame project focus, the majority of courses in this MS programencourage students to collaborate in groups.

Game development minors programsThe Viterbi School of Engineering offers an undergraduate

minors program in game development under the InformationTechnology Program. The program consists of three minors:video game programming, video game design and management,and 3D animation. This program adds six courses of gamedevelopment material to the undergraduate curriculum. Duringthe 2004-2005 school year, this program had about 600 stu-dents enrolled. Students in this program learn the industry-standard tools for the production of games through smallin-class projects.

Sponsored projectsSponsored game development projects form part of both the

GamePipe Laboratory’s educational program and its researchand development component. By offering an organized pipelinefor student intern labor, similar to that used in the game devel-opment industry, USC attracts significant funding from indus-try and governmental agencies. Having the ability toexperiment with and build full games and game prototypes isa capability unique to the USC educational program.

Educational Programs

interactive games and their application. The labo-ratory’s mission is to research and develop interac-tive games technologies that dramatically shortenthe production timeline, provide supporting tech-nologies for increasing the complexity and inno-vation in produced games, and enable the creationof serious and entertainment games for governmentand corporate sponsors.

The GamePipe Laboratory performs the researchand development required for next-generationgame production. It produces serious games andgame prototypes as sponsored projects. It also linkswith, directs, and develops educational programs ingame development and game-related research andbuilds cross-campus interdisciplinary teams to solvenext-generation game technology and developmentproblems.

B y performing the research and developmentrequired for the future of interactive gamesin a university environment, developers have

the potential to influence not just the future of inter-active entertainment but also the future of interac-tive training, education, and simulation—and,ultimately, the future of all serious-game develop-ment and deployment. With the formation of theGamePipe Laboratory and allied educational pro-grams, USC has taken the first step toward thatfuture. �

References1. N. Durlach and A. Mavor, eds., Virtual Reality: Sci-

entific and Technological Challenges, Committee on

Virtual Reality Research and Development, NationalAcademy Press, 1995.

2. M. Zyda and J. Sheehan, eds., Modeling and Simu-lation: Linking Entertainment & Defense, NationalAcademy Press, 1997; http://books.nap.edu/cata-log/5830.html.

3. M. Zyda et al., “From Viz-Sim to VR to Games: HowWe Built a Hit Game-Based Simulation,” Organiza-tional Simulation: From Modeling & Simulation toGames & Entertainment, W.B. Rouse and K.R. Boff,eds., Wiley Press, 2005, pp. 553-590.

4. MOVES Institute and the US Army, America’s ArmyPC Game—Vision and Realization, 2004;http://gamepipe.isi.edu/~zyda/pubs/YerbaBuenaAABooklet2004.pdf.

5. W. Gibson, Neuromancer, Ace Books, 1984.6. M. Zyda and D. Bennett, “The Last Teacher,” 2020

Visions, US Dept. of Commerce, 2002; http://gamepipe.isi.edu/~zyda/pubs/2020Visions.pdf.

Michael Zyda is director of the USC ViterbiSchool of Engineering’s GamePipe Laboratory,located at the Information Sciences Institute,Marina del Rey, California. His research interestsinclude computer graphics; large-scale, networked3D virtual environments; modeling human andorganizational behavior; interactive computer-generated story, modeling and simulation; andinteractive games. Zyda holds a lifetime appoint-ment as a National Associate of the NationalAcademies. He received a DSc in computer sci-ence from Washington University, St. Louis. Con-tact him at zyda@isi.edu.

Join a community that targets your discipline.

In our Technical Committees, you’re in good company.

www.computer.org/TCsignup/

Looking for a community targeted to your area of expertise? IEEE Computer SocietyTechnical Committees explore a variety of computing niches and provide forums fordialogue among peers. These groups influence our standards development and offerleading conferences in their fields.

JOIN A THINK TANK