6
Vis Comput (2008) 24: 935–940 DOI 10.1007/s00371-008-0293-1 ORIGINAL ARTICLE Future scenarios of mixed reality: the INTUITION roadmap scenarios Angelos Amditis · Ioannis Karaseitanidis · Matthaios Bimpas · Roland Blach Published online: 26 August 2008 © Springer-Verlag 2008 Abstract INTUITION is a Network of Excellence that aims to integrate the European research efforts on the scientific and technological field of Virtual and Mixed Reality. To per- form that, a series of activities have taken place in order to gather knowledge regarding actors and research profiles, projects and research results, products and patents. Having a clear view of research needs and technology trends the Net- work has envisioned the research goals that need to be pur- sued within the years to come. The starting point is a set of visionary scenarios which set out the picture for the tech- nological and scientific advances that need to take place. Within this paper a set of indicative scenarios on a higher and descriptive level are provided and the way they con- tribute to the roadmap definition is explained. With this re- port we want to share these scenarios and our initial thoughts to stimulate a broader discussion and invite people from all relevant backgrounds to enter the knowledge creation process. The paper is a collective production of the INTU- ITION Consortium. A. Amditis · I. Karaseitanidis ( ) · M. Bimpas Institute of Communication and Computer Systems, 9 Iroon Politechniou Street, Zografou, Athens, Greece e-mail: [email protected] url: http://i-sense.iccs.gr A. Amditis e-mail: [email protected] M. Bimpas e-mail: [email protected] R. Blach CC Virtual Environments, Fraunhofer IAO, Nobelstr. 12, 70569 Stuttgart, Germany e-mail: [email protected] url: http://vr.iao.fhg.de Keywords Virtual environments · Augmented reality · Mixed reality · Technology roadmap · Technology scenarios 1 Introduction The vision of the INTUITION Network is a complete in- tegration of VR/AR technologies and application-specific VR/AR concepts into general access strategies to computer based data, information or knowledge systems. This in- cludes representation as well as manipulation which can be summarized as spatial interaction concepts where users ex- perience a sense of presence in VEs and of virtual objects in the real world. The ultimate goal is a complete sensory stimulation which comprises visual, aural, haptic, olfactory and multi- modal spatial interactions that immerse the user in computer- generated (virtual) environments. The application opportu- nities of these technologies are very broad. To achieve these goals, especially to create a continuous impression of a sur- rounding environment, the content presented to the users has to be adapted to the interface requirements (namely, 3D data which can be rendered/presented in real time). In particular, this implies the capability to simulate the physical laws that rule the real world taking into account the time constraints that allow the system users to interact with the virtual world. This process, and the integration of this technology in ap- plication workflows, is also considered part of the research field. This general vision will be illustrated by four scenarios which describe the integration of VR/AR technology.

Future scenarios of mixed reality: the INTUITION roadmap scenarios

Embed Size (px)

Citation preview

Page 1: Future scenarios of mixed reality: the INTUITION roadmap scenarios

Vis Comput (2008) 24: 935–940DOI 10.1007/s00371-008-0293-1

O R I G I NA L A RT I C L E

Future scenarios of mixed reality: the INTUITION roadmapscenarios

Angelos Amditis · Ioannis Karaseitanidis ·Matthaios Bimpas · Roland Blach

Published online: 26 August 2008© Springer-Verlag 2008

Abstract INTUITION is a Network of Excellence that aimsto integrate the European research efforts on the scientificand technological field of Virtual and Mixed Reality. To per-form that, a series of activities have taken place in orderto gather knowledge regarding actors and research profiles,projects and research results, products and patents. Having aclear view of research needs and technology trends the Net-work has envisioned the research goals that need to be pur-sued within the years to come. The starting point is a set ofvisionary scenarios which set out the picture for the tech-nological and scientific advances that need to take place.Within this paper a set of indicative scenarios on a higherand descriptive level are provided and the way they con-tribute to the roadmap definition is explained. With this re-port we want to share these scenarios and our initial thoughtsto stimulate a broader discussion and invite people fromall relevant backgrounds to enter the knowledge creationprocess. The paper is a collective production of the INTU-ITION Consortium.

A. Amditis · I. Karaseitanidis (�) · M. BimpasInstitute of Communication and Computer Systems, 9 IroonPolitechniou Street, Zografou, Athens, Greecee-mail: [email protected]: http://i-sense.iccs.gr

A. Amditise-mail: [email protected]

M. Bimpase-mail: [email protected]

R. BlachCC Virtual Environments, Fraunhofer IAO, Nobelstr. 12, 70569Stuttgart, Germanye-mail: [email protected]: http://vr.iao.fhg.de

Keywords Virtual environments · Augmented reality ·Mixed reality · Technology roadmap · Technologyscenarios

1 Introduction

The vision of the INTUITION Network is a complete in-tegration of VR/AR technologies and application-specificVR/AR concepts into general access strategies to computerbased data, information or knowledge systems. This in-cludes representation as well as manipulation which can besummarized as spatial interaction concepts where users ex-perience a sense of presence in VEs and of virtual objects inthe real world.

The ultimate goal is a complete sensory stimulationwhich comprises visual, aural, haptic, olfactory and multi-modal spatial interactions that immerse the user in computer-generated (virtual) environments. The application opportu-nities of these technologies are very broad. To achieve thesegoals, especially to create a continuous impression of a sur-rounding environment, the content presented to the users hasto be adapted to the interface requirements (namely, 3D datawhich can be rendered/presented in real time). In particular,this implies the capability to simulate the physical laws thatrule the real world taking into account the time constraintsthat allow the system users to interact with the virtual world.This process, and the integration of this technology in ap-plication workflows, is also considered part of the researchfield. This general vision will be illustrated by four scenarioswhich describe the integration of VR/AR technology.

Page 2: Future scenarios of mixed reality: the INTUITION roadmap scenarios

936 A. Amditis et al.

2 Scenarios

We have chosen four from over 52 scenarios developed inthe INTUITION working-groups, which outline the appli-cation of VR/AR technology clearly. The report covers onesocial and three professional scenes, mainly located in prod-uct development and marketing.

2.1 Scenario on product development and maintenance

Peter, an engineer, works in a company for mechatronic sys-tems. When he enters his office in the morning, the computersystem identifies him and registers his attendance. The sys-tem can support him later with routing information to thedisplay/terminal where he is actually located and also theintegration of local interaction devices and infrastructure.

His work environment is fully integrated in the company’sIT environment. The whole building is interconnected withthe computer network and can utilize all displays and out-put devices which are spread all over the building. WhenPeter moves around the building, he is able to bring up hispersonal desktop or 3D-workspace everywhere. All displayscan show more than one view which makes them inherentlycapable for stereoscopic projection.

He enters a meeting where developers discuss a com-plaint of a product by a client for which he is responsible.The sound properties of the product do not meet the re-quired specifications. The new product is a complex electro-mechanical engine. To understand its operation and com-municate it to the other developers of different professionalbackgrounds, a visual and aural 3D-simulation is used. Thecustomer is remotely participating inside a shared virtualenvironment. To understand the problem, various simulationresults and testing protocols are superimposed, such that theteam can discuss the functions of the product. All partici-pants have a correct and individual aural and visual view onthe product.

Paul, Peter’s colleague, is a field engineer, on his way tothe customer. The routing planner projects the guiding in-formation on the front windshield and highlights significantlandmarks for orientation on the route. A tracking system isable to match and highlight the silhouette of objects for hisview. The examination of the system is used with a mobileaugmented reality system where additional information andassembly hints are shown. In this case he has to consult withPeter. He establishes a link to Peter’s workspace where bothhave accurate spatial view on the machine and they can dis-cuss the problem as if they were both in the same room.

Mary, the controller, contacts Peter about some budgetissues in his project. The various multidimensional busi-ness data are represented in 3D and accessed via a 3D-interaction space. With a gesture Mary picks out the prob-lematic data and shows these to Peter who has his personal3D-view on the dataspace.

Jane, an industrial designer, works on new housings forPeter’s project. To find an early shape, she works with intel-ligent clay which provides haptic feedback and establishesimmediately a digital 3D-model. A projection system aug-ments spatially right visual surface material which providesa more realistic look than the rough clay surface. Peter isable to connect to her workspace and present the new de-signs to a potential customer.

2.2 Scenario on social life and leisure

Peter’s daughter is on her way to an archaeological exhi-bition of an ancient city in the local museum. Schools areusing more and more media based and situated learning en-vironments in cooperation with public bodies. An exhibitionarea where visitors are spatially immersed in images, soundand smell, was set up in the museum. She brought her mo-bile assistant on which her homework and an access key tothe immersive system were uploaded. In the virtual recon-struction she should find the market place and record and an-alyze the scene. The exhibition has been created in anothermuseum and is now virtually exhibited worldwide. There arealso common spaces where visitors around the world canmeet.

In the afternoon the family has an appointment in thekitchen planning studio. On Peter’s PDA, they have a floorplan of the kitchen and some preliminary planning, whichwill be uploaded in the planning system. At home they havealready chosen a model series and the appropriate furniture.They are only unsure about the final look and want to havean optimal realistic 1:1 impression. An immersive projectionspace with a high performance visual system is available atthe kitchen planning studio. The planning result is saved asdata and its impressions are also recorded as a movie whichPeter can take back on the PDA or download from the web.

Peter’s son is participating in an online game where thegamer can participate virtually at a real car racing. Thedrivers in the cars use see-through displays where the virtualdrivers are superimposed and are informed if they producean accident, which will be accounted for. The virtual par-ticipants have the opportunity to challenge professional racedrivers without endangering them. The employed interfacecan range from classical joypads to physical car mock-ups.Similar systems exist for e.g. sailing, skiing, etc., with inter-faces that encourage physical exercise.

In the evening, the kids use to go out at interactive clubspaces where large spatial group interactions are connectedwith dance and music. Movement of the dancers can shapevisuals and music in such a way that every performance be-comes a unique experience.

2.3 Scenario on engineering

A fluid dynamics engineer is evaluating the behavior of aconcept vehicle exterior. By “blowing” a virtual smoke wind

Page 3: Future scenarios of mixed reality: the INTUITION roadmap scenarios

Future scenarios of mixed reality: the INTUITION roadmap scenarios 937

around a virtual body she can freely walk around and posi-tion it as she likes. The system can be controlled via naturallanguage. Meanwhile, she can get aerodynamics figures inthe preferred format: tables, graphs, even color maps on andaround the vehicle body. Data appear on a virtual screen thatfollows the engineer in the room. For this activity the en-gineer wears lightweight goggles and her position, postureand gesture are acquired by remote motion tracking systems.An audio system reproduces the wind noise generated bythe body; at will the engineer can superimpose engine androlling noise.

The engineer discovers a critical point in the frontwindshield pillar—already validated by the ergonomicsdepartment—and, by manipulating in a natural way the ve-hicle body, achieves eventually the desired performance,at the price of changing both the styling and the externalvisibility. A “virtual ergonomist,” an application running inbackground, checks the ergonomic validity of the alternatesolution and gives the green light.

Instead, no agent can assess the impact on styling, mak-ing necessary the direct evaluation by the stylist; therefore,the engineer is not allowed to validate her solution. Whileshe has the freedom either of exploring alternate solutions,the engineer must contact the stylist for negotiating. Thestylist is remotely located, at a conference in a similar room.The two actors can see and talk to each other, and bothcan see (and share) their preferred representation. Respec-tive reasons are explained, aided by examples; eventually thestylist creatively explores and finalizes a new solution. Dur-ing this meeting, some solutions are discarded because the“virtual ergonomist” has raised a warning. After the meet-ing, each actor validates the solution and the concept is for-warded to the management meeting for styling approval.

2.4 Scenario on design management

Giovanna and her team of five designers, who are collocatedat the industrial park’s recently opened collaboration center,are preparing for a milestone meeting with their main cus-tomer. Instead of flying the entire team down, the milestonemeeting is organized as a distributed session, by connect-ing the co-workspace installation at the collaboration centerand the matching facilities at the customer’s headquarters inChina with a large, interactive display for visualization andvirtual tools for collaboration.

During the preparations, one of the test engineers sud-denly notices a mismatch between the results of two testsused to discover complex load conditions. By facilitatingcontributions from external experts participating using a mo-bile setup or similar advanced meeting facilities, the teamcan quickly connect to various resources to solve the prob-lem. They decide to check if it is possible to run a new setof simulations to determine if it is necessary to change thedesign or not.

While her five designers are busy preparing the presen-tation and different physical and digital models they aregoing to discuss with their customer, Giovanna enters the“External Team Resources” section of their digital projectworkspace by interacting directly with a large, interactivedisplay. The optical tracking setup installed around the dis-play allows her to use her fingers to interact directly withthe virtual screen. She notices that Tom’s status is “personaldevice / online,” and calls him by using her finger to presshis virtual picture on the display. Giovanna explains to Tomthe situation, and asks him if it is possible to quickly checkif the new load conditions will require any design changesto prevent failure. “I’ll see what I can do,” says Tom andleaves the session. Giovanna had met Tom three years ear-lier through the LinkedIn community, and has nothing butgood things to say about him.

Twenty-five minutes later, it therefore came as no sur-prise when the communication system announced “Incom-ing call from Tom, asking for video mode with shared view-ing of simulation results, please respond with preferred com-munication mode.” Giovanna responds “accept, engage infull collaboration.” The system adapts to her request and op-timizes cameras and displays to support shared interactionwith complex graphics. Tom appears in full size on the maincommunication screen wearing his see-through 3D stereoglasses. Tom says “I have good news for you. As you cansee on the shared display, the new load conditions do notrequire a redesign. You have the protocol files in the sharedspace.” Giovanna thanks him for helping out and closes thevideo session to continue preparing for their meeting. Theymight even have time for lunch and a coffee before the meet-ing, she thinks.

3 From scenarios to roadmap

The described scenarios illustrate the future use of technol-ogy based on today’s VR/AR technology and research. It iscan be seen that a few themes are always immanent in allscenarios. These can be summarized as follows:

Ubiquitous access to the virtual world in parallel to thephysical real world wherever and whenever the user needsit. Sometimes it is an exact spatial superposition (3D sup-ported assembly), sometimes it is only loosely related tothe physical environment (touristic information access), andsometimes the virtual environment is completely unrelated(entertainment). The spatial access is seamlessly integratedin the media technology. Collaboration is naturally inbuilt.The challenge for the future is to provide access technolo-gies which can blend virtual and physical environment spa-tially everywhere and be connected to everyone. These vi-sions contain not only the development and improvement

Page 4: Future scenarios of mixed reality: the INTUITION roadmap scenarios

938 A. Amditis et al.

Fig. 1 Possible VR/ARevolution in the upcommingyears

of immersive technology but also the need of infrastruc-ture for immediate access to computation power and contentdata.

• Convergence of immersive interface technology withother highly interactive and multimodal interaction tech-nologies.

• Integration of interaction modalities as speech, gesturerecognition, haptic and tactile feedback, etc., concurrentlysuch that the user can choose which mode is most appro-priate.

• Unobstructed spatial interaction devices.• Mobile technology.• New interaction paradigms.• Online and offline remote collaboration.• Content creation and management tools, as e.g. multi-

domain correlated product representations.• Simulation technology for complex environment behav-

ior, as e.g. for embedded real time simulation of productperformance, incorporating where necessary human be-havior simulation.

If we condense these reoccurring attributes to the follow-ing six characteristics, we can roughly describe where spe-cific technology is today and how it might evolve in the fu-ture, here shown very general for VR and AR:

• Mobility: Wearable and ergonomic (weight, dimensions);wireless, low power.

• Connectivity: online/offline, synchronous/asynchronous,bandwidth and latency. This is the technical foundationfor seamless collaboration.

• Sensor and display: fidelity and bandwidth or resolution.• Complexity: data complexity which can be processed, be-

havioral and dynamic complexity which can be processed.• Integration: data, process and workflow integration.

• Interaction: Which and how many modes are accessible tointeract, and are they intuitive and ergonomic? To whichextent or depth can the virtual environment be manipu-lated by users?

These characteristics capture different qualities of systemand application configurations in hardware and software,and can be used for their comparison. They can be split upin more detail which might be helpful for a deeper analysis.Obviously the existence of parallel realities, where the phys-ical is only one among others but not necessarily the mostimportant one, will change definitely the way we live andwork. This is very difficult to anticipate from our presentpoint of view. Nonetheless, the provision of sophisticatedvirtual environments has the potential to already experimentwith the look and feel of future and emerging technologies.

Based on the set of envisioned scenarios the Networkidentified a set of barriers (technological, scientific, orga-nizational, etc.) that need to be overcome. Barriers havebeen categorized in pure technology, interaction relatedissues, integration topics and socio-economic issues. Anindicative list is provided at [1], while for a completereference one should address [2]. Alongside drivers forchange, which are the dominant factors that lead to tech-nology and applications, evolution was under investiga-tion. These are mainly the continually increasing indus-trial requests (from design and early prototyping to train-ing applications and maintenance), the investment increasein novel technologies and interfaces, and the need for ad-vancement in information representation and data integra-tion. The technology itself has strongly matured within thelast few years, enabling the development of systems and ap-plications that bring 3D in everyday life scenarios in severalcases.

The knowledge of scenarios, barriers and drivers allowsus to create a Roadmap providing a series of research ar-

Page 5: Future scenarios of mixed reality: the INTUITION roadmap scenarios

Future scenarios of mixed reality: the INTUITION roadmap scenarios 939

eas and topics of interest in each one, where advancement isrequired or is under way either in a short, medium or longtime scale. Roadmaps have been successfully used in otherresearch and development fields to envision and illustrate fu-ture developments and their reflections [3, 6]. The goal is todiscuss discrete advancements needed, their interrelations,likelihood and influence on other parameters. The final re-sult is a time-scheduled weighted matrix of discrete stepsto be undertaken. INTUITION has already prepared its firstRoadmap. However, this is nothing more than a snapshotof the technological and scientific fields at a specific timepoint. Thus, a continuous updating and evolution, accord-ing to technology trends and achievements, research con-cepts and products, fields of research and state-of-the-art, isneeded. The updating procedure should provoke consensusamong the research communities to a large extent and allowroom for discussion and disagreements.

4 Conclusions

Mixed Reality technologies are being increasingly usedin industrial and other scenarios. Technology is becomingmore robust and affordable, making researchers envision avariety of scenarios of future use. The INTUITION Con-sortium, the largest European community on VR and ARresearchers, has come up with a list of indicative future sce-narios for use that are mapped along the technological bar-riers and drivers for change. The scenarios indicate in a de-scriptive and simple way the use of the technology withindifferent life aspects within the next 20 years. Needs andwishes for the scenarios may be technically translated in or-der to form a set of distinguished technological advances tobe pursued. These consist of a research Roadmap and IN-TUITION has worked on providing such. This will be con-tinually updated through a collaborative Knowledge Basethat will act both as information repository and as a brain-storming point of reference amongst the VR/AR commu-nity. The INTUITION Knowledge Base is a powerful col-laborative workspace based on wiki-technology [5] whichhelps to structure knowledge but at the same time pro-vides a collaborative workspace for creating new knowl-edge. The user friendliness of a wiki front-end should fa-cilitate information and knowledge sharing, exchange anddebate. Wiki technology has been also used as a base forontology building [4] which is also an important outcomeof knowledge creation processes. The tool is backed upby other information repositories on technology products,research themes, training courses, best practice guidelineswhich have been created and populated during the INTU-ITION project. It will be publicly available from March2008.

Acknowledgements The authors need to emphasize that the paper isa product of the collective knowledge of the INTUITION Consortium.The authors would like to acknowledge the contribution of the 11 in-dividual INTUITION working-groups in the scenarios and roadmapdevelopment.

References

1. Amditis, A., Bimpas, M., Blach, R.: Towards networked and struc-tured VR European research area: intuition network of excellenceand future research challenges. MMI-Interaktiv 12, 43–54 (2007)

2. Blach, R., Dangelmeier, M., Bimpas, M., Frangakis, N., Walzack,K., Gravez, P., Bourdot, P.: ERA structuring plan and future re-search activities. INTUITION Deliverable (2006)

3. Groenveld, P.: Roadmapping integrates business and technology,Res. Technol. Manag. 40(5) (1997)

4. Hepp, M., Bachlechner, D., Siorpaes, K.: Harvesting Wikiconsensus—using Wikipedia entries as ontology elements. In: Pro-ceedings of the 1st Workshop: SemWiki2006—From Wiki to Se-mantics, co-located with the 3rd Annual European Semantic WebConference (ESWC 2006), June 12, Budva, Montenegro (2006)

5. Leuf, B., Cunningham, W.: The Wiki Way: Quick Collaboration onthe Web. Addison-Wesley, Reading (2001)

6. Phaal, C., Farrukh, J.P., Probert, D.R.: Collaborative technologyroadmapping: network development and research prioritisation. Int.J. Technol. Intel. Plan. 1(1) (2004)

Angelos Amditis has obtained theDiploma in Electrical and ComputerEngineering from National Techni-cal University of Athens—NTUA(Greece) in 1992, Ph.D. in Elec-trical and Computer Engineering(Telecommunications) from NTUA(Greece) in 1997. He is a Senior Re-searcher (Research Associate Pro-fessor) of the Institute of Communi-cation and Computer Systems andthe writer of several journal andconference papers and a co-authorof three books.

Ioannis Karaseitanidis holds aB.Sc. in Electrical and ComputerEngineering from NTUA (2000).He obtained his Ph.D. at the Mi-crowaves and Fiber Optics Labo-ratory (2007) concerning trackingtechniques on Virtual Reality appli-cations with emphasis to ultrasonicwave-based methods. His researchinterests include design and hard-ware and software setups for Vir-tual Reality systems, Digital SignalProcessing techniques, design andevaluation of interaction devices forVEs and collaborative concepts for

working environments mostly applying VR solutions. He has workedin several European and national projects, has published a number ofpapers in Conference Proceedings and Journals and is a member of theTechnical Chamber of Greece.

Page 6: Future scenarios of mixed reality: the INTUITION roadmap scenarios

940 A. Amditis et al.

Matthaios Bimpas holds a B.Sc. inElectrical and Computer Engineer-ing from July 2000 at DECE/NTUAand a Ph.D. in Microwaves Labo-ratory of DECE/NTUA on designand implementation of innovativeradar sensors and antenna design.He was professor in the Electron-ics Department of the University ofPedagogical and Technical Educa-tion in Athens from 2000 until 2002,while he was teaching assistant inundergraduate and graduate courseson Microwaves and Telecommuni-cation Electronics. He is currently

professor of Sensor Theory and Antennae in the Department of Air-craft Engineers of the Hellenic Air Force University.

Roland Blach is a senior scien-tist and works since 1995 at Fraun-hofer IAO in the field of VR. Heis one of the architects of the VRsystem Lightning and has partic-ipated in many industrial and re-search projects. His research inter-ests are software architecture for in-teractive realtime systems, 3D in-teraction, projection-based displaysystems and immersive informationvisualizations. He serves actively inthe scientific reviewing process onvirtual reality, e.g. for Eurographics,IEEE VR or the IEEE 3DUI Sympo-

sium. He is also involved in the technical management of the EuropeanINTUITION Network of Excellence on VR/AR technology and alsoresponsible for the VR technology working-group at INTUITION.