Building virtual reality fMRI paradigms: A framework for presenting immersive virtual environments

  • Published on

  • View

  • Download

Embed Size (px)


<ul><li><p>Journal of Neuroscience Methods 209 (2012) 290 298</p><p>Contents lists available at SciVerse ScienceDirect</p><p>Journal of Neuroscience Methods</p><p>journa l h omepa g e: www.elsev ier .com</p><p>Basic Neuroscience</p><p>Building virtual reality fMRI paradigms: A framewvirtual environments</p><p>Charles M nieMichael a Department f acultyb Klinik fr Neu ny</p><p>h i g h l i g h t s</p><p> We develop informatical concepts, which allow the creation of own VR-fMRI paradigms. We develo We embed We valida Subjects in</p><p>a r t i c l</p><p>Article history:Received 12 FeReceived in reAccepted 23 Ju</p><p>Keywords:Virtual realityVirtual enviroStimulus preseReal-time fMRBraincomput</p><p>1. Introdu</p><p>In the laing (fMRI) presenting </p><p> CorresponGermany. Tel.:</p><p>E-mail add1 Institut F</p><p>versitt MagdeTel.: +49 391 6</p><p>0165-0270/$ http://dx.doi.op neuroinformatical techniques which provide real-time VR-fMRI studies. an easy-to-handle integration concept for virtual environment les.</p><p>te the application in a real-time VR-fMRI study with spatial memory topic.dicate higher interaction and more attention than in common fMRI studies.</p><p> e i n f o</p><p>bruary 2012vised form 18 May 2012ne 2012</p><p>nmentsntationIer interface</p><p>a b s t r a c t</p><p>The advantage of using a virtual reality (VR) paradigm in fMRI is the possibility to interact with highlyrealistic environments. This extends the functions of standard fMRI paradigms, where the volunteer usu-ally has a passive role, for example, watching a simple movie paradigm without any stimulus interactions.From that point of view the combined usage of VR and real-time fMRI offers great potential to identifyunderlying cognitive mechanisms such as spatial navigation, attention, semantic and episodic memory,as well as neurofeedback paradigms. However, the design and the implementation of a VR stimulusparadigm as well as the integration into an existing MR scanner framework are very complex processes.To support the modeling and usage of VR stimuli we developed and implemented a VR stimulus appli-cation based on C++. This software allows the fast and easy presentation of VR environments for fMRIstudies without any additional expert knowledge. Furthermore, it provides for the reception of real-timedata analysis values a bidirectional communication interface. In addition, the internal plugin interfaceenables users to extend the functionality of the software with custom programmed C++plugins. The VRstimulus framework was tested in several performance tests and a spatial navigation study. Accordingto the post-experimental interview, all subjects described immersive experiences and a high attentionalload inside the artical environment. Results from other VR spatial memory studies conrm the neuronalactivation that was detected in parahippocampal areas, cuneus, and occipital regions.</p><p> 2012 Elsevier B.V. All rights reserved.</p><p>ction</p><p>st years, several functional magnetic resonance imag-studies used virtual reality (VR) environments whentheir stimuli. Virtual reality refers to an articial</p><p>ding author. Permanent address: Hegelstrasse 6, 39104 Magdeburg, +49 391 40 59061; fax: +49 391 40 59061.ress: (C. Mueller).r Biometrie und Medizinische Informatik, Medizinische Fakultt, Uni-burg, Leipziger Str. 44, 39120 Magdeburg, Germany.7 13537; fax: +49 391 67 13536.</p><p>computer-generated environment, with which users act and inter-act as if in a known real environment. Furthermore, users canexperience things that would otherwise be very difcult or evenimpossible in a magnetic resonance scanner or with an electroen-cephalograph.</p><p>The big advantage of virtual environments lies in the presen-tation of realistic stimuli. Instead of passively watching a simplemovie stimulus, subjects can interact actively with the paradigm,for example, navigating and exploring an articial environment.Referred to this, several virtual environment studies have been usedto investigate the role of the hippocampus and the parahippocam-pal area in topographical, spatial and episodic memory processes(Aguirre, 1998; Maguire et al., 2006; Mellet et al., 2010). In</p><p> see front matter 2012 Elsevier B.V. All rights reserved.rg/10.1016/j.jneumeth.2012.06.025uellera,, Michael Luehrsa, Sebastian Baeckea, DaLuchtmannb, Johannes Bernardinga,1</p><p>or Biometrics and Medical Informatics (IBMI), Otto-von-Guericke-University, Medical Frochirurgie, Otto-von-Guericke-University, Leipziger Str. 44, 39120 Magdeburg, Germa/ locate / jneumeth</p><p>ork for presenting immersive</p><p>la Adolfa, Ralf Luetzkendorfa,</p><p>, Leipziger Str. 44, 39120 Magdeburg, Germany</p></li><li><p>C. Mueller et al. / Journal of Neuroscience Methods 209 (2012) 290 298 291</p><p>addition, the location of so-called place cells in the human brainwas researched with virtual environments (Aguirre et al., 1996;Doeller et al., 2010).</p><p>Another example of how virtual environments are used in neu-rosciences a(HBI). By usown brain acases, fMRImeans as fatime (TR) (Get al., 2009to control t(Yoo et al., 2cingulate co(DeCharms</p><p>Similarlyments in threduction o2007), and virtual realiaspect of thence, whichthe essentia</p><p>Alongsidreality in nimplementatational meframeworkstudies usiapplicationthe technicsciences descientic grenvironmentain only odictate the to implemeronment stiparadigms. few mouse the applicawhich fulltual reality support reafor exchangPosse, 2001</p><p>2. Materia</p><p>2.1. Experim</p><p>For the pmodal stimsystems. Se</p><p>The rstresented bysystem prothe subject.</p><p>The secocase by theimage datasystem andbody MRI scSiemens MeMagdeburg</p><p>coil (Siemens Medical Systems, Erlangen, Germany) was used forimaging. The vendors EPI BOLD sequence and the correspondingreconstruction programs were modied to export each volumedataset immediately after acquisition and internal motion correc-</p><p> real1999ftwaputn prronitic re</p><p> last ore, wBrainis syssly, w</p><p> realmmu011)a anz, 2 G</p><p> to threal-n prois valof visal coind</p><p>tor oed ous cossovimagc scar sys</p><p> stim</p><p> simftwasionahite</p><p>uctednto ore Automgraphmpotes sionanmenractei, andand teracriggeitiateo cov</p><p> Inc.d prnon-ion Grograork viroWe re neurofeedback studies and humanbrain-interfacesing operant conditioning, subjects learn to regulate theirctivation with the aid of a neurofeedback signal. In such</p><p> image data need to be analyzed in real-time, whichst as they are acquired, i.e., within a single repetitionembris et al., 2000; Weiskopf et al., 2007; Scharnowksi). Specic examples include a classication algorithmhe movement of an avatar in a two-dimensional maze004) or reducing the pain intensity in the right anteriorrtex with neurofeedback training and a virtual ame</p><p>, 2005)., several psychological studies used virtual environ-eir therapies: burn pain therapy (Hoffman et al., 2000),f claustrophobic symptoms (Garcia-Palacios et al.,therapy of posttraumatic stress disorder based on thety exposure therapy (Rothbaum et al., 1999). The basicese psychological therapies is the immersive experi-</p><p> helps to evoke relevant neuronal activity and providesl feeling of being inside of a real world.e these benets, the main difculty of using virtualeuroscientic experiments are the time-consumingtion, the combination of neuroscientic and compu-thods, and the integration into an existing acquisition. In spite of the increasing number of neuroscienticng virtual reality, few virtual environment stimuluss can handle such experiments in their entirety. Sinceal complexity and the required knowledge in computermand high personnel and nancial resources, severaloups work with virtual environment stimuli using staticts from computer games. These game frameworks con-ne virtual environment, whose technical restrictionsdesign of the neuroscientic experiment. Our aim wasnt an adaptive, exible, and extendable virtual envi-mulus application that includes several scenes for fMRIThe user should be able to import a VR scene with just aclicks and present this scene to a subject. Furthermore,tion should provide a highly realistic stimulus outputls the immersive requirements necessary for using vir-in psychological therapies (Regenbrecht et al., 1998). Tol-time fMRI experiments we integrated functionalitiesing data values with external applications (Mathiak and).</p><p>ls and methods</p><p>ental infrastructure</p><p>resentation of virtual environments we used a multi-ulus environment concept that comprises three basice Fig. 1 for a detailed overview of this concept.</p><p> basic system is the presentation system which is rep- the virtual environment stimulus application. This</p><p>vides visual and auditory stimuli and presents them to</p><p>nd is the data acquisition system represented in our magnetic resonance imaging scanner which produces</p><p> for the data analysis system. Implementation of the all test experiments were conducted on a whole-anner (3T Trio, Software Version Numaris Syngo VA35,dical Systems, Erlangen, Germany) at the University of, Clinic for Neurology. An eight-element phased array</p><p>tion inet al., This soble combetweea synchmagne</p><p>TheTherefTurbo analystaneouFor theTCP coet al., 2</p><p>Dat3.0 GHnectedof the nicatioanalystation personRAM, Wprojecmountstimulnet croof the speciscanne</p><p>2.2. VR</p><p>Thetwo sodimenand arcconstrthem isoftwathe cusas geoond cocalculadimenenvirocal chastimulacters and intem. Tand intion. TTrinigyto builus for the Viscode pFramewtual en2.8.1. time to the host computer of the MRI scanner (Posse; Weiskopf et al., 2004, 2007; Hollmann et al., 2008).re function performs the image export to any accessi-er in the local area network. For the synchronizationesentation system and data acquisition system we usedzation signal that is sent from the host computer on thesonance scanner to the presentation computer.basic system is represented by the data analysis system.e used the commercial real-time data analysis software</p><p>Voyager (Brain Innovation, 2012). To ensure that datatem and stimulus presentation were processed simul-e distributed both systems on two personal computers.</p><p>-time data analysis transfer, a bidirectional client/servernication protocol was implemented and tested (Luehrs.alysis was performed on a computer (Pentium IV,B Random Access Memory (RAM), Windows XP) con-e local area network via 100 Mbit/s. For the rst tests</p><p>time fMRI framework we embedded the TCP commu-tocol into a Turbo BrainVoyager plugin to transfer dataues in real time to our stimulus application. The presen-ual and auditory VR stimuli also took place on a separatemputer (stimulus computer, Pentium IV, 3.0 GHz, 3 GBows XP). Visual information was projected with a videon a transparent screen and viewed via a 45 mirrorn the receiver coil. The statistics computer and the VRmputer were connected directly via a 100 Mbit/s Ether-er cable. Apart from the interface for the real time exporte data, which had to be implemented for the vendor-nner system, all components were independent of thetem.</p><p>ulus application</p><p>ulation of a virtual environment generally requiresre components. The rst is a virtual scene or a three-l model representing the pure environment, e.g. objectsctural structures. For the creation of our virtual scene we</p><p> several single three-dimensional models and mergedne comprehensive model using the professional designtodesk, Inc., 3ds Max. Furthermore, this process allows</p><p> modeling and integration of structural elements, suchical landmarks or human character models. The sec-nent is the so-called game engine framework, whichall interactions between the subject and the three-l model. It also simulates human behaviour and realistictal conditions, such as collision detection, the physi-ristics of the specic environment, visual and auditory</p><p> the behavioral patterns of computer-simulated char-their movements. In addition, building event-markerstive elements is supported by a virtual event sys-r event classes observe the main character movements</p><p> predened animations relative to the current posi-er these comprehensive functions, we integrated the</p><p> Vision Game Engine, version 7.6.4. It is typically usedofessional computer games and has been provided tocommercial and scientic use. All important parts ofame Engine were directly embedded in our C++sourcemmed with Microsoft Visual Studio 2008, Microsoft.NET2.5. To simulate the physical effects inside the vir-nment we used the free Nvidia, Corp., PhysX, versionextracted the necessary software libraries only and</p></li><li><p>292 C. Mueller et al. / Journal of Neuroscience Methods 209 (2012) 290 298</p><p>Fig. 1. The technical infrastructure for the real-time fMRI setting. The components inside of the box depict the vendor-specic measurement system. During the dataacquisition process, the image data will be motion-corrected and transformed by the vendor specic image calculation environment. Afterwards the reconstructed imagedata will be transferred in real time to the host computer. Host computer, presentation system and data analysis system are connected via 100 Mbit local area network(LAN). To analyze the MRI image data in real time the data analysis system takes access on the image data at the host computer. At the same time the presentation systempresents the virtual environment to the subject in the MRI scanner. The synchronization signal connects the host computer with the presentation system and synchronizesthe experiment protocol.</p><p>integrated them into our virtual environment stimulus framework.For implementing the graphical user interface, the TCP commu-nication interface, our plugin interface, and for the several datamanagement functions we used the open source version of theuser interface framework Nokia, Corp., Qt, version 4.6.3. To keepthe virtual extendablearchitectureface repres</p><p>second is represented by the application logic layer, which con-trols the main process of the Vision Game Engine. The third andlast is dened by the communication layer, which governs thecommunication processes, e.g. TCP communication protocol, syn-chronization signal and Microsoft DirectX interface for Universal</p><p>us bransplici-thr</p><p>Fig. 2. The sofhighest tier init. The secondand the plugininterface are loenvironment stimulus application well organized and we designed our software as a multi-layered three-tier</p><p> software. In this hierarchical concept the user inter-ents the highest layer in the software framework. The</p><p>Serial BTo t</p><p>ulus apa multtware architecture of the VR stimulus application. The software design is based on the cl the hierarchy is represented by the user interface layer, which includes all graphical co</p><p> is the application logic layer, which includes the MRI experiment data storage, the gam management. The third is the communication layer. The TCP communication, MRI triggecated here.utton-press event control. For further details, see Fig. 2.fer the real time fMRI data analysis values to the VR stim-ation, e.g. for neurofeedback paradigms, we developedead and bidire...</p></li></ul>


View more >