The Influence of Rendering Styles on Participant Responses in Immersive

Virtual Environments

Mel SlaterInstitució Catalana de Recerca i Estudis Avançats (ICREA)

@ Universitat Politècnica de Catalunya&

Department of Computer Science, UCLhttp://www.lsi.upc.edu/~melslaterhttp://www.lsi.upc.edu/~moving

www.icrea.es

Global Illumination – Who Cares?

• Why spend effort and resources on ‘global illumination’ (eg, ray tracing, radiosity, photon mapping,…)?

• Scientific and practical question?• What is its impact on people who are in

immersive virtual environments?• Brings together my two strands of research

• Presence in virtual environments• Real-time global illumination

Outline

• Introduction • Immersive VEs and Presence

• Real-time Ray Tracing and Presence• An experiment

• Virtual Light Field • Real-time global illumination for the Cave

• Conclusions

Immersive Virtual Environment• Displays in all sensory systems• Fully encloses participant in the displays• Tracks the head, limbs, body• determines the optical, auditory... sensory

data as a function of (head) tracking

Immersive Virtual Environment

• ‘Cave’ – immersive projection system

Immersion: the technology• Displays

• Field of view (visual), stereo, resolution• Frame rate• Multiple sensory modalities (visual, auditory, haptics,

olfactory)• Content, level of realism

• Interaction• Head-tracked (body tracking), capability for interaction• latency

• General• Correlation between sensor data and overall proprioception• Transparency of devices (weight, cables, etc)

Reported Presence…• ‘The sense of “being there”’ (Held & Durlach,

Sheridan, Zeltzer: premier issue of PRESENCE, 1992)

• ‘A perceptual illusion of nonmediation’ (Lombard and Ditton, 1997)

• ‘A mental state in which a user feels physically present within the computer-mediated environment’ (Draper & Kaber, 1998)

• ‘The subjective experience of being in one place or environment, even when one is physically situated in another’ (Witmer & Singer, 1998)

Presence• Successful substitution of real sense

data by computer generated sense data• ‘Successful’ – response is as if the

sense data were from a real source• ‘Response’ –

• Low level physiological → high levelcognitive and emotional

• Includes verbal responses about ‘being there’

Presence• The evidence that presence occurs is strong• People tend to respond automatically in a realistic

manner• In spite of knowing for sure that what they are

experiencing is unreal

The ‘Pit Room’ ExperimentsLong history of usingthis environment for experimentation: eg

•Slater, Usoh, Steed (1999)

•Meehan et al (2002)

•Zimmons and Panter (2003)

Fear of Heights• Meehan/Insko (2002)

exposed 10, 52 and 33 subjects in 3 different studies.

• Heart rate increasewhen in the pit room.

• Static haptics further significantly increased heart rate.

• Heart rate correlatedwith subjective self-report of presence level.

Visual Realism

• In a between-groups study utilising the ‘pit room’Zimmons et al. (2003) used 5 levels of rendering:

• From wire frame to radiosity (global illumination)• All subjects showed increased heart rate when approach

the precipice in all conditions• No significant differences between the

conditions in heart rate or reported presence.• So, from the point of view of presence

is visual realism not important?

Outline

• Introduction • Immersive VEs and Presence

• Real-time Ray Tracing and Presence• An experiment

• Virtual Light Field • Real-time global illumination for the Cave

• Conclusions

Does Ray Tracing Help?• What do we get with ray tracing?

• Shadows (umbras only)• Reflections

• Real-time ray-tracing (parallelism + GPU) offers chance to investigate impact of dynamic changes (shadows+reflections) on presence

Experimental Design• 33 people• Displayed with V8 HMD + polhemus

tracking• Pit room rendered in two ways

• Real-time Ray Tracing (RT) with reflections and shadows of crude virtual body

• Ray Casting (RC) ~ OpenGL shading (no reflections/shadows)

• Both between 13-15 fps

Experimental Design• Participants divided into two groups

• 16 experienced first RT and then RC• 17 experienced first RC and then RT• Both groups had relaxation periods for ‘baseline’

readings

• This design is both between groups• Consider 1st results only

• Within groups• Consider comparisons between the two results• (with high danger of interference between the two

experiences!)

Scenario:

VIDEO 1:12, 2:08

Measured Variables

• Standard presence questionnaire• Skin conductance• Heart rate and respiration• Demographic variables – age, gender• Background

• Programming knowledge, game playing, prior experience of VR, etc

Between Group Analysis

• Reported presence is significantly associated with:

• Ray tracing (higher than for RC)

• Prior VR experience

Within Group Analysis

Physiological Responses

• Participants experienced 3 parts to the experiment• A baseline period – just standing looking

around (2 minutes)• First exposure (RT or RC) (3 minutes)• Second exposure (RC or RT) (3 minutes)

• Baseline allows incorporation of natural differences between people

• Recorded electrodermal activity & ECG

Electrodermal Activity

• A change in electrical properties of the skin, associated with sweat, which typically indicates arousal.

• Arousal in response to events leads to skin conductance responses (SCR) (which also occur spontaneously)

• The number of SCRs increases with the level of arousal

• EDA sampled at 32Hz

Skin Conductance Responses

Example from one participant in the baseline period

SCR Between Groups Analysis

• The number of SCRs per unit time should follow a Poisson distribution (random events in time)

• If there is no difference in arousal between the 2 conditions then the means should be approx the same

• Can use log-linear regression to regress the number of SCRs on Condition + other ‘nuisance’variables

• Baseline number of SCRs• Gender, age, VR experience, game playing, etc..

Result• The analysis showed that the following variables were

significantly associated with number of SCRs• Baseline number of SCRs (+ve)• Age (+ve)• Gender (higher for females)• Prior exposure to VR (-ve)• Programming knowledge (+ve)• Subjective report of physiological responses (+ve)• Condition (RT higher than RC)

• Also there was a strong positive association with the questionnaire based presence score

• n = 29 (some Skin Conductance data not usable)

Interpretation

• Taking into account other factors• The evidence suggests that the RT condition led

to greater arousal than the RC condition• From this alone we cannot determine the

associated valence

ECG Wave Form

• ECG waveform during first few seconds of baseline for one participant

Heart Rate and HRV

• ECG recordings give the raw waveforms sampled at 256Hz

• Extract the QRS complexes from the raw signal

• Determine the RR intervals• Derive HR (beats per minute)• HR variability (various measures)

Heart Rate and HRV

• Interpretation – under conditions of stress• HRV should be lower• HR should be higher

• Simple overall measure is to take response as• Y = HR / SD(HR)• Higher values indicate lower HRV and/or

higher HR• We should see a difference between RT and RC

HR and HRV Analysis

• As before regress Y on • Baseline-Y• Condition (RT / RC)• Other ‘nuisance’ variables

• age, gender, previous VR, etc

Results HR and HRV

• The analysis showed that Y was associated with• Baseline-Y (+ve)• Gender (-ve)• Status (+ve for postgrads)• Condition (higher for RT compared to RC)

Overall Interpretation• Taking into account other factors

• Greater sympathetic nervous system arousal in RT condition

• RT condition had signs that would be associated with higher stress than RC condition

• RT associated with higher subjective presence score than RC

• Unknown whether this is the effect of dynamic shadows and reflections or the impact of overall greater realism

Outline

• Introduction • Immersive VEs and Presence

• Real-time Ray Tracing and Presence• An experiment

• Virtual Light Field • Real-time global illumination for the Cave

• Conclusions

Virtual Light Field

• The Virtual Light Field (VLF) is based on a discretisation of ray-space for representation and walkthrough of a virtual scene. It also uses this ray-parameterisation for propagation of radiance

• Global illumination solution for diffuse + specularsurfaces

• Easy extension for BRDFs

• Uses a partition of a hemisphere over a scene• Each ray direction spawns a field of parallel rays (PSF)

Parallel Subfields

• For any discretiseddirection, consider the set of rays parallel to it

• Parallel subfield of rays (PSF)

• Rays originate from a 2D grid perpendicular to the direction

Discretising Ray-space :Placing rays in space - Tiling

• Subdivide the grid of rays into Tiles

• Each Tile contains a number of Cells

• Rays (=Cells) in a Tile have coherence with respect to the objects they intersect

Discretising Ray-space :Ray-Polygon intersections

• Cells in a Tile represent parallel rays that intersect polygons in the scene

• These intersections for the Tile can be found by OpenGL rasterisation

• Tiles store a list of intersected polygons

• Neighbouring rays are likely to intersect the same polygons

Propagation and Rendering

• Starting from light sources energy is propagated through the scene

• At rendering time each ray segment has an associated radiance

• Can be rendered using much the same method as traditional light fields

• But now global illumination for virtual scenes

A Faster VLF

• The CPU implementation of the VLF could spend days propagating simple scenes (100s polygons)

• Due to clipping operations in tile-tile transfers• Also because of the O(n2) complexity using

OpenGL visibility determination

Point sampling

• The new approaches uses point sampling• Adaptively chosen sampling density

proportional to the projected area of a face onto the PSF plane

• This yields a speed-up of ~5x

Incremental visibility

• By pre-sorting polygons in the PSF directions, visibility can be pre-calculated

• The pre-sorting step employs a BSP subdivision step to optimise sorting

• For scenes >10,000 polygons the sorting step takes less time than a propagation iteration

• Incremental visibility yields a speed-up in excess of 10x (for a total of ~50x)

• Propagation scaling becomes O(n)

VLF-GPU

• Incremental visibility and adaptive point sampling have been ported to a GPU

• Uses a sparse tiled data structure suitable for the GPU

• Exploits fp16 textures and FBOs (framebufferobjects) for efficient render-to-texture functionality

• GPU offers a speed-up of ~10x (total: ~500x)

Scaling of propagation time with scene complexity

0

50

100

150

200

250

300

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Polygons

Tim

e(s

ecs)

Propagation times

GPU-VLF• Propagation times with >10K polygons

are minutes• Similar running times to Coombe’s* GPU

radiosity, but with specular reflections too… another example

• An XVR renderer is in the works for use with HMD and CAVETM

* Coombe, G.; Harris, M.J. & Lastra, A. “Radiosity on graphics hardware”GI '04: Proceedings of the 2004 conference on Graphics interface, Canadian Human-Computer Communications Society, 2004, 161-168

XVR• XVR is a platform independent virtual reality

system developed at the PERCRO lab in Pisa (PRESENCCIA Project)

• VLF being ported to run under XVR• Next experiments will be in the Cave with

capability of global illumination beyond ray tracing

• Unravel the issues about whether it is simply greater dynamism (shadows and reflections)

• Or overall more realism• www.vrmedia.it

XVR Demo

• Grotto demo• Bugatti

Outline

• Introduction • Immersive VEs and Presence

• Real-time Ray Tracing and Presence• An experiment

• Virtual Light Field • Real-time global illumination for the Cave

• Conclusions

Conclusion• Computer graphics is for … people• There is the aim for greater realism, faster speed,

better animations – the technical and engineering side.

• There is also the ‘so what?’ side – this is the scientific side

• What impact does what we do have on the people who use it.

• ‘Presence’ is one such issue (appropriate to immersive systems)

The Virtual Light Field Team

Pankaj Khanna Jesper Mortensen Insu Yu

Department of Computer Science, UCLvlfgroup@cs.ucl.ac.uk

Acknowledgements

• UK EPSRC Project “Presence in the Virtual Light Field” – for funding

• FET PRESENCCIA Integrated Project• Christoph Guger, Gtec, Austria, for help with

physiological signals analysis.