Interscopic User Interface Conceptsfor Fish Tank Virtual Reality Systems

Frank Stenicke∗ Timo Ropinski† Gerd Bruder‡ Klaus Hinrichs§

Visualization and Computer Graphics (VisCG) Research GroupDepartment of Computer Science

Westalische Wilhelms-Universitat Munster

ABSTRACT

In this paper we introduce new user interface concepts for fish tankvirtual reality (VR) systems based on autostereoscopic (AS) dis-play technologies. Such AS displays allow to view stereoscopiccontent without requiring special glasses. Unfortunately, until nowsimultaneous monoscopic and stereoscopic display was not possi-ble. Hence prior work on fish tank VR systems focussed either on2D or 3D interactions.

In this paper we introduce so called interscopic interaction con-cepts providing an improved working experience, which enablegreat potentials in terms of the interaction between 2D elements,which may be displayed either in monoscopic or stereoscopic,e.g., GUI items, and the 3D virtual environment usually displayedstereoscopically. We present a framework which is based on a soft-ware layer between the operating system and its graphical user in-terface supporting the display of both mono- as well as stereoscopiccontent in arbitrary regions of an autostereoscopic display. The pro-posed concepts open up new vistas for the interaction in environ-ments where essential parts of the GUI are displayed monoscop-ically and other parts are rendered stereoscopically. We addresssome essential issues of such fish tank VR systems and introduceintuitive interaction concepts which we have realized.

Keywords: fish tank VR, autostereoscopic displays, interscopicuser interfaces

Index Terms: H.5.1 [Information Interfaces and Presentation]:Multimedia Information Systems—Artificial, augmented and vir-tual realities; H.5.2 [Information Interfaces and Presentation]: UserInterfaces—Graphical user interfaces (GUI), Interaction styles

1 INTRODUCTION

In recent years virtual environments (VEs) have become more andmore popular and widespread due to the requirements of numer-ous application areas. Two-dimensional desktop systems are oftenlimited in cases where natural interfaces are desired, for example,when navigating within complex 3D scenes. In such cases virtualreality (VR) systems using tracking technologies and stereoscopicprojections of three-dimensional synthetic worlds support better ex-ploration of complex datasets. Although costs as well as the effortto acquire and maintain VR systems have decreased to a moder-ate level, these setups are only used in highly specific applicationscenarios within some VR laboratories. In almost each human-computer interaction process – even when 3D tasks have to be ac-complished, for instance, in CAD, CAE, or medical analysis – VR

∗e-mail: fsteini@math.uni-muenster.de†e-mail: ropinski@math.uni-muenster.de‡e-mail: g brud01@math.uni-muenster.de§e-mail: khh@math.uni-muenster.de

systems are rarely applied ([4]), even not by experts and least of allby ordinary users.

One reason for this is the inconvenient instrumentation requiredto allow immersive interactions in such VR systems, i.e., the user isforced to wear stereo glasses, tracked devices, gloves etc. Further-more the most effective ways for humans to interact with synthetic3D environments have not finally been resolved ([4, 8]). Deviceswith three or more degrees of freedom (DoFs) may provide a moredirect interface to 3D manipulations than their 2D counterparts, butusing multiple DoFs simultaneously still involves problems for bothdevelopers as well as users. And as a matter of fact 2D interac-tions are performed best with 2D devices usually supporting onlytwo DoFs ([14, 23]). Hence 3D user interfaces are often the wrongchoice in order to accomplish tasks requiring exclusively or mainlytwo-dimensional control ([4, 14]).

Because of their suitability for 2D and menu-based interactiontasks desktop systems with keyboard and mouse are the de factouser interface since long time. This user interface can even be usedto perform 3D interaction, for instance, via 3D widgets ([4, 10]).However, as mentioned before there is an obvious need for inter-acting with 3D datasets in a more intuitive way than supported bystandard desktop-based environments ([9]). Most software systems,in particular 3D modeling applications, but also scientific applica-tions, include 2D user interface elements, such as menus, texts andimages, in combination with 3D content. While 3D content usu-ally benefit from stereoscopic visualization providing an improveddepth perception, 2D GUI items often do not require stereo display.Therefore, interactions between monoscopic and stereoscopic ele-ments, which we refer to as interscopic interactions, have not beeninvestigated with special consideration of the interrelations betweenthe elements.

For the combination of 2D and 3D content, fish tank VR systems([31]) support both desktop-based as well as immersive interaction.These systems enhance desktop systems via a stereoscopic displayon a conventional display. For immersive interaction head track-ing is used to realize head coupled perspective stereoscopic pro-jection. Furthermore, 3D input devices are available for intuitiveinteraction. Mulder et al. have stated the following benefits of fishtank VR systems ([18]). In comparison to large projection-basedVR systems, e.g., CAVE ([11]), fish tank VR systems require lesscomponents, i.e., they are easier to set up, calibrate, and transport.Moreover, fish tank VR systems can provide many pixels per de-gree in the viewers field of view (FoV). Since fish tank VR systemsrequire less specialized hardware, these systems are less expensivein both initial costs and maintenance. Another important factor isversatility, which means that smaller fish tank VR systems can beused as ordinary desktop computer systems, larger ones as displaymedia for presentations. Thus fish tank VR systems have the po-tential to become more common in the future. Because of theirdesktop-based nature fish tank VR systems can be interfaced withstandard desktop devices such as mouse and keyboard. Of course,also 3D input devices, e.g., tracked gloves and space mouse, areusable in fish tank VR systems. For example, pointing devices can

27

IEEE Virtual Reality Conference 2007March 10 - 14, Charlotte, North Carolina, USA1-4244-0906-3/07/$20.00 ©2007 IEEE

be used to control the mouse cursor on the screen surface of thedisplay by projecting the device’s positional and orientational datato 2D positions on the screen ([32]).

For a long time fish tank VR systems had the same problems asprojection-based VR, i.e., users were forced to wear stereo glassesand head trackers. On current autostereoscopic (AS) displays userscan see 3D data without wearing any instruments, because bothstereo half images are spatially separated; the corresponding halfimages are projected to the correct eye using, for instance, lenticu-lar rasters or LCD barriers ([12]). Thus the user is able to perceivea stereoscopic image in a fixed area called sweet spot. When theAS display features a head tracker in order to rearrange the rasterrespectively barrier, or when multiple sweet spots are supported,the user can even move in front of the display. Unfortunately, theseparation of the stereo half images influences viewing of mono-scopic content in such a way that the most essential elements of theGUI, such as menus, images or texts, are distorted. Although thereare some displays allowing to switch off the LCD barriers in orderto display monoscopic content on the display, until now it was notpossible to display monoscopic and stereoscopic content simultane-ously. Hence simultaneous viewing is possible only by using an ad-ditional regular display to show the monoscopic content. But onlyfew applications support rendering of a stereoscopic window on adifferent display. Nevertheless, problems arise from decoupling in-teraction and visualization; interactions with 2D GUI elements haveto be performed on the 2D screen, whereas the stereoscopic visual-ization have to be viewed on a different display.

Although, current stereo-in-a-window ([6, 29]) systems showstereoscopic content in one window time-sequentially or using fil-tering techniques, these visualizations are restricted to only onerectangular window, while stereoscopic glasses are still required.The interaction with stereoscopic content using two-dimensionalstrategies involves further problems, for example, monoscopic rep-resentation of the mouse cursor disturbs stereoscopic perception,and precise interactions, for example with widgets or handles, arenot possible. For these reasons, most prior work on fish tank VRsystems either focussed on 2D or 3D interactions only.

In this paper we introduce new user interface strategies for fishtank VR systems using AS display technologies. The concepts arebased on a framework implemented as a software layer betweenthe operating system (OS) and its graphical user interface. Henceit is possible to display arbitrary shaped areas of the GUI eitherin a monoscopic or in a stereoscopic way. We address the essentialissues of such fish tank VR systems and explain intuitive interactionparadigms which we have developed on top of the framework.

The remainder of this paper is structured as follows. Section 2summarizes work related to our concepts. Before we introduce themain interscopic interaction concepts, we briefly describe imple-mentation details and technical aspects of the framework in Sec-tion 3. In Section 4 we present interscopic interaction conceptswhich address problems of the interaction process in the describedfish tank VR setup. Section 5 concludes this paper and gives anoverview about future work.

2 RELATED WORK

In the past much work has been done in order to provide hardware-based approaches for the interaction in fish tank VR system en-vironments. In 2000, the mUltimo3D group at the Heinrich-Hertz-Institute in Germany built an AS display system consisting of a gazetracker, a head tracker and a hand tracker. These were combinedwith an autostereoscopic display for viewing and manipulating ob-jects in 3D ([16]). The head tracker gives the user a look-aroundcapability, while the gaze tracking activates different applicationson the desktop. The hand tracker enables the user to navigate andmanipulate objects in 3D space. Similar approaches support nat-ural interactions with stereoscopic content by tracking the users

hand and fingers with magnetic fields ([30]) or optical-based solu-tions ([3]). These systems rather address tracking technologies forfish tank VR systems than advanced interactions or visualizationsin these environments.

In recent years, many software systems have been proposedwhich extend existing 2D desktop environments to so called 3Ddesktops. These approaches provide a virtual 3D space in which 2DGUI elements such as windows are replaced by three-dimensionalrepresentations ([27, 20, 1, 21, 2]). Hence more space is available todisplay certain information. Although these environments providea fancy visualization, it has not been investigated in how far theyimprove the interaction process, since they force the user to per-form 3D interactions where 2D interactions are intended. Currentdevelopment indicates that also graphical user interfaces for operat-ing systems, which are essentially of a two-dimensional nature willevolve to 3D and include depth information.

At present concepts for hardware-based approaches have beenproposed to display monoscopic and stereoscopic content simul-taneously on one AS display ([24]). However, interaction conceptshave not yet been developed for these displays. Because of the enor-mous costs these devices are not commercially available at presentand do only exist as prototype solution.

Due to the lack of simultaneous display of monoscopic andstereoscopic content most interaction approaches only propose im-provements for interactions either in 2D using monoscopic displayor in 3D using stereoscopic display, but they do not combine bothworlds. There are some VR approaches which examine hybrid in-terfaces combining 2D and 3D interaction using different displayor interaction technologies ([5, 28]). For example, Benko et al.propose in [5] techniques to grab monoscopically displayed objectsfrom a projection screen in order to view them stereoscopically us-ing a head mounted display (HMD). However, an instrumentationof the user is still required to allow this gestural-based interaction.

The aim of our research is not to debate the validity of desktop-based interaction concepts nor VR-based interaction technologies,but to explore in how far these concepts can be combined in order toprovide new user interfaces and paradigms for fish tank VR system.

3 INTERSCOPIC USER INTERFACE FRAMEWORK

To provide a technical basis for the visualization and interactiontechniques introduced later on in this paper, we briefly discuss theimplementation of our interscopic user interface framework ([22]).Though the concepts can be easily transferred to virtually any cur-rent graphics desktop system, our prototype implementation hasbeen developed for the windows operating system. It deals witharbitrary 3D applications either driven by OpenGL or DirectX. Toallow simultaneous viewing we need to modify the monoscopiccontent in order to make it perceivable on AS displays and gen-erate a stereo pair out of the 3D content. Since these are diverseimage processing operations first we have to separate the 2D fromthe 3D content. To achieve this separation, our technique acts asan integrated layer between the rendering application and the OS.By using this layer we ensure that the operating system takes careabout rendering the 2D GUI elements in a native way (see Figure 1(step 1)).

In the following subsections we first describe how to appropri-ately process 2D and 3D content in a graphics desktop environment,before explaining how to merge these for the display on various ASdisplays.

3.1 Processing 2D ContentTo achieve a correct display of 2D content on AS displays it hasbeen processed with respect to the corresponding structure, e.g., alenticular raster, attached to the panel. The easiest case is preparing2D content for vertical interlaced AS displays ([12]), which have avertically oriented prism raster attached in front of the LCD panel.

28

application

+ objects+ models+ curves+ ...

3DGUI

+ menus+ icons+ texts+ ... G

UI

3.

operating system

GUI engine 1. rendering GUIscaling

image warping & interlacing

GU

I

2.

Figure 1: Illustration of the interscopic user interface framework showing 2D and 3D content simultaneously.

This prism raster operates as a beam splitter and ensures that thepixels displayed in each odd column are seen by the user’s left eye,while the pixels displayed in each even column are perceived withthe right eye. Usually images viewed on vertical-interlaced AS dis-plays are viewed, such that the two half images (in the odd resp.even columns) show the same scene but with a binocular dispar-ity resulting in the stereoscopic effect. However, when viewing 2Dcontent, e.g., the desktop, the two separated images perceived bythe eyes do not match, which leads to an awkward viewing experi-ence. To make this content perceivable we have to ensure that theleft and the right eye perceive the same information, resulting ina flat two-dimensional image embedded in the image plane some-times referred to as focal plane ([7, 6]). To achieve this effect withvertical-interlaced AS displays the image has simply to be scaledin the horizontal direction by a factor of two (see Figure 1 (step2)). This scaling ensures that on both the odd and the even columnsthe same information is displayed. To support other AS displays,e.g., multiple view systems ([13]) which need up to nine differentviews of the same scene, a corresponding image processing patternhas to be applied. However, since all AS displays have to displaymore than one view on the screen and we want the user to perceivethe same image regardless of the user’s position, the operating sys-tem has to render the 2D content with a resolution being lower thanthe original screen resolution of the AS display. In case n differentviews can be displayed by an AS display, we have to render only1n th of the pixels needed for the current resolution. Thus we exploit

a virtual desktop having 1n th the resolution of the native AS display

resolution. In case of vertical-interlaced AS displays this is a virtual

desktop having 12 the resolution of the AS display (see Figure 1).

Resolution / Interpolation Evaluation summary

800×600 4.01024×768 bilinear 3.241024×768 bicubic 3.261280×960 bilinear 1.921280×960 bicubic 1.71

1600×1200 1.0

Table 1: ASD setups for the reading task evaluation.

We have performed a usability study in order to evaluate to whichdegree we can scale the resolution of the virtual desktop. If theresolution is more the half of the resolution of the target display,scaling the content yields slightly different information for both halfimages. However, since differences in both images are marginal,the human vision system can merge the information to a final imagewhich can be viewed comfortably by the user.

We have performed several reading tasks on an AS display us-ing different screen resolutions for the virtual display and different

interpolation methods when scaling the content, i.e., bilinear vs.bicubic. The users had to reveal the quality, depth perception, eye-strain and further properties on a five-point Likert scale, where 1corresponds to a negative evaluation, while 5 corresponds to thepositive one. As expected users reveal the quality of the displaybest for the 800×600 resolution and worst for the native resolutionof the display. However, because the 800×600 resolution restrictsthe provided display size, we decided to apply the 1024×768 res-olution for the virtual display, which yields in the same resolutionfor the 2D information after the merging process. Since the bicubiconly yields slightly better results, while requiring more calculationresources we apply the bilinear interpolation.

To exploit this virtual desktop we had to develop an appropri-ate display driver. This display driver ensures that the windows OSannounce an additional monitor with the necessary resolution andmirrors the desktop content into this screen. For implementing thisdisplay driver we have used the driver development kit (DDK) pro-vided by Microsoft.

The overall process for preparing 2D content is depicted in step1 and step 2 shown in Figure 1. To allow proper viewing of 2Dcontent on the AS display, we render these elements into the virtualdesktop having at least a quarter of the resolution of the AS display(step 1). Later on the generated image is scaled uniformly by acorresponding factor before it is written to the main desktop (step2). As mentioned above, we could also have used a virtual desktophaving, for instance, half the resolution and scale the result non-uniformly by a factor of two along the horizontal axis.

In cases where a different AS display is used, the content cannotbe simply scaled but it has to be displayed according to the lensesraster. This can be done easily using our approach since, the layerenables control about each pixel usually displayed by the OS. Sincethis is an operation being computationally slightly more complexthe frame rate decreases, though allowing still real-time renderingand processing of the content.

In addition to adapt the content to be perceived on the AS dis-play, further image processing techniques can be applied to the GUIelements in order to enhance perception and interaction behaviouras well as supporting an improved working experience. These tech-niques are described in detail in Section 4.

3.2 Generating Stereoscopic ImagesSince only a few 3D applications natively support stereoscopicviewing on certain AS displays, in most cases we have to adapt alsothe 3D content in order to generate stereocopic images. There aretwo techniques for making an existing 3D application stereoscopic.The first one is to trace and cache all 3D function calls and executethem twice, once for each eye. Despite of some issues with contextchanges this works properly for many applications; we have tested,e.g. 3D Studio Max, Maya, Cinema4D, AVS Express, Google Earthetc. However, applications making heavy usage of multipass ren-

29

(a) (b)

Figure 2: Google Earth example with (a) original depth values havinga bad depth separation and (b) depth values after applying the rangetransformation.

dering were problematic in this context. For some of the conceptsdescribed later in Section 4 we have used this approach. In addi-tion we decided to use image warping techniques ([17]) to generatestereoscopic images (see Figure 1 (step 3)). This technique per-forms a reprojection of the monoscopic image depending on thevalues stored in the depth buffer. Image warping has the drawbackthat not all of the scene content visible from both eyes is presentin a single monoscopic image. Especially planes almost perpen-dicular to the view direction suffer from this effect. However, weuse a simple pixel filling approach similar to the one described in([15]). This is a scanline based image processing, which can beintegrated directly into the image warping technique in order to en-hance the performance. In each half image we set the color at apixel position for which no sufficient information is available to thelast proper color encountered on the scan line. In practice this leadsto very convincing results, were the errors cannot be spotted any-more. Most images shown in this paper have been generated usingthis technique.

Since image warping takes into account the depth values of ascene, the image quality is directly dependent on the depth distri-bution. With existing 3D applications this becomes a problem es-pecially when depth values are not equally distributed in the depthbuffer. Figure 2 shows a screenshot of Google Earth with the ac-companying depth image as well as two depth histograms generatedby our application. In the left depth histogram it can be seen thehead-up-display (HUD) elements at the lower-left and the upper-right are in terms of depth values far in front of the rest of the scene.Since this would lead to a stereoscopic image with a deep overalldepth but with relatively flat scene objects, we apply a range trans-formation and obtain the depth histogram shown in Figure 2 (b).This transformation is similar to a grey value transformation as usedin image processing. We analyze the depth value distribution, iden-tify peaks and stretch them to spread the range form an offset valueclose to the near clipping plane till the far clipping plane. As it canbe seen the range transformation can be modified using the GUI.This allows for example to eliminate the depth interval between ob-jects, or make adjacent object even overlap in depth. Additionally,the stereo effect can be improved by interactively adapting the eyeseparation and the focal length, i.e., the distance from the user tothe focal plane. The adapted image is illustrated in Figure 3.

Similar to the preparation of the 2D content the stereoscopy ef-fect can also be applied when using virtually arbitrary AS displays.Only the pattern into which the half images are rendered has to beadapted. For vertical interlaced AS displays this is the typical stripepattern, while for lenticular multiple view displays a more complexpattern has to be applied ([13]). With more sophisticated displayshaving an own image processor, e.g., the Philips 42-3D6C01 ([19]),we simply transmit a monoscopic image with accompanying depthinformation and let the display generate the appropriate half im-ages.

3.3 Merging Mono- and Stereoscopic ContentTo separate the 2D and the 3D content, we have to know whichwindow areas are used for stereoscopic display. This can be eitherdetermined manually or automatically. When using the manual se-lection mechanism, the user is requested to add a 3D window or aregion and selects it to be displayed stereoscopically with the mousecursor. In contrast to display the entire screen stereoscopically thishas the benefit, that not all 3D windows are used to generate stereo-scopic images. Manual selection is beneficial, for example, whenhaving a 3D modeler providing multiple 3D views showing an ob-ject from different locations. In this case one may only want tohave the perspective 3D view stereoscopic but not the orthographicside views. Furthermore, as described in Section 4.2.3 arbitrary 3Dobjects can be rendered stereoscopically.

The final merging step of 2D and 3D content is depicted by step3 in Figure 1. An obvious problem arises, when 2D and 3D contentareas overlap each other. This may happen when either a pull-downmenu, a context menu or the mouse cursor overlaps a 3D canvas.In this case the separation cannot be performed on the previous3D window selection process only. To properly render the mousecursor, context menus and pop-up windows which may appear ontop of the 3D canvas we apply a masking technique. This is forexample important, when dealing with 3D graphics applications,whereas context menus provide convenient access to important fea-tures. When merging 2D and 3D content the mask ensures that onlythose areas of the 3D window are written to the display, which arenot occluded by 2D objects.

Figure 3: Example screenshot of Google Earth application and theinterscopic user interface framework on a vertical interlaced AS dis-play.

Figure 3 shows a resulting image in the described interlacedstereoscopic mode. The content of the large window is shown instereoscopic. In addition we have assigned depth values to bothwindows, which makes them to appear at different distances to theuser (see Section 4.1.1). The task bar and the desktop with its iconsare rendered monoscopically, i.e., they have a zero parallax.

4 INTERSCOPIC INTERACTION CONCEPTS

When considering factors that have led to the widespread accep-tance of conventional display systems, the synergy between the dis-play and the interaction devices and the corresponding techniquesis essential. Desktop systems have shown their benefits for manyproblem domains including 2D as well as 3D interaction tasks.Hence there is no reason to throw out 30 years of 2D interface

30

(a) (b)

(c) (d)

Figure 4: Autostereocopic images showing desktop with Maya modeling application (a) for the left and (b) for the right eye respectively a desktopwith the Google Earth application (a) for the right eye and (b) for the left eye. To perceive a stereoscopic effect the images can be viewed from adistance of approximately 5 inch with eyes focusing at (top) infinity respectively (bottom) crossed eyes.

work. However, as mentioned before fish tank VR systems haveproven the capability to increase the interaction performance and,moreover, they have the potential to be accepted by users as newuser interface paradigm for specific tasks as well as for standarddesktop-based interactions.

Nevertheless it is important to address the benefits of the pre-sented fish tank VR setup and to further enhance the interac-tion with novel concepts in such a way that the system providesa demonstrable improvement in comparison to using an ordinarydesktop system. Hence we propose concepts which provide a focusof attention and a user-centered guidance model, which supportsthe user during an interaction process. We enable the user to focuson the current task and provide affordances which guide the userduring the interaction. For this purpose we use the functionalitiesof the software framework presented in Section 3 that allows us topresent aribtrary elements of the user interface either in a mono-scopic or stereoscopic way.

For the design of interscopic interaction techniques, we have for-mulated a set of rules to be observed in order to provide intuitiveand helpful interaction paradigms. First of all we decide to maintain2D tasks in 2D, we provide and use depth information only whennecessary or when it supports the user. This includes to keep 2Dwindows aligned to the view plane because we believe that thereis no need for changing the orientation of windows which wouldresult in disturbed perception of data contained in the window, e.g.,texts or images. Affordances already provided by the graphical userinterface, for instance, by buttons or sliders, should be emphasizedin order to guide the user. Moreover, the user’s attention should befocused on the desired tasks as well as on important system infor-mation. Finally we want to use the possibilities of our concepts toallow a better immersion and fusion between the user interface andthe real world using novel visual approaches.

4.1 Visual EnhancementsWe introduce several visual enhancements in order to guide the userduring the interaction. Hence it is easier for the user to focus onrelevant aspects or certain regions of the screen and to handle ap-plication or system specific tasks.

4.1.1 Window ArrangementWhen interacting either with an application consisting of multi-ple windows or when working with several applications simulta-neously, the desktop includes numerous windows, which partiallyor completely overlap. 3D desktop replacements ([27, 20, 1, 21, 2])try to address the complex representation of this situation by di-minishing the limiting space of the desktop by means of providinga virtual 3D space to arrange windows. For example, the windowsmay be arranged on a spherical shape, oriented freely or they maybe located behind each other. As mentioned above, we believe thatit is important to maintain the orientation of the windows alignedto the desktop surface in order to have a sufficient viewing quality.Moreover this constrain helps to reduce the DoFs and supports thearrangement process ([26]).

However, the windows have a priority given by the system orthe application in terms of the sequence in which they have to beprocessed; a system alert message box, for instance, has highest pri-ority. Furthermore the user can implicitly assign the priority whenaccessing the windows, i.e., the longer a window has not been inthe focus the lower gets its priority. We use this information andarrange these windows with respect to this priority. Hence a cer-tain depth is assigned to each window so that it is arranged in astereoscopic way such that the first level window is the one whichappears closest to the user; the user can optionally define how depththe windows should be arranged. In addition we add shadows to allwindows to further increase the depth perception.

31

Figure 5: Interscopic user interface with several mono- as well asstereoscopic windows arranged with different depths. The image isrendered in red-cyan anaglyph mode.

The top figure of Figure 4 shows two example stereoscopic halfimages of the 3D modeling software Maya for the (a) left eye and(b) right eye with several windows arranged after each other. In thebottom of Figure 4 the Google Earth application is illustrated with(a) the image for the right eye and (b) the image for the left eye. Themodel inside the main window of the Maya application as well asthe terrain model inside the Google Earth window are also renderedstereoscopically. In addition, the images show the user interface ofthe software in which we have integrated the concepts presented inthis paper. Figure 5 shows an anaglyph version of the interscopicuser interface with several mono- as well as stereoscopic windowsarranged with different depths.

The opportunity to change the depth of windows allows the useralso to change the depth of each window manually by using themouse wheel. This can be exploited for example to examine thecontent of a window. Usually, in ordinary 2D desktop environ-ments if the content of a window cannot be scaled the user wouldapproach the head to the screen in order to investigate the data. Asmentioned in Section 1 when using an AS display this is limitedbecause the user has to stay in the sweet spot in order to maintain acorrect stereoscopic image without any interference. Thus insteadof moving the head to the display towards the window, the windowcan be moved from the display towards the direction of the head.

Alternatively, all windows belonging to the same applicationmay be displayed stereoscopically with the same depth. This sup-ports the user especially if several applications are active simulta-neously. Switching to another application corresponds to interactwith the windows at a different depth.

4.1.2 3D GUI Items

In a graphical user interface the usage of affordances showing theuser how to interact with certain objects is essential when providingan intuitive interface ([25]). We use the presented technology toemphasize existing affordances and to introduce new ones. GUIitems such as buttons etc. – even when 2D – already have a three-dimensional appearance provided by borders and shadows. In thecontext of an OS these items are interpreted as windows, which canbe further emphasized by using stereoscopic representation for thecorresponding GUI elements. Thus, buttons appear as 3D objectsattached to the display’s surface indicating the user to press them.

When the user moves the mouse cursor over the shape of the in-

(a) (b) (c)

Figure 6: 3D depth cursor moved over two windows.

teraction handle, e.g., a button, the mouse cursor is also renderedstereoscopically with the corresponding depth of the underlyingGUI element (compare to Section 4.2.1). Furthermore we use an-other effect to emphasize an affordance; we decrease the stereo-scopic effect of the button which is located under the mouse cursorindicating an already initiated press action which further shows theuser the affordance of a possible clicking at the button.

4.1.3 Window Extending Stereoscopic ContentCurrent stereo-in-a-window solutions based on time-sequentialstereoscopy have the drawback, that stereoscopic content extendingthe window size gets cut-off, resulting in a distorted stereoscopicimage. Thus, the stereoscopic effect gets lost and the immersiongets disturbed. With our framework we can hijack each stereo con-tent based on display lists and render this information anywhereon the display either monosopically or in stereoscopically. Thiscan be done since we can control each pixel of the user interfaceafter the described capturing process. Thus, we are able to gener-ate a stereoscopic window in which the 3D content can extend thegiven window edges, such that the objects appear to come out ofthe window and appear ahead of monoscopic content outside thestereoscopic window such as the 2D desktop background with itsicons etc. To further increase this effect we are also able to rendershadows onto the monoscopic areas behind the stereoscopic objectswhich extend the window edges. Moreover the stereoscopic objectscan be dragged out of a stereoscopic window and they can be placedanywhere, in particular on the monoscopic desktop, while still ren-dered stereoscopically.

4.2 Interscopic InteractionsIn the previous section we have pointed out some concepts how weemphasize affordances using visual enhancements in order to im-prove interscopic interaction and to advance the experience whenworking in fish tank VR systems. In the following subsectionswe consider interscopic interaction techniques which support theuser when interacting with 2D and 3D content. It has been shownthat using 3D interaction paradigms for inherently 2D interactiontasks is disadvantageous ([10, 14]). The more the user has to con-trol, the more the user has to specify and think about the corre-sponding tasks. Therefore we constraint the interactions to thosewhich are similar to the two-dimensional proceedings when inter-acting in desktop environments. The main interface for 2D interac-tion is the mouse cursor which may be interfaced with an ordinarymouse, a space mouse or tracked gloves by projecting the perceivedDoFs data to the AS display’s surface by means of ray-casting ap-proaches. Application specific virtual input devices such as a virtualhand are controlled depending on wether the developer has imple-mented the interface within a corresponding 3D application.

4.2.1 3D Depth CursorAn often named drawback, when interacting with stereoscopicrepresentations using desktop-based interaction paradigms is the

32

monoscopic appearance of the mouse cursor, which disturbs theperception of the stereoscopic scene. Therefore we provide twodifferent strategies for displaying the mouse cursor. The first oneexploits a 3D mouse cursor which hovers over both 3D objectswithin stereoscopic windows as well as over 3D GUI items as men-tioned above. Thus the mouse cursor is always visible at top ofthe objects’ surface. When moving the cursor over the surface ofa three-dimensional object, the user gets an improved perceptionof the depth of such an object serving as an additional shape cue.The alternative is to display the cursor always at the image plane.In contrast to ordinary desktop environments the mouse cursor getsinvisible when it is obscured by another object extending out of thescreen. Thus the stereoscopic impression is not disturbed by themouse cursor, indeed the cursor is hidden during that time.

Figure 6 shows a sequence of stereoscopic half images of amouse cursor that is moved along the surface of two windows hav-ing different depths. In Figure 6 (b) a shadow of the window withlower depth values can be seen on the mouse cursor which is ar-ranged behind the window.

4.2.2 Monoscopic Interaction lens

Many two-dimensional as well as three-dimensional applicationsprovide interaction concepts which are best applicable in two di-mensions using 2D interaction paradigms. One example, are 3Dwidgets ([10]) which reduce degrees of freedom manipulated si-multaneously. Since these interaction paradigms are optimizedfor 2D interaction devices and monoscopic viewing we propose amonoscopic interaction lens, through which two-dimensional inter-actions can be performed without loosing the entire stereoscopiceffect. Therefore we attach a lens at the position of the mouse cur-sor. The content within such an arbitrary lens shape surrounding themouse cursor is projected at the two-dimensional plane defined bythe lens shape. The user can change the depth of the lenses content,which is usually displayed with zero parallax, i.e., at the image orfocal plane. Thus the user can focus on the given tasks and tools toperform 2D or 3D interactions in the same as done on an ordinarymonoscopic display. This can be used to read text on a stereoscopicobject, or to interact with 3D widgets often used in numerous 3Dapplications.

Figure 7: Monoscopic interaction lens used to enable interaction witha 3D translation widget in 3D Studio Max

Figure 7 shows a monoscopic interaction lens used to controla 3D translation widget of 3D Studio Max. The content withinthe disc shaped lens centered around the mouse cursor is renderedmonoscopically, while the rest of the 3D model is rendered stereo-scopically.

4.2.3 Stereoscopic LensThe opposite approach to the monoscopic interaction lens describedin the previous section is to render objects within a specific lens inthe stereoscopic mode. We use this feature in the following way.The user can activate a stereoscopic rendering for arbitrary objectsby clicking at them when this functionally is activated. Afterwardsthe corresponding object is rendered stereoscopically while otherobjects within the same window stay in a monoscopic render mode.Furthermore the user can change the stereo parameters in order toimprove the exploration of desired objects.

Figure 8 (a) shows a set of virtual 3D objects in a Maya scene,each rendered monoscopically. After the user has select the secondobject from the left, only this object is rendered stereoscopicallyand the user can adjust the stereoscopic settings using the GUI ofour framework. Thus, this object of interest is emphasized and inthe focus of the user’s attention.

5 CONCLUSION AND FUTURE WORK

In this paper we have introduced interscopic interaction conceptsfor fish tank VR systems based on AS display technologies and wehave described technical aspects about the interscopic user interfaceframework. The proposed techniques have proven the capabilityto increase the interaction performance in such setups. Moreover,these strategies have the potential to be accepted by users as newuser interface paradigm for specific tasks as well as for standarddesktop-based interactions since they enhance the working experi-ence, in particular, when simultaneous interaction with monoscopicand stereoscopic content is intended.

Although, we have not performed a usability study for the inter-scopic interaction concepts, which has to be conducted in the futurework, we have surveyed subjects which have tested the interscopicuser interface. The results of the survey indicate that the subjectsare highly motivated to use the described fish tank VR system, sinceas they remarked an instrumentation is not required and the userslike the experience of using the system. In particular, they evaluatedthe window arrangement as very helpful when they need to identifythe active window. The usage of the monoscopic interaction lenshas been revealed as very useful because the subjects prefer to in-teract in a way which is familiar for them from working with anordinary desktop system.

In the future we will perform a usability study in which we com-pare how effective the described concepts are in comparison to theusage of an ordinary system, for example, when performing sev-eral 3D modeling tasks. We will integrate further functionalityand visual enhancements using more stereoscopic and optionallyphysics-based motion effects. Moreover, we will examine furtherinterscopic interaction techniques, in particular, for domain-specificinteraction tasks, and evaluate in how far they improve the interac-tion in fish tank VR systems.

ACKNOWLEDGEMENTS

The authors wish to acknowledge the Stanford 3D scanning reposi-tory for providing the 3D bunny and dragon datasets. Furthermore,Autodesk deserves thanks for providing free evaluation versions ofthe applications to test the concepts proposed this paper.

REFERENCES

[1] 3DNA. http://www.3dna.net/.

[2] A. Agarawala and R. Balakrishnan. Keepin’ It Real: Pushing the

Desktop Metaphor with Physics, Piles and the Pen. In Proceedings

33

(a) (b)

Figure 8: 3D Maya scene with (a) four objects rendered monosopically and (b) after selecting one object this object is rendered stereoscopically.

of the SIGCHI conference on Human Factors in computing systems,

pages 1283–1292, 2006.

[3] Z. Y. Alpaslan and A. A. Sawchuk. Three-Dimensional Interaction

with Autostereoscopic Displays. In A. J. Woods, J. O. Merritt, S. A.

Benton, and M. R. Bolas, editors, Proceedings of SPIE, StereoscopicDisplays and Virtual Reality Systems, volume 5291, pages 227–236,

2004.

[4] R. Balakrishnan. A Grant of 3D. Keynote speach Symposium on 3D

User Interfaces, 2006.

[5] H. Benko, E. W. Ishak, and S. Feiner. Cross-Dimensional Gestural

Interaction Techniques for Hybrid Immersive Environments. In Pro-ceedings of the Virtual Reality, pages 209–216. IEEE, 2005.

[6] P. Bourke. Autostereoscopic Lenticular Images

(http://local.wasp.uwa.edu.au/ pbourke/stereographics/lenticular/).

[7] P. Bourke. Calculating Stereo Pairs

(http://astronomy.swin.edu.au/ pbourke/stereographics/stereorender/),

1999.

[8] D. Bowman, E. Kruijff, J. LaViola, and I. Poupyrev. 3D User Inter-faces: Theory and Practice. Addison-Wesley, 2004.

[9] W.-S. Chun, J. Napoli, O. S. Cossairt, R. K. Dorval, D. M. Hall, T. J. P.

II, J. F. Schooler, Y. Banker, and G. E. Favalora. Spatial 3-D In-

frastructure: Display-Independent Software Framework, High-Speed

Rendering Electronics, and Several New Displays. In A. Woods, M. T.

Bolas, J. O. Merritt, and I. E. McDowall, editors, In Stereoscopic Dis-plays and Virtual Reality Systems XII, volume 5664, pages 302–312,

2005.

[10] D. B. Conner, S. C. Snibbe, K. P. Herndon, D. C. Robbins, R. C.

Zeleznik, and A. van Dam. Three-Dimensional Widgets. In Sympo-sium on Interactive 3D Graphics, 1992.

[11] C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. Kenyon, and J. C. Hart.

The CAVE, Audio Visual Experience Automatic Virtual Environment.

Communications of the ACM, pages 64–72, June 1992.

[12] N. A. Dodgson. Autostereoscopic 3D Displays. In Computer, vol-

ume 38, pages 31–36, 2005.

[13] N. A. Dodgson, J. P. Moore, and S. P. Lang. Multi-View Autostereo-

scopic 3D Display. In International Broadcasting Convention, pages

497-502, 1999, 1999.

[14] A. J. Hanson and E. Wernert. Constrained 3D Navigation with 2D

Controllers. In Proceedings of Visualization ’97, pages 175–182.

IEEE Computer Society Press, 1997.

[15] P. Kozankiewicz. Fast Algorithm for Creating Image-based Stereo

Images. In Proceedings of WSCG, pages 59–66, 2002.

[16] J. Liu, S. Pastoor, K. Seifert, and J. Hurtienne. Three Dimensional

PC toward novel Forms of Human-Computer Interaction. In Three-

Dimensional Video and Display Devices and Systems SPIE, 2000.

[17] W. R. Mark. Post-Rendering 3D Image Warping: Visibility, Recon-struction, and Performance for Depth-Image Warping. PhD thesis,

University of North Carolina at Chapel Hill, 1999.

[18] J. D. Mulder and R. van Liere. Enhancing Fish Tank VR. In Proceed-ings of Virtual Reality, pages 91–98. IEEE, 2000.

[19] Philips 42-3D6C01. (http://www.inition.co.uk).

[20] Project Looking Glass. (http://www.sun.com/software).

[21] G. Robertson, M. van Dantzich, D. Robbins, M. Czerwinski,

K. Hinckley, K. Risden, D. Thiel, and V. Gorokhovsky. The Task

Gallery: A 3D Window Manager. In Proceedings of the SIGCHIconference on Human factors in computing systems, pages 494–501,

2000.

[22] T. Ropinski, F. Steinicke, G. Bruder, and K. Hinrichs. Simultaneously

Viewing Monoscopic and Stereoscopic Content on Vertical-Interlaced

Autostereoscopic Displays. In Poster-Proceedings of the 33rd Interna-tional Conference on Computer Graphics and Interactive Techniques(SIGGRAPH06), 2006.

[23] T. Salzman, S. Stachniak, and W. Sturzlinger. Unconstrained vs. Con-

strained 3D Scene Manipulation. In 8th IFIP International Confer-ence on Engineering for Human-Computer Interaction, volume 2254,

2001.

[24] SeeReal Technologies. Next Generation Display Technology

(http://www.seereal.com/en/products/nextgen/nextgen.pdf).

[25] B. Shneiderman. Designing the User Interface: Strategies for Effec-tive Human-Computer Interaction. Addison-Wesley, 1997.

[26] G. Smith, T. Salzman, and W. Sturzlinger. 3D Scene Manipulation

with Constraints. In B. Fisher, K. Dawson-Howe, and C. O’Sullivan,

editors, Virtual and Augmented Architecture, pages 35–46, 2001.

[27] SphereXP. (http://www.spheresite.com).

[28] Z. Szalavari and M. Gervautz. Using the Personal Interaction Panel

for 3D Interaction. In Proceedings of the Conference on Latest Resultsin Information Technology, page 36, 1997.

[29] H. Tramberend. A Display Device Abstraction for Virtual Reality

Applications. In Proceedings of Afrigraph, pages 75–80, 2001.

[30] C. van Berkel. Touchless Display Interaction. In SID 02 DIGEST,

2002.

[31] C. Ware, K. Arthur, and K. Booth. Fish Tank Virtual Reality. In

Proceedings of CHI, pages 37–42, 1993.

[32] C. Ware and K. Lowther. Selection using a One-Eyed Cursor in a Fish

Tank VR Environment. Transactions on Computer-Human Interac-tion, 4(4):309–322, 1997.

34