Using Tweek to Create Graphical User Interfaces in Virtual Reality

  • View
    18

  • Download
    0

Embed Size (px)

DESCRIPTION

Using Tweek to Create Graphical User Interfaces in Virtual Reality. Patrick Hartling IEEE VR 2003. Overview. Problem description Goals for Tweek software Design overview Implementation overview Future work. Problem Description. Interaction in VR is primarily spatial - PowerPoint PPT Presentation

Transcript

  • Using Tweek to Create Graphical User Interfaces inVirtual RealityPatrick HartlingIEEE VR 2003

    VR Juggler http://www.vrjuggler.org/

    OverviewProblem descriptionGoals for Tweek softwareDesign overviewImplementation overviewFuture work

    VR Juggler http://www.vrjuggler.org/

    Problem DescriptionInteraction in VR is primarily spatialNot all interactions map well to this methodAlphanumeric inputFine-grained interactionList selectionPer-application custom interfacesMap abilities of specific devices to interactionTypically use 3D input devicesWhy abandon effective 2D interaction techniques in 3D space?

    VR Juggler http://www.vrjuggler.org/

    MotivationWhy have GUIs in virtual reality?GUIs are everywhere in the computing worldMost computer users understand how to use a GUIWhy not have GUIs in VR?

    VR Juggler http://www.vrjuggler.org/

    Issues with GUIs inVirtual Reality (1/2)PositivesTypically more flexible and more extensible than other input devicesMore input methods means less overloading of button presses, gestures, etc.More portable between VR systems than hardware input devicesFamiliarity from desktop experience

    VR Juggler http://www.vrjuggler.org/

    Issues with GUIs inVirtual Reality (2/2)NegativesNo direct physical correspondenceIn VR, a user could grab an object with his/her hand or use a GUINo GUI in real world for moving the objectRemote users may not see GUI [Park00]Use of the interface may be difficultCollision detectionPlacementImmersive interfaces have to re-invent GUI technology

    VR Juggler http://www.vrjuggler.org/

    Previous Work (1/3)Many areas of previous workWidget sets for VRPDAs as input devicesFloating menus and heads-up displaysVirtual User Interface (VUI)Originally developed at Iowa State University [Heath98]3D immersive user interfaceMay be an open source release from Fuel Tech

    VR Juggler http://www.vrjuggler.org/

    Previous Work (2/3)Java Interface to Virtual Environment (JAIVE)Used Java interface on handheld computer [Hill00]Communicated with C++ applicationsInterface could be extended from C++ using JAIVE protocol

    VR Juggler http://www.vrjuggler.org/

    Previous Work (3/3)PDA in a CAVE systemKent Watsen, Naval Postgraduate SchoolExperimented with use of Palm PDA in CAVE [Watsen99]3DI GroupDoug Bowman, Virginia TechInvestigating differences in system interaction techniques between VE displaysActive on 3DUI mailing list

    VR Juggler http://www.vrjuggler.org/

    TweekFramework for the development of cross-platform, cross-language, scalable GUIsGUIs communicate with remote applicationsGUIs can be use on the desktop, on a PDA, or in an immersive VR environmentFeaturesGUI panels are dynamically loaded componentsPanels use standard 2D GUI widgetsDynamic interface update and extensionMulti-language capabilities

    VR Juggler http://www.vrjuggler.org/

    GoalsAllow GUI scalabilitySame GUI runs on the desktop, a PDA, and a projection-based VR systemMust provide an easily extendable framework for cross-platform, cross-device GUIsInteract with VR applications written in any programming languageFully separate the GUI from the applicationReuse existing GUI technologyWe do not want to re-invent scrollbars, menus, etc.Avoid cluttering immersive space

    VR Juggler http://www.vrjuggler.org/

    Same GUI, Multiple Uses (1/2)PDA in Projection SystemOffers force feedbackHigh-resolution displayBack-lit display is neededWidely availableA person with a PDA can walk into the VR system and get the GUI dynamically

    VR Juggler http://www.vrjuggler.org/

    Same GUI, Multiple Uses (2/2)Desktop simulationStandard GUI interface used to control VR software in simulator modeImmersive 3D spaceVRJ VNC incorporates 2D GUI into 3D visualizationModeled after 3Dwm (www.3dwm.org) use of VNC and other VR VNC viewers

    VR Juggler http://www.vrjuggler.org/

    Design Overview (1/3)Distributed Model/View/Controller

    Cube rotationCube translation

    Network Communication

    Controller

    Model

    View

    VR Juggler http://www.vrjuggler.org/

    Design Overview (2/3)Observer design patternSimplified MVC: subject, observerVR application is the subjectMaintains user-defined state informationPer-application customizationsGUI is the observerProvides a view of subjects stateGUI interactions manipulate application stateBenefitsDistinct separation of application and GUIAllows multiple GUIs to observe the same application

    VR Juggler http://www.vrjuggler.org/

    Design Overview (3/3)Two-way communication between application and GUIApplication can send data to the GUIGUI flexibilityA GUI can load any component it needs to make interaction more effectiveNot limited to direct manipulation of application state

    VR Juggler http://www.vrjuggler.org/

    Implementation OverviewLanguage toolsC++ VR applicationsJava-based GUI composed of JavaBeansBeans can be pushed to the GUI while the user is in the virtual spaceDynamic extension of interfaceCORBA communication between C++ and Java

    VR Juggler http://www.vrjuggler.org/

    Example Tweek InterfaceMulti-purpose toolNavigationMap visualizationData visualizationInformation storage to spreadsheet-friendly format

    VR Juggler http://www.vrjuggler.org/

    VRJ VNC: Embedding 2D GUIs in 3D SpaceEmbed any 2D GUI in 3D worldUses VNCCan be added to existing VR Juggler applications

    VR Juggler http://www.vrjuggler.org/

    VRJ VNC: Presentations in VRBring PowerPoint presentation into the 3D environment

    VR Juggler http://www.vrjuggler.org/

    VRJ VNC: Embedding TweekTweek Java GUI is just another window on a desktopSet VNC desktop size to give the desired embedded frame sizeInteraction with the GUI is translated to the desktop via VNC

    VR Juggler http://www.vrjuggler.org/

    Future WorkUse Java Applets to allow Tweek to be used from a web browserMake use of a familiar interfaceEasy access to application GUI for remote demonstrations

    VR Juggler http://www.vrjuggler.org/

    Online Information3DUIhttp://www.mic.atr.co.jp/~poup/3dui.htmlTweekhttp://www.vrjuggler.org/tweek/Contains publications, guides, and programmer references

    VR Juggler http://www.vrjuggler.org/

    References[Park00] Park, K. et al. Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 1012, 2000.[Heath98] Heath, D. Virtual User Interface (VUI): A Windowing System for VR. In Proceedings of 2nd Immersive Projection Technology Workshop, Ames, Iowa, May 1112, 1998.[Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Masters thesis, Iowa State University, Ames, IA, 2000.[Watsen99] Watsen, K. A Handheld Computer as an Interaction Device to a Virtual Environment, In Proceedings of 3rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 1011, 1999.

    Controller is an amorphous concept that exists on both sides of the network communiation. It performs different functions depending on which side it is on.Separation of the GUI and the application allows the GUI to be re-used effectively. A GUI running in a desktop environment can communicate with an application running in simulator mode on the desktop or one running in a full-scale immersive VR system.