Using Tweek to Create Graphical User Interfaces in Virtual Reality
Using Tweek to Create Graphical User Interfaces in Virtual Reality. Patrick Hartling IEEE VR 2003. Overview. Problem description Goals for Tweek software Design overview Implementation overview Future work. Problem Description. Interaction in VR is primarily spatial - PowerPoint PPT Presentation
Using Tweek to Create Graphical User Interfaces inVirtual RealityPatrick HartlingIEEE VR 2003VR Juggler http://www.vrjuggler.org/OverviewProblem descriptionGoals for Tweek softwareDesign overviewImplementation overviewFuture workVR Juggler http://www.vrjuggler.org/Problem DescriptionInteraction in VR is primarily spatialNot all interactions map well to this methodAlphanumeric inputFine-grained interactionList selectionPer-application custom interfacesMap abilities of specific devices to interactionTypically use 3D input devicesWhy abandon effective 2D interaction techniques in 3D space?VR Juggler http://www.vrjuggler.org/MotivationWhy have GUIs in virtual reality?GUIs are everywhere in the computing worldMost computer users understand how to use a GUIWhy not have GUIs in VR?VR Juggler http://www.vrjuggler.org/Issues with GUIs inVirtual Reality (1/2)PositivesTypically more flexible and more extensible than other input devicesMore input methods means less overloading of button presses, gestures, etc.More portable between VR systems than hardware input devicesFamiliarity from desktop experienceVR Juggler http://www.vrjuggler.org/Issues with GUIs inVirtual Reality (2/2)NegativesNo direct physical correspondenceIn VR, a user could grab an object with his/her hand or use a GUINo GUI in real world for moving the objectRemote users may not see GUI [Park00]Use of the interface may be difficultCollision detectionPlacementImmersive interfaces have to re-invent GUI technologyVR Juggler http://www.vrjuggler.org/Previous Work (1/3)Many areas of previous workWidget sets for VRPDAs as input devicesFloating menus and heads-up displaysVirtual User Interface (VUI)Originally developed at Iowa State University [Heath98]3D immersive user interfaceMay be an open source release from Fuel TechVR Juggler http://www.vrjuggler.org/Previous Work (2/3)Java Interface to Virtual Environment (JAIVE)Used Java interface on handheld computer [Hill00]Communicated with C++ applicationsInterface could be extended from C++ using JAIVE protocol VR Juggler http://www.vrjuggler.org/Previous Work (3/3)PDA in a CAVE systemKent Watsen, Naval Postgraduate SchoolExperimented with use of Palm PDA in CAVE [Watsen99]3DI GroupDoug Bowman, Virginia TechInvestigating differences in system interaction techniques between VE displaysActive on 3DUI mailing listVR Juggler http://www.vrjuggler.org/TweekFramework for the development of cross-platform, cross-language, scalable GUIsGUIs communicate with remote applicationsGUIs can be use on the desktop, on a PDA, or in an immersive VR environmentFeaturesGUI panels are dynamically loaded componentsPanels use standard 2D GUI widgetsDynamic interface update and extensionMulti-language capabilitiesVR Juggler http://www.vrjuggler.org/GoalsAllow GUI scalabilitySame GUI runs on the desktop, a PDA, and a projection-based VR systemMust provide an easily extendable framework for cross-platform, cross-device GUIsInteract with VR applications written in any programming languageFully separate the GUI from the applicationReuse existing GUI technologyWe do not want to re-invent scrollbars, menus, etc.Avoid cluttering immersive spaceVR Juggler http://www.vrjuggler.org/Same GUI, Multiple Uses (1/2)PDA in Projection SystemOffers force feedbackHigh-resolution displayBack-lit display is neededWidely availableA person with a PDA can walk into the VR system and get the GUI dynamicallyVR Juggler http://www.vrjuggler.org/Same GUI, Multiple Uses (2/2)Desktop simulationStandard GUI interface used to control VR software in simulator modeImmersive 3D spaceVRJ VNC incorporates 2D GUI into 3D visualizationModeled after 3Dwm (www.3dwm.org) use of VNC and other VR VNC viewersVR Juggler http://www.vrjuggler.org/Design Overview (1/3)Distributed Model/View/ControllerCube rotationCube translationNetwork CommunicationControllerModelViewVR Juggler http://www.vrjuggler.org/Design Overview (2/3)Observer design patternSimplified MVC: subject, observerVR application is the subjectMaintains user-defined state informationPer-application customizationsGUI is the observerProvides a view of subjects stateGUI interactions manipulate application stateBenefitsDistinct separation of application and GUIAllows multiple GUIs to observe the same applicationVR Juggler http://www.vrjuggler.org/Design Overview (3/3)Two-way communication between application and GUIApplication can send data to the GUIGUI flexibilityA GUI can load any component it needs to make interaction more effectiveNot limited to direct manipulation of application stateVR Juggler http://www.vrjuggler.org/Implementation OverviewLanguage toolsC++ VR applicationsJava-based GUI composed of JavaBeansBeans can be pushed to the GUI while the user is in the virtual spaceDynamic extension of interfaceCORBA communication between C++ and JavaVR Juggler http://www.vrjuggler.org/Example Tweek InterfaceMulti-purpose toolNavigationMap visualizationData visualizationInformation storage to spreadsheet-friendly formatVR Juggler http://www.vrjuggler.org/VRJ VNC: Embedding 2D GUIs in 3D SpaceEmbed any 2D GUI in 3D worldUses VNCCan be added to existing VR Juggler applicationsVR Juggler http://www.vrjuggler.org/VRJ VNC: Presentations in VRBring PowerPoint presentation into the 3D environmentVR Juggler http://www.vrjuggler.org/VRJ VNC: Embedding TweekTweek Java GUI is just another window on a desktopSet VNC desktop size to give the desired embedded frame sizeInteraction with the GUI is translated to the desktop via VNCVR Juggler http://www.vrjuggler.org/Future WorkUse Java Applets to allow Tweek to be used from a web browserMake use of a familiar interfaceEasy access to application GUI for remote demonstrationsVR Juggler http://www.vrjuggler.org/Online Information3DUIhttp://www.mic.atr.co.jp/~poup/3dui.htmlTweekhttp://www.vrjuggler.org/tweek/Contains publications, guides, and programmer referencesVR Juggler http://www.vrjuggler.org/References[Park00] Park, K. et al. Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 1012, 2000.[Heath98] Heath, D. Virtual User Interface (VUI): A Windowing System for VR. In Proceedings of 2nd Immersive Projection Technology Workshop, Ames, Iowa, May 1112, 1998.[Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Masters thesis, Iowa State University, Ames, IA, 2000.[Watsen99] Watsen, K. A Handheld Computer as an Interaction Device to a Virtual Environment, In Proceedings of 3rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 1011, 1999.Controller is an amorphous concept that exists on both sides of the network communiation. It performs different functions depending on which side it is on.Separation of the GUI and the application allows the GUI to be re-used effectively. A GUI running in a desktop environment can communicate with an application running in simulator mode on the desktop or one running in a full-scale immersive VR system.