User Interfaces and User Centered Design Techniques for Augmented Reality and Virtual Reality

  • Published on

  • View

  • Download


UIs and User Centred Design Techniques for AR + VRWhats AR and VR?!Augmented Reality and Virtual reality.AR: technology that layers over our everyday VR: technology that transports us to a different worldBut They Must be Years Away?! do we design applications for AR and VR?IxD for System vs IxD for FrameworkDesigners are even confronted with the obstacle of a methodology or framework to design the system itself and quickly iterate through prototypesSteps are being taken to address these concerns Few applications / frameworks although commercial + open source are helping Evaluating VR UIs Goals FormationNavigate the WorldLocate ObjectsPosition for InteractionDeciding on ActionManipulating ObjectsRecognise FeedbackInterpret FeedbackDecide on New ActionVR Related Issues with EvaluationObjects can obscure and may break interaction cycleDifferent modes of design for navigation and for environment driven VR systemsExpert users see the modes blend togetherFeedback should be multisensoryCant see things off screen or behind a wall etc. Environmental cues are keyEvaluation TechniquesWalkthrough each phase step by stepAsk necessary questions along the way Aim to uncover breaks in affordancesQuestions guided to create generic design principles (GDPs)Collect design issues and virtual environment featuresPrioritise them for developmentUCD Tips for AR Install on a familiar deviceChoose a design scenarioRun in an appropriate settingBuild for two hands if on mobileChoose the right audienceChallenge users with mental flow not with physical strainIssues OverallInformation is often weakly exhibited that link design problems with design solutions in VR and AR.Even some of Normans interaction evaluation techniques can break down when dealt with navigating 3D space.Need a tailored methodology for dealing with emerging issues in hardware and interaction.Virtual RealityUses, Current Tech and DesignUses of VREducationWith the leaps in technology, virtual reality can be used to transport people to other planets, tourist destinations and the many jungles and oceans on earth.Video GamesVirtual Reality allows players to be transported to other worlds and puts them in the middle of the action! MedicalVirtual reality can allow surgeons to move throughout the body and diagnose problems that patients have. Virtual reality is also being used for therapy for PTSD veterans and phantom limb syndrome.source: openmedical.orgsource: immersivevreducation.comsource: military.com Technology available todayDesigning for VRUser Interface design in VRWhen designing interfaces in a virtual reality, there are some considerations which must be taken into account.Is there a motion controller used in conjunction with the VR system?What is the nature of the experience? Who is the interface being designed for? What are the perceptual limitations of the user? Perceptual LimitationsPerceptual limitations occur in the areas that users find interaction difficult in their field of view.305560201240comfortablecomfortable72102PeripheralPerceptual LimitationsPerceptual limitations occur in the areas that users find interaction difficult in their field of view.305560201240comfortablecomfortable72102Peripheral0.5mRegion where convergence can occuri love this graph :) although does the person on the right not have their nose in the wrong spot or am i reading the graph wrong? Theyre nose is in the right position! its a side viewHow do we design UIs with these limitations in mind?How do we design UIs with these limitations in mind?Is this a good UI design or bad UI design in VR?Augmented RealityUses, Current Tech and DesignUses of ARNotificationAR headsets can notify you of social media, texts or email as you go about your daily life.Video GamesAR allows players to have their world transformed in front of their very own eyes!NavigationAR allows people to find their destination in a non- obtrusive way. source: google.comsource: nintendo.comAR Technology available todayDesigning for ARUser Interface design in ARWhen designing interfaces in a augmented reality, the considerations taken into account are similar to VR.But there is one thing that is the utmost importance! Obscurity / Opacity.User Interface design in VRContinue 10mUser Interface design in VRUser Interface design in VRCase Studies1.Design and Evaluation of Menu Systems for Immersive Virtual Environments - Bowman & Wingrave 20012.Experimental Evaluation of User Interfaces for Visual Indoor Navigation - Moller et al. 2014Key Interaction Tasks VRNavigationSelectionManipulationSystem ControlTULIP Menu vs Floating Menu vsPen and Tablet MenuPinch GlovesHardware for TULIP MenuPinch Gloves consist of a flexible cloth gloves augmented with conductive cloth sewn into the tips of each of the fingers. When two or more pieces of conductive cloth come into contact with one another, a signal is sent back to the host computer indicating which fingers are being pinchedVirtual Research V8 head-mounted display (HMD) and the head and both hands are tracked using a Polhemus Fastrak tracking systemEvolution of TULIPScrolling MenuThree Up MenuScrolling MenuThree Up MenuPilot StudyEvaluating these two menu designs, users had to change a virtual object to match a target object.3 parameters could be controlled: the objects shape, color, and texture. Each of these corresponded to a top-level menu.There were 3 shapes to choose from, 8 colors, and 6 textures these corresponded to second-level menu items.Test 4 users - Think Aloud - Informal ResultsPilot ResultsNeither design satisfied the desired requirements.Users did prefer the Scrolling Menu BUT realized tasks could be completed with less steps using the Three Up Menu.Three Up Menu hides options if there is more than 3. Scrolling Menus prompt users to incorrectly attempt to push palms.Lack of feedback when items selected and fatigue from hands being raised.RE-DESIGNT hreeU pL abelsI nP almSummative EvaluationCompare Ease Of UseCompare Ease Of LearningEfficiencyComfort of all 3 MenusSummative Evaluation26 users participated Repeated object matching taskCompleted a questionnaire containing demographic information and information about their experience with computers and VEsSame equipment used again with added stylus for floating menus and pen and tablet menus30 trials of each menu and no help provided after initial briefingFloating MenusPen and Tablet MenusResultsAppears that the gloves were the hardest to learn initially, but performance was at reasonable levels for all three types within five trials.Reason times for the pen and tablet menu are initially poor is that users were not told they needed to look at the tablet in their handComfort LevelsThe main drawback of the pen and tablet system is the discomfort it causes users, which might be alleviated by adding an ergonomic handle. Reflection On StudyCombining the efficiency, comfort, and preference information, it appears that both the pen and tablet menu and the TULIP menus performed well in the evaluationFifteen users expressed a preference for the TULIP interface, while nine preferred the pen and tablet, and only two preferred the floating menus This evaluation reiterated some important heuristics from the traditional human-computer interaction literature. Menu systems for VEs need to have good feedback, affordances, and constraints, and items and their actions should be visible.Evaluation of User Interfaces for Visual Indoor NavigationAndreas Moller et al.Implemented a novel UI for visual localization, consisting of Virtual Reality (VR) and Augmented Reality (AR) views that actively communicate and ensure localization accuracy.Beneficial for large buildings and navigating your way around.The concept consists of a panorama-based view as a complement to Augmented Reality and proposes different visualizations for motivating users to record good query images.Good query images are importantComparing AR vs VRThe users would hold the phone up as seen in the figure and look at the phone in order to see the augmentation i.e. items superimposed onto their real life surroundings.The virtual reality displays pre-recorded images of the environment (downloaded from a server) that are arranged to a 360 panorama on the mobile deviceEnhancing Visual LocalizationVisual localization can be very dependent on how the device is being held.Four indicator types were proposed which were:Text HintBlurColour ScaleSpirit LevelInitial TestingA questionnaire-based survey with mockup videos were used for early testing. Users did not actually travel through a building. From initial user evaluations there was an inconsistency. Majority of people stated that the VR mode helped them orient themselves even if the location estimate of the system was incorrect yet the subjects still claimed they prefer the AR mode. Users preferred the Text Hint and Spirit Level as a means to prompt user to provide a better quality of image.PrototypeA prototype was built using Android 2.3Had both the AR and VR modesWizard Of Oz technique - experimenter would play a roleThe navigation interface on the subjects device was implemented with OpenGL ES 2.0.For the automatic trigger, they used a FAST feature detector from the OpenCV framework for Android to detect the number of features in the cameras live image. ExperimentsSo the goal of the experiment was to verify results from earlier mock-ups. Three experiments took place to test:Efficiency, perception and convenience of AR and VR under different accuracy conditions.Effectivity of UI elements specific to vision-based localization.Convenience and distraction of object highlighting. 12 participants took part most of which were students but none were involved in the project. HardwareSubjects used a Samsung Galaxy S II (4.3- inch screen, 8 megapixel camera)The WOz app ran on a Samsung Nexus S (4-inch screen). Both devices had a screen resolution of 480800 pixels. UIs Implemented (subject)UIs Implemented (WoZ)Experiment 1- Navigation using AR and VRSubjects performed a navigation task in a university building on a path of 220 meters length, using both the AR and the VR mode. Each user traversed the path eight times and was asked to rely only on the given instructions. Navigation instructions were fed into the subjects phone by the experimenter (Wizard of Oz). Sent the appropriate panoramas in VR mode (and directional arrows in AR mode) to the subjects phone using the WOz interface.Users were encouraged to think allowedResultsSubjects were in average 25 seconds faster to reach their destination with VR (averagely 2:39 minutes for the 220 m path) than with AR (averagely 3:04 minutes). The experiment also proved that AR was worse with regards to orientation errors. Subjects found carrying the phone more convenient in VR. The required upright position for carrying the phone in AR was physically constrainingExperiment 2 Vision Specific UI ElementsSubjects performed a navigation task on the path shown in figure, but in opposite direction as in Experiment 1, so that the path was not already too familiar. Three times during the walk, a relocalization procedure was simulated.The experimenter triggered a spirit level visualization to appear on the subjects device. This indicator told subjects to collect enough features for relocalization. ResultsReliable localization requires 100 to 150 features in the image. While the indicator was visible, the average number of detected features per frame rose from 42 to 101.The experiment also showed that subjects preferred the lower carrying position for VR mode, compared to the upright pose for AR modeExperiment 3 Object Highlighting MethodsThere were two ways to highlight objects: Frame and Soft HightlightThe algorithm was optimized to detect square, feature-rich objects out of a uniform background. This applies to, e.g., a poster on a wall.Subjects pointed at the posters using both highlighting visualizations. Feedback was afterwards collected by a questionnaire.ResultsOn a Likert scale from -3 to +3, subjects indicated that Frame drew more attention to the poster (M = 3) than Soft Highlight (M = 1). The semi-transparency of Soft Highlight complicated readability of text on the poster. Regarding distraction, the visible contours of the Frame visualization were perceived as unstable.Reflection on ExperimentsVR mode turned out to be advantageous in several ways. Contrary to initial feedback, AR UI appeared far more appealing in theory. AR UI does have it strengths though; It can help to improve feature collection using the feature indicator i.e. spirit level prompt to probability of reliable re-localization.Subjects reported that the frequent updates of the panorama images in VR mode were partly irritating, especially when not permanently looking at the screenFurther studies are required in the field to strengthen these UI concepts, particularly with regards to AR and seeking more reliable localization.Soooo whats the activity? Considering the revised User Centred Design Model, were going design some augmented reality applications. NavigationNotification SystemPhoto and Video Messaging Systemthanks!ANY QUESTIONS?ReferencesAlger, M., 2015. Visual Designs for Virtual Reality.Alger, M., n.d. VR Interface Design Manifesto.Alger, M., n.d. VR Interface Design Pre-Visualisation Methods.Bowman, E., Wingrave, C. (2001) "Design and Evaluation of Menu Systems for Immersive Virtual Environments", in Proceedings Of The Virtual Reality 2001 Conference (VR'01) (VR '01), IEEE Computer Society.Broll, W., Shafer, L., Hollerer, T., Bowman, D., 2001. Interface with angels: the future of VIR and AR interfaces. IEEE Computer Graphics and Applications 21, 1417. doi:10.1109/38.963455De Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., Poulovassilis, A., 2010. Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. British Journal of Educational Technology 41, 6985.Denis, J.-M., 2015. From product design to virtual reality. Google Design.DesigningInteractions_8.pdf, n.d.Faaborg, A., n.d. Designing for virtual reality and the impact on education. TEDx Talks.Hsu, C., Shiau, H., 2013. The Visual Web User Interface Design in Augmented Reality Technology. Editorial Preface 4.ReferencesJang, D., Kim, J.-S., Li, K.-J., Joo, C.-H., 2011. Overlapping and synchronizing two worlds. Proceeding GIS 11 Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems 493496.Kunz, A., Wegener, K., 2013. Towards natural user interfaces in VR/AR for design and manufacturing. Eidgenssische Technische Hochschule Zrich, Institute of Machine Tools and Manufacturing.Leap Motion VR Best Practices Guidelines.pdf, n.d.rabedik, n.d. AR Screen Hackathon Project.Mller, A., Kranz, M., Diewald, S., Roalter, L., Huitl, R., Stockinger, T., Koelle, M., Lindemann, P. (2014) "Experimental evaluation of user interfaces for visual indoor navigation", in Proceedings Of The SIGCHI Conference On Human Factors In Computing Systems (CHI '14), ACM: New York, NY, USA.Samsung Developer Connection, 2014. VR Design: Transitioning from a 2D to 3D Design Paradigm.Sutcliffe, A., n.d. Multimedia and Virtual Reality: Designing Multisensory User Interfaces. Pyschology Press, 2003.Sutcliffe, A, & Deol Kaur, K 2000, Evaluating the usability of virtual reality user interfaces, Behaviour & Information, n.d.Vent-Olkkonen, L., Posti, M., Koskenranta, O., Hkkil, J., 2014. ReferencesInvestigating the Balance between Virtuality and Reality in Mobile Mixed Reality UI Design User Perception of an Augmented City. Proceeding NordiCHI 14 Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational 137146.Wesolowski, M., n.d. Designing Next-Gen Virtual Reality Gaming Experiences., A, & Deol Kaur, K 2000, 'Evaluating the usability of virtual reality user interfaces', Behaviour & Information Technology, 19, 6, pp. 415-426, Academic Search Complete, EBSCOhost, viewed 12 February, (2016) User Testing As A Design Driver : Looksery Created A Product For Users, Not Designers: Looksery Created A Product For Users, Not Designers | UX Magazine [online], available: [accessed 20 Feb 2016], (2016) User Testing As A Design Driver : Looksery Created A Product For Users, Not Designers: Looksery Created A Product For Users, Not Designers | UX Magazine [online], available: [accessed 292Feb 2016].1


View more >