Upload
sebastien-kuntz
View
112
Download
5
Embed Size (px)
DESCRIPTION
MiddleVR is a generic VR plugin. SEARIS is the "Software engineering and architecture for realtime interactive systems" workshop for IEEEVR.
Citation preview
Sébastien ‘Cb’ Kuntz @Cb_VRGeek
Philosophy and architecture!SEARIS 2014
SEARIS: Software engineering and architecture in interactive systems, an IEEE VR workshop.
When I was in engineering school, we had software architecture classes and projects. Without any experience, this is a very abstract topic When I was an intern at Discreet (Autodesk, 3DS Max, Flame, Inferno, movies sfx), was very impressed with the “architect”, and wanted to be like him ! Turns out it takes a lot of experience to be a good architect, so share my experience architecting MiddleVR and show that the philosophy that you choose, consciously or not, actually drives your architecture
Plan
• Introduction!
• Origin of MiddleVR!
• Goals!
• Demo!
• Philosophy & Architecture!
• Conclusion
(French railways)
“Creating Virtual Reality Games:!
The Fundamentals”
Founder & President
Sébastien ‘Cb’ Kuntz
Also board member of the national French VR association AFRV (http://www.afrv.fr) and founded the VR Geeks association (http://www.vrgeeks.org)
And it seems I have been interested in head-mounted displays for longer that I remember.
Zone hunter (Virtuality-1994)
I fell in love with VR on a field trip in London when I was a teenager. The experience was far from perfect: the HMD was bulky, one display was blurry, but being able look around and shoot where I wanted incredible. I quickly decided I wanted to make a career in VR.
FRENCH RAILWAYS - SNCF
• Polhemus magnetic tracker, treadmill, 5DT Glove • C++ / OpenSceneGraph / VRPN / OpenAL
Fiacre: train the operators at maintenance of equipments on the railway tracks. I was in charge of all the VR engine and application development !(sorry I can’t put SNCF videos online)
FRENCH RAILWAYS - SNCF
Simurat: check wagons before they leave on the tracks
VIRTOOLS
After 4 years I was hired by Virtools as their VR lead engineer. Virtools was a great tool to create 3D applications. Simple to learn thanks to the building blocks programming scheme, and many non-programmers, like designers and even farmers, created great applications.
VIRTOOLS VR PACK
I was in charge of adding VR functionality's to Virtools engine, coding, writing documentation, support customers
I often say that we are at the prehistory of VR. Last week was the Big bang of VR. I could test Oculus DK2, Sony Morpheus and Valve’s HMD. The future is coming quickly.
• VR tools to simplify VR
Virtual reality
Presence in a virtual world
Interesting because: - natural interactions: no interface to learn. if we want to look under a table, no button to press, just move your body. - natural reactions & emotions : if I throw a ball at you, you will have the reflex to grab it. if you are afraid of heights, you will have this fear in VR.
Ridiculous
If it’s real VR, you probably
look ridiculous !
• Build hardware
To create and maintain presence, you need to ...
• Build software
• Build user experience
Most people forget about taking time to build the user experience. Test with users !
The challenges of creating a VR software
We already have great hardware, but we don’t know how to make the most of it.
“VR is mainly a software problem”!!Jaron Lanier! !
Software is like a fuel for your car. Without any fuel, you will not go very far, even if you have a Porsche.
Device management
Physical device!!
Driver Application
An application needs to get the data from a physical device through a driver.
Device management
Tracker 1! Driver Tracker 1
Application
Tracker 2!!
Driver Tracker 2
The problem is that each driver gives access to similar data in a different way so you need to have specific part of programs for similar drivers.
code for one driver if you switch to another device, another driver:
This is a completely different code which gives exactly the same information but for another device.
And there are a lot of devices out there.
Correct perspective
(c) Johnny Lee
https://www.youtube.com/watch?v=Jd3-eiid-Uw !3D applications assume that the user is in front of the screen. If you move in front of the screen, it looks like a 2d painting !If the application knows where the user is, the screen becomes a window on the virtual world (perceived depth without stereoscopy)
Correct perspective
In general 3D engines allow you to do that but you have to do all the maths by yourself Necessary for every big VR wall, anytime stereoscopy, HMDs, Caves... !
Stereoscopy
Even after you have computed your stereo images, you have different ways to display them. It can be active stereo, passive stereo, autostereo etc.
Warping & Blending
Then if you have a curved screen, your perspective will have to be distorted.
Warping & Blending
Warping is about getting a correct perspective for screens that are not planar (curved) Blending is about managing the overlapping regions between several projectors.
Graphics cards limitations
• Number of outputs!
• Processing power
multiple graphics cards
• Pro!
• More outputs!
• More processing power!
• Con!
• 3D engines are not optimized for multi-pipe
Cluster
• Pro!
• More outputs!
• More processing power!
• Con!
• 3D engines don’t support clustering
• No seams across displays require 3 layers of sync!
• 3D objects synchro (Scenelock)!
• New images display (Swaplock)!
• Left/Right eyes (Genlock)
Cluster
Interactions
• Many different 3D user interactions!
• Navigation!
• Selection!
• Manipulation!
• Draw!
• System control!
• etc.
Interactions are dependent on hardware
For example navigation will be different for different VR systems
Menus
Menus are also a big issue in VR.
Haptics (force feedback)
If you need haptics, you will have to have a physics engine that run at 1000hz and interface with specific hardware.
Deployment
• Modify application to!
• Change drivers!
• Change displays!
• Change interactions
So to create a complete VR application, you have to manage all of this !
• Platform for creating interactive 3d applications!
• Widely used!
• 2.5 million registered developers
• High-quality rendering!
• Deferred renderer!
• Occlusion culling!
• Realtime shadows!
• Ambient occlusion!
• Light mapping
Unity 3D Features• Programming!
• C#, Javascript!
• Profiler!
• Physics!
• NVidia PhysX!
• Cloth!
• Soft/rigid bodies!
• Ragdolls!
• Cars
3D formats !
FBX, Collada, 3DS, OBJ…!
Audio engine!
Networking!
Terrain
Editor
• Simple to use!
• Create your own tools inside Unity
Asset store
• Buy / sell assets!
• 3D models!
• Textures!
• Shaders!
• Scripts!
• Editor plugins
Desktop
Unity is a great tool but it doesn’t natively support VR systems easily.
• Build hardware
To create and maintain presence, you, the author, need to ...
• Build software
• Build user experience
Our goal is to minimize the time you spend on building the software so that you can focus on building the user experience.
In my mind, VR is not that difficult. Doesn’t have to be. A part is that the hardware is hard to use, but a bigger part is that the software is hard to use mostly by design.
In 2010 sent an e-mail to Roland Blach asking for all the SEARIS papers and studied them attentively
Software Engineering and Architectures for Realtime Interactive Systems Working Group
Existing VR software
• VRPN!
• OpenTracker!
• VRJuggler!
• Virtools!
• Worldviz!
• …
VRPN
In VRPN, you first have to edit a configuration file, understandable by computer scientists only. Then you have to understand how it works: callbacks, terminology is also a bit obscur (analog..) VRPN is only solving the devices management, and as we have seen, there are a lot of other issues when creating a VR app.
VR Juggler
VR Juggler needs to understand a lot of concept: proxy device? derive of off OpenGL::App ? My main problem is that you have to bend your application to their design
VIRTOOLS VR PACK
VRPack’s configuration file is even more complex than VRPN. Even working at Virtools, it took me quite a while to understand how it works ! Switching configuration files is also complicated.
Worldviz Vizard
• Based on OpenSceneGraph!
• Python
Vizard is a really nice environment but still a bit too complex to use.
Worldviz Vizard
For example this is how you would configure trackers and a Cave.
The need for better VR software tools
• Hardware getting cheaper!
• But engineers are not getting cheaper!
• Need to be more efficient and spend less time creating apps!
• Democratization!
• VR developers are not all engineers anymore
Goals
• Simplify VR!
• Simplify creation of VR applications!
• Simplify deployment of VR applications!
• Create new VR tools!
• Adapting existing 3D applications
What is ?
• Generic VR plugin (C++ SDK) !
• Functionalities ○Manage input devices
•3D trackers, keyboard, mouse, joystick ○Manage display
•Viewports, cameras, stereoscopy, cluster ○High-level building-blocks ○Interactions ○Menu
CaveLib - VRJuggler
Architecture
MiddleVR (C++)
Drivers OpenGL DirectX Network
C# Wrapper
GUI UnityLua
WrapperPython
Wrapper
GUI to easily configure any VR system quickly.
Tutorial: https://www.youtube.com/watch?v=EEKH-1EilXs
Once you have exported your application, you can simply add it to the list of Simulations in MiddleVR’s gui. Then you can pick any configuration and simply click Run and your application will be dynamically reconfigured for this configuration
https://www.youtube.com/watch?v=yyR6w43hgSo
MiddleVR Features
Supported Devices• Via VRPN :
• A.R.T • Vicon • Optitrack • Wiimote !
• Polhemus Fastrak Tracker, 3Space Tracker, Patriot Trackers,
• Ascension Flock of Bird • Intersense IS600, IS900 • 3dTech HiBall-3000 Wide Area Tracker, • Advanced Realtime Tracking Gmbh DTrack
Client, • World Viz Precision Position Tracker PPT
1.2, • Natural Point Optitrack Rigid Body Toolkit, • Phasespace optical tracking OWL, • ...
• Native drivers • Oculus Rift • Leap Motion • Kinect (Microsoft SDK) • PNI SpacePoint
Fusion • GameTrak • Razer Hydra • SpaceMouse • Motion Analysis (beta)
Haptics (force feedback)
https://www.youtube.com/watch?v=lnoE_VclnOI Haption IPSI
VR Camera• MiddleVR computes all the cameras’ parameters
automatically!
• Correct perspective!
• Scale 1
Stereoscopy
• Active stereoscopy (OpenGL Quad-Buffer)!
• Which Unity can’t natively handle!
• Passive stereoscopy
Viewports management
Viewports configuration
Simply configure your system
Configure what is happening in the real life. Seeing things happening in real life changes a lot. Can easily check - axes swap, offsets - screens are correctly placed - if tracking is covering the area !Allows you to experiment.
Navigations
We provide standard navigations: - classic joystick navigation - elastic navigation - grab the world
Homography
A nice new feature is the homography. This allows you to grab corners of your viewports and have them match the physical corners of your cave.
Cluster & Multi-pipe
• Scenelock!
• Swaplock!
• Genlock!
• Multi-pipe (soon)
Regarding clustering, one unity player is running on each computer. SceneLock: synchronise the state of all input devices and position/orientation of some specific objects. SwapLock: implemented our own mechanism Genlock: done in HW
Supported displays
• Virtually any HMD • VR-Wall, Powerwall • Workbench, Holobench • HoloStage • CAVEs • 3D TVs • zSpace • Igloo Vision • …
Goals
• Simplify creation of VR applications!
• Simplicity of adding VR into application!
• Simplicity of API!
• High-level tools!
• Interactions: Selection, manipulation, navigation, menus, GUIs
Example of device management
Driver 1
Driver 2
Device management
Replaced manual edition of cfg file by GUI Simply choose the driver from the list, don’t need to rewrite it.
Looks like english ! No need to understand what a sink is (opentracker), or forced to set callbacks.
MiddleVR
Here is the minimum code you have to write to use MiddleVR. You don’t have to modify the structure of your application to integrate MiddleVR !
Class reference
Most VR systems out there can be described with these highlighted simple, atomic, concepts
The value of Simplicity
The value of simplicity
“Perfection is achieved, not when there is nothing more to add, but when there is
nothing left to take away”!
Antoine de St Exupery
Simplicity is a huge principle for me. We don’t hesitate to remove code/feature that are covering way too much. EVEN IF ALREADY WRITTEN !! We’re a Startup, we don’t have resources, so we have to be simple No resources is a strength Lots of resources can actually be a curse
The value of simplicity
• For each feature you add!
• Test!
• Document!
• Support / maintain!
• Each feature costs !!
Each feature you add, you have to support it for the entire life of the product. Simplicity is harder to achieve, so cost more at first, but in the long run is beneficial
The value of simplicity• Huge value in simplicity !!
• User side!
• Faster to learn, less to remember!
• Developer side!
• Faster to code!
• Easier maintain!
• Less bugs!
• Business side!
• Need less coders!
• More interesting features!
• Easy download / setup > Quickly validate on user’s hardware !!
• Easier support
API in simple english• Long discussions for naming of functions, classes, variables!• Consistent api!
• SetNumButtons, SetButtonsNum, SetNbButtons > SetButtonsNb!• French not good at english!
• Analog ? Digital ? Proxy ?!• Use natural terms depending on device!
• Joystick->GetAxisValue()!• Joystick->IsButtonPressed()!• Keyboard->IsKeyPressed()!
• Why “Register” ? Why not “Add” ?!• Why so complex, want to sound very clever ?!• AddDevice, AddCamera, …!
• API in degrees, because more intuitive / simple to work with
Simplified API
• Simple but complete API!• Like legos!• Rotate!
• Node3D->Rotate( Quat )!• Node3D->Rotate( Euler )!• Node3D->Rotate( Axis, Angle)!• Quat( quat ), Quat( Euler ), Quat( Axis, Angle)
Simplified API: only one way of doing things: less to learn, less to remember Everything you need is in the Quat class, so it’s reusable in any part of the code
The value of simplicity
“Simplicity is the ultimate sophistication”! DaVinci
Fight hard to create / maintain simplicity. Engineers come and implemented what was in their mind, anticipate cases that would rarely happen. We do code reviews. Often rewrite often 3 times before acceptable.
The value of simplicity• Design user interface for the 80%!
• Don’t ruin the experience for the 20%!
• Simple things should be simple!
• Complex things can be complex!
• Put in an “advanced menu”!
• Don’t let beauty of design ruin user experience!
• Like real life architects who want their design to be beautiful but totally unusable on a day to day!
• Example of Lego!
• Simple but complete
Tell horror network socket story :) !Current tools designed for geeks/engineers (examples vrpn/virtools vr)
Optimizations
• Engineers have a strong tendency to over-optimise!
• Complexifies code!
• Code must be above all readable!
• To avoid bugs!
• To maintain maintenability!
• If you want to optimise!
• First profile your code!
• Optimize what is *really* taking time!
• not what you *think* takes time
Empathy
“The ability to understand and share the feelings of another.”!
!
Test with users
• if user don’t understand, not add more doc,!
• simplify !!
Hardware SDK are mostly too complicated
!Even Oculus !!!!! Less time to implement haptics !!!
Drivers
• Why not just a few simple functions with sensible defaults ?!
• Init()!
• Update()!
• GetValue()!
• Destroy()!
• And add more functions for more advanced
HW manufacturers don’t know how to write an API
Documentation
• User Guide!
• Class references
Of course the ideal tool would not need documentation, I honestly like to write documentation. It helps spot errors, inconsistencies. If it’s hard to document, the feature must probably must be simplified instead of writing a more sophisticated doc !!
Goals
• Simplify deployment of VR applications!
• GUI to configure VR systems!
• Dynamic reconfiguration of application!
• Ship with standard configurations!
• Abstractions
Deployment
• Abstractions!
• Devices!
• 3D Nodes (User)!
• Interactions (soon)
Device management
Driver Tracker 1
Application
Driver Tracker 2
** The problem is that each driver gives access to similar data in a different way so you need to have specific part of programs for similar drivers.
Device abstraction
Virtual Device Application
Driver Tracker 1
Driver Tracker 2
A virtual device only stores data. For tracker, position+orientation Drivers write data in the virtual device. Allow to change driver without application (PORTABILITY) --- Replay allows you to playback a recorded working session. It enables quality insurance because you can give the same input to your application and check that the output is always the same.
Device management
Driver
Virtual Device Application
Simulation
Replay
Replay allows you to playback a recorded working session. It enables quality insurance because you can give the same input to your application and check that the output is always the same.
Display
• Cameras & viewport created dynamically
User• Human-centric rather than techno-centric
Rather than accessing devices, we could simply ask the position of the user’s head, or hand ! This allows to write scenarios like : What does the user see ? Is the hand close to ... ?
Virtual Device 1
Hand (3D Node) Application
Virtual Device 2
Inverse Kinematics
User abstraction
Interactions• MiddleVR includes high-level interactions!
• Navigation!
• Selection!
• Manipulation!
• … More to come!
• Future: Interactions abstractions!
• Dynamically change interactions based on hardware
Interactions abstraction?
Grab-air
Navigation
Application
Point
Steering
Selection
Manipulation
Raycasting
Go-go
Hand
Homer
Hand
WIM
• How to create portable interactions ?!
• Navigations can be portable!
• Except collisions!
• Generalise what is generalisable!
• + Custom part for engine dependent!
• Collisions!
• Highlights!
• Gizmos
Interactions
Menus
• Ability to create menus and GUIs in HTML5 (soon)
Create immersive HTML5 GUI
HTML 5 gui in VR
C++ Reflexion
• Also for configuration file!
• Cluster synchronisation!
• Accessing drivers specific information
Property is simple couple of getter/setter. In header write naturally Set/GetParent !One line to add property in source code.
Working with multiple 3D engines
• Different coordinate systems!• vrCoordinateSystem( Front, Up, Right )!
• +/- X,Y,Z!• OpenGL:!
• Front= -Z!• Up = +Y!• Right = +X!
• DirectX:!• Front=+Z!• Up=+Y!• Right=+X
Working with multiple 3D engines
• C++!
• Compiling!
• STL, Boost!
• Strings (Wide strings)!
• Link issues!
• Different compilers (VC2005/2008/2010/2012)!
• Different languages!
• C wrapper!
• Lua wrapper
!
• Simplicity!• Creation!• Deployment!
•Adaptability!•Hardware!• Software
Our future
• Games/apps development:
Conclusion
Talked about the philosophy and architecture You see how deeply the philosophy impacts the architecture Simplicity at its core which has a huge value for everybody, but has to fight hard for it.
http://www.imin-vr.com/download !
Thank you ! Questions ?
@Cb_VRGeek @imin_vr