Emerging Applications of VR Electrical and Computer Engineering Dept

Emerging Applications of VREmerging Applications of VR

Electrical and Computer Engineering Dept.

Emerging applications of VREmerging applications of VR

In manufacturing (especially virtual prototyping, assembly verification, ergonomics, and marketing);

In robotics (programming, teleoperation, space robotics);

In data visualization (volume visualization, oil and gas exploration, volumetric displays);

Other areas.

(UK VR Forum survey, 2000)

VR penetration in non-medical fieldsVR penetration in non-medical fields

% - companies using,% - companies using,

Experimenting orExperimenting or Considering VR Considering VR

Classes of VR applicationsClasses of VR applications


Main benefits of using VRMain benefits of using VR


VRVR in Manufacturingin Manufacturing

GHOST GHOST Free FormFree Form Application Application

Free Form sculpting is a new type of Human-

Computer interaction; It functions as a convenient GUI that sits on top of the

GHOST library;

The GUI is based on static and dynamic bar menus “dynabars”; It allows model export to CAM machines for rapid prototyping, as well as to animation packages for computer character animation. Requires two Quad Core processors, 4 to 16 GB RAM, high-end hardware graphics acceleration, 1024x768 screen resolution (or better).

Free Form SculptingFree Form Sculpting

GHOST SDKGHOST SDK

PHANToM DriversPHANToM Drivers

FreeForm Initial screenFreeForm Initial screen

Menu barMenu bar

Tool barTool bar

DynabarDynabarStatus barStatus bar

Work areaWork area borderborder

The center block is “digital clay”

FreeForm wire-cut modeFreeForm wire-cut modeWire Cut MenuWire Cut Menu

Sketch planesSketch planes

Free Form wire cutFree Form wire cut

Drawing on the cut planeDrawing on the cut plane

Clay after Outside wire cutClay after Outside wire cutClay after Inside wire cutClay after Inside wire cut

Cut Outside Cut Outside buttonbutton

Cut Inside Cut Inside buttonbutton

FreeForm carving toolsFreeForm carving tools

Carving a holeCarving a hole Carving a cornerCarving a corner

Getting carried away….Getting carried away….Smoothing while carvingSmoothing while carving

Example – Making of Saint FruitionExample – Making of Saint Fruition

Artist’s sketchArtist’s sketchRough initial clay modelRough initial clay model


Finished body (clay)Finished body (clay)

Wire cut arrow for support (clay)Wire cut arrow for support (clay)

Finished support (clay)Finished support (clay)


5 ft Aluminum statue5 ft Aluminum statue Digital clay statueDigital clay statue


Key frame animation of statueKey frame animation of statue Statue model used forStatue model used for animation (Maya)animation (Maya)

Textured statue used forTextured statue used for animation (Maya)animation (Maya)

Link to VC 9.1 on book CD

Another stage in product development when the prototype is made of several parts; University of Washington developed the “Virtual Assembly Design Environment” (VADE) to verify CAD design assemblies; Parts geometry and attributes are imported from CAD into VADE then the assembly is analyzed and robots are programmed

Assembly verificationAssembly verification

Collision detection through swept volumesCollision detection through swept volumesDesign modification in VRDesign modification in VR

Parts making up car exterior have varying tolerances. Tighter tolerances are more esthetically pleasing but also cost more. What is good enough? Inspection is done in inspection rooms using stripped lights. Same can be done on a virtual car ahead of real production

Assembly verification – Car body tolerancesAssembly verification – Car body tolerances

Virtual Inspection roomVirtual Inspection roomReal inspection roomReal inspection room

Assembly verificationAssembly verification

Unacceptable tolerances discoveredUnacceptable tolerances discovered in the virtual inspection roomin the virtual inspection room

Researchers in UK developed the Visualization of the Impact of Tolerance Allocation (VITAL) and tested it on a prototype Rover R75; They constructed several models with various tolerances; by shining the virtual car body with stripped light looking at discontinuities;

Ergonomic AnalysisErgonomic Analysis

Jack is a an intelligent agent homanoid used in ergonomic analysis; the Task Analysis Toolkit computes lower-back effort and energy consumption – relates to worker fatigue


Ergonomic Analysis - continuedErgonomic Analysis - continued

Once a prototype is done, it has to be tested for ease of use (ergonomic analysis); One such product is VirtualANTHROPOS developed in Germany to test the ease of use of tractor cabins


An avatar controlled by the user is interacting with the virtual cabin while the system computes joint discomfort levels using ERGONAUT (an ergonomic analysis tool)


Another use of VirtualANTHROPOS is to visualize reach envelopes; The user can drive the avatar in real time using a wireless body suit;


Personnel TrainingPersonnel Training

Training in airplane maintenance – task has a cognitive component (manuals) and a tool/part manipulation component – and they are sequential.


Task-related information is placed directly in the scene using augmented reality. Results in faster information retrieval and enhanced associative memory. System uses vision-based tracking to recognize worker’s view and places text in relation to objects;


Training system detect removal of cover and labels parts underneath; then it detects the cap was removed and changes dynamically the text to “4. Press to test”. If the test fails then additional areas of interest are highlighted (“Filter Bypass”)

VR Marketing ApplicationsVR Marketing Applications

CitrCitrööen uses virtual en uses virtual showroomsshowrooms



VR applications in Robotics/manufacturing relate to several areas: CAD design and robot programming, making the process more intuitive; Teleoperation (control at a distance) alleviating problems related to poor visibility and large time delays; Multiplexed teleoperation, acting as a filter of particular robot kinematics;

Robots are also used in VR in haptic interfaces (discussed earlier

in our course).

Robotics ApplicationsRobotics Applications

The “multi-modal teaching advisor helps novice operators program welding paths for industrial robots; It runs on a PC networked with trackers and laser range finder; Calculates the difference between the pre-computed (optimal) path and user’s input on the teach pendant. This is presented graphically on the user’s HMD

Robot programmingRobot programming

Robotic programming - continuedRobotic programming - continued

Research done in Germany for off-line robot programming done in VR, with the programmer immersed in the task he is programming; The programmer specifies the trajectories, and the simulation performs optimized collision detection; Validation is done at run time when the real robot is controlled using the same computer and real sensor data is used to fine-tune the VR-generated program.

VR Robotics Applications - continuedVR Robotics Applications - continued

(Burdea, 1999)

Operator VR GUIOperator VR GUI Degraded video feedbackDegraded video feedback

Research at University of Tokyo for the teleoperation of robots in smoke-filled remote environments. Over-imposes the visual scene from the remote robot with the virtual scene of a kinematically identical robot. Thus VR acts as a guide to allow teleoperation.

Teleoperation with large time delaysTeleoperation with large time delays

Research at NASA developed a VR-based teleoperation to allow operation despite large time delays. Works by controlling a “phantom robot” which responds instantaneously to the operator. Allows preview of the move, before it is executed.

Teleoperation with large time delays - continuedTeleoperation with large time delays - continued

Research at NASA drove the Mars rover using VR-based teleprogramming; This was used to send high-level macro commands based on the simulation of a virtual rover on a virtual Mars surface. This overcame a 20 minute time delay!

Supervisory controlSupervisory control

Researchers in Germany developed a way to naturally controlling robots through avatars; The users is immersed in VR and sees a scene with avatars to which he is mapped; He interacts through gestures (measured by a sensing glove)

Supervisory controlSupervisory control

A real robot then interacts with the remote real environment; If the task is visual inspection then real images from the remote site can be overlaid on the virtual scene, and thus seen by the user;

Real remote robotReal remote robot Virtual robot with viewfinderVirtual robot with viewfinder

VR Robotic TeleoperationVR Robotic Teleoperation

Research at Jet Propulsion Lab (California) allows the teleoperation of a remote robot indirectly by controlling a motion-guide trajectory, which the robot is then constrained to follow.

Motion guideMotion guide

Task lineTask lineRemote robot armRemote robot arm

Text mode task editorText mode task editor

VR Robotics Applications - continuedVR Robotics Applications - continued

Research at University of Paris to allow multiplexed (one-to-many) teleoperation of kinematically dissimilar robotic arms; VR acts as a high-level filter masking the detailed slave robot configuration; Translator then converts user actions to robot actions.

Remote robot armsRemote robot armsOperator task-level GUIOperator task-level GUI

Supervisory control - continuedSupervisory control - continued

INFORMATION VISUALIZATIONINFORMATION VISUALIZATION

Represents the transformation of abstract data into 3D scenes; The information visualization pipeline allows the user to control the view to the scene using an input device and select an area of interest; The data extraction loop is asynchronous, so as to maintain interactivity. It reads user input from a FIFO buffer; Time-varying data represent a complex case, as user time may not coincide with the time clock used in visualizing the time-dependent data.

Oil and Gas Exploration and Well ManagementOil and Gas Exploration and Well Management

Oil and Gas Exploration and Well ManagementOil and Gas Exploration and Well Management

Volumetric graphicsVolumetric graphics

In this class we learned about surface-based (polygonal or spline) rendering only. This leaves the interior of virtual objects hollow; Volume graphics renders the surface as well as the interior of objects, called “voxels.”

Surface rendered object Surface rendered object Same object rendered volumetriclySame object rendered volumetricly

voxelsvoxels

Ray casting to create a 2-D image from volumetric dataRay casting to create a 2-D image from volumetric data

RaysRays

View PlaneView PlaneTri-linear interpolationTri-linear interpolation

Calculate gradients and lighting Calculate gradients and lighting

Volumetric graphics advantagesVolumetric graphics advantages

Much richer dataset; Objects appear more real; Can be displayed on same displays as surface-based models

Volumetric graphics disadvantagesVolumetric graphics disadvantages

Much larger memory requirement (order of GB); Requires special boards to be real-time; Require special displays if auto-stereoscopy is desired; Much less used today – small software support base.

Volumetric graphicsVolumetric graphics


Volumetric Graphics HardwareVolumetric Graphics Hardware

Consist of volumetric rendering boards and of volumetric displays; VolumePro 1000 is a graphics accelerator sold by TeraRecon Inc. renders 512 x 512 x 512 voxels at 30 frames/second using a Mitsubishi chip.

For surface geometry data it works together with the graphics board installed on the same PC

VolumePro Rendering PipelineVolumePro Rendering Pipeline

VolucePro 1000 processed the 3D data through ray casting from a view plane. Rays pick up color and opacity information by tri-linear interpolation to the nearest lattice point Gradients are then computed

VOLUMETRIC DISPLAYSVOLUMETRIC DISPLAYS

Early models used LED matrix panel that translates back-forth on rails; User can see stereo with bare eyes. Due to eye inertia, the image appears to float in space; But they had low resolution, noisy, monochrome (red LEDs); Did not have a 360º viewing area

LEDLEDObject sectionObject section

Volumetric DisplayVolumetric Display

According to Wikipedia, a Volumetric Display is defined as a device that “creates 3-D images by emission, scattering or relaying of illumination from well defined regions in the (x, y, z) space.”

Almost all Volumetric Displays are autostereoscopic.

Currently, Volumetric Displays are only available to major companies, the military, and certain schools for academic purposes. They are still unavailable for the general public.

There are two main types of Volumetric Displays: Swept-Surface Displays, and Static Volume Displays.

Rely on the phenomena known as persistence of vision.

Persistence of VisionPersistence of Vision

Persistence of Vision occurs when a human being is exposed to an image for a very short amount of time, usually nanoseconds.

The result is around a millisecond of sight. This occurs because the chemical reaction time it takes for a human to process an image is much longer than the speed of light.

Swept-Surface DisplaysSwept-Surface Displays

Also known as Swept-Volume Displays

Rely on a rotating source or surface.

Project a 2-D image that constantly changes based on the orientation of the rotating source or surface.

An example: the Perspecta, created by Actuality Systems, Inc.

Perspecta by Actuality Systems, Inc.

Unlike Swept-Surface Displays, Static Volume Displays do not rely on motion.

Some use pulsating, concentrated, Infrared Lasers that pulse around 100 times per second to establish floating balls of plasma in space which cause a clicking sound.

The exact point is known as a voxel (similar to a pixel would be in 2-D, except 3-D)

Others use lasers to cause visible radiation with a gas, liquid, or solid.

Static Volume DisplaysStatic Volume Displays

Static Volume DisplaysStatic Volume Displays

Use slideable mirrors and lenses to redirect the location of the voxel that are usually built in.

Burton Inc, Keio University and AIST has made a true 3D display.

DisadvantagesDisadvantages

Suck up a lot of bandwidth

Unable to deal with occlusion and opacity without limiting the FOV: Image appears distorted

The cost of a volumetric display is much greater than that of its closest competitors, such as virtual retinal displays (retinal scanning devices). They use hundreds of times more light and power.

How far are we from Star Wars

A volumetric display device is a graphical display device that forms a visual representation of an object in three physical dimensions, as opposed to the planar image of traditional screens that simulate depth through a number of different visual effects

The Graphics Lab at the University of Southern California has designed an easily reproducible, low-cost 3D display system with a form factor that offers a number of advantages for displaying 3D objects in 3D. The display is:

autostereoscopic - requires no special viewing glassesomnidirectional - generates simultaneous views accomodating large numbers of viewersinteractive - can update content at 200Hz

The display consists of

•A high-speed video projector

•A spinning mirror covered by a holographic diffuser

•A FPGA circuitry to decode specially rendered DVI video signals

•A standard programmable graphics card to render over 5,000 images per second of interactive 3D graphics, projecting 360-degree views with 1.25 degree separation up to 20 updates per second

ANISOTROPIC SPINNING MIRROR

3D display uses an anisotropic holographic diffuser bonded onto a first surface mirror.

Horizontally, the mirror is sharply specular to maintain a 1.25 degree separation between views.

Vertically, the mirror scatters widely so the projected image can be viewed from multiple heights.

http://gl.ict.usc.edu/Research/3DDisplay/

About Encoding

The FPGA takes each 24-bit color frame of video and displays each bit sequentially as separate frames . Thus, if the incoming digital video signal is 60Hz, the projector displays 60×24 = 1,440 frames per second. To achieve even faster rates, we set the video card refresh to rates of 180-240Hz.

Sony RayModeler

http://blog.discover.sonystyle.com/360-degree-3d-prototype-takes-gaming-to-new-levels

There is not much information and no paper for the device. However, we can see the refresh in the video, reveals that is kind of TV technique

We can only guess from what we can see. In the 2 snapshots from the SONY website. We can see RayModeler did not create 3D scene just 360 show pictures to 360 degree. Maybe kind of raster- based view control

(Favarola et al., 2001)

DMD-based volumetric displayDMD-based volumetric display

Projection Projection DMD engine DMD engine

motormotor

CondenserCondenser lenslens

lamplamp

ColdColdmirrormirror RotatingRotating

mirrorsmirrors

Translucent screenTranslucent screen

fold fold mirrormirror

The display produces 200 disk-shaped slices each refreshed at 20 Hz; Resolution 768x768, 8 colors; 10”-diameter spherical image;360º x 180º viewing angle.

Actual system assembly Actual system assembly

(Favarola et al., 2002)

Auto-stereoscopic 3-D Display produced by Actuality Systems

(www.actuality.com)

Comparison of Volume DisplaysLight Field Display Sony RayModeler Perspecta

Core Structure Spinning Mirror Special arranged LED

Rotating Screen

Shape Sphere Cylinder Sphere

resolution 768*768*288 96*128 768*768*198

Contradiction high high low

Color BW by 1 face mirror Multicolor by multi-face mirror

Colors 8 Color

Fresh Rate 20 Scenes/sec 24 Scenes/sec

Interactive handle Gesture, Game pad Contact, Gesture

price Claim to be a cheap Solution

New Interaction Techniques with Volumetric DisplaysNew Interaction Techniques with Volumetric Displays

(Balakrishnan et al, 2001)

Depth perception studies done at University of Toronto (2006) Depth perception studies done at University of Toronto (2006)

Compared desk-top mono, stereo and stereo with head tracking with depth perception in a volumetric display Tasks were 1) judge the position of a sphere in a cube; 2) decide if two objects were going to collide or pass by each-other

http://www.dgp.toronto.edu/~tovi/#Publications

Selection techniques study done at University of Toronto (2006) Selection techniques study done at University of Toronto (2006)

The display was this time only volumetric, but object selection technique varied Within a cluttered virtual world a ray technique will intersect several objects not just the target object; Selection by point cursor, depth ray, lock ray, flower ray and smart ray (below) Movement times were largest for the smart ray and smallest for depth ray; However error rates for new techniques were smaller (13% depth ray, 11% lock ray and flower ray and 10% smart ray) vs. point cursor (21%).

http://www.dgp.toronto.edu/~tovi/#Publications

http://charafantah.wordpress.com/2008/01/06/running-out-http://charafantah.wordpress.com/2008/01/06/running-out-of-time-%D9%85%D9%81%D9%8A%D8%B4-of-time-%D9%85%D9%81%D9%8A%D8%B4-%D9%88%D9%82%D8%AA/%D9%88%D9%82%D8%AA/

http://www.youtube.com/watch?v=8KaQmn2VTzshttp://www.youtube.com/watch?v=8KaQmn2VTzs

http://www.youtube.com/watch?v=He2QTpelAjEhttp://www.youtube.com/watch?v=He2QTpelAjE

http://en.wikipedia.org/wiki/Persistence_of_visionhttp://en.wikipedia.org/wiki/Persistence_of_vision

http://en.wikipedia.org/wiki/Volumetric_displayhttp://en.wikipedia.org/wiki/Volumetric_display

SourcesSources

Documents

Emerging Applications of VR Electrical and Computer Engineering Dept