Physical I/O devices Part 2: haptic output and other non-visual … · 2012-10-21 · • Haptic output needs to be correlated with video and audio stimuli. • Vibrations of gamepad

Department of Signal Processing

Physical I/O devicesPart 2: haptic output and other

non-visual displaysSGN-5406 Virtual Reality 2012

Atanas Boev

based on material byStanislav Stankovic and Ismo Rakkolainen

1

SGN-5406 Virtual Reality 2012


Outline2


Physical I/O devices

Part 1: Input (haptic sensors)

Passive Active

Nofeedback

Movementsupport

Intentionalfeedback Tactile Kinesthetic

VestibularHaptic

Olfactory

Part 2: Output (non-visual displays)

End-effector

Forcefeedback

3D “fly” mouseWiimotePS MoveKinectVR Gloves3D ProbesAccelerometersEtc…

KeyboardMouseJoystickTouch screensSteering wheel3D “desk” mouse(SpaceNavigator)Etc…

Combination ofinput+output

Surgery simulatorNovint FalconCyberGrasp glove

Virtual keyboard +force feedback


DISPLAYS IN GENERALOutput

3



Output devices

• Present the artificial computer generated stimuli to the user.

• Provide the information about the state of VR environmentto the user.

• Provide feedback about the results of user’s actions.

• Work with human sensory organs:• Visual system• Auditory apparatus• Haptic, (tactile, kinesthetic)• Rarely with other senses – olfactory (smell), taste, etc.


Displays

Display = Output Device

Video display:• Computer Screen• Projector• HMD – Head Mounted Display• 3D Screens

Audio display:• Speakers• Headphones

Haptic display:• Tactile displays• Force Feedback Devices


HAPTIC DISPLAYSDisplays

6



Skin senses (reminder)

• Tactile – sense of touch• Kinesthesis – sense of joint position and effort

• E.g. carrying heavy load affects both tactile (pressureon hands) and kinesthetic (muscle effort) senses.

• The tip of the finger• The most sensitive• Even 20 nm movement

• Termoception – sense of hot and cold (15-45°C, 0.001°/s)



Types of haptic displays

• Tactile - provides artificial stimuli (touch,vibration) to receptors in our skin

• Kinesthetic – provides artificial sense offorce• End-effector – limits the natural movement, thus

providing feeling of solid objects / passive resistance• Force feedback – applies force, gives sense of active

movement / impact



Multimodality

• Haptic output needs to be correlated with video andaudio stimuli.

• Vibrations of gamepad paired with events in thegame (character getting hit).

• Vibration of a mobile phone while phone is ringing.

• Vibration of the device when a virtual key is pressedon the touch screen.


Haptic display controller



TACTILEOutput devices

11



Actuators

• Actuators, devices by which artificial stimuli ispresented to human skin.

• Actuators present some force to a region skin.

• Types of actuators:• Blader actuators – pneumatic, hydraulic• Vibrator actuators – electromechanical• Pin actuators – electromechanical• Piezoelectric devices.• Electro-active polymers.• Etc.


Bladder Actuators

• Pockets that can be expanded and contracted:• By controlling the flow of air (pneumatic)• By controlling the flow of liquid (hydraulic)

• Strategic placement of the pockets creates thesensation of pressure on different areas of theparticipant’s hand and body


Vibrator Actuator

• Most often found in mass market products:• Game controllers• Mobile phones• Portable consoles• Data Gloves• Simulator seats

• Easier to control then other types of devices.• Very robust and easy to implement.• Can’t convey the sense of surface texture or shape of the object.• Offer a limited range of effects.

• Low-frequency speakers (subwoofers) can also be used as a vibratorydisplay.


Case study - Haptic Compass

• Belt with 12 vibration devices.• At any given moment the device facing north vibrates.• Augmentation of human senses. Humans do not have an explicit compass

sense. This device gives acute awareness of sense of direction.

• See• Udo Wächter, University of Osnabrück• http://feelspace.cogsci.uni-osnabrueck.de/


Pin Actuators

• Small pin arrays placed on each finger.

• Height of pins controlled electronically.

• Textures are detected by pressure variationsacross the fingertip over time.

• Good for displays for blind people


Case study - interactive terrain

Northrop Grumman’s TerrainTable.

Array of 4600 pins push up touch-sensitive silicone screen.

Overhead projector projects 2D map.

Military applications.


Temperature Actuators

• Can very rapidly present temperature fluctuations,typically to finger tips.

• Danger of tissue damage.• Limits must be adjusted to the safe range of temperatures tolerated by

human skin.

• Not the ”real” temperature of the simulated objects.

• Use scenarios rare, not many practical implementations.


Senseg E-Sense

• Tactile feedback on touch screens.

• Screen is covered by a grid ofelectrically activated elements.

• Tixel a tactile pixel.• Tixels generate a controlled electric field which

extends several millimeters above the surface.• Exploits electro-sensory phenomenon.

See: http://senseg.com/technology/system-architecture


Senseg E-Sense

• Ultra-low electrical current ispassed into the insulatedelectrode – the tixel.

• Electrical charge creates a smallattractive Coulomb force to fingerskin.

• By modulating this attractiveforce, any number of touchsensations can be generatedfrom vibrations, clicks, texturedsurfaces, etc.


Aurborne tactile display

• SIGGRAPH 2008 EmergingTechnologies: Airborne UltrasoundTactile Display• Mid-air tactile sensations by means of

airstreams generated by ultrasonicactuators

• SIGGRAPH 2009 Emerging Tech:Touchable Holography• See: http://www.youtube.com/watch?v=Y-

P1zZAcPuw



KINESTHESICOutput devices

22



End-effector Devices

• Force Feedback – End-effector Devices.

• End-effector devices are a special class of forcefeedback devices.

• Users limbs, hands, arms, legs are in contact withmachinery.

• Input and output device.• Movements on hands serve as input.• Device provides feedback through the active force.

• Generally linked to mechanical tracking sensors.


Telesurgery devices

• Stanford Research Institute,SRI International.

• M7 Surgical Robot.

• Developed for NASA.

• Can perform operations inZero gravity.

• Compensates for unwantedmovements in Zero-G.


End-effector Devices

• UNC Nanomanipulator.

• Microscop image on screen.

• Mechanical arm with 6DoF.

• Forcefeed back.

• See:http://cismm.cs.unc.edu/tag/nanomanipulator/


Force Feedback Data Glove

CyberForce system by CyberGloveSystems.

Mechanical system that exerts forceon hand and arm.

Sense of weight and inertia whilepicking up a "heavy" virtual object

Feel resistance of a simulatedobjects.

http://www.cyberglovesystems.com/products/cyberforce/overview


Components of End-effectorDisplays

Mechanical trackers.

Force generating device.

Technologies:• Electronic motors – one for each DoF,• Hydraulics,• Pneumatics


Features of End-effector Devices

• Often operates also as an input device, potentially providingresistance to input controls.

• Mechanical movement sensors are generally incorporated directlyinto the system. Mechanical tracking is generally very fast andaccurate.

• Typically operate with respect to single point in the virtual world.

• Number of DOFs: 1-6.

• Tactile displays e.g. vibrator actuators can be mounted within theend-effector.

• Can also be constructed to enhance the user’s force.


Full Body Devices

End-effector devices which affect user’swhole body.

Two mayor types:• Exoskeletons

(wearable robotics)• Surround platforms


Exoskeletons

• Exoskeletons – Meaning external skeleton.

• Can work with whole body or just with some parts.

• Exoskeletons are not just I/O device.• Applications:

• Military:• Enhance the power of user’s body or limbs. Increase the load a person can

carry or prologue the time a person can endure the load.• Medical:

• Restore mobility in paralyzed limbs.• Telepresence:

• Operate machinery at a distance, in environments dangerous for humans.


Exoskeleton Examples

• Novint XIO game controller.

• Partial exoskeleton.

• Arm exoskeleton.

• Mass market product.


Exoskeleton Examples

• Hybrid Assistive Limb.

• Cyberdine Inc. And Tsukuba Uni.Japan.

• HAL detects bioelectrical impulsessent from brain to limbs, using a set ofelectrodes on the skin.

• Possible applications:• Rehabilitation,• Physical training,• Help for disabled people,• Rescue services,• Entertainment (as input device)


VESTIBULAROutput devices

33



Vestibular Displays

• Work with human vestibular apparatus (sense of equilibrium).

• Work by physically moving the user.

• Artificial sense of equilibrium, acceleration and orientation that user isexpected to feel during a real motion.

• Strong relationship between the vestibular and visual systems of humans.

• Prevents (or causes) simulator sickness.

• Makes better immersion.

• Typically used in specialized simulators (flight simulators).


Vestibular displays

• D-BOX Motion Code movie seat.

• Moving seat platform – 4 to 6DoFdepending on model.

• D-BOX Motion controller device.

• Movie theaters and amusement parks.

• http://www.d-box.com


Galvanic Vestibular Stimulation

• Vestibular apparatus works on the sameprinciple as human auditory system.

• Works on the same principle as Cochlearimplant.

• Sending specific electric messages to anerve in the ear that transmits balanceinformation.

• Can alter one’s perception of balance.• See also: lecture VR4.3

• Side effect are still not known.

http://www.youtube.com/watch?v=ZlNfBrXYYTc


Locomotion Platforms

• User needs to remain within certain limited physical space.• User needs to have an impression of physical movement in

virtual space.• Simulate the physical movement of the user in VR environment,

while keeping user in the same physical position.

• Examples:• Omni-directional treadmills.• CirculaFloor.


Omni-directional treadmill

• Two treadmills placed on top ofeach other.

• Movement along X and Y axes.

• Platform provides the sensationof moving ground.

• Suspension system to hold theuser in place.


CirculaFloor

• Several movable tiles.

• With each step a newsegment moves infront ofuser’s feet.

• Movement of the segmentcounters the movement ofuser while maintaining theillusion of motion.

See: http://youtu.be/rYsvB2y2Ero


Haptic Rendering

• Rendering – process of generating of artificialstimuli, based on the current state of VRenvironment.

• Needs to be fast, up to 1000 Hz.

1. 3D objects & properties loaded from the database.2. Collision detection. Only colliding objects are

passed on.3. Compute collision forces, smoothing, mapping.4. Haptic texturing (vibrations, temperature, etc.).5. Present output to the user through haptic display.


OLFACTORYOutput devices

41



Olfactory displays


Digital signals in software code trigger the generator to emitprecise amounts of the appropriate aroma.


Olfactory displays

• Artificial smell simulation• Mostly at research, prototype phase, some niche products

• Film: Scent of Mystery (1960), with accompanying smells• Sensorama 1960, Smellizer 1984

• Enhanced cosmetics, perfume, food advertising, aromatherapy

• Aromas can be used to enhance the experience and trigger fear,excitement and other emotions

• Multi-sensory magazines (Esquire cover, smell etc.)

• Companies• http://www.structuralgraphics.com/sensory-effects/• http://www.aerome.com/• http://www.aromajet.com/• http://www.digiscents.com/• http://www.trisenx.com/intro.html



Case study – ScentScape

• ScentScape™ specifications:• Provides 20 basic scents per

cartridge• Scent cartridges last 200 hours or

more in heavy use, depending onpersonal settings

• ScentScape™ scent cartridges willbe available in standard andmedia-specific versions

• Separate "volume control" to adjustoverall smell strength for personalpreference


See: http://www.scentsciences.com/products/scent_scape.html

Documents

Physical I/O devices Part 2: haptic output and other non-visual … · 2012-10-21 · • Haptic output needs to be correlated with video and audio stimuli. • Vibrations of gamepad