Upload
phungthuy
View
215
Download
0
Embed Size (px)
Citation preview
VIRTUAL REALITY
Input devices. Technologies for the
Curs 2012/2013
direct interaction
Motivation
• Virtual Reality systems use, apart from conventional hardware, a specific hardware which is worth to know.
• In general this hardware is not well known, but:
2
• In general this hardware is not well known, but:– It spends a big part of the cost of a complete VR
system.
– The success or not of the system can depend on the correct choosing of these devices.
Motivation (ii)
• Moreover, the market of virtual reality hardware presents the following characteristics:
– A few distributors (and not well trained!)– Dificulties for testing before buying.
3
– Dificulties for testing before buying.– Dificulties to valuate the product from the
documentation.– Market in continuous evolution (products and
manufacturers).
� Importance of the training in VR input systems.
Overview
1. Concepts and classification 2. Obtaining position (trackers)3. Data gloves
4
3. Data gloves4. Other input devices5. Kinect6. Haptic devices (input / output devices)
1. Concepts
• Input devices (sensors): capture the participant actions (for example the head movements) and send this information to the computer which is in charge of the interactive simulation.
• They are considered as virtual reality devices when:
5
– They use the paradigm of the implicit interaction, or
– They give 3D input to the system.
• They are related, but not considered from virtual reality:
– Devices for the aquisition of 3D models.
Classification• Implicit interaction:
– Trackers
– Data gloves
– Voice recognition
– Kinect
6
• 3D Input:– Mouses 3-6DOF.
– Space balls, etc.
• Haptic devices (Input-Output)
• Capture (used to create virtual worlds)– 3D scans
Trackers
Trackers
• Objective– The tracking input systems are sensors which are in
charge of capture the position and/or orientation of a real object.
• Possibilities
8
• Possibilities– Only position (3 DOF)– Only orientation (2-3 DOF)– Position and orientation (6 DOF)
Applications
• Head-tracking, eye-tracking
– Images eye-referenced
– Navigation based on head movements
9
Applications
• Hand-tracking, device-tracking
– Direct manipulation of objects (multiple metaphors).
– Navigacion hand based (together with gloves).
– Signs language. Comunication with other users.
10
Applications
• Face-tracking, Body-tracking
– Interaction based on avatar (games)
– Capture off-line of complex sequences (sports)
11
About position
The position can be done by two ways:
• Absolute position (x, y, z): the position is given in knonw units with respect to a fixed coordinated system.
• Relative position (dx, dy, dz): the position is given by a
12
displacement with respect to the initial position or to the one before.– Require calibration (should be in a known position at the initial).
– The error is acumulated.
– Non exact.
About orientation
• Two reference systems are defined:– The global one (fixed, referenced for example to the emitter)– The local one (dynamic, referenced to the object)
• We are interested on knowing the rotation (GT matrix) which transforms the global RS in the local RS.
13
which transforms the global RS in the local RS.• The rotation can be represented as follows:
– Direction cosines– Euler angles– Angle axis representation– Spherical coordinates, polar coord– Quaternion
Direction cosines
• The orientation is represented by 9 values:ux, uy, uz << Unit vector in the direction of X vx, vy, vz << Unit vector in the direction of Ywx, wy, wz << Unit vector in the direction of Z
14
• The rotation matrix is obtained by having the 3 vectors in columns.
• Is the more complete and intuitive representation.• Is very redundant (9 reals!)
Euler angles
• The orientation is represented by 3 values:x, y, z << Rotation angles with respect each axe
• The rotation is obtained by:
Rotx, Roty, Rotz
15
x y z
• In order to avoid the ambiguation the following should be fixed:
– an order to apply the rotations (XYZ, XZY...) – an interpretation (alibi, alias).
• Is very intuitive.
Euler angles (ii)Two interpretations or ways to visualize a rotations composition Rx , Ry , Rz:
• Alibi:All rotations are done with respect a one of the three axes of a Global Reference System which is fixed, in this
16
Reference System which is fixed, in this order Rx, Ry, Rz.
• Alias: The rotations are done with respect to a Local Reference System attached to the object, in the reverse order: Rz , Ry , Rx.
Euler angles (iii)
• Problems:– The rotations are not defined in a unique form:
(z, x, y) = (90, 45, 45) = (45, 0, -45)
17
(z, x, y) = (90, 45, 45) = (45, 0, -45)
– The cartesian coordinates are independent among them; the Euler angles no.
Euler angles (v)
• Sometimes, some mnemotechnics are used for the angles, which come from the aeronautic:
yaw (azimuth, twist)
18
yaw (azimuth, twist)
pitch (elevation, tilt)
roll (somersault)
Angle axis representation
• The orientation is represented with four values:x, y, z << Unit vector (arbitrary rotation axe) angle << Rotation angle with respect to the vector
19
• The rotation is obtained by rotating angle grades around the axe determined by the vector (x, y, z).
Spherical coordinates
• The orientation is represented by 3 values:long, lat << Longitud/latitud of a unit vectorangle << Rotation angle with respect to the vector
20
• Is equivalent to the angle axis representation, but the unit vector is represented in spherical coordinates.
Quaternion
• The orientation is represented with four values:x, y, z << Unit vector (arbitrary rotation axe) w << Rotation with respect to the vector
• The rotation is obtained by rotating 2*acos(w) around the axe determined by the vector (x, y, z).
21
axe determined by the vector (x, y, z).• Is a notation very similar to the angle axis
representation; it has advantages in animation, because it’s easy to interpolate between two orientations.
• Is less intuitive.
Technologies
Technology of trackers:– Magnetic– Optic
22
– Optic – Acoustic – Mechanic– Innertial
Magnetic trackers
How they work:• Are based on the attenuation of
the orthogonal magnetic fields.• Each system includes:
23
- Emitter: generates the magnetic field.- Receptors: detect the magnetic field.- Control unit.
Emitter:– Is fixed. Defines the global RS.– It has three orthogonal bobines
through which the electric power
Magnetic trackers
24
through which the electric power is passing in a rotative way.
– Each bobine produces a magnetic field perpendicular to the others (sequential in time).
Sensor:– They are fixed to the object
you want to follow.– They have 3 bobines, but in
Magnetic trackers
25
– They have 3 bobines, but in this case the mesurement the electrical power induced by the magnetic field which envolves the sensor.
– They are usually tied to the central unit by a cable.
Control unit:– Gives power to the emitter– Read data from the receptors– Solve an equacions system:
Magnetic trackers
26
– Solve an equacions system:• Variables: 6 (3 position + 3 orientation)• Equations: 9 (3 lectures/field * 3 fields) • Is undetermined (hemisphere).
– Send the data (via RS-232, for ex.)
Advantages
• Are the most used.
• Do not require “line-of-sight”
• Very precise.
Magnetic trackers
27
• Very precise.
• Multiple receptors; small.
Inconvenients:
• Area of action limited (2 and 10 meters)
• Sensitive to ferromagnetic surfaces
• Sensitive to other magnetic fields
• Cables
Optical trackers
How they work:
• Based on image processing.
• Different sort of:– Estereo cameras (2 or more known
cameras)
28
cameras)
– Làsser
– Other techniques of image processing (siluet...)
Estereo cameras (2 or more known cameras)– For each point p to be registered:
• It detects the position in each image (x, y), (x’, y’) (correspondence problem).
Optical trackers
29
(correspondence problem).
• A point (x,y,z) is obtained by intersecting 2 straight lines.
– Usually require markers:• Pasives: color points
• Actives: LEDs
Advantages
• They can register a big quantity of points
• Relatively cheap
Inconvenients
Optical trackers
30
Inconvenients
• Line-of-sight
• Limited work space to the FOV of cameras
• Overhead (or dedicated processor).
• Latency time variable.
How they work:• Based on mesuring the propagation time (time-
of-flight) of acoustic signals (ultrasons) generated by loudspeakers and registered by
Acoustical trackers
31
generated by loudspeakers and registered by mycrophons.
• Different sort of:– Fixed emitters, mobile receptors– Mobile emitters, fixed receptors
• Several combinations:– A mobile emitter and three fixed receptors:
• Enough to calculate position (3 DOF).• Intersection of three spheres.
Acoustical trackers
32
• Intersection of three spheres.
– Three emitters and three receptors: • Enough to calculate position and orientation (6 DOF)• The orientation is trivial from the 3 points (for ex. origin, point
in X axe and point in Y axe).
Fixed receptors
Mobile Transmittersdistance1
Acoustical trackers
33
distance2
distance3
• Advantages– Very reduced cost
• Inconvenients– Line-of-sight– The signals are distorted very easily.
Acoustical trackers
34
– The signals are distorted very easily.– Sensitives to ambient noise, eco...– Latency proporcional to the distance (ex. 3 meters, 10ms
(minimum).– Not very precise.
Mecanical trackers
How they work• Articulated structure with
potenciometres attached to the joins.• Each potenciometre mesure the
angle of rotation.
35
angle of rotation.• The base point is fixed.• The other side point is free• The position and orientation of the
free part: concatenation of translations and rotacions
Advantages• Fast (almost no latency)• Very precise • Electronic very simple.
Mecanical trackers
36
• Electronic very simple.
Inconvenients• Require to fix the object to the free part of the structure.• Limits the freedom of movements.• Cost depends on the mechanical structure.
Innertial trackers
How they work• Based on small devices which mesure
the acceleration which they have been submitted.
• They detect acceleration (accelerometres) and the rotatory
37
• They detect acceleration (accelerometres) and the rotatory orientation (giroscops).
• By integring over the time the acceleration we obtain the velocity; by integring the velocity we obtain the position.
Advantages:
• Unlimited area of action. Do not require cables and are not tied to a fixed RS.
Inconvenients:
Innertial trackers
38
• The position is relative.
• The errors are accumulative.
• Non exact to mesure absolute position.
• Sensitives to the movement velocity!
Data Gloves
Data gloves
Function• Allow to detect the position of the hand fingers, expresed
by the flexion angles of each finger.• Three angles per finger (except the thumb - 2).• They are used combined with trackers.
40
• They are used combined with trackers.
Aplications• Direct manipulation of objects.• Intuitive interaction (navigation, menus)• Interaction based in signals
Technologies• Optical fiber• Mechanic• Resistence
Data gloves
41
• Resistence• Optic
Gloves of optical fiber
How they work• Atenuation of the light which
goes through the optical fiber.• Each finger: one or two circuits
of optical fiber especialy treated.
42
• The attenuation of light depends on the curvature.
• Diference between emmitted light and received light.
Advantages• Very precise• Ergonomic• Veri fastInconvenients:
Gloves of optical fiber
43
Inconvenients:• Non exact.• Require constant calibrations.• The curvature of the fiber do not always correspond to
the one on the fingers (friction).
Mechanical gloves
How they work• Similar to mechanical trackers.• The potenciometres are
incorporated in a structure similar to an exoesquelet.
44
similar to an exoesquelet.• Other models mesure the
elongation of a telescopic tube.
Advantages• Exacts and precise
• Very fast
• Possibility of force-feedback.
Mechanical gloves
45
• Possibility of force-feedback.
Inconvenients• Expensive
• Non ergonomic
Resistence gloves
How they work:• Resistive ink: special cover which offers electrical
resistence variable depending on the elongation. • Technology of the popular PowerGlobe of MattelAdvantages:
46
Advantages:• Very cheapInconvenients:• Non precise.
Optical gloves
How they work:
• Based on LEDs or passive markers, captured by video cameras.
Advantages:
47
• No cables.
• Ergonomic.
Inconvenients:
• Line-of-sight
• Lack of precision. High latency.
Other devices
• Microphon– Voice recognition– Distance interaction (teleconference)
• Devices of aquisition of 3D models
48
• Devices of aquisition of 3D models• Devices of 3D input
Devices of 3D input
2 DOF:– With base surface:
• Classical mouse, classical joystick
– Without base surface:• Track ball, joystick PSX
49
3-6 DOF: – With base surface:
• Space ball: displacement and angular moment
– Without base surface:• 3D balls, Stylus, 3D joysticks...
Kinect
Microsoft Kinect
• New controller for Microsoft’s Xbox 360• Full-body tracking, face and voice recognition• Low cost (99 Euros as of April 2011)
51
Main components
52
Main components
Video• Color CMOS camera• Infrared (IR) CMOS camera• Infrared projector - 830nm, 60mW laser diode.
Audio
53
Audio• Four microphones
Tilt control• Motor• Accelerometer (3-axes)
Processors & memory• PrimeSense chip PS1080-A2• 64 MB DDR2 SDRAM
Image sensors
• Color camera: 640x480 sensor, 640x480@30fps output• IR camera: 1280x1024 sensor, 640x480@30fps output• Operation range (depth sensor) = 0.4m - 3.5m • FOV = 58°H, 45°V, 70°D • Spatial resolution (@ 2m distance) = 3mm
54
• Depth resolution (@ 2m distance) = 1cm
Depth sensing
• The IR emitter projects an irregular pattern of IR dots of varying intensities.
• The IR camera reconstructs a depth image by recognizing the distortion in this pattern.
55
Connecting the Kinect to a PC
Trivial for “old” Kinect models (those sold separately): USB-2 standard connector
56
Further info on Kinect’s hardware
http://www.ifixit.com/Teardown/Microsoft-Kinect-Teardown/4066/1http://openkinect.org/http://www.freepatentsonline.com/7433024.pdf
57
Drivers
libfreenet low-level data(unofficial) color & depth map
Kinect OpenNI + NITE high-level data Appl.
58
Kinect OpenNI + NITE high-level data Appl.dirvers + Sensor kinect gestures / poses
Windows drivers
libfreenect
• www.openkinect.org• Unofficial, open source driver (GLP2)• Linux, Windows, OS X• Raw data (color map, IR map, depth map)
59
• Raw data (color map, IR map, depth map)• Installation: trivial
OpenNI + NITE + SensorKinect
• OpenNI– SDK for natural interfaces; open source– www.openni.org
• NITE– OpenNI Plugin for gesture/pose recognition; closed source
60
– OpenNI Plugin for gesture/pose recognition; closed source– www.openni.org
• SensorKinect– Driver for the Kinect (open source)– github.com/avin2/SensorKinect
OpenNI
• Framework for modules/sensors providing depth maps, color maps, scene maps, gesture recognition, user pose (skeleton)
• www.openni.org
61
OpenNI
OpenNI aims at abstracting developers from sensor hardware (Kinect, PrimeSensor…) and CV algorithms
(scene analysis, gesture recognition, pose recognition…)
62
NITE
Middleware for pose/gesture recognition
63
NITE
• Joint positions: given in real world coords (mm)• Joint orientations: 3x3 rotation matrix representing the rotation of
joint’s local coords w.r.t world coords (the first column is the direction of the joint’s +X axis in world coordinates, and so on).
64
Another related sensor
65
Haptic devices
• Touch: feeling caused when the skin receives a mechanical, heating, chemical or electrical stimulus
The sense of touching
67
– Thermoreceptors: they respond to the heating changes
– Mecanoreceptors: they respond to a mechanical action (force, vibration, sliding)
– Nocioreceptors: they transmit pain/ache
• Sensorial accomodation: it quantifies the variation on time of the electrical discharge number generated by a certain receiver in
The sense of touching (II)
68
number generated by a certain receiver in response to a stimulus.– slow accomodation– fast accomodation (glasses, gloves)
• Spatial resolution: The reception field is defined as the area where a stimulus can excite the receiver (beteween 1-2 mm2 and 45 mm2)
The sense of touching (III)
69
45 mm )– it depends of the body part (lower leg and leg
around 45 mm2, while hand fingers between 1 and 2 mm2)
– directly related with how big is the area of the brain that interpret the stimulus (higher spatial resolution, bigger area in the brain)
The sense of touching (IV)
70
The sense of touching (V)
71
• Time resolution: minimum time needed to be able to distinguish between two consecutive successes– aproximately around 5ms, much shorter than the
one of the view, 25ms (“hand is much faster than
The sense of touching (VI)
72
one of the view, 25ms (“hand is much faster than eye”)
– But in order to recognize the order the stimuli have been produced it requires 20 ms aprox.
– Welch i Warren (1986): response to a touching stimulus = 0.11 s, but to recognize braille code response = 0.87-1.56 s
• Propioception: sense in charge of inform of the position the body parts have
• Cinestesia: feeling that gives information of their movement
The sense of touching (VII)
73
their movement– the resolution is mesured in grades and indicates
the difference of angle which is non perceived as a change of position in a body join (the hip has the biggest precision, while the foot fingers the smallest)
• Passive touching: feeling when an object touch the skin (only stimulates the cutaneous receivers, non owned movement).
• Active touching: feeling produced when we
The sense of touching (VIII)
74
• Active touching: feeling produced when we touch in an active form an object (push buttons, slide over surfaces, …)– active better to identify objects by touching– Haptic perception: capacity to identify solids by
touching them.
Requirements to simulate touching
• The most used contact element is the hand• We need the force range a user can produce
and feel (values between 16N and 102N, different for man/woman)
75
different for man/woman)• We have to distinguish between the
maximum force for a short time or a continued force
• Sensitive bandwidth: frequence to what the touching stimuli is noticed, propioceptives and cinestetics
• Active bandwidth: velocity with which one can respond to the stimuli
Requirements to simulate touching (II)
76
• Active bandwidth: velocity with which one can respond to the stimuli
– the sensitive is much bigger than the active– for the human finger, the active is between 1 and
16 Hz while the sensitive is between 30 Hz and 10 kHz
– having a frequency of 1000 Hz we can simulate almost all the touching efects
Haptic systems
• The haptic feedback is related with the sense of touching (stimulus of contact) and the propioceptive and cinestetic
• The most important characteristic is the bidireccionality
77
bidireccionality
User Virtual environmentTouch
Audio
Video
ForceVelocity
ForceVelocity
Info. sensors
Info. actors
• Haptic system: allow the user to touch, feel and manipulate virtual objects by restitution tactile and cinestetical.
– it consists of: haptic interface + virtual environment
Haptic systems (II)
78
– it consists of: haptic interface + virtual environment
User Virtual environmentHaptic device
Haptic interface
Virtual coupling
Haptic system
REAL VIRTUAL
• Haptic devices:
Haptic systems (III)
79
Haptic rendering
• The objective of the haptic rendering is to allow the user to touch, feel and manipulate virtual objects by using a haptic interface
80
Requirement:
frequency = 1000 Hz
Haptic rendering (II)
• When the device position is detected colliding with the virtual object, a force is produced to move the device to the object surface
81
Haptic rendering (III)
• Existing interaction methods:
82
Haptic rendering (IV)
• Interaction with an only point:
Polygonal models, flat and smoth faces
Polygonal models with effects of surface (friction, textures,…)
83
Haptic rendering (VII)
• Interaction with an object:
The collision detection is more complicated
84
It requires position and orientation
Resultant forces over different points
Acceleration compute techniques are needed (bounding boxes, voxels, …)
Deformable objects?
Applications• Medicine:
– Surgery simulation– Tele-medicine– Training– Patient rehabilitation pacients for neurologic
problems• Mechanical design:
85
• Mechanical design:– Pieces assembling
• Entertainment:– Paint in 3D– Characters animation
• Scientífic:– Geophysical data analysi– Molecular manipulation
VIRTUAL REALITY
Input devices. Technologies for the
Curs 2012/2013
direct interaction