Embed Size (px)
Citation preview
Consistent Visual Information Processing
Axel Pinz
EMT – Institute of Electrical Measurement and Measurement Signal Processing
TU Graz – Graz University of Technology
[email protected]://www.emt.tu-graz.ac.at/~pinz
“Consistency”• Active vision systems / 4D data streams
• Imprecision
• Ambiguity
• Contradiction
• Multiple visual information
This Talk: Consistency in
• Active vision systems:– Active fusion– Active object recognition
• Immersive 3D HCI:– Augmented reality– Tracking in VR/AR
AR as Testbed
Consistent perceptionin 4D:
• Space– Registration– Tracking
• Time– Lag-free– Prediction
Agenda
• Active fusion
• Consistency
• Applications– Active object recognition– Tracking in VR/AR
• Conclusions
Active Fusion
fusion, contro lin teraction
w orld
w orlddescription
sceneselection
scene
exposure
im age
im age proc.,segm entation
im age
description
grouping,3D m odeling
scenedescription
integration
Simple top level decision-action-fusion loop:
Active Fusion (2)
• Fusion schemes– Probabilistic– Possibilistic (fuzzy)– Evidence theoretic (Dempster & Shafer)
Probabilistic Active Fusion
N measurements, sensor inputs: mi
M hypotheses: oj , O = {o1, …, oM }
Bayes formula:
),...,(
)()|,...,(),...,|(
1
11
N
jjNNj
mmP
oPommPmmoP
Use entropy H(O) to measure the quality of P(O)
)(log)()(1
j
M
j
j oPoPOH
Probabilistic Active Fusion (2)Flat distribution: P(oj )=const. Hmax
• Measurements can be:• difficult,• expensive,
• N can be prohibitively large, …
Find iterative strategy to minimize H(O)
Pronounced distribution:P(oc ) = 1; P(oj ) = 0, j c H = 0
)(log)()(1
j
M
j
j oPoPOH
Probabilistic Active Fusion (3)
Start with A 1 measurements: P(oj|m1, … ,mA), HA
Iteratively take more measurements: mA+1, … ,mB
Until: P(oj|m1, … ,mB), HB Threshold
Summary: Active Fusion
• Multiple (visual) information, many sensors, measurements,…
• Selection of information sources
• Maximize information content / quality
• Optimize effort (number / cost of measurements, …)
Information gain by entropy reduction
Summary: Active Fusion (2)
• Active systems (robots, mobile cameras)– Sensor planning– Control– Interaction with the scene
• “Passive” systems (video, wearable,…)– Filtering– Selection of sensors / measurements
Consistency
• Consistency vs. Ambiguity– Unimodal subsets Ok
• Representations– Distance measures
Consistent Subsets
Hypotheses O = {o1 ,…, oM }
Ambiguity: P(O) is multimodal
Consistent unimodal subsets Ok O
• Application domains
• Support of hypotheses
• Outlier rejection
Benefits:
Distance Measures
Depend on representations, e.g.:
• Pixel-level SSD, correlation, rank• Eigenspace Euclidean• 3D models Euclidean• Feature-based Mahalanobis, …• Symbolic Mutual information• Graphs Subgraph isomorphism
Mutual Information
Shannon´s measure of mutual information:
O = {o1 ,…, oM }A O, B O
I(A,B) = H(A) + H(B) – H(A,B)
Applications
• Active object recognition– Videos– Details
• Tracking in VR / AR– Landmark definition / acquisition– Real-time tracking
Active vision laboratory
Active Object Recognition
Active Object Recognitionin Parametric Eigenspace
• Classifier for a single view
• Pose estimation per view
• Fusion formalism
• View planning formalism
• Estimation of object appearance at unexplored viewing positions
Applications
Active object recognition– Videos– Details
Control of active vision systems
• Tracking in VR / AR– Landmark definition / acquisition– Real-time tracking
Selection, combination, evaluation Constraining of huge spaces
Landmark Definition / Acquisition
corners blobs natural landmarks
What is a “landmark” ?
Automatic Landmark Acquisition
• Capture a dataset of the scene:– calibrated stereo rig
– trajectory (by magnetic tracking)– n stereo pairs
• Process this dataset– visually salient landmarks for tracking
Automatic Landmark Acquisitionvisually salient landmarks for tracking
• salient points in 2D image• 3D reconstruction• clusters in 3D:
– compact, many points– consistent feature descriptions
• cluster centers landmarks
Processing Scheme
Office Scene
Office Scene - Reconstruction
Office Scene - Reconstruction
Unknown Scene
Real-Time Tracking
LandmarkAcquisition
Real-Time Tracking
• Measure position and orientation of object(s) • Obtain trajectories of object(s)
• Stationary observer – “outside-in” – Vision-based
• Moving observer, egomotion – “inside-out”– Hybrid
• Degrees of Freedom – DoF– 3 DoF (mobile robot)– 6 DoF (head and device tracking in AR)
Outside-in Tracking (1)
stereo-rigIR-illumination
• wireless
• 1 marker/device:3 DoF
• 2 markers: 5 DoF• 3 markers: 6 DoF
devices
2D B
lob
Tra
ckin
gE
pipo
lar
Geo
met
ry3D
Cor
resp
onde
nce
3D O bjects and Pose
2D Backpro jection
Epipolar G eom etry
C onstra in ts
3D C orrespondence
3D Prediction
B lob D etection
Tile Q uantisation
Prediction
B lob D etection
Tile Q uantisation
Prediction
W orkspace
O bject M odels
LEFT IM AG E R IG H T IM AG E
Outside-inTracking (2)
Consistent Tracking (1)
• Complexity– Many targets– Exhaustive search vs. Real-time
• Occlusion– Redundancy (targets | cameras)
• Ambiguity in 3D– Constraints
Consistent Tracking (2)
• Dynamic interpretation tree– Geometric / spatial consistency
• Local constraints– Multiple interpretations can happen– Global consistency is impossible
• Temporal consistency– Filtering, prediction
Consistent Tracking (3)
Hybrid Inside-Out Tracking (1)
• 3 accelerometers• 3 gyroscopes• signal processing• interface
Inertial Tracker
Hybrid Inside-Out Tracking (2)
• complementary sensors
• fusion
Summary: Consistency in
• Active vision systems:– Active fusion– Active object recognition
• Immersive 3D HCI:– Augmented reality– Tracking in VR/AR
Conclusion
Consistent processing of visual informationcan significantly improve
the performance ofactive and real-time vision systems
Acknowledgement
Thomas Auer, Hermann Borotschnig, Markus Brandner, Harald Ganster, Peter Lang, Lucas Paletta, Manfred Prantl, Miguel Ribo, David Sinclair
Christian Doppler Gesellschaft, FFF, FWF, Kplus VRVis, EU TMR Virgo
