Upload
kimi
View
18
Download
1
Embed Size (px)
DESCRIPTION
ISR – Institute of Systems and Robotics University of Coimbra - Portugal. WP5: Behavior Learning And Recognition Structure For Multi-modal Fusion Part I. Relationship of the WP3,4 and 5. WP3 ( Sensor modeling and multi-sensor fusion techniques ) Task 3.3 Bayesian network structures - PowerPoint PPT Presentation
Citation preview
University of Coimbra
ISR – Institute of Systems and RoboticsUniversity of Coimbra - Portugal
WP5: Behavior Learning And Recognition
Structure For Multi-modal Fusion
Part I
University of Coimbra
Relationship of the WP3,4 and 5
WP3(Sensor modeling and multi-sensor fusion techniques )
Task 3.3Bayesian network structures
for multi-modal fusion
WP4(Localization and tracking techniques as applied to humans )
Task 4.3Online adaptation and learning
WP5Behavior learning and recognition
• Trackers results, • Events detected , • ids on re-identification situations
•Levels of the fusion(pixel, feature or decision level) ,•Bayesian structures for the implementation of the scenarios of WP2
University of Coimbra
Proposal
Applying a multi-modal Occupancy Grid and Multi-layer Homography to reconstruct human silhouette with stationary sensors.
(Indeed using Silhouette (Multi-layer Homography), Texture and Range information to build an Occupancy Grid)
Stationary Sensors (Structure):
• Image Data
• Range Data
• Sound Source Data
University of Coimbra
Inertial Compensated Homography
Projecting a world point on a reference plane in two phases
iI
xiW I
iC
X
Y
Z
iIW
iIW H
wiI
Hrefπ
xiI
xw
XHX
XHX
iIW
wi
refref
i
i
iIW
Iππ
II
W
Real
camera
Virtual camera
'iCi
i
C
C R
[Luiz2007]
Gravity
xrefπ
First step: Projecting real world point on the virtual image plane
Second step: Projecting form virtual image plane on a common
plane
refπ
University of Coimbra
Inertial Compensated Homography
iI
xiW I
iC
X
Y
Z
iIW
iIW H
wiI
Hrefπ
xiI
xw
333231
232221
131211
1
hhh
hhh
hhh
and
wi
ref
i
i
i
Iπ
CC
IW
H
KRKHReal camera
Virtual camera
'iCi
i
C
C R
[Luiz2007]
Gravity
xrefπ
Infinite homography
Homography between two planes
Camera calibration
matrix
Rotation between virtual and real
camera (given by IMU)
00
).1(10
).1(01
y
x
v
v
wi
ref
Iπ H
refπxrefπ
v
xiW I
iIW
refπ
University of Coimbra
Image Registration
iI iC
X
Y
Z
i
refI
π H
xiI
xrefπ
xw
[Luiz2007]
Gravity
XHHX i
iwi
refref II
WI
ππ
jC
kC
kI
jIxjI xkI
j
refI
π H k
refI
π H
refref Gπ
refπ
University of Coimbra
Extending A Virtual Plane To More
Plane to image homography:
Vanishing points for X,Y and Z directions
Vanishing line of reference plane
normalized
[Khan2007]
Vanishing point of reference direction
Scale factor
Scale value encapsulating and z
iI iC
xiI
jC
kC
kI
jIxjI xkI
]|[
]ˆ[
23 refrefref
iW
l
iW
πππI
ZYXπI
v0H
lvvvH
z
refπlπ
X
YZ
Nπ
University of Coimbra
Relationship Between Different Planes In The Structure
iI
kC
xiI
Homography between views i and j, induced by a plane i parallel to ref
having the homography of reference plane:
])|[1
1])(|[( 33 refπrefπ
πI
IπI
I
ref
refπ
ref
ref
iW
jW
l
iW
jW
v0Iv0HH
[Khan2007]
iC
jC
kI
jIxjI xkI
refπlπ
X
YZ
Nπ
Vanishing point of reference direction
Scale value
Homography of reference plane
University of Coimbra
Image & Laser Geometric Registration
iI iC
X
Y
Z
xiI
xrefπ
xw
[Luiz2007-Hadi2009]
Gravity
XTX i
i
refref LL
πLπ
jC
kC
kI
jIxjI xkI
jL
Lπ ref x
refref Gπ
iL
refπ
University of Coimbra
Registering LRF Data In a Multi-Camera Scenario
4..0j| jII
xTHu i
i
LL
W
[Hadi2009]
Reprojection of LRF data on the image (blue points)
Image planes
Projection of points observed by LRF
Transformation matrix between camera and LRF obtained by calibration
Projection of points observed by LRFon the image plane
+Result
A set of cameras and laser range finder
cN...0j| uu jC
cN...0j| jII
1310
tRT ii
i
LW
LW
LW
cN...0j| jHH
Camera projection matrix
Ima
ge
Ra
ng
e d
ata
University of Coimbra
Image & Laser & Sound Geometric Registration
refref Gπ
iI iC
X
Y
Z
xiI
xrefπ
xw
Gravity
jC
kC
kI
jIxjI xkI
jL
Lπ ref x
1M 2M
PA()
[JFC2008, 2009]
iL
refπ
University of Coimbra
Bayesian Binaural System for 3D Localisation
–Binaural sensing
•For sources within 2 meters range, binaural cues alone ( interaural time and level interaural time and level differences – ITD differences – ITD , quasi frequency-independent, and ILDs , quasi frequency-independent, and ILDs L(fL(fcc
kk)))) can be used to fully localise the source in 3D space (i.e. volume confined in azimuth , elevation and distance ).
• If the source is more than 2 meters away the source can only be localised to a volume (cone of confusion) in azimuth.
1m 2m
Binaural cue information +
+
Distance
Elevation
Azimuth θ Azimuth θ only
Z
University of Coimbra
Bayesian Binaural System for 3D Localisation
Subset of [JFC2008, 2009]
CSCL(fck)
Z
University of Coimbra
Direct Auditory Sensor Model: (Direct Auditory Sensor Model: (DASM))(Bayesian learning through HRTF calibration using (Bayesian learning through HRTF calibration using
ITDs ITDs and ILDs and ILDs L)L)
AzimuthAzimuth
ElevationDistance
Binary variable denoting “Cell C occupied by sound-source”
Binary variable denoting “Cell C occupied by sound-source”
Inverse Auditory Sensor Model: (Inverse Auditory Sensor Model: (IASM))
Bayesian Binaural System for 3D Localisation
Bayes Rule
Auditory Saliency MapAuditory Saliency Map
Solution: cluster local saliency Solution: cluster local saliency maxima points (i.e. cells with maxima points (i.e. cells with
maximum probability of occupancy, maximum probability of occupancy, 1 per sound-source)1 per sound-source)(front -to-back confusion effect avoided by (front -to-back confusion effect avoided by
considering only frontal hemisphere estimates)considering only frontal hemisphere estimates)
University of Coimbra
Bayesian Binaural System for Localisation in Azimuth Planes of Arrival
Direct Auditory Sensor Model: (Direct Auditory Sensor Model: (DASM))(Bayesian learning through HRTF calibration of (Bayesian learning through HRTF calibration of
interaural time differences – ITDs interaural time differences – ITDs ))
Inverse Auditory Sensor Model: (Inverse Auditory Sensor Model: (IASM))
-90º 90º
Bayes RuleBayes Rule
Auditory Saliency MapAuditory Saliency Map
Solution: cluster local saliency Solution: cluster local saliency maxima planes of arrival (PA) maxima planes of arrival (PA)
per sound-sourceper sound-source
PA()(front -to-back confusion effect avoided by (front -to-back confusion effect avoided by
considering only frontal hemisphere estimates)considering only frontal hemisphere estimates)
University of Coimbra
Demos on Bayesian Binaural System
(Arrival Direction of Sound Source)
Two Talking persons A walking person
University of Coimbra
Image & Laser & Sound Occupancy Grid
refref Gπ Image, Range and Sound
Occupancy Grid
& Fusion
2π
Nπ
X
refπ1π
ZY
2G
NG
X
refG1G
Discretization
YZ
University of Coimbra
Bibliography
University of Coimbra
Bibliography
•Franco, J. & Boyer, E. Fusion of Multi-View Silhouette Cues Using a Space Occupancy Grid Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), 2005
•Christophe Braillon, Kane Usher, C. P. J. L. C. & Laugier, C. Fusion of stereo and optical flow data using occupancy grids Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2006.
•Saad M. Khan, P. Y. & Shah, M. A Homographic Framework for the Fusion of Multi-view Silhouettes Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, 2007
•R. Eshel, Y. M. Homography Based Multiple Camera Detection and Tracking of People in a Dense Crowd Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, 2008
•Conference (Arsic2008) D. Arsic, E. Hristov, N. L. Applying multi layer homography for multi camera person tracking Distributed Smart Cameras, 2008. ICDSC 2008. Second ACM/IEEE International Conference on, 2008
•Francois Fleuret, Jerome Berclaz, R. L. & Fua, P. Multi-Camera People Tracking with a Probabilistic Occupancy Map IEEE transactions on Pattern analysis and Machine Intelligence, 2008
University of Coimbra
Bibliography•Sangho Park, M. M. T. Understanding human interactions with track and body synergies (TBS) captured from multiple views Computer Vision and Image Understanding, 2008
•Yuxin Jin, Linmi Tao, H. D. R. N. & Xu, G. Background modeling from a free-moving camera by Multi-Layer Homography Algorithm Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on, 2008
•Luiz G. B. Mirisola, Jorge Dias, A. T. d. A. Trajectory Recovery and 3D Mapping from Rotation-Compensated Imagery for an Airship Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems San Diego, CA, USA, Oct 29 - Nov 2, 2007, 2007
•Mirisola, L. G. B. & Dias, J. Tracking from a Moving Camera with Attitude Estimates ICR08, 2008
•Batista, J. P. Tracking Pedestrians Under Occlusion Using Multiple Cameras Image Analysis and Recognition, Springer Berlin-Heidelberg., 2004, 3212/2004, 552-562
•Joao Filipe Ferreira, Pierre Bessière, K. M. C. P. J. L. C. L. & Dias, J. Bayesian Models for Multimodal Perception of 3D Structure and Motion
•C. Chen, C. Tay, K. M. & C. Laugier (INRIA, F. Dynamic environment modeling with gridmap: a multiple-object tracking application 9th International Conference on Control, Automation, Robotics and Vision, 2006. ICARCV '06., 2006
University of Coimbra
Bibliography
•J. F. Ferreira, P. Bessière, K. Mekhnacha, J. Lobo, J. Dias, and C. Laugier, “Bayesian Models for Multimodal Perception of 3D Structure and Motion,” in International Conference on Cognitive Systems (CogSys 2008), pp. 103-108, University of Karlsruhe, Karlsruhe, Germany, April 2008.
•C. Pinho, J. F. Ferreira, P. Bessière, and J. Dias, “A Bayesian Binaural System for 3D Sound-Source Localisation,” in International Conference on Cognitive Systems (CogSys 2008), pp 109-114, University of Karlsruhe, Karlsruhe, Germany, April 2008.
•Ferreira, J. F., Pinho, C., and Dias, J., “Implementation and Calibration of a Bayesian Binaural System for 3D Localisation”, in 2008 IEEE International Conference on Robotics and Biomimetics (ROBIO 2008), Bangkok, Tailand, 2009.
•Hadi Aliakbarpour, Pedro Nunez, Jose Prado, Kamrad Khoshhal and Jorge Dias. An Efficient Algorithm for Extrinsic Calibration between a 3D Laser Range Finder and a Stereo Camera for Surveillance, ICAR2009.
University of Coimbra
Institute of Systems and Robitcs
http://paloma.isr.uc.pt