Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Smart Cars for Safe Driving
Prof. Dr. Dariu M. GavrilaEnvironment Perception
Group Research and Advanced Engineering
XXXII Jornadas de Automática, Sevilla, 9-9-2011
2
We originally thought Machine Intelligence would look like
1956 "Forbidden Planet"
Robby the Robot (Flickr)
3
Then more recently, some suggested it would be more like
2004 iRobot 1983-2009 Terminator
4
when in fact, Machine Intelligence is already with us, and has a familiar embodiement
5
Driver Assistance in the current Mercedes Benz E-Class
Speed Limit
Adaptive
High Beam
PRE-SAFE
Lane Keeping
Blind Spot
Nightview
Plus
Attention
6
•Technology is rapidly expanding the capabilities of modern vehicles.
•One breakthrough development over the past few years is the emergence of
driver assistance systems.
•Use of sensor systems which continuously monitor vehicle surroundings and
interior, provide information to the driver, and even perform vehicle control.
•Help drivers operate their vehicles in a safe, comfortable, and energy-
efficient manner.
• Enables market differentiation for vehicle manufacturers
Driver Assistance
7
What got us here: Sensors
Better and cheaper.
Radars Cameras Laser Scanners
8
What got us here: Computational Power
CPU performance over time
MFlo
ps
in m
yve
hciles
10
Processing Power over Time
time
GFLOPS/MIPS
1990 2000 2010
100
200
300
400
500
ASIC
FPGA (Xilinx)
GPU (NVidia)
CPU (Intel)
Transputer/x86 P4
Core2Duo
G92
G80
G70
NV40
Virtex 5
Spartan3
Virtex 4
Tyzx
IQ2
Standford engine
3DIP
Processing Power over Time
time
GFLOPS/MIPS
1990 2000 2010
100
200
300
400
500
ASIC
FPGA (Xilinx)
GPU (NVidia)
CPU (Intel)
Transputer/x86 P4
Core2Duo
G92
G80
G70
NV40
Virtex 5
Spartan3
Virtex 4
Tyzx
IQ2
Standford engine
3DIP
*1,78/a
106
Prognosis 2030:
optimistic (1.78/a): 100 PFlops
pessimistic (1.41/a): 1 PFlops
(Still) exponentially increasing.
9
Next Challenge: Active Pedestrian Safety
Pedestrian are the most vulnerable traffic
participants. Children are particularly at risk.
Driver inattention and/or bad visibility are
important accident causes.
Worldwide fatalities of
pedestrians, bicyclists,
and motorcyclists (2006)
Source: Bosch Accident Research
10
Why is it difficult?
Large variation in pedestrian appearance (viewpoint, pose, clothes).
Dynamic and cluttered backgrounds.
Pedestrians can exhibit highly irregular motion.
Real-time processing required.
Stringent performance requirements (especially for emergency maneuvres).
11
Pedestrian System Architecture
Object
Classification
Object
Classification TrackingTracking
Driver Warning /
Vehicle Control
Driver Warning /
Vehicle Control
Path Prediction
&
Risk Assessment
Path Prediction
&
Risk Assessment
Obstacle Detection(Stereo, Flow, Radar)
Obstacle Detection(Stereo, Flow, Radar)
The benefit of object classification:
•improved detection reliability vs. obstacle detection only
•better path prediction: taking advantage of prior knowledge of
object class motion and additional object class-specific cues
•allows object class-specific driver warning and vehicle control strategies
D. M. Gavrila and S. Munder. Multi-Cue Pedestrian Detection and Tracking from a Moving Vehicle. IJCV 73(1), 2007.
S. Munder, C. Schnörr and D.M. Gavrila. Pedestrian Detection and Tracking Using a Mixture of View-Based Shape-Texture Models.
IEEE Trans. on Intelligent Transportation Systems, vol.9, nr.2, pp.333-343, 2008.
C. Keller, T. Dang, A. Joos, C. Rabe, H. Fritz, and D.M. Gavrila. Active Pedestrian Safety by Automatic Braking and Evasive Steering,
IEEE Trans. on Intelligent Transportation Systems, 2011
12
A. Wedel, C.Rabe, T. Vaudrey, T. Brox, U.Franke, D.Cremers.
“Efficient Dense Scene Flow from Sparse or Dense Stereo Data”. ECCV 2008.
stereo
stereo
optical flowoptical flow
ltyxI )1,,( −
ltvyuxI ),,( ++
ltvyuxI ),,( ++=
rtvyddduxI ),,( ++++=
rtvyddduxI ),,( ++++=
rtydxI )1,,( −+
3D Position and Motion for Every Pixel (Scene Flow)
Joint Optimization
Motion
time
t
time
t-1
13
Scene Flow
14
Pedestrian Classification – Experimental Studies
What features? E.g. Chamfer, Haar wavelets, HOG, and Local Receptive Field
What pattern classifier? E.g. SVM, Neural Networks
How to combine pattern classifiers? E.g. Cascading, Parallel (Sum/Max/Mixture)
How to deal with occlusion?
Haar wavelets + AdaBoost cascade
[Viola & Jones, 2005]
HOG features + linear SVM
[Dalal & Triggs, 2005]
Local receptive fields + NN
[Wöhler & Anlauf, 1999]
15
Daimler Pedestrian Benchmark Data Sets
1. Mono Pedestrian ClassificationS. Munder and D. M. Gavrila. An Experimental Study on Pedestrian Classification.
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.11, pp. 1863-1868, 2006.
2. Multi-Modal / Occluded Pedestrian ClassificationM. Enzweiler, A. Eigenstetter, B. Schiele and D. M. Gavrila. Multi-Cue Pedestrian Classification with
Partial Occlusion Handling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
3. Mono/Stereo Pedestrian DetectionM. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments.
IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp.2179-2195, 2009.
C. Keller, M. Enzweiler, and D. M. Gavrila. A New Benchmark for Stereo-based Pedestrian Detection.
Proc. of the IEEE Intelligent Vehicles Symposium, Baden-Baden, Germany, 2011.
Training: 14400 peds. / 15000 non-peds.
Test: 9600 peds. / 10000 non-peds. All 18x36 pixel.
Available for download (Google)
Training: 15660 peds. / 6744 non-ped images
Test: 21790 images with 259 ped. trajectories
>130.000 samples (intensity, dense stereo,
dense flow), 48x96 pixel
1. 2. 3.
16
An intriguing question …
ROC performance improves with
enlarged training set. No saturation
effects (even) for N = 12.800
In fact, doubling training size
matters more than selecting
the best feature-classifier
combination.
How many image examples are needed to learn pedestrian appearance?
Manually labeling humans in images is time-consuming and tedious!Can we do better?
17
Generating Virtual Pedestrians
Shape
variation
Texture
variation
M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008.
18
Mixed Generative-Discriminative Classification Framework
Enlarged training set significantly improved classification
performance (30% less false positives at equal true positive rate)
Meanwhile, current pedestrian classifier on-board vehicle uses
more than 1.5 million samples (“real” and “virtual”)
M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008.
19
Pedestrian Detection - Daytime (Videoclip)
20
Pedestrian Detection – Nighttime (Videoclip)
21
Now with dense stereo …
22
EU WATCH-OVER (2008)85%
50 km/h
10
Pedestrian Recognition Performance (Historical Perspective)
Correctly recognizedpedestrians
Number of falsely recognized pedestrian trajectories per hour
100%
50%
10 10000 100100
EU SAVE-U (2005)
65%40 km/h
EU PROTECTOR (2003)
40%
600
30 km/h
N.B. # False alarms per hour << # Falsely recognized trajectories per hour
We need to get
somewhere here Source: EU Final Review WATCH-OVER
23
lateral position
(m)
longitudinal position
(m)
featu
re
dim
ensi
on
lateral position
longitudin
al posi
tion
(m)
Pedestrian Path Prediction by Trajectory Matching
Our approach
Use higher order model; match learned
trajectory “snippets” (segment of fixed length).
QRLCS (Hermes et al. IV’09) metric computes
similarity after alignment (translation/rotation).
Use of additional motion features.
Path prediction by extrapolation of matched
trajectory snippets (non-param. regression).
Use of particle filter representation.
C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.
main mode
traj.
prediction
aligned snippet distribution
system trajectory
State-of-the-art path prediction: Kalman filter-
ing based on position detected bounding box.
Problem: first-order model does not capture
non-linearities well during sudden motion
changes.
24
Path Prediction
C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.
25
Action Classification (Crossing or not)
Predicting the correct pedestrian’s
action with accuracy 80% is
reached:
•570 ms before a possible standstill
by the human (cyan).
•180 ms before a possible standstill
by the proposed system (black).
•only after the possible standstill by
the IMM-KF (pink).
Motion features help.
1 Frame ≈ 45ms
C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.
26
World Premiere (2009): Automatic Evasion on Pedestrians
27
Automated Test Driving (Videoclip)
Source: Daimler Testing Department
28
Understand What Is What
Localize and classify objects in the environment
Background
Street
Pedestrian
Sky
Vision
Moving vehicle
Source: U. Franke
29
Driver Monitoring
Head / Face / Gaze Tracking Mindlab
Head / Face tracking using stereo vision
and Active Appearance Models
Driver intention estimation based on
head motion, gaze, and vehicle trajectory
Online EEG analysis of driver mental
state (work load, fatigue)
Use to objectively evaluate driver
assistance systems (Attention, IHC)
30
Automation Systems: Gradually Getting There
Not certifiable todayTraditional
driving
All on
Today‘s
ACC
Assisted 1st
Feet off
Short
takeover times
Assisted 2nd
Hands off
Moderate
takeover times
Autonomous 1st
Eyes off
Ability to
drive empty
Autonomous 2nd
Body out
Source: R. G. Herrtwich
31
Final Remarks
Driver assistance is experiencing a breakthrough: a first major deployment of
machine intelligence technology (sensing, reasoning, acting in physical environment).
Computer vision and machine learning play a central role.
Trend towards increased actuation of safety systemsDriver information � driver warning � “soft” vehicle actuation / driver-initiated “hard”
vehicle actuation � automatic “hard” vehicle actuation
Environment Perception is still the bottleneck. Need to
• recognize a wider set of traffic objects classes with better classification performance
• localize objects more accurately in 3D (perform segmentation and classification jointly).
• handle adverse visibility conditions
Future systems will fuse data from lots of sensors and build a precise 3D-
representation of the 360° car surrounding.
The progress in environment perception, driver monitoring, communication as
well as in precise 3D map data will bring us close to our vision of
accident free driving.
32
The best is yet to come!
Questions ?