Upload
peter-labaziewicz
View
43
Download
1
Embed Size (px)
Citation preview
TI Information – Selective Disclosure TI Information – Selective Disclosure
Perceiving and understanding the world for ADAS and Autonomous TU-Automotive ADAS & Autonomous October 3- 4, 2016
Peter Labaziewicz
Texas Instruments
http://events.tu-auto.com/autonomous/2016/Public/Sessions.aspx
TI Information – Selective Disclosure
Perceiving and understanding the world
One of the most critical functions a vehicle will have in the age
of ADAS and autonomous driving is to accurately perceive and
understand the world around it.
2
TI Information – Selective Disclosure
Autonomous driving steps
3
Perceive Understand Act
Sensors
Sensor Fusion
Machine Learning
CNN is not be all end all
Processing
Challenges & Approaches
TI Information – Selective Disclosure
Accurately perceiving the world
4
Local Perception Perceive local dynamic
environment at present time
Eg. pedestrians, vehicles etc.
• Onboard Sensors
Localization Determine the vehicle
position relative to a global
coordinate system
• High-resolution mapping
with precise GPS & IMU
inputs establishes location
of stationary environment
Communication • V2V, V2E, V2I
World
Perception
TI Information – Selective Disclosure
Attributes of sensor sets - passive sensors
5
Limitations:
• Passive sensors are affected by the environment
• The 3rd Dimension - Depth
Mono cameras have difficulty in
sensing distance
BUT - Stereo cameras can easily
and robustly detect distance
Cameras Passive sensors – Sense reflected or emitted radiation
• Visible image cameras: Operate in the visual spectrum
• Infra-Red image cameras: Operate outside the visual
spectrum such as Near IR or Thermal (Far IR)
TI Information – Selective Disclosure
Examples
• Ultrasonic: Cheap, commonly used for short range distance measurement
• Radar: Cost & size coming down due to 77GHz CMOS radar advances
• Lidar: Cost & size coming down due to advances in solid state LiDAR design
• Time of Flight: High resolution & accuracy, short range (up to10’s meters)
• Structured Light: High resolution & accuracy, short range
Attributes of sensor sets - Active sensors
6
Active sensors emit radiation and measure responses of reflected signals • Able to obtain measurements anytime, under any weather condition
• Provide depth information
• Range is limited by emitted energy
TI Information – Selective Disclosure
Challenges to accurately perceive the world
7
Lighting Rain
Snow Fog/Smog
Night
Dirt
TI Information – Selective Disclosure
Challenges to accurately understand the world
8
Google car examples
Daniel Rosenband, Google - Hot Chips 2016
conference presentation (http://www.hotchips.org/ )
Construction
Scale
Material properties
Repeated structures
TI Information – Selective Disclosure
Comparison of sensing technologies
9
Camera Radar LiDAR
Object Detection + +
Pedestrian Detection +
Weather Conditions - +
Lighting Conditions - + +
Dirt - + -
Velocity + -
Distance - Accuracy Stereo + +
Distance - Range +
Data Density +
Classification + -
Fusion
+
+
+
+
+
+
+
+
+
TI Information – Selective Disclosure
Sensor fusion
10
For increased accuracy and robustness under a wide variety of conditions,
data from multiple sensor types of complementary modalities needs to be
fused to view the same scene
TI Information – Selective Disclosure
Algorithm view of fusion system
6x-10x
Cameras
6x-10x
Radar
1x-4x
LIDARs
Thermal/IR
8x-12x
Ultrasonic
GPS IMU
Sensor Processing
Sensor Processing
Sensor Processing
Sensor Processing
Sensor Processing
Perception
Perception
Perception
Perception
Perception
FUSION • Sensor Fusion
• Localization
• Mapping
Maps
PLANNING AND CONTROL
• Path planning
• Motion planning
• Vehicle controls
• Acceleration
• Brake
• Steering
FEEDBACK TO DRIVER
• Visualization/Display
• Warnings
Driver monitoring
Object information
• Location
• Velocity
• Type
Raw data
• Pixels
• Point clouds
TI Information – Selective Disclosure
Balancing all the key requirements
12
COMPUTE PERFORMANCE
BANDWIDTH
I THINK WE HAVE
STRUCK THE RIGHT
BALANCE
Power/Thermal
TI Information – Selective Disclosure
Heterogeneous processing architecture
13
ARM/DSP are needed
for High-level vision stages
of the algorithm. • ARM Cortex Axx:
• Scalable RISC
• Data Fusion
• Memory Coherency
DSP: • VLIW SIMD+MIMD
• Data Fusion
EVE Vector Coprocessor: • High Bandwidth
• Pixel Operations
• SIMD Parallelism
• Energy Efficiency
Hardware Acceleration: • High Bandwidth
• Pixel Operations
• HW Acceleration
• Configurable
TI Information – Selective Disclosure
6x-10x
Cameras
6x-10x
Radar
1x-4x
LIDARs
Thermal/IR
8x-12x
Ultrasonic
GPS IMU
Sensor Processing
Sensor Processing
Sensor Processing
Sensor Processing
Sensor Processing
Perception
Perception
Perception
Perception
Perception
FUSION
• Sensor Fusion
• Localization
• Mapping
Maps
PLANNING AND CONTROL
• Path planning
• Motion planning
• Vehicle controls
• Acceleration
• Brake
• Steering
FEEDBACK TO DRIVER
• Visualization/Display
• Warnings
Driver monitoring
Efficient mapping of compute to heterogeneous cores
HWA (ISP, …)
Vector Processor (EVE)
DSP
ARM
GPU
TI Information – Selective Disclosure
Distributed Systems • State of the art today
• Processing on the edge – processor
close to sensor
• Easy to manage - clear ownership
• Power and size constraints
Hybrid “Centralized” System • High speed RAW data transmission to node
processor
• Aggregate common sensor nodes into Central
Fusion ECU (each processor node transmits
object data to fusion ECU)
• Easy to manage - clear ownership
• Easily scalable
• Sensing units can be very small
• Power and size can be relaxed
Fully Centralized Processing • Massive compute can be used
• Complex and difficult to scale to lower
cost vehicles
• Difficult to manage multi-vendor
collaboration on one chip
• Multiple vendor systems on one chip
raises safety & responsibility issues
ADAS/Autonomous vehicle system architectures
SV
Distributed
Interior Camera
Parking Camera
LR Radar
Front Camera
MultiMode
Radar
Hybrid Centralized
Fusion
SV SV SoC
Fusion
Fully Centralized
?
TI Information – Selective Disclosure
Scene Understanding
16
Feature
Extraction Classification
Scene
Labeling
Object
Detection
• Multiple sensors
• Sensor Fusion • Distance
• Velocity
• Edges
• HoG
• Etc.
• Traditional • SVM, KNN Adaboost,
LDA, HMM etc.
• Deep Learning
• CNN
Understanding http://mscoco.org
TI Information – Selective Disclosure
Deep Learning - CNN is the ultimate classifier
• Imitates the brain with neural networks that gather information and react to it independently
• Given vast amounts of annotated data the computer learns how to solve the problem
• After a training process on a powerful compute platform, can be implemented in an embedded processor
• Require relatively simple MAC processors with very high compute & bandwidth requirements
– This makes CNN a good candidate for specialized HWA
17
Trained
Features “Trained”
Classifier
Car
“hello”
34818
Cascaded layers
Machine Learning Machine Learning
Hierarchical representation Mathematical model Domain independent
Convolutional neural networking Huge Data
Training
offline
Embedded Output
TI Information – Selective Disclosure
But - CNN is not the be all end all
18
Intriguing properties of neural networks, by Christian Szegedy, Wojciech Zaremba,
Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus (2013)
• Neural Networks do fail, and for mysterious reasons
Correctly classified Incorrectly classified Image perturbation
Nguyen A, Yosinski J, Clune J. Deep Neural Networks are Easily Fooled: High
Confidence Predictions for Unrecognizable Images. In Computer Vision and
Pattern Recognition (CVPR ’15), IEEE, 2015.
Images that are unrecognizable to humans, but that state-of-
the-art DNNs trained on ImageNet believe with 99.6%
certainty to be a familiar object
For Autonomous driving failure is NOT an option!
Rich data sets from multiple sensors can dramatically
increase classifier accuracy and can dramatically
simplify classification
TI Information – Selective Disclosure
Putting it all together
19
You need to perceive the world around you
• Multi-modal sensor fusion provides robust local perception in all conditions
• Cameras, Radar and LiDAR will be the key complementary sensing technologies for object fusion
You need to understand the world around you • Robust image understanding is essential for autonomous driving
• Machine learning is the tool that enables understanding
• CNN is not the only answer to all problems
You need to process the data • Partitioning all functions within reasonable processing, bandwidth & power constraints is a challenge
• Heterogenous architectures provide an efficient approach
• TI TDAx SoC’s provide an optimized solution for ADAS and Autonomous Driving
Visit ti.com/ADAS for more information.