The Engineering Challenges of
Autonomous Driving
October 11, 2018
2
OUR PANELISTS
Speakers:• Phil Magney
Founder & PrincipalVSI Labs
• Matthew LinderAV Solutions EngineerVSI Labs
• Chris PoschDirector of Engineering – AutomotiveFLIR Systems
Moderator:• Kris Keach
Design News
2
Engineering Challenges of Automated Driving
Technical research for those designing or developing automated vehicle technologies
• Applied research on AV technologies since 2014
• Offers technical advisory and research services to support R&D & planning for automotive, suppliers and technology industry
• VSI’s applied research includes functional examinations on HW/SW components or systems.
• VSI has it own fleet for research purposes
• Skilled in contemporary automotive and AV technologies including connectivity, simulation, programming, computer vision, neural networks (AI), and ISO 26262
4
Who is VSI
Copyright 2018 – VSI Labs
Opening Remarks
• Automation = Safety
• Born from active safety systems
• Partial automation (1-3) as in series production becomes a big selling feature over the next 10 + years before fleet automation starts to dent private ownership
• High automation (L4/5) starting to emerge in next few years
• Gradual deployment in major metros but within a highly constrained operating domain!
• Infrastructure will generally lag for many years… some elements will go to the cloud!
5Copyright 2018 – VSI Labs
The Tech Behind Automated Vehicle Systems
SensorsProcessorsSoftware
6
7Copyright 2018 – VSI Labs
Vehicle Setup
NW/CAN Interfacing
AV Domain Controller
Lidar Stack
Radars Camera &
Vision
Control Systems
Calibrations
Cabin Sensing
Localization Assets
Connectivity
Lane and Object
Detection
Object Positions & Movements
3D Object & Classification
Lane models. Object-based Localization
OTA/V2XDriver
Readiness
The Componentry of Automated Vehicles
Copyright 2018 – VSI Labs
IMU/GNSS
Odometry &Correction
By-Wire Control Systems
Lane KeepingSign rec.
Pedestrian
ACCObject
Detection
Environment Mapping
Positing and Trajectory
Precision Lane Level
Localization
Firmware Updates
SPaT conn.
Driver Engagement & Attention
Timing and Synchronize
Level of Automation L2+
8
Automation Pipeline
Latitudinal Control
Longitudinal Control
Image Sensors• Camera(s)• Thermal • Scanning LiDAR
Ranging Sensors• Radar• Flash LiDAR• Ultrasonic
Localize & Plan • Sensor fusion • Localization• Path planning
Control Algorithms• Open interface• Control
Algorithms• PID Control
Vehicle Tasking
Event Planning
Odometry • IMU• RTK• Precision map
Behavior• Trajectories/Velocities • Detection/Avoidance• Maneuver planning• Fail safe plan
Path Planning
Control Fail Safe
Environmental Modeling
Sensor Fusion
9Copyright 2018 – VSI Labs
Automated Vehicle Systems – Passive Camera Vision
• Best sensor for classifying objects and scenes• Poor at range and speed• Computationally expensive
10Copyright 2018 – VSI Labs
Automated Vehicle Systems – Thermal Camera Vision
• Good sensor for identifying humans in challenging light conditions
• Performs similar in day or night and can see in total darkness
• Good for cluttered environments, especially pedestrians.
• Best when coupled with other sensors
11Copyright 2018 – VSI Labs
Passive Active
Source: FLIR
Automated Vehicle Systems – LiDAR Imaging
• LiDAR is best for 3D perception & environmental modeling• Limited performance in poor weather• Very expensive still… some promising developments though!
12Copyright 2018 – VSI Labs
Automated Vehicle Systems – Radar
• Radar is best for “ranging” – judging distances and movements of its “targets”
• 4 dimensions (elevation, range, doppler and angle)
• Complimentary for sensor suites
13Copyright 2018 – VSI Labs
Copyright 2018 – VSI Labs
Developing Automated Vehicle Systems – The “AV Stack”
14
Challenges & Limitations
Improving Performance & Safety
15
• Sensors• Camera, Radar, Lidar, etc.
• Localization Assets • Localization Assets (Maps and Metadata)• RTK Support
• Inertia Measurement (Odometry)• Adjusted Control Algorithms
• V2X• V2V (car to car), V2I (signals and roadside),
V2P (to pedestrians)
• Redundancies
• Fail Safe Strategies
• Tele-Operation
How AVs will See in Poor Weather?
16Copyright 2018 – VSI Labs
Virtual Infrastructure
• Physical Infrastructure will transition to virtual infrastructure
• Painted lanes lines give way to MAP-based lane models
• Signaled intersections give way to digital Signal Phase and Timing (SPaT)
• Road signs give way to virtual signs (as this is stored as metadata in machine readable form)
• Lanes become reconfigurable without physical gates or dedicated lanes
• Construction zones transmit electronically (smart signs)
17Copyright 2018 – VSI Labs
18
Limitations of Vision-based Automation
Copyright 2018 – VSI Labs
Actual trajectory
Proper trajectory
• A recent accident in Mountain View shows limitations of a vision based system.
• Poor lane markings and vast differences in surface reflectivity caused this accident.
• A precision map (with an HD lane model), could have eliminated this accident.
Limitations of Vision Based Solutions (Tesla Accident)
19Copyright 2018 – VSI Labs
Localization
Improving Performance & Safety with Software
20
• Road model (ADAS Map) • Topology
• Routing
• Speed attributes, etc.)
• Lane Models • Lane geometry
• Polylines
• Trajectories
• Localization Layer • Landmarks
• Signs, barriers, poles etc.
• Edges and boundaries
• Voxels
• Confidence Index
21
MAP-based Localization Assets
Copyright 2018 – VSI Labs
• Point Cloud Localization -- compare a 3D point cloud from a Lidar sensor scan to an existing 3D point cloud. Begin the alignment process.
• Processor Heavy – You have to match every point using Iterative Closest-Point (ICP) algorithm which is very taxing from a compute standpoint.
22
Localization With High Definition Maps
Copyright 2018 – VSI Labs
• When executing point-cloud to point-cloud localization, the points in the point-cloud are unclassified points.
• In landmark-based approaches the objects are classified.
• The downside is that it's much more difficult and expensive to create these maps.
23
Source: HERE
Localization With High Definition Maps
Copyright 2018 – VSI Labs
• GNSS / Real Time Kinematics (RTK)
• Supplements the GPS known positions with correction stations.
• Does not work well in dense urban areas as you need line of site because of GPS fallout.
• You still need a lane model.
24
Source: VBOX
Localization With Correction Services
Copyright 2018 – VSI Labs
Sensor Stack
IMU
GPS
LiDAR
Localization Algorithms
Transform
Source: HERE
Positioning
Odometry
Camera
Radar
AV Processing Stack
By Wire Control Messages
Lane Detection Object Avoidance
Localization With High Definition Maps
Localization using precision maps requires real time processing of objects, images, scenes or other patterns against an embedded map
25Copyright 2018 – VSI Labs
Developing Automated Vehicles
Testing & Simulation
26
Developing Automated Vehicles -- Defining Operating Domains
• USDOT-NHTSA federal guidance for Automated Driving Systems (ADS) require developers to defined their solution according to Operational Design Domains (ODD).
• A SAE Level 2, 3 or 4 vehicle could have one or multiple ODDs – e.g. geo-fenced urban, divided highways, automated parking, traffic jam assist, etc.
• AVs should be developed, tested and validated against all scenarios that could happen within the ODDs.
• Widely understood that AVs require millions if not billions to be proven safe enough!
27Copyright 2018 – VSI Labs
Developing Automated Vehicles – Filling in the Gaps
28
All States and Scenarios
Field Tests/Real World Data Collection
Simulation
Test Tracks
Algorithm Improvement
It is recommended to utilize detailed simulations to develop the systems, test-track tests to validate components and full-vehicles, and field tests to verify the real-life system robustness.
Copyright 2018 – VSI Labs
Simulation & Safety Validation Trends: End-to-end Toolchain
Single supplier’s end-to-end simulation toolchain integration with AV stack
• An integrated solution with simulation tools necessary to test multiple scenes and edge cases
• Can speed up development and testing dramatically
• Includes predefined scenarios used by safety agencies such as Euro NCAP
Copyright 2018 – VSI Labs
Simulation & Safety Validation Trends: Standardized IF
• A standardized interface specification for simulations
• Soon virtual evaluation of automated driving functions will be required
• Test/simulation software tools must be certified
• New players on the automotive simulation market still have gaps: (i.e. vehicle dynamics and powertrain models)
Copyright 2018 – VSI Labs
Closing Remarks
• Automated vehicles requires a precision localization for performance and safety purposes. • Precision maps achieve this objective as the metadata in the maps enable the vehicle to better understand its position against
ground truth.
• Automated vehicles could benefit from correction services of one form or another but this is not seen as practical as the hardware and software is expensive!
• Also, does not work well in dense urban environments.
• The industry has tried brute force to solve the compute challenge but realizing this is not efficient either • Companies are developing ASICs to cope with the huge compute requirements pushing some of the functional load to an
optimized ECU.
• Environmental sensors by themselves with cannot handle all situations• For example, long range radar is vital but you have to tune out non-doppler activity (static objects for example)
• Cameras get confused easily, especially with AI-based solutions • You need anomaly checkers, or deterministic rules to filter out the anomalies
• Challenging light conditions or cluttered environments are problematic
• Infrastructure is lagging • And may always be behind where it should be!
• Some infrastructure is going to cloud (mapping assets, localization assets, signage, etc.)
31Copyright 2018 – VSI Labs
Thermal Cameras in ADAS and AV
A U TO N O M O U S V E H I C L E T E C H N O L O G Y
W E B I N A R
Proprietary - Company Confidential ©2018 FLIR Systems Inc. Information and equipment described herein
may require US Government authorization for export purposes. Diversion contrary to US law is prohibited.
Thermal Cameras Make ADAS & AV Safer
Thermal provides a redundant imaging solution to fill the visible-camera performance gaps.
Ther
mal
Cam
eras
will
sig
nif
ican
tly
hel
p in
all
thes
e ca
ses
Proprietary - Company Confidential ©2018 FLIR Systems Inc. Information and equipment described herein
may require US Government authorization for export purposes. Diversion contrary to US law is prohibited.
Reliably Classify in Cluttered Environments
Free starter thermal image dataset - over 14,000 annotated thermal images
Proprietary - Company Confidential ©2018 FLIR Systems Inc. Information and equipment described herein
may require US Government authorization for export purposes. Diversion contrary to US law is prohibited.
www.flir.com/adas
Best sensor for pedestrian detection & reliable
classification
• Performs similar in day or night
• See in total darkness - 4x headlights
• Not blinded by the sun, shade, tunnels, etc.
• Sees through most of fog
• Deep learning can be readily applied to
thermal camera data
Collect data with FLIR ADK™ and FREE thermal starter dataset.
Person 0.84
38
FLIR Systems Inc.
• FLIR Thermal infrared cameras reliably classify objects – especially humans in a cluttered environment
• FLIR thermal infrared cameras see in challenging lighting conditions – flog, sun glare, complete darkness
• FLIR has the only auto-qualified thermal sensor and it is installed in >500K vehicles
Developing Imaging Solutions That Enhance Perception and Awareness
See in Total DarknessEnhanced Long Range
ImagingSee Through Obscurants
Non-Contact Temperature
Reliably ClassifyPeople & Animals
The World Leader in Thermal Infrared Imaging
40
THANK YOU!
• Please be sure to complete the feedback survey
• Thank you for attending today’s Design News
Editorial Event:
The Engineering Challenges of Autonomous Driving
40