Principles of Modeling for Cyber-Physical Systems
Instructor: Madhur Behl
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 1
Module 3: Automotive CPS Data driven modeling
Slides credits:- Urs Muller
Everything that moves will go autonomous
2
Cars Trucks Carts Drones3
AutonomousSystems
Safe
Secure
Connected
SelfLearning
Trusted
Resilient
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 4
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 5
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 6
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 7
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 8
1.35 million deaths worldwide due to vehicle crashes
94% of crashes involve human choice or error in the US.
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 9
3 millionAmericans age 40 and older are blind or have low vision
79%of seniors age 65 and older living in car-dependent communities
42 hourswasted in traffic each year per person
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 10
Localization and Mapping
Scene Understanding
Trajectory Planning and Control
Human Interaction
Where am I ?
Where/who/what/why of everyone/everything else ?
Where should I go next ? How do I steer and accelerate ?
How do I convey my intent to the passenger and everyone else ?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 11
Localization and Mapping
Scene Understanding
Trajectory Planning and Control
Human Interaction
Where am I ?
Where/who/what/why of everyone/everything else ?
Where should I go next ? How do I steer and accelerate ?
How do I convey my intent to the passenger and everyone else ?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 12
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 13
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 14
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 15
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 16
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 17
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 18
20Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected]
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 21
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 22
Localization and Mapping
Scene Understanding
Trajectory Planning and Control
Human Interaction
Where am I ?
Where/who/what/why of everyone/everything else ?
Where should I go next ? How do I steer and accelerate ?
How do I convey my intent to the passenger and everyone else ?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 23
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 24
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 25
Detailed three-dimensional maps that highlight information such as road profiles, curbs and sidewalks, lane markers, crosswalks, traffic lights, stop signs, and other road features.
Where am I?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 26
What's around me?
Scan constantly for objects around the vehicle—pedestrians, cyclists, vehicles, road work, obstructions—and continuously read traffic controls, from traffic light color and railroad crossing gates to temporary stop signs.
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 27
What will happen next?
Predict the movements of everything around you based on their speed and trajectory
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 28
What should I do?
Determine the exact trajectory, speed, lane, and steering maneuvers needed to progress along the route safely
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 29
HD Maps: Localization
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 30
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 31
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 32
Localization: Scan Matching
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 33
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 34
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 35
Localization and Mapping
Scene Understanding
Trajectory Planning and Control
Human Interaction
Where am I ?
Where/who/what/why of everyone/everything else ?
Where should I go next ? How do I steer and accelerate ?
How do I convey my intent to the passenger and everyone else ?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 36
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 37
Localization and Mapping
Scene Understanding
Trajectory Planning and Control
Human Interaction
Where am I ?
Where/who/what/why of everyone/everything else ?
Where should I go next ? How do I steer and accelerate ?
How do I convey my intent to the passenger and everyone else ?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 38
Localization and Mapping
Scene Understanding
Trajectory Planning and Control
Human Interaction
Where am I ?
Where/who/what/why of everyone/everything else ?
Where should I go next ? How do I steer and accelerate ?
How do I convey my intent to the passenger and everyone else ?
End-
to-e
nd le
arna
ble
Deep
Neu
ral N
etw
orks
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 39
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 40
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 41
Training the Neural Network
Left camera
Center camera
Right camera
Random shift and rotation
Adjust for shift and rotation
CNN -Back propagationweight adjustment
Recorded steering
wheel angle
Network computed steering command
Desired steering command
Error
End-to-End Driving: PilotNET
DrivingWith a single front-facing camera
Center camera CNN
Network computed steering command Drive-by-wire
interface
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 44
Machine intelligence is largely about training data.
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 45
Image courtesy: Cognata
When’s a pedestrian not a pedestrian? When it’s a decal.
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 46
One car ? or Multiple cars ?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 47
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 48
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 49
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 50
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 51
There is a bus right next to you!!Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 52
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 53
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 54
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 55
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 56
How can we ensure that an autonomous vehicle drives safety upon encountering an
unusual traffic situation ?
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 57
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 58
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 59
How can CNNs help us drive ?
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 60
Autonomous Driving: End-to-End
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 61
Autonomous Driving: End-to-End
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 62
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 63
Autonomous Driving: End-to-End
Principles of Modeling for CPS – Fall 2019 MADHUR BEHL [email protected] 64
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 65
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 66
Principles of Modeling for CPS – Fall 2019 Madhur Behl [email protected] 67
68
THE BASIC IDEALearn from human drivers
Training data -
Human actuator commands
Record data from lots of humansdriving their cars:Ø Sensor dataØ Actuator data
Sensordata
Error (training) signal
CNN actuatorcommands
CNN
69
EARLY EXAMPLES
Neuralnetwork
PixelsSteering
commands
Of end-to-end learning
ALVINN, CMU, late 80es(Pomerleau et Al.)
Lane following with a small 2-layer fully connected network and low-resolution video input
30x32 pixel
DAVE, Net-Scale/NYU, 2004(LeCun et Al.)
Off-road obstacle avoidance using a convolutional network (ConvNet)
149x48 pixel
70
TRAINING EXAMPLES
225K images
Label: turn right Label: go straight
Label: turn right Label: turn left
71Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 71
72Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 72
73Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 73
74Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 74
75Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 75
76Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 76
77Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 77
78Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 78
79Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 79
8080Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected]
81Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 81
82Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 82
83Principles of Modeling for CPS – Fall 2019
Madhur Behl [email protected] 83
84
TRADITIONAL DECOMPOSITIONNecessary approach when data and compute power are limited
Hand-designedfeatureextractor
ClassifiersSemantic
abstraction (cost map)
Pathplanner Controller
Actuatorcommands
Trained fromdata
Domain knowledge(from human experts)
Hand-designed
Learned
Camera
Deep learning –feature extractor andclassifier trainedsimultaneously
85
EXAMPLE: ROAD FOLLOWING
Good quality lane markers, good driving conditions
Traditional lane detection-based systems expected to work well
Poor quality lane markers
Lane detection-based systems struggle
End-to-end learning empowers the network to use additional cues
86
LEARNED ROAD FOLLOWING (PILOTNET)
Learned featureextractor(ConvNet)
Learnedcontrol
Actuatorcommands
Domain knowledge• Recover from off-center
and off-orientation• Prevent training on too
much straight road
Hand-designed
LearnedTraining data
Single front-facing
camera
Highway, local, residential – with or without lane markings
Domain knowledge• Network architecture
Examples• Data collected from
human drivers
Both blocks trained simultaneously
No explicit object detection nor path planning
→ Maps pixels directly tosteering
87
TRAINING THE NEURAL NETWORK
Left camera
Center camera
Right camera
Random shift and rotation
Adjust for shift and rotation
CNN -Back propagationweight adjustment
Recorded steering
wheel angle
Network computed steering command
Desired steering command
Error
88
DRIVINGWith a single front-facing camera
Center camera CNN
Network computed steering command Drive-by-wire
interface
89
VISUALIZATIONWhat the network pays attention to