PERCEPTION FOR AUTOMATED AND ASSISTED DRIVINGppniv15.irccyn.ec-nantes.fr/material/session2/Darms/... · 2015-10-01 · PERCEPTION FOR AUTOMATED AND ASSISTED DRIVING . DR.-ING. MICHAEL

PERCEPTION FOR AUTOMATED AND ASSISTED DRIVING

DR.-ING. MICHAEL DARMS GROUP RESEARCH / HEAD OF SENSORS AND FUSION IROS – 28.09.2015

2 Volkswagen AG | Konzernforschung

ENVIRONMENT PERCEPTION

Motivation

• Highly automated driving functions impose increased requirements on the performance of the perception module

• Perception module has to cope with contradictory requirements (comport vs. safety systems)

• Development of a modular and extensible perception architecture

• Environment model has to be independent of specific ADAS function

• Implementation of unified interface to function

Goals


Representation of the world around the vehicle

Static Environment

Road / Lanes

Dynamic Objects

Road Graph

WORLD MODEL


OBJECT ESTIMATION

Mono Camera Laser Radar

• Estimating of moving/ movable objects and dynamic model

• Benefit from individual sensor detection capabilities


GRID ESTIMATION

Ultrasonic Front Camera

2D Layer Fusion

• Estimate Free Space

• Estimate Occupied Space

• Exclude Movable/ Moving Objects


ROAD ESTIMATION

Highway Lane Markings Other Vehicles

Rural Roads Lane Markings Other Vehicles Curbs, sward

Inner City Lane Markings Other Vehicles Curbs Traffic Lights, Trafic Signs Digital Maps

• Interpret where in the world vehicles should drive using feature cues from the environment


SCENE ESTIMATION

RoadGraph • Output of the perception modules is

integrated into one model • Roadgraph is main interface to

driving functions Challenges • Fusion of road estimation and

context knowledge into unified, consistent and comprehensive model

• Scene Estimation for urban areas, especially in complex intersection scenarios

• Handling of traffic participants that do not behave as expected


V-CHARGE: AUTONOMOUS VALET PARKING AND CHARGING FOR E-MOBILITY

Automated valet parking and charging • no time-consuming search for parking spots any more • driverless valet service • no human intervention Fully automated driving • in mixed-traffic scenarios • in indoor and outdoor parking lots and parking garages

without GPS


V-CHARGE: AUTONOMOUS VALET PARKING AND CHARGING FOR E-MOBILITY Close-to-series sensors

Stereo- Camera

Area- view

Ultra- sonic

Car2x


OBJECT ESTIMATION USING FISH-EYE CAMERAS

University of Parma, VisLab


GRID ESTIMATION USING FISH-EYE CAMERAS

C. Häne, T. Sattler, M. Pollefeys, Obstacle Detection for Self-Driving Cars Using Only Monocular Cameras and Wheel Odometry


GRID ESTIMATION WITH ALL SENSORS

06/25/2015 22

Ultrasonic Grid Stereo Grid

Motion Stereo Grid Fused Grid


GRID ESTIMATION WITH ALL SENSORS

06/25/2015 22

Ultrasonic Grid Stereo Grid

Motion Stereo Grid Fused Grid


ROAD ESTIMATION: KNOWING WHERE THE ROAD IS VIA LOCALIZATION

14

Setup


APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

15

Survey Run

Furgale et al., ICRA2014



16

Survey Run

Online Run

Survey Run


17 Volkswagen AG | Konzernforschung 17




18




19




20



October 13, 2013 15:15


November 8, 2013 10:26


December 6, 2013 13:50


July 31, 2014 15:45



25

Summary

Mapping / Fusion Localization

Database

Summary map for localization

Localization output & raw data

Full-scale map

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis



26

Find the most useful Landmarks:

Most observed landmarks from a single session:

Landmarks observed over the most different sessions:

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis


APPROACH: INTERPRETING WHERE THE ROAD IS – ROAD ESTIMATION

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis Mühlfellner et al.: Summary Maps for Lifelong Visual Localization, JFR, 2015

19.0

9.

04.1

0.

11.1

0.

17.1

0.

23.1

0.

31.1

0.

07.1

1.

08.1

1.

12.1

1.

19.1

1.

03.1

2.

06.1

2.

10.1

2.

12.1

2.

14.0

1.

04.0

4.

14.0

4.

25.0

5.

28.0

4.

30.0

4.

05.0

5.

14.0

5.

21.0

5.

26.0

5.

28.0

5.

11.0

6.

30.0

6.

02.0

7.

16.0

7.

23.0

7.

30.0

7.

31.0

7.

100%

75%

50%

25%

0%

1 2 3 4 5 8 9 6 7 10

11

12

Full Multi-Session Map Summary Map

Date, Arrows Indicate Mapping Sessions

% S

ucce

ssfu

l Loc

aliz

atio

ns

Success Rate


APPROACH: INTERPRETING WHERE THE ROAD IS – ROAD ESTIMATION

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis Mühlfellner et al.: Summary Maps for Lifelong Visual Localization, JFR, 2015

19.0

9.

04.1

0.

11.1

0.

17.1

0.

23.1

0.

31.1

0.

07.1

1.

08.1

1.

12.1

1.

19.1

1.

03.1

2.

06.1

2.

10.1

2.

12.1

2.

14.0

1.

04.0

4.

14.0

4.

25.0

5.

28.0

4.

30.0

4.

05.0

5.

14.0

5.

21.0

5.

26.0

5.

28.0

5.

11.0

6.

30.0

6.

02.0

7.

16.0

7.

23.0

7.

30.0

7.

31.0

7.

1e7

1e6

1e5

1 2 3 4 5 8 9 6 7 10

11

12

Full Multi-Session Map Summary Map

Date, Arrows Indicate Mapping Sessions

Land

mar

ks T

otal

(Log

. Sca

le)

Number of Landmarks per Session


CES DEMO DRIVE 2015

− About 900 km on Highways − 5 Journalists as drivers using the automated vehicle

Bakersfield / 5.1.2015 / 7:00

San Francisco / 4.1.2015 / 11:00

Las Vegas / 5.1.2015 / 16:00

CANV / 5.1.2015 / 12:00 Interstate I5 / 4.1.2015 / 15:00


1 4

5

2x Laser-scanner

4x Ultrasonic 1x 3D-video-camera

4x Top View 4x Mid-range-radar

2x Long-range-radar

1x Stock GPS 2x Short Range Radar

2

2

3

3

4

2

1 2 3 4 5 6 7 8

6

7

8

1

TECHNOLOGY – TODAY


KEY CHALLENGE: WHERE IS THE ROAD?

Simple Marking features, simple lane geometries

Medium Marking features, more complex geometries (smaller radius of curvature, not parallel geometries)

Complex Arbitrary features, arbitrary lane geometries

Robust and highly available marking features low demands

Marking features with more complex geometries medium-level demands

No clear road markings High demands on the scene interpretation

31

Töpfer, Spehr et al: Efficient Scene Understanding for Intelligent Vehicles Using a Part-Based Road Representation


INPUT AND OUTPUT OF THE ROAD ESTIMATION • Spatial and temporal reasoning

Features

Patches

Dig. map

Result

Lanes

Road Estimation

Spat

ial a

nd te

mpo

ral

reas

onin

g

Standards

A-priori

Sensors

Road model

Credits Spehr et al


INTEGRATION OF A-PRIORI KNOWLEDGE

• Standards: • Guidelines for building roads, freeways and street • Use of country-dependent policies (e.g. in Germany „Straßenbaunormen DIN

EN 1423 und DIN EN 1424“)

• Deviations are modeled with appropriate probability distributions

• Digital maps:

• Information beyond the detection range of the sensors • Used for integrating the expected road geometry and topology during

spatial reasoning.


SENSOR SETUPS

• High-level sensor information (Lanes) • Covers a wide range in the vehicle’s environment • Trajectories of other vehicles • Lane output of external preprocessing units

• Low-level sensor information (Patches/ Features) • Spatially restricted features in the vehicle’s environment such as

boundary features • But also features like grid cells (occupied yes/no), color values of a

camera image


HIERARCHICAL MODEL OF A SINGLE SCENE

35

The graphical model comprises hidden random variables 𝒙𝒙𝑖𝑖 and observations 𝒚𝒚𝑖𝑖 • Hidden variables represent parts and sub-parts of a scene encoded by the root node • Variables are continuous, multi-dimensional, and multi-modal

Edges encode probabilistic dependencies between pairs of variables

2-Lane Road Scene

Features

Right Lane Left Lane

Lane-Segment Lane-Segment

Patches


INFERENCE DEPTH-FIRST MESSAGE PASSING bottom-up = generating a

hypothesis top-down = verifying the

hypothesis Example: 11. Verifying the road hypothesis (top-down)

36


IS THE ROAD DRIVEABLE – DISTANCE TO STOP

Velocity

50 km/h

70 km/h

100 km/h

130 km/h

Deceleration

2 m/s2 54 m 103 m 205 m 342 m

10 m/s2 23 m 38 m 66 m 101 m

0

100

200

300

2 4 6 8 10

Ove

rall

Stop

ping

Dis

tanc

e in

m

Deceleration in m/s2

Geschwindigkeit in km/h

50

70

130

100

t1=0,3s r0=10ms-2 / 0,7s


Sensors G

atew

ay

Vehicle

zFAS - modular & scalable architecture

CAN

/ F

LEXR

AY /

ETH

ERN

ET /

…

APPLICATION FOR SERIES PRODUCTS

http://litbimg3.rightinthebox.com/images/384x384/201401/qlppme1389320644905.jpg


Today: Release testing of todays ADAS with up to 2 million test km and 1.000 test drivers

data basis: 2.000.000 km, > 1.000 drivers; source: Dr. Markus Fach et al., Daimler AG, VDI/VW Gemeinschafts-Tagung, 2010)

Tomorrow: Increasing system complexity of future ADAS will increase diversity of relevant test scenarios Forecast for high automation: 100 Million km = 0,67 x average distance sun earth = 5,6 light minutes Costs for such release testing: several 100 Million EUR source: Prof. Winner et al., Darmstädter Kolloquium „Mensch und Fahrzeug“, 2011

Objective: Sustainable and affordable concept for test and release of future ADAS

System Activation frequency per 10.000 km km till activation Distance Warning 40 - 60 170 – 250

Breaking Assistance (BAS) Plus 0,5 - 1 10.000 – 20.000

PRE-SAFE breaking, level 1 0,1 – 0,2 50.000 – 100.000

PRE-SAFE breaking, level 2 0 -

Ensuring reliability – challenges


Extension Function range /

Security requirements

SW Module- Tests

SIL- Tests

HIL- Tests

Veh. Tests

Test levels/ Function range

Com

plex

ity

SW Module-Tests

SIL-Tests

HIL-Tests

Veh. Tests

The effort of vehicle tests rises disproportional with increasing functions range/ safety requirements

Reduction of the vehicle tests effort, through shifting the test from the street to the simulation. => Virtual Test Drive virtual

test drive

Test levels and complexity


vehicle model

sensor models

virtual camera environment model video Inter- face

image processing

sensor data fusions

application SW - longit. control

- lateral control - path planner

vehicle SW

Simulation via Virtual Test Drive


CONCLUSION

Knowledge about the location of the road is a key factor for automated driving and future driver assistance systems

Interpretation based approaches using environment sensors work well in easy to medium challenging scenarios

Using additional map information leads to more robust results

Localization techniques are currently used to solve the most complex scenarios

Drivability estimation at long ranges for high speed driving is still challenging

New sensor principles and machine learning approaches are one way for solving this topic

Testing environment perception is one of the key challenges

Shifting tests from street to simulation reduces vehicle test efforts significantly

Centralized ECUs like the zFAS help facilitating testing procedures

THANK YOU FOR YOUR ATTENTION

Documents

PERCEPTION FOR AUTOMATED AND ASSISTED DRIVINGppniv15.irccyn.ec-nantes.fr/material/session2/Darms/... · 2015-10-01 · PERCEPTION FOR AUTOMATED AND ASSISTED DRIVING . DR.-ING. MICHAEL