Road Scene Analysis Road Scene Analysis by Stereovision: a by Stereovision: a Robust and Quasi-Robust and Quasi-Dense ApproachDense Approach

Nicolas HautièreNicolas Hautière11, Raphaël Labayrade, Raphaël Labayrade22, , Mathias PerrollazMathias Perrollaz22, Didier Aubert, Didier Aubert22

11 LCPC – French Research Institute for Public Works LCPC – French Research Institute for Public Works

22 INRETS – French National Institute for Research in Transportation INRETS – French National Institute for Research in Transportation and Safetyand Safety

2/43

Outline of the Outline of the PresentationPresentation

ProblematicProblematic The The ″″v-disparity v-disparity ″″ approach approach The quasi-dense matching algorithmThe quasi-dense matching algorithm The robust and quasi-dense The robust and quasi-dense

approachapproach ExampleExample ConclusionConclusion

3/43

ProblematicProblematic

Robust detection of both the Robust detection of both the longitunallongitunal and and laterallateral positions of vertical objects by positions of vertical objects by in-vehicle stereovision.in-vehicle stereovision.

Due to real-time constraints, sparse Due to real-time constraints, sparse matching techniques are more matching techniques are more encountered in the literature, but poorly encountered in the literature, but poorly reconstruct the 3D structure.reconstruct the 3D structure.

Voting techniques (eg. Hough transform) Voting techniques (eg. Hough transform) provide a high rate of robustness:provide a high rate of robustness: The The ″v-disparityv-disparity″ approach is now widely used approach is now widely used

4/43

The Road Scene ModelThe Road Scene Model

obliqueplane

horizontalplane

verticalplane

b

Z r

Z l

ZY r

Y l

-Y

X

v

vur

u l

h

d

The road scene is assumed to be composed of: A road surface composed of horizontal and oblique planes Vertical objects considered as vertical planes

5/43

The The ″″v-disparityv-disparity″″ approach approach [Aubert, 2005][Aubert, 2005]

Robust computation of longitudinal position of Robust computation of longitudinal position of objectsobjects

Grabbing of right and left images

Computation of a sparse disparity map

Computation of ″v-disparity″ image

Global surfaces extraction

Extraction of the longitudinal position

″″v-disparity″″ image = v coordinate of a pixel towards its disparity Δ (performing accumulation from the disparity map along scanning lines)

[Aubert, 2005] ] D. Aubert and R. Labayrade, “Road obstacles detection by stereovision: the "v-disparity" approach,” Annals of Telecommunications, vol. 60, no. 11–12, 2005.

6/43

The The ″″v-disparityv-disparity″″ approach approach Computation of lateral position of objects is Computation of lateral position of objects is

problematicproblematic ″″v-disparity″ approach relies on horizontal gradientsv-disparity″ approach relies on horizontal gradients Consequently, ″u-disparity″ approach is not robust enough to Consequently, ″u-disparity″ approach is not robust enough to

compute the lateral position of objects, eg:compute the lateral position of objects, eg:

LIDAR is often used to fill this gap.LIDAR is often used to fill this gap.

u-disparity image

″″u-disparity″″ image = u coordinate of a pixel towards its disparity Δ (performing accumulation from the disparity map along scanning lines)

7/43

How to cope with this How to cope with this situation ?situation ?

Densification of the disparity map is Densification of the disparity map is a solutiona solution

Problem: dense disparity map Problem: dense disparity map schemes are still costly to implement schemes are still costly to implement

An in-between method exists: the An in-between method exists: the quasi-dense matching algorithm quasi-dense matching algorithm [Lhuillier, 2002] but has not been [Lhuillier, 2002] but has not been yet tested for in-vehicle stereovisionyet tested for in-vehicle stereovision Let’s do it !Let’s do it !

[Lhuillier, 2002] M. Lhuillier and L. Quan, “Match propagation for image-based modeling and rendering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 8, pp. 1140–1146, 2002.

8/43

The quasi-dense matching The quasi-dense matching algorithmalgorithm

Idea: Idea: propagationpropagation of the initial seeds in a way similar of the initial seeds in a way similar to a to a region growingregion growing, guided not by a criterion of , guided not by a criterion of homogeneity but by a homogeneity but by a score of correlationscore of correlation

Initial seeds are the local Initial seeds are the local maxima of ZNCC correlationmaxima of ZNCC correlation Disparity propagation if correlation is good enough in Disparity propagation if correlation is good enough in

close neighborhoods by allowing a small gradient of close neighborhoods by allowing a small gradient of disparity: disparity:

Disparity is propagated only in textured areas, i.e. Disparity is propagated only in textured areas, i.e. only if:only if:

txNxIxI 255/)(,)()(max 4

ac

b B

AC

Neighborhood of pixel a in I1Neighborhood of pixel A in I2

9/43

Initial seeds (ZNCC>0.9)

t=0.05

t=0.01reconstructed

″″v-disparity″″images not reliable !

Disparity is propagated along horizontal edges However, the method creates some correlated matching errors to which ″″v-disparity″″ approach is sensitive !

The quasi-dense matching The quasi-dense matching algorithm: examplesalgorithm: examples

10/43

Proposed solution: the Proposed solution: the robust and robust and

quasi-dense approachquasi-dense approach Idea:Idea:

1. Computation of 1. Computation of ″v-disparityv-disparity″ image and image and extraction of global surfaces extraction of global surfaces

2. Propagation of disparity except that 2. Propagation of disparity except that for each match candidate we check if it belongs to one of the planes of the ″v-disparity″ image

We add a We add a global constraintglobal constraint on the on the quasi-dense matching algorithmquasi-dense matching algorithm

11/43

t=0.05

t=0.01reconstructed« v-disparity »

images OK

By adding a global constraint on the disparity propagation, matching errors are much less numerous ! However, there are till some errors on occluded contours and periodic low textured areas

Robust and quasi-dense approach: Robust and quasi-dense approach: examplesexamples

12/43

Application: extraction of Application: extraction of lateral position of objectslateral position of objects

″″u-disparity″″ image computation is now reliable Fitting a bounding box is possible.

13/43

ResultsResultsBad Contrasted Video (Daytime Fog)Bad Contrasted Video (Daytime Fog)

Standard « u-v disparity » approach Few false detections, low detection rate

Quasi-dense approach (t=0.05) Good detection rate, lots of false detections

Robust and quasi-dense approach (t=0.05) Good detection rate, few false alarms

14/43

ConclusionConclusion

We have presented a stereovision methodWe have presented a stereovision method Based on ″v-disparity″ approach and the quasi-Based on ″v-disparity″ approach and the quasi-

dense matching algorithmdense matching algorithm Computing reliable quasi-dense disparity mapsComputing reliable quasi-dense disparity maps Detecting robustly both lateral and Detecting robustly both lateral and

longitudinal positions of objectslongitudinal positions of objects Performing well under adverse conditionsPerforming well under adverse conditions

Perspectives:Perspectives: Quantitative assessment of the methodQuantitative assessment of the method Comparison with other schemesComparison with other schemes