GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,

GM-Carnegie Mellon Autonomous Driving CRL

Title Automated Image Analysis for Robust Detection of Curbs

Thrust Area Perception

Project Lead David Wettergreen, CMUWende Zhang, GMInna Stainvas, GM

Contributors JongHo Lee, CMU

1


ScheduleCurb location Sensor location Methodology Date

On the side Bottom of the side mirror Visual appearance ~ Jan. 2014

In front The front bumper Geometric structure ~ May. 2014

In front The front bumper Appearance + Geometry with production camera

~ Nov. 2014

DeliverablesDemonstration: In-vehicle curb detection

Annual reports

2


Objectives

Develop reliable methods of detecting, localizing, and classifying features associated with curbs using in-vehicle, low-cost, monocular vision sensor

Localize curbs within a range of 5 meters with 99% accuracy

3


Approaches for Curb Detection Using Mono Camera Images

• Appearance-based image analysis (~ Nov. 2013)• Extract features

• Evaluate performance

• Geometry-based image analysis (~ May. 2014)• Structure-from-motion to estimate camera motion

• Multi-resolution plane sweeping algorithm to create 3-D point cloud

• Plane fitting to detect curb

• Combine appearance and geometric analysis (This Review)

4









5


Appearance-based image analysis

6


Edge Detection

7


Detect Curb Using HOG* Feature

* Histogram of Oriented Gradients

Input image HOG imageCurb model

8









9


Geometry-based image analysis

Input image Depth image

Ground plane estimation

3-D point cloud

10


Plane Fitting

11









12


Appearance at t+1

Schematic Overview

Input at t+1

Input at t Appearance at t

Geometry

Candidate regions

Annotate curb region

GM-Carnegie Mellon Autonomous Driving CRL 14

Appearance

- For each image, divide intom x n grids- m: image height / grid

size (pixels)- n: image width / grid size

(pixels)

Image at t


Appearance

- For each grid, classify among two classes (road, curb)- uniform Local Binary

Pattern (LBP)

Image at t


Local Binary Pattern

64 63 52

85 152 227

189 167 205

0 0 0

0 1

1 1 1

threshold

Binary: 00011110Decimal: 30

0 1 2 3 255


Appearance

- Once all the grids of two images are classified, get the intersection of them

Output at t+1Output at t Intersect


Geometry

- Green lines shows the vectors from the interesting points of image at time t (blue dots) to those of image at time t+1 (red dots)

- Calculate the 3-D points using camera matrix


Appearance + Geometry

- For each grid,- Fit the best plane using 3-D

points - Compute the normal vector- Determine the normal vector

is a road surface or a curb surface


Appearance + Geometry


Extend Curb Region

- If the appearances are similar, extend the curb region- Calculate the distance of LBPs using chi-square


Extend Curb Region


Track Curb Region

- For the next frames, tracking the appearance of the curbs- When tracking, keep checking the geometry constraint to

remove the false positives if exist

Input at t+2 Input at t+3 Input at t+4


Curved curb case

Extend Curb RegionCombine Analyses


Curb Detection using Production Camera

Image size : 480 by 640FOV: 180 degree


Test Curb Detection

Image Size: 640 x 480 (pixels)ROI: 640 x 160 (pixels)Size of grid: 20 x 20 (pixels)Number of grids: 32 x 8

Output of the appearance-based curb detection


Test Curb Detection

Remove outliers based on cluster size

Find edges using Canny operator inside candidate region


Test Curb Detection

- Fit polynomials to each segments, and check lines for similar curvatures (blue), and remove high curvatures (red) Annotate curb region on the

original input image


Future WorkApplication: Operate real-time curb detection in vehicle

~ May 2015

29


Thank you

Questions ?

30

Documents

GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,