Road Sign Detection in Autonomous VehiclesSteven Cahail

Faculty Collaborators: Dr. Michael Quinlan, Dr. Peter Stone

Departments: Department of Computer Science; FRI Autonomous Vehicle Stream; Austin Robot Technology

Velodyne HDL-64E• 64 lasers • 360 degree field

of view • 5 cm resolution

(distance) • 10 Hz field of

view update • ~1,000,000

points per second

Our Algorithm: Overview Three stages:

1. Use point-clustering algorithm on Velodyne data to identify objects that may be road signs.

2. Coordinates corresponding to these objects are mapped onto camera data, so area containing potential sign can undergo image processing.

3. Image processing is performed on sign object to determine if it’s a sign and, if so, which sign.

Other Collaborators• Vladimir Coxall, Tony Daddeo, Jesse Vera, Nick Shelton,

Adam Menz, Jonathan Long

• Austin Robot Technology: Jack O’Quinn

Existing approach to handling road signs•Location of signs must be preloaded into the car’s path. •Can we create an algorithm that uses our sensors to detect and identify road signs during operation?

Sensors•Velodyne HDL-64E LIDAR•Sony XCD-SX90CR camera•Velodyne produces a 3D point cloud mapping of the area surrounding the car. •Camera uses IEEE 1394b digital interface (FireWire) , runs at 15 FPS at SXGA Resolution

Example Velodyne Output

Step 1: Velodyne Sign Detection• sign_detect node processes velodyne data, returns point cloud coordinates corresponding to objects that could be signs• Clusters of points filtered based on position in XYZ coordinate plane: height, distance from car, position relative to car (i.e. in front and on the right side of the road).• Also filtered based on size of cluster.• If criteria is met, coordinates of points in the cluster sent to point_to_image node

sign_detect(Velodyne sign

detection)

point_to_image(3D image projectonof Velodyne data to

camera)

sign_vision(image processing

for visual sign identification)

Camera Image

Velodyne point cloud data

ROS and Algorithm Flowchart• Project is implemented in ROS (Robot Operating System), a software library that allows simplified passing of data between nodes, or programs• Also uses OpenCV library for image processing functions

A screenshot of sign_detect running. The cluster of points highlighted in red is a sign candidate and will be sent to point_to_image for further processing.

Step 2: Projection of Points onto Camera Image• 3D point cloud data must be mapped onto a 2D camera image.• Gives us (x, y) coordinates in the video image of sign locations, allowing these locations to be focused on for image processing.

The general form of a 3D to 2D projection

The projection produced by point_to_image. Note the accuracy of the detection and projection in the left image: even with several objects surrounding the sign, the projection is only onto the sign.

Step 3: Image Processing and Sign Identification

• point_to_image sends a region of the image that contains an object, which may or may not be a sign.• sign_vision first processes the image to verify that it contains a sign.• If it contains a sign, further processing is performed to determine what type of sign has been seen.

(1)First, a binary image is created:

(2) An erosion filter is then run on the image, followed by a dilation filter, producing an image containing white blobs, which can be used to determine if a sign is present.

(1)

(2)

Once an object has been accepted as a sign, a template matching process is used to determine which type of sign. First color matching is performed, and then templates of several different sizes of a sign are slid over the image until a match is found.

Example output of the sign_vision algorithm on an image from Google Streetview. The yellow bouding box indicates a warning sign was detected.

Testing and ResultsTesting was done individually on each of the component programs.

sign_detect:For sign_detect, there were two types of errors: false positives (clusters sent to point_to_image that are not signs) and false negatives (signs that are missed). False positive error percentage: ~11%False negative error percentage: ~17%

point_to_image:Projection errors were calculated by taking the distance between the actual pixel and the projected point in the camera image. If the car is moving toward a sign and the sign is near the center of the image, the error is less than 10%. Error could increase if these conditions are not met.

sign_vision:Testing for sign_vision was done on images of signs taken from Google Streetview::

Total Signs

Signs Detected

False Detection

Signs Recognized

False Recognition

Number 90 75 3 71 4

Percent 83.30% 2.40% 78.90% 4.44%

Conclusion and Future WorkAll the components of our system work independently, but further testing and development is required to enable them to work together to produce the full result – a system that can detect road signs as the car is driving. The most algorithmically complex part of this is the code for sign_vision – this is computationally the slowest part of the process and the one that is most prone to error. More work is needed here in order to make our system fully functional. In addition, improving the speed of the overall algorithm is also necessary. Currently, signs are not typically recognized until the car is too close to react to the sign. Finally, mechanisms must be added to allow the car to react to the sign once it has recognized it. If these additions can be made, the system described in this poster can be fully functional.