AN EMBEDDED LANE DEPARTURE WARNING SYSTEM Pei-Yung Hsiao1, Kuo-Chen Hung1, Shih-Shinh Huang2, *Wen-Chung Kao3, Chia-Chen Hsu1, Yao-Ming Yu4

1Dept. of Electrical Engineering, National University of Kaohsiung, Kaohsiung, Taiwan 2Dept. of Computer and Communication Engineering, National Kaohsiung First University of Science and

Technology, Taiwan 3Dept. of Applied Electronics Technology, National Taiwan Normal University, Taiwan

4Department of Computer and Communication Engineering, National Taipei University of Technology, Taiwan

ABSTRACT In this paper, we propose an embedded lane departure warning system that is composed of a robust control flow and a well-designed lane detection algorithm. The lane detection algorithm is designed based on a peak finding method to accommodate various lighting conditions, and a spatiotemporal mechanism using the detected lanes is proposed to generate appropriate warning signals. The entire system has been implemented on a portable device with embedded software.

1. INTRODUCTION

In order to improve driving safety, many works on the development of driving-assistance systems (DAS) have been done in the past decade [1]-[7]. A DAS that incorporates the digital camera and the laser range finder for collecting the environment information in the front of the car has been proposed in [1]. With the camera, it becomes possible to detect pedestrians as well as cars on the lanes in real time [2]. For further guiding the driver, the system that automatically detects the lane boundaries was presented in [3], [4]. It is possible to integrate the detected information taken from several cars and boost up the accuracy of the estimation of car locations [5]. Although these approaches may be helpful in some aspects, the system cost is very high. Hence developing a portable low cost DAS becomes an important topic. With the advance of image sensor technology as well as system-on-a-chip development, a real-time lane departure warning system (LDWS) can be realized on a portable device. The properties of such a system include low cost, flexibility, and high compatibility. To achieve these objectives, two common issues needed to be addressed for developing an LDWS: the robustness of a lane departure warning algorithm and the embedded system architecture for realizing the algorithm on the platform.

In this paper, we present an LDWS that contains a robust control flow that has been implemented on an embedded system. The system can be easily installed on the car. The algorithm of lane detection is based on the peak-finding method. In order to improve the detection performance, the captured images are first processed by a one-dimensional Gaussian filter to attenuate noise and a global edge detector to further remove undesired background information [8]. In the previous works related to LDWS, the angle based method would be the most popular one. The lane model may be approximated by parabola or other nonlinear curves. A practical method to fit curved lanes is by using piecewise linear model [9]. On the other hand, detecting lane boundaries should consider some imperfection factors caused by various lighting condition [10]. Although angle based

algorithm is one of popular methods, the detection results are usually unreliable when the scene contains some lanes with large angles. In addition, most of LDWSs only have a single warning mechanism such that the system may report a few false alarms. It is possible that the warning system sets the alarm, even if the car is still kept on the right location/direction. To remedy this drawback, the system we developed includes two detection mechanisms. The first one is according to the location of lane boundary intercepts. The alarm is set when the intercept point is close to the center of the image when the car is just locating on the boundary of two lanes. The second one is based on the evaluation of the changing rates of lane intercepts. It efficiently eliminates the false alarm when the car is just locating on a lane boundary.

In the following sections, we will first describe the details of the proposed system as well as the system optimization method in Sections 2 and 3. The experimental results and conclusion are provided in Sections 4 and 5.

2. THE PROPOSED SYSTEM 2.1 The Proposed Control Flow

Lane ?

Master/slave lanes determination

Spatial mechanism

Temporal mechanism

Capturing Video frames

Global edge detectorGlobal edge detector

Lane feature point extractionLane feature point extraction

Line segment combination Line segment combination

Converting to gray imagesConverting to gray images

Noise filtering by Gaussian filterNoise filtering by Gaussian filter

YES

YES YES

Warning

Start

NO

NO NO

Fig. 1. The control flow of lane departure warning system.

Fig. 1 shows the control flow of the proposed lane departure warning system. Following by the necessary image preprocessing stages, the system first extracts global edges by the Sobel operator. Some feature points that have maximum gradients in each horizontal scanning line. The global edges that have maximum number of features points are recognized as the master lane segments. The slave lane segments are defined as the edges

2011 IEEE 15th International Symposium on Consumer Electronics

978-1-61284-842-6/11/$26.00 ©2011 IEEE 162

with the opposite slope of the master lane that pass through the most numbers of feature points. All detected lane segments are assembly into the master and slave land boundaries. Then the system runs into the judgment stages which contain two mechanisms: spatial and temporal ones. The spatial mechanism is according to the relative location between the car and lane boundaries; while the temporal mechanism is based on the variation speed of the car location among successive frames.

2.2 The Method of Dangerous Condition Detections

md sd mdMd MdSd

Fig. 2. An example of lane location identification.

Fig. 2 shows an example of lane location identification. The parameters Md / Sd are defined as the interception distance between the master/slave lanes and boundary of the warning box. Based on the observation, we can detect the locations of the lanes based on the values of Md and Sd , and the variation speed among successive frames can be used to detect the dangerous conditions.

MdWarning Box

Dangerous Region

Md SdWarning Box

Dangerous Region

Warning Box

Dangerous Region

Time Period

Md

Time Period

SM dd +

Fig. 3. The proposed detection mechanisms: the first three figures show three locations of the car, and the final one shows the statistical data ( Md + Sd ) among successive frames.

Fig. 3 shows the concept of the lane departure warning system by spatial and temporal mechanisms. The system enables the warning determination system if the values of Md or Sd are larger than a predefined value. This is called as spatial mechanism because it examines the locations of the car within its bilateral lanes. In addition, we also define another smaller box called dangerous windows and check the interception points of the windows and the master/slave lanes. If the lanes cross the dangerous windows, it shows the car is crossing the lane boundaries. Hence the system also enables the warning determination system.

Fig. 4 shows the details of temporal mechanism built in the system. The left figure shows the two parameters for several frames. If the lane departure condition occurs, we can find that the value of a curve is increased and the other one is decreased. This chart can be also used to analyze the changing speed δ of the car locations ( ( )v t ). If the changing speed of the car location is too high, the car may be in a dangerous condition. To realize this concept, the two parameters ( Md and Sd ) are first summed up for each frame. Then the gradient of the curve is analyzed for estimate the car velocity in horizontal direction. In order to get more stable results, the statistical data of ten frames are averaged first. The changing speed of the car locations is then formulated as the difference of the averaged data ( Md + Sd ) between the previous ten and the next ten frames. If the changing speed is higher than a predefined, the system also enables the warning system.

( ) max,tdM ( ) max,tdM

( ) ( )tdtd SM +( ) ( )tdtd SM +

t

MdMdSdSd

t

( )tv( )tv

tΔtΔ tΔ2 tΔ2

δ( )2tv( )2tv

( )1tv( )1tv

tΔtΔ tΔtΔ

Fig. 4. The details of temporal mechanism.

3. System Optimization on Embedded System

In order to achieve the real-time applications, several optimization factors should be taken into consideration when implementing the algorithm on a portable device: complexity, power consumption, data and program memory access. The complexity reduction is done by using pure integer operations instead of floating point operations. Most of multiplication/division operations are replaced by data shift operations. The power reduction is achieved by adaptively enabling the external devices and dynamically switching the embedded processor into low power mode. In addition, a block of memory is allocated for multiple purposes in different routines that are not running at the same time. The concept comes from the fact that a memory block will have higher probability to be loaded into the cache memory if it is frequently used in the

163

program. The system should fully utilizes the allocated memory assigned the real-time operating system.

4. IMPLEMENTAL RESULTS

Fig. 5. Overview of lane departure warning system.

Fig. 6. The pictures of the proposed lane departure warning system.

Line 0

Line 1

Line 2

Line 3

Line 0

Line 1

Line 2

Line 3

Boundary 0Boundary 1

Boundary 0Boundary 1

Fig. 7. An example of line segment and lane boundary detection.

Table 1.

Personal Computer

Embedded System Without Optimization

Embedded System With Optimization

System Clock 1.8GHz 520MHz 520MHz

System Memory

1GB 128MB 128MB

Execution time

2ms 1034ms 132ms

The system overview of the proposed lane departure warning system is shown in Fig. 5. The entire system has been implemented on a Window CE 5.0 platform with running on an embedded microprocessor at 520 MHz clock rate. As shown in Fig. 6, the proposed system can be easily installed on the car. Fig.7 shows an example of line segment and lane boundary detection. The car departure warning mechanisms are designed by analyzing the detected lane boundaries. With the resolution of the input frames is 320 240× , the system performance can

achieve 15 frames/s. The overall accuracy achieves higher than 97.9% under day and night lighting conditions. Table 1 shows the execution time under different hardware conditions. The proposed system can reach the requirement of real-time application.

(a)

(b)

Fig.8. An example of normal condition.

Fig. 8 shows a normal case, which has no lane departure condition. The spatial and temporal mechanisms show the parameters Md and Sd are located in the safe region. As shown

in Fig. 8-(a) both Md and Sd are lower than the threshold value, hence the system does not set the alarm. Fig. 8-(b) shows the summation values of Md and Sd . Since there is no significant change in a unit time frame, the temporal mechanism does not enable the alarm.

Fig. 9 shows the case of the car is moving cross the lane boundary. As shown in Fig. 9-(a), the spatial mechanism detects that the parameters Md and Sd have out of the safe region. But

164

the temporal mechanism shown in Fig. 9-(b) does not detect any dangerous conditions, since there is no significant change in a unit time frame.

020406080

100120140160180 dmdssum

pixel

frame

(a)

(b)

Fig. 9. An example of lane departure condition with low changing rate.

5. CONCLUSION

In this paper, we have presented a robust lane departure warning system which is realized on a portable system equipped with Windows CE operation system. It achieves the objectives of low power, high performance, and easy installation. The proposed system also achieves a high detection rate for lane departure condition.

6. REFERENCES [1] Bertozzi, M., Bombini, L., Broggi, A., Cerri, P., Grisleri, P.,

Medici, P., Zani, P., “GOLD: a framework for developing intelligent-vehicle vision applications,” IEEE Transactions on Intelligent Systems, vol. 23, no. 1, pp. 69 - 71, 2008.

[2] Ogawa, G., Kise, K., Torii, T., Nagao, T., “Onboard evolutionary risk recognition system for automobiles—

toward the risk map system,” IEEE Transactions on Industrial Electronics, vol. 54, no. 2, pp. 878 - 886, 2007.

[3] Tang-Hsien Chang, Chih-Sheng Hsu, Chieh Wang, Li-Kai Yang, “Onboard measurement and warning module for irregular vehicle behavior,” IEEE Transactions on Intelligent Transportation Systems, vol. 9, no. 3, pp. 501 - 513, 2008.

[4] Toledo-Moreo, R., Betaille, D., Peyret, F., “Lane-level integrity provision for navigation and map matching with GNSS, dead reckoning, and enhanced maps,” IEEE Transactions on Intelligent Transportation Systems, vol. 11, no. 1, pp. 100 - 112, 2010.

[5] Thanh-Son Dao, Leung, K.Y.K., Clark, C.M., Huissoon, J.P., “Markov-based lane positioning using intervehicle communication,” IEEE Transactions on Intelligent Transportation Systems, vol. 8, no. 4, pp. 641 - 650, 2007.

[6] Pei-Yung Hsiao, Chun-Wei Yeh, Shih-Shinh Huang, and Li-Chen Fu, “A portable vision-based real-time lane departure warning system: day and night,” IEEE Transactions on Vehicular Technology, vol. 58, no. 4, pp. 2089 – 2094, May 2009.

[7] Agarwal, V., Murali, N.V., Chandramouli, C., “A cost-effective ultrasonic sensor-based driver-assistance system for congested traffic conditions,” IEEE Transactions on Intelligent Transportation Systems, vol. 10, no. 3, pp. 486 - 498, 2009.

[8] Ahmad, M.B. and Tae-Sun Choi, “Local Threshold and Boolean Function Based Edge Detection,” IEEE Transactions on Consumer Electronics, Vol.45, pp. 674 – 679, Aug 1999.

[9] Claudio Rosito Jung and Christian Roberto Kelber, “A Lane Departure Warning System based on a Linear-Parabolic Lane Model,” IEEE Intelligent Vehicles Symposium, pp. 891 – 895, June 2004.

[10] J. W. Lee, C.-D. Kee, and U. K. Yi, “A New Approach for Lane Departure Identification,” in Proc. IEEE Intelligent Vehicles Symposium, Columbus, USA, pp. 100–105, 2003.

165