DEVELOPMENT Assistance Systems

10 ATZ worldwide 7-8/2002 Volume 104

Image sensors have a good chance of becoming establishedalongside radar and laser-based sensor systems. They facilitatethe optical detection of objects and features in traffic spacewhich can be detected only with great difficulty or not at allusing other physical sensor principles, putting them in aposition to provide effective support for assistance systems ofthe next generation. This article by Hella KG Hueck & Co.explains the sensor characteristics as well as possible futureapplications in some detail.

1 Introduction

The importance and the share of

automotive safety systems on the one hand

and systems providing more comfort and

convenience on the other will significantly

increase in the next few years. Sensors for

measuring data in the vehicle environment

are the first links in the signal processing

chain in all such systems. This results in

two new challenges: the measurement of

physical parameters outside the vehicle

By Martin Mühlenberg

and Tilman Seubert

Fahrerassistenzsysteme

basierend auf Bildsensorik

You will find the figures mentioned in this article in the German issue of ATZ 7-8/2002 beginning on page 658.

Driver AssistanceSystems Based onImage Sensors

11ATZ worldwide 7-8/2002 Volume 104

system and, often inevitably linked with

this, the use of innovative sensor concepts.

With the aid of image-producing sensor

technology and mechanical sight, pictorial

features describing the infrastructure can

be detected in traffic space. Optical sensors

from Hella already used in standard vehicle

models, such as rain or light sensors, form

the optimum basis for integrative and

functional expansion through future image

sensor concepts. Expertise in the field of

optics, synergies within electronic and

software development as well as a

thorough understanding of integrative

questions are essential pre-requisites for

survival in the automotive product market

of future optical-image-producing systems.

Figure 1 illustrates the basic topology of

future driver assistance systems. A

distinction is made between the long-

distance range > 100 m covered by 77 GHz

radar or infrared lidar systems for functions

such as Adaptive Cruise Control (ACC), the

mid-range up to 50m and beyond covered

by image-processing methods for functions

such as Lane Departure Warning or night

vision, and a close-up range covered by 24

GHz radar or infrared scanners for

functions such as stop & go, automatic

emergency brake, pre-crash, recognition of

the blind angle and parking assistance.

Figure 2 classifies the image sensor

applications according to customer

functions, complexity and possible

moment of use: the entry functions are

comparatively easy and cover

conveniences requirements. After this the

driver assistance functions develop into

safety-related networked functions, on the

one hand increasing the complexity of the

algorithms and on the other making the

data fusion of several measuring principles

necessary.

2 Lane Tracking: One of theFirst Applications of ImageSensors in Driver AssistanceSystems

The first image sensor applications in

passenger vehicles are rear-view camera,

night vision and lane tracking. Whereas

with the first two of these image

reproduction for the driver and ergonomic

marginal conditions are at the fore, the last

mentioned feature manages without

image representation. In this last case,

image analysis and the transparent

application for the driver with regard to the

function are at the fore.

Figure 3 illustrates the most important

features of a lane tracking system:

■ The sensor captures the environment in

front of the vehicle. Accuracy and stability

in a wide range of different light and

weather conditions are the most important

features of the image quality. These are

mainly determined by optical and image

sensor chips.

■ Integration in the vehicle. Optimum

fulfilment of the customer function with

aesthetic integration and a structure and

connection technology which is as reliable

as it is economical are the main design

objectives.

■ The algorithms of image processing

mainly characterise the application. These

will be explained in the following in more

detail.

■ The interface to the driver completes

the customer function. It is here that the

largest share of customer-related

differentiation can be found.

3 Sensor Features

For applications with a view forwards, the

obvious place to install such sensors is

behind the wiped area of the windscreen.

On the sensor side, there is a lens unit and

design measures to reduce stray light

situated in front of a CMOS image sensor

chip. Primary design parameters

determined by the application(s) are

mainly the alignment of the optical axis,

the horizontal and vertical aperture angle

and the sensor resolution. The spectral

sensitivity of the image sensor, the high

variability of the spectral distribution of

the light in the environment to be recorded

(sunlight, artificial light at night) as well as

the transmission behaviour of the

windscreen must all be taken into

consideration for the optical design.

Depending on the sensitivity of the sensor

and the aperture chosen, sufficient

exposure must also be ensured at night,

Figure 4 and Figure 5. The type of dipped

beam light source used for illumination of

the scenario (halogen or gas discharge

bulb) has a decisive influence on sensor

exposure due to its different spectral

distribution. Situations at night in

particular make it clear that the

performance of a driver assistance

application based on image sensors is

determined by the limitations with regard

to the prevailing visible conditions. Thus,

for example, the viewing distance for lane

tracking at night without an additional IR

light source is limited to the range achieved

by the dipped beam light. Furthermore,

detection of neighbouring lane markings in

the dark in scenarios with more than one

lane often causes problems, resulting in

limited performance when changing lanes

and putting the search areas for image

processing into practice. On the other hand,

some conditions for feature detection are

better at night, since the retro-reflecting

property of the markings at least supplies

good image contrasts and prevents the

occurrence of shadow situations. One point

often discussed in connection with the

physical radiation properties of sensors for

automotive use is the demand for sufficient

dynamics. Since extreme exposure

situations can often occur extremely

quickly (for example, shadow situations in

a tree-lined avenue at high vehicle speed),

the required high grey value dynamics

must be related to each individual image

frame. Last but not least, the whole camera

unit is subject to great temperature

influences, which result in fading images

in the image sensor chip and material

expansion with corresponding optical

effects in the imaging unit, Figure 6. A

sturdily designed image processing system

can tolerate such effects up to a given limit

depending on the application involved,

since the image has been intended for pure

image processing and does not have to

fulfil the visual requirements of a human

observer.

4 Lane Departure Warning

Unintentional lane departure is one of the

most common causes of fatal accidents. The

reasons for this vary, and include

distraction of the driver and excess speed,

to give only two examples. A system that

detects the position of the vehicle in

relation to the lane markings and compares

this with the expected driver intention,

which can be determined by means of

changes in steering angle, activation of the

indicator light and brake pedal, can provide

additional information to the driver in

critical situations and give a warning. Such

lane departure warning systems are

already available in the commercial vehicle

sector and will enter the passenger car

sector in the next few years. Initially, such

systems will primarily be used in well-

structured scenarios with those road

markings already available. White lane

markings as pictorial patterns allow

reproducible and advantageous

recognition using appropriate image

processing operators. This requirement of

solid feature recognition must not be

underestimated, and guarantees successful

image processing in a wide range of

different lighting and weather conditions.

Edge-based image processing operators

measure the positions of the lane markings

at different viewing distances, Figure 7.

These are entered into a model-based

tracking using a Kalman filter process.

DEVELOPMENT Assistance Systems

12 ATZ worldwide 7-8/2002 Volume 104

Descriptive model parameters are the lane

geometry and the vehicle kinematics.

Special features of the infrastructure in the

course of the lane such as filter lanes and

motorway exits often require a regulation-

based feature selection as an initial

parameter component, due to the

ambiguity of the markings, Figure 8. The

requirements with regard to aperture angle

are first of all dependent on the application.

In the case of lane tracking, this would be

the detection of markings as near to the

vehicle as possible – thus describing the

position of the vehicle itself very well –

resulting in a wide-angle design of the lens.

Manufacturing and installation tolerances

as well as vehicle-dynamic structural

movements must also be taken into

account, however, in that a certain increase

in angle must occur with regard to the

design of horizontal and vertical aperture

angle.

5 Object Recognition

In contrast to the individual sensors with

their primarily specific applications, the

driver assistance systems of the future will

require an overall view. Thus, future image

sensor systems will initially detect the lane

and warn drivers if a lane is changed

unintentionally and will then in future

sensor generations certainly take over

further tasks in combination with other

sensor systems. One of these applications

will be supportive object recognition for

ACC systems, which can already be found

in the form of radar or laser sensors in some

vehicle models today. The distance and

lateral position of vehicles and objects

supplied by the ACC sensor serves as a basis

for fixing parameters for good pre-

conditioned image processing with regard

to the values to be expected. This is then

used as a basis for recognising object

contours through the image sensor system.

In this way, the search areas in images and

contour positions to be expected can be

restricted in a well-defined way. The results

are more exact lateral position and

dimension data of the objects followed by

the ACC, as well as a more accurate

estimation of the cut-in and cut-out

manoeuvres of the relevant vehicle targets.

Vehicles can be classified with regard to

their design in order to implement a model-

adaptive strategy with length regulation

and, last but not least, when coupled with

the lane tracking function, the system can

allocate lanes to the relevant vehicle

targets.

6 Further Functionalities

The present Hella light sensor uses two

independent receiver elements for

luminance detection in the environment

and directly in front of the vehicle. The

locationally resolved measurement

achieved using an image sensor optimises

the detection of tunnels and underpasses in

particular. Furthermore, such a sensor

plays an important role in the selection of

situation-adapted beam patterns. Absolute

measurements are possible, taking sensor

exposure parameters and appropriate

calibration into account. The knowledge

gained about the environment by an

image-processing-based light sensor

system leads once again to even better

settings of the parameters for the image

processing algorithms for the lane tracking

application. This mutual support and the

similarity of the required optical marginal

conditions says a lot for combining these

two applications in one image sensor

system. This is why the functional

possibilities and sensible focusing of

different applications into one image

sensor system are being pursued in the

development steps.

7 Block Structure of an Image Sensor System for Lane Tracking

Figure 9 represents the basic structure of

an image sensor system:

■ The front-end consists of a lens and an

image sensor. The image captured is / not

pre-processed and forwarded to the

computer unit via a suitable image data

bus.

■ Image processing and application

algorithms run on a digital signal processor

(DSP).

■ Communication with the vehicle

interface to the HMI can take place via CAN.

Depending on the construction space

available, the image processing system can

be partly or completely integrated in a

different module. Thus, for example,

solutions are currently under consideration

in which the image sensor front-end is

integrated in the roof control module, as

illustrated in Figure 2, whereas the

computer unit can be installed elsewhere.

In this case the image data bus is subject to

higher EMC requirements.

8 Summary and Outlook

The reliability of lane tracking has been

proved using extensive driving situations.

Based on these results, Hella is working at

high priority on concrete implementation

in an automotive product. The introduction

of a series product is to be expected in 2005.

Lane tracking is certain to be one of the first

applications. Once image sensors have

been generally accepted in vehicles, further

applications will follow. The triggering of

vehicle lighting is important in this context

in particular. The future networking of

environment sensors will then be the next

step towards a complete driver assistance

strategy. Due to the complementary sensor

characteristics, the combination of an ACC

sensor and an image sensor system creates

good pre-conditions for optimised vehicle

and object recognition, Figure 10. The aim

is to record the environment of a vehicle in

its whole complexity in order to achieve

functions which initially increase

convenience and subsequently safety. ■

DEVELOPMENT Assistance Systems