ROAD EDGE TRACKING FOR ROBOT ROAD FOLLOWING

A.D. Morgan E.L. Dagless D.J. Milford B.T. Thomas

Faculty of Engineering, University of Bristol, Bristol, UK.

The problem of navigating a robot along a road is ap-proached by means of creating and updating a simplerepresentation of the road from a sequence of images.The representation chosen is a 4-parameter model thatdescribes the width, direction and simple curvature ofthe road in a vehicle centred (X,Y,Z) world coordinatesystem. The model is created from tracking along ma-jor edge features in an image and applying constraints toselect road edge candidates. Updating consists of track-ing a set of measured edge points from frame to frame(assuming that vehicle motion is known) and using aweighted least squares process to find the 4 parametersof the road model. A number of constraint and filteringprocesses representing knowledge of how a vehicle moveson a road have been applied.

The application of computer vision techniques to the au-tomatic guidance of mobile robots is a very challengingtask. In the case of a road vehicle this is especially soconsidering the competence of human vision and the va-riety of other human skills that are used and the effortstaken to learn how to coordinate them. In addition theenviroment can become very complicated from simplewell-defined roads to complex ill-defined roads that in-clude lay-by's, junctions and roundabouts all with mov-ing potential obstacles. Initial research in the USA 1|2'3

and 4'5'6 has demonstrated the automatic navigation ofroad vehicles over simple, well-defined roads at a mod-est speed using image processing techniques that processlarge regions of the image. Research in Germany 7'8 hasshown that the use of dynamic and geometric modelsof the road and the vehicle can be used to focus theimage processing into gathering information about thescene from small windows. These techniques have beenapplied to tracking edges in very well defined roads atmuch higher speeds, an approach supported by humanfactors reasearch into driver's eye movements while ne-gotiating curved roads 10. However, it seems likely thata more competent visual navigation system would haveto include a great wealth of information about the en-viroment in the form of hierarchical models that guideand are updated by image processing of large parts ofthe image. An important component of such a systemwould be a model of the road edges.

In subsequent sections, a simple model of a pair of roadedges is described which is used to guide the tracking ofedge points through a sequence of images. An orientated'edge-finder' operator is descibed that determines theedge point positions which are the data for a weightedleast squares curve fitting process to determine the pa-

rameters of the model at each frame. Finally, a descrip-tion of the overall algorithm is given with initial resultsand conclusions.

ROAD MODEL

In computer vision, a model is an application specificrepresentation of features of interest in a scene that iscapable of predicting the form of features in an imageor the behaviour of features in a sequence. The useof a model is important as it allows the results fromprocessing of previous images to influence the currentprocessing.

A Four-Parameter Road Model

The simplest possible model of a road is a pair of straightlines in an (x,y) image coordinate system. This cruderepresentation is obviously inadequate when the road be-gins to curve but also is not convenient to accomodateedge position changes in the image due to measured ve-hicle motion. A more powerful and convenient represen-tation is to think of the road as comprising a flat surfacebounded by two road edges that are each described bya circular arc in a vehicle centred (X,Y,Z) world coor-dinate system (initially assume that the earth is flat sothat Y is a constant) as in Figure 1. In the first instanceit is reasonable to assume that the road-width is con-stant or varies slowly from frame to frame. This meansthat the left and right circular arcs having radii iZje/«and Rright respectively share a common centre (Zo, XQ).These are the four parameters of the model which arerelated by,

\^-left — -A-0) — n-left ~ (•"left ~ "O) \L)

IY y \ 2 _ p2 (7 <7 \2 (n\(Aright — -A-o) — n-right ~ (bright — ^0) (*•)

roadwidth — \ Rright — Rujt \ (3)

The radius of the circular arc determines the curvature,with a very large radius giving an almost straight sectionand the position of the centre of the circle determiningthe general direction of the arc. The lateral offset of thevehicle from the left edge for example, is obtained whenZ = 0. This 4-parameter model is capable of represent-ing the position of the robot on the road, the generaldirection of the road and a simple curvature. This rep-resentation has proved to be adequate for the majorityof road scenes that we have encountered. However, thereare situations in which this representation is inadequate.For example, a long straight section followed by a curveor a section with two or more curves. A more powerfulrepresentation would be to use cubic spline approxima-tions or other higher order polynomials to model the

179

AVC 1988 doi:10.5244/C.2.28

Figure 1: Two circular arcs with a common centreXQ,ZQ.

Figure 2: The coordinate system.

road edges. Other representations n ' 1 2 based upon a3D "ribbon" have been described that model the roadwith both a horizontal and vertical curve. The difficultywith these representations is that the uncertainty in theparameter values is increased as more parameters haveto be identified 'from the information extracted from theimage.

Using the Model

In order to relate information extracted from the imageto features in the world and to the road model it is neces-sary to be able to transform between the (x,y) image and(X,Y,Z) world coordinate systems. The transformationis direct if it assumed that the earth is locally flat andthe height and look angle of the camera are known. Fig-ure 2. shows the two coordinate systems. The followingexpressions assume that the camera is looking straightahead with a tilt angle 9. This was found to be neces-sary in order to have the road appear in the majority ofthe image.

(x,y) <= (X,Y,Z)x =

y -

(X,Y,Z) «= (x,y)

Z =

Y =

X =

FXsecOZ - Ytanfl

Ztan6)Z -Ytan6

y-Fta.nOconstant(ZcosO -YsinO)x

(4)

(5)

(6)

(7)

(8)

where F is the focal length of the camera. Under theassumptions made these transformations allow a depthand lateral offset to be assigned to each pixel in theimage.

In the situation where a road model has been initialisedfrom edge information extracted from a particular im-age (described in a later section), then the processing ofsubsequent images should take into account informationfrom the previous image. It has been assumed that theforward velocity of the vehicle can be measured whichprovides distance travelled AZ in an algorithm cycletime. The (X,Y,Z) edge point data can then be up-dated with this AZ to provide predicted edge locationsin the next image. There are two ways of updating themodel: Firstly searching for edge points in a fixed gridahead of the vehicle. In this case, to use the edge infor-mation from the previous frame it would be necessaryto interpolate a distance AZ between known (X,Y,Z)edge points to find the predicted points on the grid. Analternative method is to just update the (X,Y,Z) edgepositions with AZ to provide the predicted edge loca-tions in the next image. The effect of this is have fixededge points in the world that sweep into and then outof view as the vehicle moves forward. This scheme wasused in the algorithms described below.

EDGE TRACKING IN AN IMAGE

The appearence of road edges in an image can varygreatly depending upon such things as:

• road design, age and condition, painted lane mark-ers, kerb-stones, grass or dust verges,

• daily weather conditions such as intermittent sun-shine, shadows and poor contrast, rain and reflec-tions, and

• seasonal effects such as snow, ice, leaves, and blowndust.

Eventually it will be necessary to be able to detect roadedges under all possible conditions but in order to makeprogress with the use of edge information it is necessaryto make the assumption that the road scene is simpleand that the road edges are reasonably well defined.

An Orientated Edge-Finder Operator

The usual approach to detecting an edge in a region ofan image is to firstly highlight those pixels that couldbe edge pixels and then to group them together intoedge segments. The highlighting can be achieved us-ing convolution operators that vary in complexity, rang-ing from a simple 3x3 Sobel up to the Marr-HildrethDOG operator. The grouping process could be a testfor connected pixels or commonly a Hough Transform asused in 6. Whichever combination of methods is chosenthere is always a large number of arithmetic operationsto perform. By designing novel, special purpose com-puter architectures that are well suited to such low-levelimage processing tasks, the execution times can be sig-nificantly reduced. At present though realistic algorithmcycle times are between 10 and 1000 times greater thanvideo frame rates 3'6. One approach to detecting edges

180

that is less computationally intensive is to assume thatthe approximate location and orientation of the edge canbe predicted from previous knowledge stored as a model.Then, after correlation of an edge template over a regionpositioned by a model, search the resulting surface forthe best match. This also requires a large number ofarithmetic operations if performed in two dimensions.However, the appearence of a road edge in a region ofan image is essentially just one dimensional and its ori-entation predictable from a model. An edge templatethat is sensitive to an edge of a particular orientationneed only be correlated in a direction normal to the ex-pected direction of the edge. This approach was firstproposed in 8. The simplest edge template, [1,0,-1]using integer coefficients, is sensitive to vertical edges.The first problem with simple operators of this kind isthat they are sensitive to edges of a particular size at aparticular image resolution. The operator needs to bedesigned so that it is sensitive to the size of edge featureof interest. The second problem is that they are verysensitive to the effects of random noise in an image. Inorder to improve the signal-to-noise ratio, the operatorcan be replicated along the extent of the expected edgedirection. For example, an operator sensitive to edges ofa particular scale at 45 degrees,

000100

001000

01000

- 1

1000

- 10

000

- 100

00

- 1000

A set of such operators can be defined so that an edgeof any orientation can be detected. In addition, 8 alsosuggested a scheme whereby further unnecessary calcu-lation is avoided by forming intermediate arrays of thesums of appropriate pixels P, j as shown,

n9^29

^39

Psi D82 83

Pn P92 P93 Pi94

7988 ^89

'97 P98 P99

The final results are easily obtained from addition ofpairs of these intermediate array elements. In form-ing these intermediate arrays, further computation isavoided if the array indexing for the differently orien-tated templates is pre-computed and stored in look-uptables. The result of combining these techniques is avery fast, scaleable operator that, when scanned over aregion of an image, is capable of locating the position ofan edge. There still remains the possibility that the edgeoperator can produce a spurious result, due to the effectsof noise in the image or a variation in the road edge it-self. In this case, greater consistency can be achieved ifthe appropriately orientated operator is scanned over aregion in a number of channels. The operator results can

operator response

3 channels of a6x6 operator @ 45 degs.run across a 32x32 patch

Figure 3: Correlation in 3-channels and the correlationarrays.

then be tested for consistency since they should agree onthe position offset. Figure 3. shows these channels andthe corresponding correlation arrays obtained from scan-ning the 6x6 operator above across a road edge feature.Since it is important to design an operator that is ap-propriate for the scale of the edge feature and the imageresolution, a number of tests were made on typical roadedges in images at three different resolutions. The 6x6operator shown above was found to be very sensitive towell-defined edge features appearing at 45 degrees in a256x256 resolution image.

Using the Edge-Finder Operator

The orientated edge finder has been designed to locatea road edge whose approximate position and orientationin an image are known. This situation arises in the fol-lowing two cases,

• Register predicted edge positions - where a modelof a road has predicted the approximate positionof an edge in an image and selects the appropriateoperator to accurately locate the edge. In this way,the same edge points are tracked through a sequenceof images.

• Locate new edge points from extrapolation - wheretwo or more known edge points are used to extrapo-late the approximate position of another edge point.The appropriate mask can be selected from mea-suring the angle made in the image by these knownpoints. Linear extrapolation to find the nth edgepoint using the (X,Y,Z) coordinates of the knownedge points by,

V V J_/V V \ td\

Zn = Zn_i + (£„_! - Zn_2) (10)

WEIGHTED LEAST SQUARES FITTING

The purpose of fitting a curve to the (X,Y,Z) data pointsis to initialise or provide information to update a modelof the road.

181

Approximation to a Circular Arc

For the sets of edge points {X(i),Z(i)}iejt and{X(j),Z(j)}right it is required to find a pair of circulararcs that are constrained so that the road is a constantwidth as defined in equations (1),(2) and (3). However,these equations are not linear in the unknown parame-ters and it is necessary to reduce them to a linear func-tion. To do this, re-arrange and expand as a binomialseries:

the higher order terms in the expansion can be neglectedbecause, in general R*^> Z. Re-arrange the last equationinto a quadratic form,

X = (Xo ± R T Tjf) ± -j^Z T ol?^ ( ^

and comparing this with y = a + bx + ex2 it can be seenthat the parameters XO,ZQ and R can be determinedfrom the least squares fitting of a quadratic function.

Weighting Function

In a typical road scene under perspective transformationthere is obviously a relationship between the the positionaccuracy of an edge in the image and physical distancefrom the vehicle (assuming a constant image resolution).It is possible to include a 'weighting function' into theleast squares fitting process to account for this relation-ship. This is achieved by weighting the original sets ofedge points {X,-, Zi} according to the distance from thevehicle Zi. Another reason for introducing a weightingfunction to bias nearer points is that deviations of theroad edge appearence from a circular arc representationwill be larger the further from the vehicle. If it assumedthat the total measurement error Ax of the edger-finderoperator, due to such things as noise, camera wobbleetc., has a normal distribution N(0,a2), then the corre-sponding position error AX in the real world will alsohave a normal distribution JV(0, f(<r2)), the variance be-ing scaled by a simplification of the perspective trans-formation,

a2 (13)

where k is a normalising constant and F is the focallength of the camera. The coefficient of a2 correspondsto a 'weight' in the least squares fitting process and withappropriate choice of constant the 'weight' for the fur-thest point of interest at Zmax can be normalised to 1,increasing as the distance from the vehicle Zi decreases.

Weighted Least Squares Process

In the weighted least squares fitting process, there area number of measurements j/,-, in error by a randomamount e,- and it is required to minimise the sum ofthe squared errors,

y,- = a + bxi + ex2 + (14)

with a weighting function incorporated into the min-imisation by scaling the variance of each e; by the ap-propriate weight Wi and applying the transformatione,- = eiy/wl to the j/j's so that the variances are equal.The corresponding normal equations are obtained bypartial differentiation with respect to each parameterand after being set to 0, these are,

a J2 «W +6 J2 wi+c

+c

+c wixi i J/i

The solution to this system of linear equations (a, b, c)can be found using Cramer's rule. An estimate of thequality of the fit or the accuracy with which the param-eters have been identified can be found from the meanerror /z, the 'sum of the residual squares' So and the cor-responding variance a2, for a quadratic curve given by,

So =

a2 =

~ a

So( A T - 3 )

(15)

Filtering Using the Fitted Curve

It has been assumed that the measurement errors havea normal distribution which allows for the identificationof 'rogue' edge points in the original data set using thefollowing test,

\Xi-n\ > 3<r (16)

In this way the edge point set can be filtered with 'rogue'points being adjusted to lie on the fitted curve. Fur-ther, because 'rogue' edge points would have adverselyaffected the initial fitting procedure, the whole fittingprocess can be repeated. This recursive operation cancontinue until the variance of the residual errors stopsdecreasing.

ALGORITHM DESCRIPTION

Model Initialisation

There are two situations where a road model would needto be initialised.

Firstly, when the vehicle is stationary on the road thereis almost unlimited processing time to analyse the scene.In this case, it is most likely that a number of methodswould be used to account for failures and resolve ambi-guities in finding the road. One such method describedin 6 is to apply a Hough transform to a set of edge pixelsthat have been found from histogramming of the gradi-ent direction, in order to find dominant linear features.

Secondly, whilst the vehicle is moving a situation is en-countered that causes the normal mode of operation tofail. In this case, a fast re-initialisation process is pro-posed. Assuming that one or both of the road edges arestill in view, overlap the edge finding operator to producea very long orientated operator. Then search the result-ing correlation array for the 'brightest' n% and attemptto track along these using the extrapolation process..Possible single tracks or track pairs can then be testedfor consistency. Failure of this method would require thevehicle to stop and analyse the full scene.

These proposed initialisation methods have yet to be in-cluded and in their absence the initialisation is guided

182

manually by providing the initial coordinates of two(x,y) edges on the left and right road edges. The re-sult of this processing is to produce sets of edge pointsfor the left and right road edges in both the image andworld coordinate systems.

Normal Frame-to-Frame Mode

In subsequent frames, the following sequence of opera-tions are performed on the sets of edge points.Firstly,using the road model;

vehicle motion correction- knowledge of the vehicle'sforward motion and the algorithm cycle time allow thedistance travelled AZ to be calculated. The (X,Y,Z)edge coordinates are then updated. After transforma-tion into (x,y) image coordinates, these then form thepredicted locations of the same points in the currentframe.

extrapolating new points - because the vehicle hasmoved forward, new portions of the road edge are vis-ible and are found through the (X,Y,Z) extrapolationprocess described in the previous section. In addition,a constraint on the maximum allowable 'curvature' canbe applied to limit the extrapolation process, where themodel state in the previous cycle is used to find the max-imum.

filtering old points - as the vehicle proceeds edgepoints that are near the vehicle will disappear from viewand thus this condition needs to be tested and the edgepoints filtered.

register predicted edge positions - each predictededge position is used to position the edge finder operatordescribed in the previous section. The correct orienta-tion of this operator is chosen by using the state of themodel in the previous cycle. The new (x,y) edge pointsthat have been consistently located using this methodare then used in the next stage of updating the model.

and then to update the road model;

curve fitting in (X,Y,Z) - after transforming into(X,Y,Z) coordinates, use the weighted least squares pro-cedure to fit a curve to the sets of left and right edgepoints. As described earlier, the statistics of the resid-ual errors can then be used to identify 'rogue' pointsand filter the sets. The coefficients obtained in the finalfit for each side provide values for {Zo, Xo, R}uft and{Zo,Xo,R}righf By the earlier definition, the coordi-nates of the centres of the two circular arcs are the sameand the difference in the radii is the road width.

confidence measure - a change in the model parame-ters can be attributed to a combination of a change inthe road appearence and/or a change in the accuracyof the fit. In order to allow the model to change to anextent controlled by the quality of the information, aconfidence measure is used based upon the standard de-viation a of the residual errors in the curve fitting. Thatis, if the (7 in the current cycle is less than the previousthen use the parameters from the fit. Otherwise makea percentage change in the previous model parameters.Two reasons for a low confidence measure are firstly the

uncertainty of the edge data at larger distances from thevehicle and secondly the inadequacy of the simple circu-lar arc representation at large distances. Consequently,a maximum 'look ahead' distance related to the confi-dence measure is used.

Failure Recovery

A failure is defined as a low confidence measure from thefitting procedure or a loss of edge points. If the failureis only on one side then the previous state of the modelis used to predict the positions in the image of two edgepoints. If they are found using the edge-finder operatorthen attempt to track along the extent of the edge usingthe (X,Y,Z) extrapolation process. If they are not foundthen revert to initialisation procedure detailed above.

EXPERIMENTAL RESULTS

The algorithm described was coded in OCCAM to runon the Transputer hardware that has been designed andbuilt at Bristol University 9. The system comprises of avideo interface board that provides digitised video datafrom camera or VCR to a number of framestores. Theseare accessible by a number of 20 MHz T414 32-bit Trans-puters via a VME-like bus. An IBM compatible PC withplug-in B004 is used for program development. A num-ber of sequences of road images have been collected froma video camera mounted on the roof of a car travellingat an average speed of 20 mph around public roads inthe Bristol area.

Firstly, the initialisation procedure is manually providedwith the initial (x,y) coordinates of two points on the leftand the right road edges. These are sufficient for the(X,Y,Z) extrapolation process to track along the extentof the road edges up to a fail condition or a maximumdistance of 50 metres. The curve fitting process is thenperformed to filter the edge points and to initialise themodel. If this completes successfully on this frame thenthe normal frame-to-frame mode begins.

Initial results show the successful tracking of the sameedge points in the real world until they disappear fromview due to the vehicle moving forward and also the(X,Y,Z) extrapolation process searching for and findingnew edge points. The curve fitting process at each stageis monitored by drawing the curve corresponding to themodel parameters in the current frame. Figure 4. showsthe set of edge points and the fitted curve at a snap-shot in the sequence. The algorithm begins to fail whenthe edge features in the image become indistinct or frag-mented and when the road becomes more complex thanthe representation, for example at a short straight sec-tion followed by a sharp curve.

The overall algorithm cycle time is dependant upon thenumbers of edge points in the sets, but for a typical valueof 15 each side the cycle time is 90 millisec. which cor-responds to an update rate of 11 frames/second. Theprocessing of image data by the edge-finder operator isvery fast at nearly 2 millisec./ edge point. The trans-formations and curve fitting take a significant amountof processing time (« 30 millisec.) due to the necessityof working with real numbers. This should improve dra-

183

Figure 4- Tracking of edge points and the result of curve fitting.

matically when using T800 Transputers. In addition, wehave yet to distribute the algorithm amongst a numberof processors.

CONCLUSIONS

Early results show that the use of the circular arc modelrepresentation of a road to guide selective image process-ing is sufficient in the case of simple roads and that thealgorithms are fast and stable when the road edges arewell defined. Situations arise whereby the model fails,due to inadequacy of the representation or failure of theedge-finder operator in cases where the road edges areill-defined. On the occasions when edge information isnot sufficient or not available, other forms of image pro-cessing from work described elsewhere should be used,for example region segmentation.

In order to improve the reliability and range of operationof the edge-tracking approach, planned developments in-clude improvements to the image processing operators,an increase in the complexity of the representation, toidentify further heuristics, to implement on a transputernetwork (using IMS T800's), and test the algorithms ina closed loop using a small tracked vehicle.

Acknowledgements

This work is supported by MOD RARDE(Chertsey) un-der contract ERl/9/4/2034/085 to develop Transputerbased hardware and algorithms for robot road vehicleguidance.

REFERENCES

1. Wallace R., Matsuzaki K., Goto Y., CrismanJ., Webb J. & Kanade T. "First Results inRobot Road Following." Proc. IJCAI, Los Angeles,USA, Aug., (1985).

2. Wallace R., Stentz A., Thorpe C , MoravecH., Whittaker W. & Kanade T. "Progress inRobot Road Following." Proc. 1986 IEEE INT.Conf. Robotics & Automation. (1986).

3. Thorpe C. & Kanade T. "1986 Year End Reportfor Robot Road Following at CMU.",CMU-RI-TR-87-11, (1986).

4. LeMoigne J., Waxman A., Srinivasan B. &Pietikainen M. "Image Processing for Visual Nav-igation of Roadways.", Computer Vision Lab., Univ.Maryland, CAR-TR-1S8, July (1985).

5. Waxman A., LeMoigne J., Davis L.S., LiangE. & Siddalingaiah T. "A Visual Navigation Sys-tem." Proc. IEEE INT. Conf. Robotics & Automa-tion., (1986).

6. Waxman A., LeMoigne J., Davis L.S., Srini-vasan B., Kushner T.R., Liang E. & Sid-dalingaiah T. "A Visual Navigation System forAutonomous Land Vehicles." IEEE Journal ofRobotics & Automation, vol. RA-3, No.2, April(1987).

7. Dickmanns E.D. & Zapp A. "Guiding LandVehicles Along Roadways by Computer Vision."Proc. AFCET, Congres Automatique, Toulouse,Oct., (1985).

8. Kuhnert K. "A Vision System for Real-TimeRoad and Object Recognition for Vehicle Guid-ance." In: Advances in Intelligent Robotic Sys-tems, Proc. SPIE, vol.727, Cambridge, Mass., Oct.,(1986).

9. Milford D. & Lotufo R. in preparation, BristolUniversity Tecnical Report, (1988).

10. Shinar D., McDowell E.D. & Rockwell T.H."Eye movements in Curve Negotiation." HumanFactors, 19(1), pp63-71. (1977).

11. Ozawa S., Rosenfeld A. "Synthesis of a RoadImage as seen from a Vehicle.", Pattern Recognition,19(2), ppl23-145, (1986).

12. DeMenthon D. "Inverse Perspective of a Roadfrom a Single Image." Computer Vision Lab., Univ.Maryland, CAR-TR-210, July (1986).

184