Insect neural networks as a visual collision detection mechanism in automotive situations

Richard Stafford (1), Matthias S. Keil (2), Shigang Yue (1), Jorge Cuadri-Carvajo (2), F. Claire Rind (1)1) School of Biology, Ridley Building, University of Newcastle upon Tyne, NE1 7RU, UK. 2) Instituto de Microelectronica de Sevilla (IMSE), Centro Nacional de Microelectronica (CNM),Avda. Reina Mercedes, 41012, Sevilla, Spain

Structure of the talk Introduction Improvements to the LGMD model

Light on and light off pathways Testing the LGMD model

Methods and Results Further improvements

Biologically inspired filtering of images by lateral inhibition

Analysing the filtered images using EMD like structures

Conclusions

Locusts as collision detectors The Lobula Giant Movement Detector

(LGMD) neuron responds most vigorously when objects of certain speeds and sizes approach, as if on a direct collision course

This has been linked to a predator avoidance, gliding behaviour in flying locusts

Predator avoidance caused by the LGMD

Angular subtense of object

LGMD Spikes

Inputs and structure of the LGMD

Why use the Locust LGMD to detect automotive collisions? Evolutionary honed collision avoidance

system Efficient circuit – based on insect

neurons Neural architecture well studied Responds optimally to imminent

collisions Simulated networks respond in a similar

manner to real locust

Limitations of existing model(e.g. Rind and Bramwell, 1996; Blanchard et al., 2000)

Simulations only tested in simple closed environment

Model needs to work in real automotive situations

Biology of the LGMD is not fully used – model only responds to change in light

Structure of the talk Introduction Improvements to the LGMD model

Light on and light off pathways Testing the LGMD model

Methods and Results Further improvements

Biologically inspired filtering of images by lateral inhibition

Analysing the filtered images using EMD like structures

Conclusions

Model Improvements – Light on and Light off Pathways

Small scale spatial antagonism between the pathways helps eliminate noise caused by vibration etc.

Larger scale antagonism can interfere with collision alerts

Model Improvements – Light on and Light off Pathways and Block Sum Cells

Input Image ‘S’ units – light on ~ light

off

Block Sum CellsAllow small scale antagonism ofpathways only

Location of BSC in model

Block sum cellsoccur here

Model Improvements - Block Sum Cells

Sum light on (-ve) and light off (+ve) excitationto obtain net excitation

Excitation (+ve only) is passed

to the LGMD

from theBSC

Block sum cells obtainexcitation from a 10x10section of the array of ‘S’ units

Light on andLight off excitationfrom ‘S’ units Block Sum Cells

LGMD

Structure of the talk Introduction Improvements to the LGMD model

Light on and light off pathways Testing the LGMD model

Methods and Results Further improvements

Biologically inspired filtering of images by lateral inhibition

Analysing the filtered images using EMD like structures

Conclusions

Testing the model in automotive situations

Input video sequences8 – 25 Hz

Input via frame-grabber of Playstation images8.3 Hz

Detecting collisions Membrane

potential of LGMD is obtained from sum of BSC

If a threshold is exceeded then the LGMD produces spikes

If > 2 spikes in 3 timesteps then collision detected

Results: LGMD model

Results show % of times collision was detected even if no collision occurred

Stationary car100 %

Moving Car90 %

Head on withmoving Car100 %

Entering Tunnel0 %

General Driving0 %

Driving in closeproximity0 %

Translating cars70 %

Why do translating cars proveproblematic?

• Excitation is much higher in the LGMD for translating objects• Locust LGMD ignores translating objects partially due to differences in mathematics of object approach

Structure of the talk Introduction Improvements to the LGMD model

Light on and light off pathways Testing the LGMD model

Methods and Results Further improvements

Biologically inspired filtering of images by lateral inhibition

Analysing the filtered images using EMD like structures

Conclusions

Image Filtering by LGMD network‘S’ units only excited by objects moving in close proximity to care.g.Colliding or translating objects

No threat Threat

Input Image

‘S’ units

Analysing the biologically filtered images Analysing patterns of excitation in ‘S’ or

‘BSC’ layers over time shows: No or little excitation – no threat. LGMD does not

reach threshold Excitation moving in one direction over time – no

threat, translating object. LGMD spikes can be suppressed

Excitation moving in all directions over time – collision threat, object on collision course is expanding in all directions. LGMD spikes and produces collision mitigation response

Structure of the talk Introduction Improvements to the LGMD model

Light on and light off pathways Testing the LGMD model

Methods and Results Further improvements

Biologically inspired filtering of images by lateral inhibition

Analysing the filtered images using EMD like structures

Conclusions

Incorporation of simple Elementary Movement Detector like units (EMDs) into the model

EMD like units take input from the Block Sum Cells – simplified visual environment

One detected ‘Right’ movement and one ‘Left’ movement

If membrane potential of ‘left’ EMDs was > 5 x potential of ‘right’ EMDs at time t or time t-1 then LGMD spikes were suppressed for time t, t+1 & t+2

Location of EMD like units

BSC

EMDsSuppressionof LGMD spikes

Results: LGMD incorporating EMDs

Results show % of times collision was detected even if no collision occurred

Stationary car85 %Was 100 %

Moving Car80 %Was 100 %

Head on withmoving Car50 %Was 100 %

Entering Tunnel0 %UnchangedGeneral Driving0 %Unchanged

Driving in closeproximity0 %UnchangedTranslating cars20 %Was 70 %

Results: LGMD and EMDs

Incorporation of EMDs reduce false collision alerts

Real collision detection was also reduced EMD model was very simple. Using a more

advanced (adaptive) model may improve the responses

Non bio-inspired image analysis could also be used on the biologically filtered ‘S’ units to improve model performance

Conclusions Locust based LGMD model can be used for

automotive collision detection In some situations modifications are

needed as the LGMD’s function in automotive situations is quite different to the evolved function in the locust

The biologically filtered image can be analysed to further assess the threat of collision

Acknowledgements Project funded by Future and

Emerging Technologies Grant from European Union (LOCUST – IST - 2002-38097)

We would like to thank Marrti Soininen of Volvo Car Corporation for supplying the video footage of car crashes

Other Improvements to the LGMD model On-Off cells look at absolute change in

image Lateral inhibition has a greater potential

spread to eliminate more non threatening situations

Spiking threshold of LGMD is self variable to allow a greater range of visually complex scenes to be investigated

Model parameters tuned using a Genetic Algorithm to automotive situations

Differences between automotive collisions and predator avoidance in locusts Locusts respond to

small, fast moving predators

Final excitation, just before predator strikes, is much higher

This can be used to distinguish between different object types

Small translating objects produce less excitation than larger objects