47
Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Embed Size (px)

Citation preview

Page 1: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electrosensory data acquisition and signal processing strategies

in electric fish

Mark E. Nelson

Beckman InstituteUniv. of Illinois, Urbana-Champaign

Page 2: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

How Electric Fish Work

Page 3: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Distribution of Electric Fish

Fish tank upstairs

blackghost

knifefish

elephant-nosefish

Page 4: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electric Organ Discharge (EOD) - Spatial

Page 5: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

EOD - Temporal

Page 6: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electric Organ Discharge (EOD)

Page 7: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Principle of active electrolocation

Page 8: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

mech

an

o

MacIver, fromCarr et al., 1982

Electroreceptors

~15,000 tuberous electroreceptor organs1 nerve fiber per electroreceptor organ

up to 1000 spikes/s per nerve fiber

Page 9: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Individual Sensors (Electroreceptors)

VIN

nerve spikesOUT

Page 10: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Neural coding inelectrosensory afferent fibers

Page 11: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Probability coding(P-type) afferent spike trains

00010101100101010011001010000101001010

Phead = 0.333

Phead = 0.337 Phead =

0.333

Page 12: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Principle of active electrolocation

Page 13: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electrosensory Image Formation

Page 14: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electrosensory Image Formation

Page 15: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electrosensory Image Formation

Page 16: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Prey-capture video analysis

Page 17: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Prey capture behavior

Page 18: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Fish Body Model

Page 19: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Motion capture softwareMotion capturesoftware

Page 20: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

MOVIE: prey capture behavior

Page 21: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Electrosensory Image Reconstruction

Page 22: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Voltage perturbation at skin :

Estimating Daphnia signal strength

waterprey

waterpreyfish ar

rE

/21

/133

electrical contrastprey volume

fish E-field at prey

distance from prey to receptor

THIS FORMULA CAN BE USED TO COMPUTE THE SIGNAL AT EVERY POINT ON THE BODY

SURFACE

Page 23: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

MOVIE: Electrosensory Images

Page 24: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

System Capabilities

Electric fish can analyze electrosensory images to extract information on target

direction (bearing) distance size shape composition (impedance)

Page 25: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Distance Discrimination

Page 26: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Distance Discrimination

Page 27: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Shape Discrimination

Page 28: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Shape Discrimination

Page 29: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Shape Generalization

Page 30: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Shape “completion”

Page 31: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Impedance Discrimination

Page 32: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

How Do They Do It? Electric fish analyze dynamic 2D electrosensory images on the body surface to determine target direction, distance, size, shape and

composition (impedance)

Fish might perform an inverse mapping from 2D sensor data to obtain a dense 3D neural representation of world conductivity sensor data 3D conductivity action

Alternatively, fish might use sensor data to directly estimate target parameters sensor data target parameters action

Page 33: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Parameter estimation

(bearing)

Page 34: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Parameter Estimation (cont.)

Page 35: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Dynamic Movement Strategies

Fish are constantly in motion not a single, static ‘snapshot’ dynamic, spatiotemporal data stream

With respect to target objects in the environment, fish body movements simultaneously influence the relative positioning of the sensor array the electric organ effector organs (e.g. mouth)

Page 36: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

MOVIE: Electrosensory Images

Page 37: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Active motor strategies: Dorsal roll toward prey

Page 38: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Probing Motor Acts

chin probing back-and-forth (va et vient )

lateral probing

tangentialprobing

stationaryprobing

Page 39: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Fish exploring a 4 cm cube

Page 40: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

CNS Signal Processing Strategies

Multi-scale filtering spatial and temporal

Adaptive background subtraction tail-bend suppression

Attentional ‘spotlight’ mechanisms local gain control

Page 41: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Multiple Maps

Page 42: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Multi-scale Filtering

INPUT

(from skin receptors)

Centromedial map High spatial acuity Low temporal acuity

Centrolateral map Inter spatial acuityInter temporal acuity

Lateral map Low spatial acuityHigh temporal acuity

tem

pora

l

inte

grat

ion

bothspatial

integration

HINDBRAIN PROCESSING

PERIPHERALSENSORS

Page 43: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Adaptive Background Subtraction

Page 44: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Adaptive Background Subtraction

Page 45: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Attentional ‘spotlight’ mechanism

Page 46: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Summary Fish can evaluate direction, distance, size, shape and composition of target objects

How? model-based parameter estimation based on 2D image

analysis, not full 3D reconstruction presumably some sort of (adaptive) (extended)

(unscented) Kalman-like algorithm extensive pre-filtering (virtual sensors?)

self-calibrating, adaptive noise suppression, multi-scale spatial and temporal signal averaging

dynamic control of source and array position

Page 47: Electrosensory data acquisition and signal processing strategies in electric fish Mark E. Nelson Beckman Institute Univ. of Illinois, Urbana-Champaign

Acknowledgements

Colleagues Curtis Bell (OHSU) Len Maler (Univ. Ottawa) Gerhard von der Emde (Univ. Bonn)

Nelson Lab Members Ling Chen, Rüdiger Krahe, Malcolm MacIver

Funding Agencies NIMH, NSF