1

Sensordatafusion

Egils SviestinsSaabTech Systems

Fusion levels (JDL model)

Level 1ObjectsLevel 1Objects

Level 2SituationsLevel 2

SituationsLevel 3

IntentionsLevel 3

Intentions

Level 4ProcessLevel 4Process

Sources

3

Terminologi

Sensordata-fusion

Informations-fusion

Sensor-data

Andradata

Objekt Situationer Avsikter

StyrningOptimering

StyrningOptimering

4

Modeller

• Mätningar/information räcker inte

• Modeller krävs!• Matematiska:

– exempel

• Idéer om verkligheten/”mentala” modeller– Begränsat av naturlagar, ekonomiska lagar, mänsklig förmåga etc.

• Mätningar/information snävar in möjligheterna

an)bruspåverk med el(kastparab )(2

2

tdtd

wgx

+=

12

3

5

Från verkligheten...

Rån = stöld e.d. som utförs under hot om våld

Context

Data processing: Improvement or Destruction?

Raw information

Meaningful information

Sensor User

8

Synkanalen (hypotetiskt!)

TolkningLinjerYtor

Pixels Linjer, ytor Fysiska kroppar

Extraktionav kroppar

Erfarenhet

Begrepp

9

Hörselkanalen (hypotetiskt!)

Tolkning

Erfarenhet

TonerTransienter

FrekvensAmplitud

TonerTransienter Ljudkällor Ord mm.

410Hz, 63 dBk Mänsklig

röst Kaffeaaaf

Sortering

10WSC

Early fusion...

... or late?

11WSC

Seeing (hypothetical)

PixelsLinesSurfaces

Physicalbodies

Knownobjects

Pixels LinesSurfaces

12

Artskilda sensorer

Kaffedags

Kaffekask

Kaffetax

??

13

Tidig fusion - för och emot

• Mindre risk för tvetydigheter• Osäkerheter kan lättare beskrivas statistiskt - Bayes teori

kan användas

• Mindre robust m a p systematiska fel• Svårt hantera artskilda källor

14

Inte så enkelt...

15

Fusionsprincip i hjärnan?

16WSC

The Radar Data Processing Chain

ExtractorReceiver Tracker

Raw video Plots (R,az) Tracks (#,x,y,vx,vy,...)

A12

A07

Steps in Tracking

18WSC

The Tracking Cycle

Pred iction

Ass

ocia

tion Updating

Initiation

TerminationMeasurements

Filtering techniques

• Linear regression (least squares batch processing) (hardly used in this context)

• (70’s) Alpha-Beta• (80’s) Adaptive Kalman• (90’s) Interactive Multiple Model (IMM) • (2000’s ?) Non-linear filtering?

Linear regression

t

x

How to handlemaneuveringtargets???

Alpha-Beta filtering~, &~ ,

$ , &$ ,

x x

x xx

predicted position speed

updated position speedmeasured positionm

~ $ &$

&~ &$x x x

x x

= +

=

T

( )$ ~ ~

&$ &~~

x x x x

x xx x

= + −

= +−

α

β

m

m

T

Prediction step

Updating step$, &$x x

~, &~x x

new ifand

$ , &$

. .x x

α β= =0 5 0 5

xm

α and β are tuning constantsbetween 0 and 1

α=β=0: Measurement has no effectα=β=1: History has no effect

Kalman filtering

Like a-b-filter, but:Automatically optimizes a and bBest weighting between history

and measurementOutput includes estimated accuracy

Current state & uncertainties+

Measurement & uncertainties=

New state & uncertainties

Probability densities

x

x.

Prediction

Measurement

Update

IMM States

&&&& , &&x l

x s x tl s l t s t

⋅ =

⋅ ⋅ =⋅ = ⋅ = ⋅ =

white noise

white noise0

&& ; &x x uu

= ⋅ =0 0= vertical unit vector

Dynamics

&& &&

&& ( &

x u x llx u x

⋅ ⋅ ==⋅ ×

and white noise longitudinal unit vector

) = 0

&ωω

==

0 turn rate

( , , , & , & , & )x x x x x x1 2 3 1 2 3

Linear Kalman filter

( , , , & , & , & , )x x x x x xnd

1 2 3 1 2 3

2

ω

order Extended Kalman

( , , , & , & , & )x x x x x x1 2 3 1 2 3

Linear Kalman filter

State Vector and Filter Type

( , , , & , & , & )x x x x x x1 2 3 1 2 3

Linear Kalman filterUniformHorizontal Motion

Speed Changes

Slow Turns

Fast Maneuvers

IMM structure

Input

Transition

Merging

Propagation

Updating

Averaging& Output

UH

UH UH UH UH

UH

UH

UH

X

SC ST FM

SC SC SC SC

SC

SC

SC

ST ST ST ST

ST

ST

ST

FM FM FM FM

FM

FM

FM

26

Bayes teori

p(H )

p(H )

p(H )

1

2

3

p'(H )

p'(H )

p'(H )

1

2

3

Observation z

∝ i

i

p'(H )

p(z|H )p(H )i

27

Associering

• M målspår, N plottar: hur koppla samman?– OBS! Falska/saknade plottar, falska/saknade målspår

• Närmaste granne?• Närmaste granne i statistiskt avstånd?• Global optimering statistiskt avstånd

(minimera )?• Söka globalt mest sannolika koppling?

Hur man än gör kan det bli fel. Motiverar multihypotes

∑ 2d

• Clusters with M measurements and N tracks• Form hypotheses like

• Calculate probabilities for each hypothesis, e.g.

Measurement-to-track association

( ) ( )P p z x p P p z x Pd s d d1 3 3 2 1( )−

H z x z z x x( , , , )1 3 2 3 2 1→ → ∅ → ∅ →

LPQ association: Plot & Track clusters

∗

∗

∗∗

∗∗ ∗

∗

∗

∗

∗

Track predictedposition andsearch bin

Plot

Cluster with3 plots and 2 tracks

OH103

Bayesian track initiation

Given a tentative track. Two hypotheses:H0: Track is falseH1: Track is genuineCn=p(H1): Credibility at scan n

Obtained measurement z. Spurious plot density ps.

( ) ( )( ) ( )[ ]

p H z C

p z H p H

P p z x P p C

n

d d s n

( )1 1

1 1

1

=

∝

= + −

+ ( ) ( )( )

p H z p z H p H

p Cs n

( )0 0 0

1

∝

= −

Initiation by Credibility

uRequired: Fast initiation and low false track rateu Sequential hypothesis testinguCredibility C ≈ likelihood that a potential track is

genuine

cred

C

1 2 3 4 5 6 7 8 Scan #

0

1

32

Andra sensorer

• Bildalstrande– TV– FLIR (Forward Looking Infrared)– Millimetervågsradar– SAR (Synthetic Aperture Radar)

• Icke bildalstrande– Störbäringsavtagare– Signalspaning– IRST (Infrared Search & Track)– Akustiska/Hydroakustiska sensorer– GPS

Decentralized Multi-Radar Tracking

Tracking

TrackingPlots

Plots

Trackcorrelation& merging

System tracks

Centralized Multi-Radar Tracking

System tracks

Multi-radartracking

Plots

Plots

Filling coverage gaps1. 2.

3. 4.

Two radarsCoverage gap Red single

radar tracklost andreinitiated

DecentralizedMRT may giveconfusing picture

Centralized MRTperforms well

Disadvantages of centralized multi-radar tracking

• More sensitive to bias errors– Bias compensation required

• Difficult to distribute CPU load on several processors– But not impossible

• Existing data links often do not supply plot level data– Sometimes requires hybrid solutions

• Sensors sometimes include extensive processing– Sometimes requires hybrid solutions

Strobes only

150 km

Crossings

Reasons for Multi-Sensor Tracking

• Radars can be jammed• Protective need to keep radars silent• Radars don’t always give best target detection• May support target identification

Target Type Identification

• Based on– Direct observations– ESM / IRST measurements– Kinematics

• Each track carries a vector with probabilities of possible target types.

• Requires a library of target type characteristics

MST+ scenario

42

Example

T1 T2 T 3 T4 T 5

M1 M2 M3 M4 M5 M6 M7

T 1

T 2

T 3

T 4

T 5

3 3 3 1 1 3 3 3 3 13 3 3 2 2 3 3 3 3 23 3 3 4 5 3 3 3 4 53 3 3 4 5 6 3 3 3 43 3 3 4 5 6 7 6 6 7

Lockheed F16Lockheed F16 Mirage 2000Mirage 2000 Lockheed U2Lockheed U2 MiGMiG--2525 MiGMiG--2929

Kinematic typingOffline: Create Target Type Database

• Max altitude• Min/Max speed as function of altitude• Max climb rate as function of altitude• Max distance from base• Max linear/turn acceleration as function of altitude

Step 1 - Collect flight data

• Max altitude• Min/max velocity as function of altitude• Max climb rate• Max distance from base• <Max linear/turn acceleration as function of altitude>• Utilise meteorological data if available

NewProbability

Vector[p´(F16),...]

Step 2 - Update Probability Vector

CollectedFlight Data

Target TypeDatabase

PreviousProbability

Vector[p(F16),...]

Bayes’ Rule

46

Avrundning• Sensordatafusion - uppgifter om enskilda objekt baserat (mest) på

sensordata• Bygger oftast på matematiska modeller och

Bayesiansk hypotesprövning• Många svåra områden återstår

– Sensorer som ger knepiga data– Svårtolkade scenarier (t ex mark och undervatten)– Gemensam lägesbild (distribuerad fusion)– Fusion av starkt artskilda sensorer– Integration med infofusion