Applied Mathematics in Defense ApplicationsAndrea Bertozzi

Department of MathematicsUniversity of California, Los Angeles

Topics• Multisensor fusion• Image and human event fusion for statistical density estimation• Automated boundary tracking for autonomous robots and high dimensional

imagery• Collaborative searching through swarming• Diffuse interface methods in imaging• Segmentation with corners• Imaging through turbulence• Direct sparse deblurring• Geographic profiling• Crime hotspots• Gang violence data• Predicting crime

Data Fusion – Multiple sensors

• Pan Sharpening – panchromatic (greyscale) higher spatial resolution, multiband – lower spatial resolution – IKONOS and QUICKBIRD satellite

• Hyperspectral sharpening – panchromatic obtained separately (may not be perfect match) – hyperspectral can have hundreds of bands – contain material information

• Human Event data – events in space and time fused with geographical data (e.g. residential burgalaries)

• Point sensor data – mobile sensor data

Data Fusion and Segmentation– Multiple sensors

• Pan Sharpening – panchromatic (greyscale) higher spatial resolution, multiband – lower spatial resolution – IKONOS and QUICKBIRD satellite

• Hyperspectral sharpening – panchromatic obtained separately (may not be perfect match) – hyperspectral can have hundreds of bands – contain material information

• Human Event data – events in space and time fused with geographical data (e.g. residential burgalaries)

• Point sensor data – mobile sensor data

Panchromatic signal is not a linear combination of isolated bands

Recent pansharpening techniques

• IHS• Brovey• PCA • Wavelet Fusion • First variational approach: ’A Variational

Model for P+XS Image Fusion’, Ballester, Casselles, Igual, Verdera, 2006

Intensity Hue Saturation Results

Assumes panchromatic is a linear combination of spectral bands.

Variational Wavelet PansharpeningMichael Moeller, Todd Wittman, ALB, preprint

Wavelet matching – data must be registered to dyadic scaling (pansharpening)

Full VWP variational problem

Alternate VWP – avoids switching from wavelet to physical space

Numerical results

Hyperspectral data fusion

Michael Moeller, Todd Wittman, and Andrea L. Bertozzi, A Variational Approach to Hyperspectral Image Fusion, Proc. SPIE Conference on Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XV. Orlando, Florida. April 2009.

Spatial detail inherited from master image – spectral detail from AVIRIS data

Spectral preservation - examples

Spectral angle is preserved

George Mohler, Andrea Bertozzi, Tom Goldstein, Stan OsherFast TV regularization for 2D Maximum penalized likelihood estimation,

preprint 2009

• Method for estimating non-smooth probability densities• Important for estimating threat level based on event data and

other intel.• TV based regularization allows for best estimation of densities

with spatial discontinuities.• Computationally challenging in multi-D• Challenge solved using Split Bregman L1 minimization

technique.• Tested using V-fold Cross Validation with large 2D data sets.• Tested on data from LAPD for residential burglaries.

Maximum Penalized Likelihood Estimationbasic problem

• Estimate probability density u(x) from point data x1, x2, x3, etc.

• General approach for regularizer R(u).

• For discontinuous densities, choose R = TV

Example from San Fernando ValleyData courtesy of LAPD

• Point process data for residential burglaries• No residences in area in middle

Actual data TV method (new) kernel estimation (old)

TV method is much closer to real problem, does not bleed threat level into region where threat is not active. Can be fused with other types of data, such as spatial visual, infrared, LIDAR etc as long as one has a model to incorporate this into the problem.

Density Estimation for Sparse DataLaura Smith, Matthew Keegan, Todd Wittman, ALB UCLA

• Point data of individual events that come from a background source

• Examples – human event activity – burglaries – what is the probability of event as a function of space?

• Data is sparse – want to fuse with other information e.g. overhead imagery

Improving Density Estimation by Incorporating Spatial Information, preprint 2009.

Overhead imagery vs. human events

San Fernando Valley Burglaries

Orange County Coastline

Experimental Validation of Cooperative Environmental Boundary

Tracking with On-board Sensors

A. Joshi, T. Ashley, Y. Huang, and A. L. Bertozzi, Experimental validation of cooperative environmental boundary tracking with on-board sensors, American Control Conference, St. Louis, MO, June 2009, pp. 2630-2635.

Control Algorithms for Boundary Tracking

• UUV-gas bang-bang type steering controller• Time-corrected algorithm• Robotic path planning – • Hsieh et al Amer. Contrl. Conf. 2005• Jin and ALB, IEEE CDC 2008• Joshi et al Amer. Contrl. Conf. 2009

25

Bang-bang type steering Control Law

• UUV-gas algorithm:

26

Bkz

Bkz

ku

ref

ref

)(,boundary inside if

boundaryon if

)(,boundary inside if

0

)(

boundary indicating )(for threshold

angle turning

step-at time control steering theinput to )(

control steering

,

kzB

kkz

u

where

ref

Time-corrected Steering Control Law

• Time-corrected algorithm:

• Includes time difference between crossing points on boundary,

• Reduces to the bang-bang type controller when,

27

Bkz

Bkz

t

t

tu

)( when

boundary, inside if

)( when

boundary, outside if

2/)2(

2/)2(

)(

ref

ref

2

0t

Decision Algorithm CUSUM Filter

• Upper : indicates teal tape

• Lower : indicates black tape

29

0

0

))1()(,0max(

0)(

k

k

kUcBkzkU

u

0

0

)1()(,0min(

0)(

k

k

kLcBkzkL

l

noisy be toassumed is )(

onaccumulati of rate thegcontrollin parameters are and

kz

cc lu

Single Vehicle Implementation• A Kalman inspired pre-filter was used to weakly damp

the noisy signal.

• Essentially a simple proportional model with the empirical gain factor from Kalman filtering a previous data set.

30

049.0

data filtered-pre )(ˆ

,

)(ˆ)1(ˆ

)(ˆ)(

Gain

kz

where

InnGainkzkz

kzkzInn The Gain value is in fact the

convergent Kalman gain (this is expected in view of the fairly constant, though high, noise co-variance of the testbed)

Single Vehicle Implementation

• The sharp features when on the black tape cause the decision algorithm to slip-up occasionally

• Pre-filtering reduces these errors considerably

31

Single Vehicle Implementation

Vehicle speed 0.3m/s Left raw data -> CUSUM Right raw data -> prefilter -> CUSUM

32

Cooperative boundary tracking

• The global control law held admirably

33

The time difference between data points is 24 s

A. Chen, T. Wittman, A. Tartakovsky, and A. L. Bertozzi Image segmentation through efficient boundary sampling, in SAMPTA '09, Marseille, May 18-22, 2009.

W. Liu, M.B. Short, Y. E. Taima, and A. L. Bertozzi,

Multiscale Collaborative Searching Through Swarming, preprint.

35

• Sensor data processing– Kalman filter to reduce noise– CUSUM filter to check threshold

• Movement control of agents– Models for search phase, target locating phase– Target location estimated

• Performance– Average time to locate a target– Average error in estimate of location– Number of false registers

• Scaling properties– Estimate for swarm size, measured by diameter– Optimal swarm diameter (analytical approximation, upper bound)

Collaborative Searching through SwarmingW. Liu, M.B. Short, Y. E. Taima, and A. L. Bertozzi

Introduction 2

36

• Overview of the algorithm

Collaborative Searching through SwarmingW. Liu, M.B. Short, Y. E. Taima, and A. L. Bertozzi

Movement Control of Agents

• Search phase (with Levy flight)

37

• Location estimation

• Target locating phase

Collaborative Searching through SwarmingW. Liu, M.B. Short, Y. E. Taima, and A. L. Bertozzi

Sensor data processing

• Agent sensor reading

38

• Kalman filter to reduce noise

• CUSUM filter to detect threshold

Collaborative Searching through SwarmingW. Liu, M.B. Short, Y. E. Taima, and A. L. Bertozzi

Performance

• Simulations of a 20 by 20 dimensionless board, 32 agents, target sensing radius 1.0, 200 trials

• Divide-and-conquer and whole region search strategies

• Average time to locate a target and average location estimate error measured

• Larger swarms are more accurate, multiple smaller ones more efficient

39

Collaborative Searching through SwarmingW. Liu, M.B. Short, Y. E. Taima, and A. L. Bertozzi

Scaling Properties

• Swarm diameter D scales with inter-agent distance

• For 25% of agents to sense before deciding to locate,

• Optimal D maximizes separation between the center of the swarm and the target location

• The average time to locate a target is

40

Diffuse interface methods

Ginzburg-Landau functionalTotal variation

Cahn-Hilliard Inpainting

Bertozzi, Esedoglu, Gillette, IEEE Trans. Image Proc. 2007, SIAM MMS 2007Patent pending. Transitioned to NGA for road inpainting. Transitioned to InQtel for document exploitation.Continue edges in the same direction – higher order method for local inpainting.Fast method using convexity splitting and FFT

H-1 gradient flow for diffuse TVL2 fidelity with known data

Wavelet Allen-Cahn Image Processing

• Dobrosotskaya, Bertozzi, IEEE Trans. Image Proc. 2008, Preprint subm. IFB.• Transitioned to NGA for road inpainting. Transitioned to InQtel for document exploitation.• Nonlocal wavelet basis replaces Fourier basis in classical diffuse interface method.• Analysis theory in Besov spaces. • Gamma convergence to anisotropic TV. H-1 gradient flow for diffuse TV

L2 fidelity with known data

Convex Splitting SchemesSchoenlieb and Bertozzi, submitted

Basic idea:

Art is to choose Ec to give an implicit problem that is easy to solve- e.g. Ec is H1 semi norm – can be solved using FFT- in wavelet case Ec is wavelet Laplace operator

Contraints on Ec and Ee so that splitting is unconditionally stable

Proof of convergence of splitting schemes for various higher order inpainting methods.

Segmentation with Corners

Image Snakes (KWT ‘88)

Droske & Bertozzi – geometric corner snakes2009

Chan-Vese 2001

CV with corners 2009

Idea – segmentation requires a regularizationIt is analogous to denoising. CV, Snakes reduce length of curve.Removes corners as well as noise.Instead regularize with the “curve” analogy of TV – nonlinear penalization of curvature-based functional.Low Curvature Image Simplifiers (Tumblin & Turk SIGGRAPH 2000, Bert. And Greer CPAM 2004)Extend to curve evolution using either Lagrangian framework or Level sets.

Marc Droske and Wenhua Gao

Higher-order feature-preserving geometric regularization, SIAM J. Img. Sci. 2010.

Imaging through turbulence

Images taken at China Lake – courtesy of Alan Vannevel and Gary HewerWhen you image at a kilometer anisplanatic effects are relevant – we need betterdeblurring and deconvolution methods.Often imformation is known about the image – the difficulty is to extract features.

morning afternoon

Direct Sparse DeblurringLou, Bertozzi, Soatto, submitted 2009

Blurry data ROF deblurring Our method

Uses training data

Dictionary based

Inverse problem not solved

Fit data to blurred dictionary then directly unblur

Solves problem of amplifying noise with solution of inverse problem

Geographic ProfilingGeorge Mohler and Martin Short

Estimation of probabilities

2004 San Fernando Valley Data

2007 Los Angeles Burglary Data

M. B. Short, M. R. D'Orsogna, V. B. Pasour, G. E. Tita, P. J. Brantingham, A. L. Bertozzi, and L. Chayes, A statistical model of criminal behavior, M3AS: Mathematical Models and Methods in Applied Sciences, special issue on Traffic, Crowds, and Swarms, volume 18, Supp., pages 1249-1267, 2008.

G. Mohler, M. Short, P. Brantingham, F. Schoenberg, and G. Tita, Self-exciting point process modeling of crime, submitted.

Gang violence DataMike Egesdal, Chris Fathauer, Kym Louie, and Jeremy Neuman, Statistical Modeling of Gang Violence in Los Angeles, to appear in SIURO online.

Gang violence Data

Burglary 2005 Robbery 2007

Courtesy of George Mohler

AcknowledgmentsONR grant N000141010221, Information Fusion of Human Activity, Social Networks, and Geography Using Fast Compressive Sensing, co PIs Stan Osher, Jeff Brantingham, George Tita (UCI).

ONR grant N000140810363 Geometry Based Image Analysis and Understanding, 1/08-12/10.

ARO grant (STIR) W911NS-09-1-0559 Mathematical modeling of insurgent activities as compared to urban street crime, 10/09-6/10.

ARO MURI grant 50363-MA-MUR Spatio-temporal event pattern recognition subcontract from Brown University, Boris Rozovsky, PI.

The Department of Defense.

NSF grant DMS-0914856 Algorithms for Threat Detection (ATD): adaptive sensing and sensor fusion for real time chemical and biological threats, (jointly funded by DTRA).

NSF grant DMS-0601395 Research Training Group in Applied Differential Equations and Scientific Computing, 6/06-5/11 (supports undergraduate research).