Slide 1Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Maritime Situational AwarenessBig data,

machine learning, and human-machine teaming

Dr Catherine Warner

Director, Centre for Maritime Research and Experimentation

Slide 2Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Maritime Situational Awareness Foundations

DATAMulti-sensorMulti-modal

RoboticsM&S

PROCESSINGSignal ProcessingMachine Learning

Deep Learning

H-M TEAMINGMulti-INT Information

FusionKnowledge elicitation

Information Visualization

© allaboutais.com

© ESA 2020

Slide 3Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Big data 5V

Volume

Velocity

Variety

VeracityValue

Satellite dataGlobal surveillance systems

Scalable processing infrastructures

Machine LearningDeep Learning

Surveillance streams

Information fusion Data interoperability

M&S

Multi-INT IFUncertainty modelling

Data quality modelling

Multi-sensor dataNoisy data What data?

Maritime Cybersecurity

RoboticsSpace

© IEEE 2020 (Balduzzi, 2013)

Slide 4Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

AIS databases: Terrestrial and Satellite

Big data sets at CMRE

Sonar data: Sea mines (imagery)

Submarines (echoes)

Satellite imagery:MultispectralSentinel-1-2-3 data

© ESA 2020

Oceanographic:Ambient noise

METOC

Rapid Environmental Assessment (REA)

Slide 5Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

PROCESSING:

DATA FUSION,

SIGNAL PROCESSING,

MACHINE LEARNING,

DEEP LEARNING

Automatic Target Recognition

Long-term ship prediction and Anomaly Detection

Automatic port recognition

Maritime Patterns of Life

Slide 6Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Maritime traffic graph

[data are property of CAN DRDC]

Long-term ship prediction (6/12/24 hours)

Mean-Revertingmodels (Ornstein-Uhlenbeck) with

historical dataenable

accurate prediction of future ship positions

> 10X accuracy w.r.t. traditional models

Slide 7Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Anomaly detection using Mean-Reverting model

Ship motion prediction applied to anomaly detection

Detection of anomalous vessel behavior typical of smuggling, waste dumping

[Data from Journal of Fisheries]

5 days of AIS tracks. In red, a cargo vessel with sharp changes in direction, suspected to have had close

encounters with other ships.

Slide 8Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Estimation of sea port operational area from AIS data

AIS report on Rotterdam sea port(Jan-Mar 2015)

Data driven port-area estimate from the data on the left

Kernel Density Estimator from AIS

Accurate estimates of constantly evolving port areas to support data-driven decision making

Slide 9Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Evolution of port visits and strait transits, clustered by ship type, automatically computed from AIS data on a

user-defined port shape

All traffic

Cargo traffic

Tanker traffic

Passenger traffic

Fishing traffic

Other traffic (not belonging to the other categories)

International Maritime Exercise - IMX19

Slide 10Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Copernicus Sentinel-2 multi-spectral images of 4 vessel classes, processed by CMRE. Left: spectral signature. Right: RGB composition

VLCC

RORO-VehicleCarrier

COSCO Fleet

LNG-Spherical Moss

Automatic ship classification with satellite data

Training data set

Performance of the trained Convolutional Neural Network

Slide 11Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

HUMAN-MACHINE

TEAMING:

GAMING,

MULTI-INT IF,

INFORMATION

VISUALIZATION

Gaming and knowledge elicitation Information Visualization

Multi-INT fusion and uncertainty modelling

?

Maritime Visualization

Genoa

Slide 12Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

CMRE uses gaming techniques to investigate the impact of information quality on belief state

Analytical games for knowledge acquisition

Reliability game

RISK GAME: Impact of uncertainty, imprecision, conflict

RELIABILITY GAME: Impact of source factors (quality and type)

15 Nations participating; validated as an engaging and efficient methodology

20

17

MSA

Gam

es

Evaluation of behavioral analysis

Risk game

MARISA trial

Slide 13Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Multi-source IF framework for reasoning under uncertainty (valuation networks)

Detects and help solving information conflict

Uses Maritime Situational Indicators

?

Multi-INT fusion for automated reasoning under uncertainty

Vessel destination estimation

Slide 14Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Interactive statistics about maritime patterns-of-life

In use at NATO MARCOM

Maritime Patterns-of-Life Information Service - MPoLISPort visits over time Port visits by ship type

Port visits by flag

Density map:All port visits by all vessels with AIS

Slide 15Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Maritime Traffic 2019 vs 2020 (during Corona virus pandemic)

March 2019 March 2020

All

Mar

itim

e T

raff

ic

Italian Riviera

Denmark

Copenhagen

Genoa

Slide 16Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

• Maritime Situational Awareness is built on a foundation of:– valuable data

– accurate computation

– human-centric design and appropriate communication

• Future work – Hybrid sensor networks

– Robust predictive and classification methods

– Scalable computational approaches

– Interoperable solutions integrating diverse data and software

– Improved human-machine teaming, for the development of explainable AI solutions

Summary and Future Directions

Slide 17Virtual Maritime Security Regimes Roundtable 2020 – 30 April - Dr Catherine Warner, Director, CMRE

Questions? catherine.warner@cmre.nato.int

This research is sponsored primarily by:

NATO Allied Command Transformationand including:

EC H2020 projects: datAcron, RANGER, MARISA, INFOREUS ONRCAN DRDC