Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
CNN-based ship classification method
incorporating SAR geometry information
Shreya Sharma, Kenta Senzaki, Yuzo Senda, Hirofumi Aoki
Data Science Research Laboratories
NEC Corporation
Published in SPIE Remote Sensing 2018https://doi.org/10.1117/12.2325282
2 © NEC Corporation 2015 NEC Group Internal Use Only2 © NEC Corporation 2019
Motivation
An accurate ship classification technology is needed
Ship classification enhances the performance of maritime surveillance
Helps in quick identification of vessels involved in illegal activities
© Nature Magazine
© NOAA/CC BY 2.0
3 © NEC Corporation 2015 NEC Group Internal Use Only3 © NEC Corporation 2019
Ship Monitoring from Space
▌3 major sources of information:
Automatic Identification System (AIS)
Optical imagery
Synthetic Aperture Radar (SAR) imagery
SAR images are now preferable for ship classification
AIS
OpticalSAR
• all weather
• day and night
Requires
equipment on
each ship
4 © NEC Corporation 2015 NEC Group Internal Use Only4 © NEC Corporation 2019
Existing SAR Ship Classification Methods
1. Hand-crafted Features (HCF)-based
2. Convolutional Neural Network (CNN)-based
CNN Class
© JAXA
Bounding
Box
© JAXA
Feature*
extractor
Classifier
(SVM) Class
Pros
• Intuitive features
Cons
• Requires expert
knowledge of ships
Pros
• Does not require
expert knowledge
Cons
• Requires huge
training data
*Length, Area, Intensity, … etc.
These methods classify a ship based on its appearance
5 © NEC Corporation 2015 NEC Group Internal Use Only5 © NEC Corporation 2019
Problem
𝜃2= 40°
𝜃1= 30°Appearance of a ship varies with SAR geometry
Same ship viewed
under different angles © JAXA
© JAXA
Appearance information is not sufficient to achieve robust classification
𝜃1 𝜃2
SAR
Example:
Azimuth
Range
6 © NEC Corporation 2015 NEC Group Internal Use Only6 © NEC Corporation 2019
Relationship between Appearance and SAR geometry
Incident angle (θ) is a key SAR geometry information which
directly affects the appearance of a ship
θ changes the imaging order of the major scattering points
Incident angle indicates how appearance of a ship varies
O
𝜃1
𝜃2
A
B
C A
B
C
A1B1 C1 A2 B2
Radar
Shadow
SAR Image Plane
OB<OA<OC OA<OB
Different
appearance!
Toy Example:
7 © NEC Corporation 2015 NEC Group Internal Use Only7 © NEC Corporation 2019
Proposed Solution
Use incident angle as an additional information in a CNN
Helps the CNN to combine feature information and
geometry information in feature space
CNN can follow SAR geometry changes
CNN Class
© JAXA
MetadataExtract
Incident Angle
© JAXA
Incident angle
information
8 © NEC Corporation 2015 NEC Group Internal Use Only8 © NEC Corporation 2019
Representation of Incident Angle Information
BinningOne-hot
encoding
Angle
Info.Incident Angle
…
Binning and one-hot encoding reduces the real-valued angles to
discrete labels which accelerates CNN training
EX) Bin 1: [25°- 27°)Bin 2: [27°- 29°)
Bin 8: [39°- 40°)
…
EX) Bin 1: 1 0 0 0 0 0 0 0
Bin 2: 0 1 0 0 0 0 0 0
Bin 8: 0 0 0 0 0 0 0 1
…
9 © NEC Corporation 2015 NEC Group Internal Use Only9 © NEC Corporation 2019
Network of Proposed Method
SAR Image
FC
120x1
BinningSAR
Metadata
Extracting
Incident
angle
CONV
18@9x9POOL
18@6x6
CONV
36@5x5 POOL
36@4x4
CONV
120@4x4
CNNCNN
CNNCNN
CNN
N
One-hot
Encoding
* Bentes, C., Velotto, D. and Tings, B., "Ship classification in terrasar-x images with convolutional neural networks,"
IEEE Journal of Oceanic Engineering 43(1), 258-266 (2018)
Feature Information*
Ship Class
3x1
FC
120x1
Flattening
+
3000x1
Merged
3120x1
Combine
Incident Angle Information
120x1
8x1
10 © NEC Corporation 2015 NEC Group Internal Use Only10 © NEC Corporation 2019
Evaluation
Test 1: Classification performance
• Accuracy
• F-measure
Test 2: Dependence on training data size
11 © NEC Corporation 2015 NEC Group Internal Use Only11 © NEC Corporation 2019
Experimental Set-up
Container
Bulk-
carrier
Tanker
Satellite Sentinel-1
Resolution 20m
Polarization HH
Image size 128x128
No. images 200/class
Ground truth AIS +
Marine
Traffic
*Huang, L et al., "OpenSARShip: A dataset dedicated to Sentinel-1 ship interpretation," IEEE Journal of Sel. Top. in App. Earth Obs. and Rem. Sen. 11(1), 195-208 (2018).
© ESA
© ESA
Dataset: OpenSARShip* Specifications
HCF 10 Features + SVM
CNN w/o incident angle
Conventional Methods
© ESA
© ESA
© ESA
© NOAA
12 © NEC Corporation 2015 NEC Group Internal Use Only12 © NEC Corporation 2019
Network Training and Testing
• Five-fold cross-validation
• Training data split into: 80%, 66%, 50%, 25%, 20% of
full dataset to evaluate the effect of training data size
• 10 initial random seeds
Training data: 80% Testing data: 20%
Full Dataset
13 © NEC Corporation 2015 NEC Group Internal Use Only13 © NEC Corporation 2019
Result 1: Classification Accuracy (Overall)
63.00
73.2074.25
60.00
65.00
70.00
75.00
HCF CNN Proposed
Accura
cy [
%]
Proposed method outperforms the conventional methods
Results are averaged over 10 initial seed values
14 © NEC Corporation 2015 NEC Group Internal Use Only14 © NEC Corporation 2019
Result 2: Classification Accuracy (Each class)
0.7
0.57
0.61
0.65
0.72
0.81
0.68
0.75
0.82
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
Container Bulk-carrier Tanker
Accura
cy [
%]
HCF CNN Proposed
Accuracy of bulk-carrier and tanker has improved
15 © NEC Corporation 2015 NEC Group Internal Use Only15 © NEC Corporation 2019
Result 3: F-measure (Overall)
0.63
0.72
0.75
0.60
0.65
0.70
0.75
0.80
HCF CNN Proposed
F-m
easure
Proposed method achieves the best overall f-measure
16 © NEC Corporation 2015 NEC Group Internal Use Only16 © NEC Corporation 2019
Result 4: F-measure (Each class)
0.74
0.56
0.6
0.69
0.72
0.77
0.71
0.74
0.79
0.5
0.55
0.6
0.65
0.7
0.75
0.8
Container Bulk-carrier Tanker
F-m
easure
HCF CNN Proposed
Proposed method outperforms in bulk-carrier and tanker
17 © NEC Corporation 2015 NEC Group Internal Use Only17 © NEC Corporation 2019
Result 5: Effect of Training Data Size
65
67
69
71
73
75
20 25 50 66 80
Accu
racy [
%]
Training Data [%]
CNN Proposed
Consistently
high accuracy
Proposed method requires less training data for high accuracy
18 © NEC Corporation 2015 NEC Group Internal Use Only18 © NEC Corporation 2019
Conclusion
▌A CNN-based ship classification method incorporating SAR
geometry information is proposed
▌The proposed method uses incident angle information to
separate feature information and geometry information
▌The proposed method outperforms the CNN without incident
angle information by 1.05% and HCF method by 11.25%
▌The proposed method achieves best f-measure for bulk-carrier
and tanker but fails in container
▌The proposed method requires 25% less training data as
compared to the conventional CNN method
19 © NEC Corporation 2015 NEC Group Internal Use Only19 © NEC Corporation 2019
Appendix
20 © NEC Corporation 2015 NEC Group Internal Use Only20 © NEC Corporation 2019
A1: AIS + Synthetic Aperture Radar (SAR)
Legal
May be illegal
Vessel detected by SAR only
Vessel detected by both SAR and AIS
A SAR image can be used in conjunction with AIS data to detect
illegal vessels in ocean
21 © NEC Corporation 2015 NEC Group Internal Use Only21 © NEC Corporation 2019
A2: Hand-crafted Features*
Feature Formula
Length 𝐿
Width 𝑊
Perimeter 2 × (𝐿 +𝑊)
Area 𝐿 ×𝑊
Shape Complexity 𝑃/4𝜋𝐴 P: Perimeter
Compactness 𝑃/2𝜋𝐿
Elongatedness 𝐿/𝑊
Aspect Ratio 𝑊/𝐿
Centroid X 𝑀10
𝑀00𝑀𝑖𝑗: Image Moment
Centroid Y 𝑀01
𝑀00𝑀𝑖𝑗: Image Moment
*Lang, H., Zhang, J., Zhang, X. and Meng, J., "Ship classification in SAR image by joint feature and classifier selection,"
IEEE Geoscience and Remote Sensing Letters 13(2), 212-216 (2016).
Very High Resolution SAR Change Detection
with Siamese Networks
Shreya Sharma, Masato Toda
Data Science Research Laboratories
NEC Corporation
Published in the 66th Conference of the Remote Sensing Society of Japanhttp://www.rssj.or.jp/eng/activity/meeting-of-the-remoto-sencing-society-of-japan/66th_spring/
23 © NEC Corporation 2015 NEC Group Internal Use Only23 © NEC Corporation 2019
Motivation
Example:
Change Detection
System
Trends in the target area
Comparison
Multi-temporal SAR Images
Change Detection Maps
To develop change detection technology using Earth Observation data
For quick and detailed monitoring of important economic areas
GroundTruth
24 © NEC Corporation 2015 NEC Group Internal Use Only24 © NEC Corporation 2019
Optical Image v/s SAR Image
Optical Image SAR Image
• Nadir-view imaging• Passive sensing – day only• Cannot image over clouds
• Side-view imaging• Active sensing – day/night• Can image over clouds
© DLR© Google Earth
25 © NEC Corporation 2015 NEC Group Internal Use Only25 © NEC Corporation 2019
SAR Change Detection Methods
ClassifyClassify
Compare
Change Map
Compute DI*
Threshold
Change Map
*Difference Image
Not useful when object is too small
to be classified
Methods
Post classification based
Pixel-to-pixel difference based
26 © NEC Corporation 2015 NEC Group Internal Use Only26 © NEC Corporation 2019
Low Resolution v/s Very High Resolution (VHR) SAR CD
Pixel-to-pixel difference method works well for low resolution SAR
We need a change detection method for VHR SAR
Low Resolution
HighResolution
Image 1 Image 2 Change Map
© DLR © DLR
© ESA © ESA
27 © NEC Corporation 2015 NEC Group Internal Use Only27 © NEC Corporation 2019
Problem In VHR SAR Change Detection
▌Camera Jitter
causes co-registration error
▌Speckle
characteristic property in SAR
causes noisy background
▌Camouflage
non-defined shape and boundary
difficulty to detect small and moving objects
Image 1
Image 2
Low change detection accuracy
Pixel-to-pixel difference method cause many false changes in VHR
© DLR© DLR
© DLR© DLR
© DLR
28 © NEC Corporation 2015 NEC Group Internal Use Only28 © NEC Corporation 2019
Solution
Feature-to-Feature difference method is robust to the conditions
Siamese network has been widely used for feature comparison
Patch 𝑋1
Patch 𝑋2
Conv
Conv
Poo l
Conv
FC
FC
Conv
Conv
Poo l
Conv
FC
FC
Share Weights
64x3x3 64x3x3 2x2
𝑓1
𝑓2
64x3x3 128 128Image 1
Image 2
However, Siamese network is not yet implemented for VHR SAR
Comparison
29 © NEC Corporation 2015 NEC Group Internal Use Only29 © NEC Corporation 2019
Proposed Siamese Network for VHR SAR
Patch
Extraction
Conv
Conv
Pool
Conv
FC
FC
Conv
Conv
Pool
Conv
FC
FC
Euclidean
Distance
Thresholding
Share Weights
𝑋1
𝑋2
𝒇𝟏
𝒇𝟐
Change Map
SAR Image 1
SAR Image 2
Siamese Network
Trained
Network
Distance
Calculation
TRAINING
TESTING
Euclidean distance-based loss is used for feature comparison
Compute
LossY true
Y pred
30 © NEC Corporation 2015 NEC Group Internal Use Only30 © NEC Corporation 2019
Experimental Evaluation
▌Application Parking Lot Monitoring
▌Baselines: PCA + Kmeans [1]
Sparse AE + Kmeans [2]
▌Evaluation Metrics f-measure
ROC-AUC
[1] T. Celik: Unsupervised change detection in satellite images using principal component analysis and k-means clustering, IEEE Geoscience and Remote Sensing Letters, vol. 6, no. 4, pp. 772-776, 2009.
[2] M. Gong., H. Yang, and P. Zhang: Feature learning and change feature classification based on deep learning for ternary change detection in SAR images, ISPRS Journal of Photogr. and Remote Sensing, no.129, pp.212-225, 2017.
Satellite TerraSAR-X
Resolution 1m
Polarization HH
No. of parking
sites
5
No. of
images/site
10
No. of
pairs/site
9
Patch sizes 10x10, 16x16
Ground Truth
(GT)
Manual
interpretation
Specifications
31 © NEC Corporation 2015 NEC Group Internal Use Only31 © NEC Corporation 2019
Dataset
Site 1
Site 2
Site 3
Site 4
Site 5
t0 t1 t2 t9 t10
Pair 1 Pair 2 Pair 9
…
…
…
…
…
…
…
Test
All images are copyrighted to DLR
32 © NEC Corporation 2015 NEC Group Internal Use Only32 © NEC Corporation 2019
Result [1/4] : f-measure
0.32
0.59
0.41
0.72
0.62
0.76
0.66
0.76
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Pair 1 Pair 2
Comparison of f-measure for 2 test pairs
PCA-K SAE-K Siamese-10 Siamese-16
Siamese networks outperforms the conventional methods
+0.25
+0.4
33 © NEC Corporation 2015 NEC Group Internal Use Only33 © NEC Corporation 2019
Result [2/4] : ROC-AUC
0.640.68
0.75
0.81
0.88 0.890.93 0.93
0.4
0.5
0.6
0.7
0.8
0.9
1
Pair 1 Pair 2
Comparison of ROC-AUC for 2 test pairs
PCA-K SAE-K Siamese-10 Siamese-16
Siamese networks outperforms the conventional methods
+0.18+0.12
34 © NEC Corporation 2015 NEC Group Internal Use Only34 © NEC Corporation 2019
Result [3/4] : Change Maps
GT PCA-K SAE-K Siamese-10 Siamese-16
PCA-K SAE-K Siamese-10 Siamese-16GT
Test Pair 1
Siamese networks produce visually better change maps
Test Pair 2
35 © NEC Corporation 2015 NEC Group Internal Use Only35 © NEC Corporation 2019
Result [4/4] : Colorized Change Map
Siamese-16Siamese-10
SAE-KPCA-KGT
Siamese networks reduce the number of false positives
Legend TP, TN, FP, FN
36 © NEC Corporation 2015 NEC Group Internal Use Only36 © NEC Corporation 2019
Conclusion
▌Proposed Siamese neural network for change detection in VHR
SAR
▌Proposed network is trained with Euclidean distance loss function
▌Evaluated for parking lot monitoring using 1m resolution SAR
images
▌Proposed network outperforms the conventional methods both in
f-measure and ROC-AUC
▌Proposed network produces visually better change maps
▌Proposed network reduces number of false positives
▌Future work
reduce number of false negatives
ternary change detection