Image Representation Alternatives for the Analysis …...Image Representation Alternatives for the Analysis of Satellite Image Time Series Corneliu Octavian Dumitru, Gottfried Schwarz,

Image Representation Alternatives for the Analysis of Satellite Image Time Series

Corneliu Octavian Dumitru, Gottfried Schwarz, and Mihai Datcu Remote Sensing Technology Institute, German Aerospace Center (DLR)

www.DLR.de • Chart 1 MultiTemp2017 > 28.06.2017

How can we represent the traffic conditions on the A10 near Bruges? • Did the early morning traffic jam dissolve?

• What is needed for image interpretation and understanding? • Feature extraction, image classification, or semantic catalogues?

www.DLR.de • Chart 2

© Panoramio by Paulo Yuji Takarada

MultiTemp2017 > 28.06.2017

Presenter

Presentation Notes

Imagine that we’d like to automatically analyze what is going on in a motorway using image time series to see, for example, if the early morning traffic jam dissolved. What do we need for the interpretation and understanding of the images? Do we need feature extraction, image classification, or semantic catalogues?

Proposed Approach


Mul

ti-re

solu

tion

patc

hes

Quick-looks

Texture

Spectral

Deep learning

Features

Active learning

Clusters Semantics catalogue

Performance measures

Data analytics Classification maps

Image content

- New semantics - Hints to changes - High level information

Graphs

Patches Features Classification Annotation Knowledge Graph

Selection of Use Cases / Applications

Sate

llite

Im

ages

Ex

tern

al

Data

MultiTemp2017 > 28.06.2017

Presenter

Presentation Notes

We developed a modular system approach for the interpretation and understanding of image time series which includes quantitative analytics. Selecting methods and algorithms in every module of this system depends on the application case. When the application case is decided, the relevant images and external data are collected. The satellite images are then tiled into smaller patches which can have different sized based on the image properties and application. These patches are stored for visual analytics in the Quick-look database. In the next module, the image patches are described by extracting their primitive features. Selecting the feature extraction method depends on the data and the application. Could be texture, spectral or even computed based on deep learning methods. These features are stored in the feature database. In the next step higher level spatial grouping is performed through classification methods which could be fully automatic or interactive. The computed classes are stored in the cluster/class database. In order to assign a human understandable meaning to the classes, annotation is done in the next step to generate semantic labels and their relations. These labels are stored in the semantics database. For understanding what is going on in the image time series, semantic classification of the images is not enough. We should know how these semantics relate to each other overtime. When they appear and when turn into another one. This information is given by a knowledge graph which will be stored into a database. Having these information, we derive to analytics such as image content, classification maps, data analytics and statistics, and performance measures such as precision/recall.

Typical examples with selected satellite image time series (SITS) • Typical use cases and applications

• Coastal environmental monitoring, land cover/use changes, disaster monitoring, forest management, traffic monitoring, etc.

• We concentrated on: • Monitoring coastal environments:

• Wadden Sea (in the Netherlands) • Danube Delta (in Romania)

• Rapid mapping for disaster monitoring: • Sendai (in Japan) and surrounding areas affected by a tsunami • Elbe river (in Germany) affected by floods

• Datasets • Sentinel-1A medium resolution SAR images:

• Monitoring coastal environments • TerraSAR-X high-resolution SAR images

• Rapid mapping


Presenter

Presentation Notes

There are wide variety of use cases and applications for SITS, for example, coastal environmental monitoring, land cover/use changes, disaster monitoring, forest management, traffic monitoring, etc. In our work, we concentrated on two application: First, monitoring coastal environments by studying Wadden sea in Netherland and Danube delta in Romania which are UNESCO’s natural heritage. The second application is Rapid mapping for which we analyzed the tsunami in Sendai and its surrounding area in Japan, and the flood in Elbe river in Germany. For these analysis, two main datasets have been used, the medium resolution Sentinel-1A SAR images for coastal environment monitoring; and the high-resolution TerraSAR-X SAR images for rapid mapping.

Proposed Image Handling

• Multi-scale patch cutting for change detection applications • Optimal patch size and number of levels depends on:

• Selected application, instrument type, image parameters (e.g., resolution, pixel spacing), and type of features to be used [1]

• For Sentinel-1 the patch size is 128x128 pixels • For TerraSAR-X is 160x160 pixels


[1] C.O. Dumitru, S. Cui, and M. Datcu, “Validation of Cascaded Active Learning for TerraSAR-X Images”, in Proc. of IIM 2015, Bucharest, Romania, 2015.

MultiTemp2017 > 28.06.2017

ploughed agriculture land low density residential area

before before

after after

Presenter

Presentation Notes

After selecting the satellite images they are tiles into smaller patches. In our system, a Multi-scale patch cutting has been performed. Where the Optimal patch size and the number of levels depends on: The selected application, instrument type, image parameters (e.g., resolution, pixel spacing), and type of features to be used [1] For Sentinel-1 the patch size is 128x128 pixels For TerraSAR-X is 160x160 pixels Here you can see some image examples extracted from pre-event and post-event TerraSAR-X tsunami datasets.

Proposed Feature Extraction and Classification

• Feature extraction • Enhanced rotation-invariant feature extractor

• Gabor filter bank results sorted by magnitude • Deep learning

• Deep convolutional neural network for SAR images [2]

• Classification • Active learning classification based on a cascaded learning method [1]

• Using a Support Vector Machine

• The advantages of cascaded learning 1) Reduction of data volume to be classified from one level to the next level 2) User can select up to which level the data shall be classified and annotated


[1] C.O. Dumitru, S. Cui, and M. Datcu, “Validation of Cascaded Active Learning for TerraSAR-X Images”, in Proc. of IIM 2015, Bucharest, Romania, 2015. [2] M. Gong, J. Zhao, J. Liu, Q. Miao and L. Jiao, "Change Detection in Synthetic Aperture Radar Images Based on Deep Neural Networks," IEEE Trans. on NNLS, 27(1), pp. 125-138, 2016.

MultiTemp2017 > 28.06.2017

Presenter

Presentation Notes

For the feature extraction two methods have been used: The first method is an enhanced rotation-invariant feature extractor which obtained by sorting the Gabor filter bank results by their magnitudes. The second method is based on a deep convolutional neural network developed for SAR data. For the classification module an active learning classification approach based on a cascaded learning method [1] which uses a Support Vector Machine has been used. The advantages of cascaded learning are: Reducing the volume of data to be classified from one level to the next level. The user can select up to which level the data shall be classified and annotated.

Proposed Annotation

• Annotation and semantics • Hierarchical semantic

annotation scheme for high-resolution SAR images, with 3 hierarchical levels and with a total of 150 categories [3]

• The uppermost level 3 categories describe details of man-made infrastructure, while the categories describing natural environments do not have level 3 refinements.

• For medium-resolution SAR images (e.g., Sentinel-1) our proposed annotation scheme will not cover the very detailed level (such as TerraSAR-X).


[3] C. O. Dumitru, G. Schwarz, and M. Datcu, “Land Cover Semantic Annotation Derived from High-Resolution SAR Images”, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(6), pp. 2215-2232, 2016.

MultiTemp2017 > 28.06.2017

Presenter

Presentation Notes

We developed a hierarchical semantic annotation scheme for high-resolution SAR images, with 3 hierarchical levels and with a total of 150 categories. The uppermost level 3 categories describe details of man-made infrastructure, while the categories describing natural environments do not have level 3 refinements. For medium-resolution SAR images (e.g., Sentinel-1) our proposed annotation scheme will not cover the very detailed level (such as TerraSAR-X).

Proposed Knowledge Graphs

• Knowledge graphs • The scientific goal is to select image data combined with additional information and to

generate from them higher-level interpretation results • The linking can be understood as an upwards translation of binary data into content-

related information. • In contrast, current applications of linked data are mostly limited to the integration of

geographical data with query systems.


Presenter

Presentation Notes

We also need knowledge graphs for higher level descriptions of image content changes.

Typical Outputs

• Classification maps for SITS • Used for analyzing the temporal evolution of the

affected areas


Airport runways

Aquaculture

Bridges

Channels

Debris

Flooded areas

Industrial buildings

Medium-density residential areas

Mountains

Ocean

Ploughed agricultural land

Shores

MultiTemp2017 > 28.06.2017

From Center for Satellite Based Crisis Information (ZKI) Available: https://www.zki.dlr.de/

Presenter

Presentation Notes

As an output, classification maps can be generated for image time series. These maps will be used to analyze the temporal evolution of the affected areas in disaster monitoring. Figure left Comparative time series classification maps based on data taken prior to the Sendai tsunami (left column), one day after the March 11, 2011 tsunami (center-left column), two months after the tsunami (center-right column), and three months after the tsunami (right column). The flooded area is illustrated in yellow. You can see how the coastal area covered by flood in the second image and how the flood drained afterwards. This is consistent with the ZKI change map of the region at that time which you can see in the right figure. Figure right ZKI change map

Typical Outputs

• Data analytics • Used for detecting changes or to see the distribution of the retrieved categories


Presenter

Presentation Notes

Data analytics results can be used to detect any changes or to see the distribution of the retrieved categories Figure left Here you can see the Quantitative analysis of the tsunami-affected areas. Or in the right Figure The Diversity of categories identified from the pre- and post-event tsunami images acquired before and one day after the event. For example, the aquaculture which had covered about 3% of the are, after the tsunami totally destroyed and therefore, disappeared.

Typical Outputs

• Performance measures • Such as classification accuracy of the extracted features


Presenter

Presentation Notes

Attainable classification accuracies by Gabor feature vectors for a selected number of categories from our dataset.

Typical Outputs

• Image content information • For detecting anomalies or changes between new data and given reference data from a

database


Airport runways → Flooded areas

Aquaculture → Debris

Bridges → Flooded areas

Channels → Flooded areas

Industrial areas → Flooded areas

Medium-density residential areas → Flooded areas

Ocean → Debris

Ploughed agricultural land → Flooded areas

Unchanged

Debris → Ocean

Debris → Aquaculture

Flooded areas

Flooded areas → Airport runways

Flooded areas → Channels

Flooded areas → Industrial areas

Flooded areas → Medium-density residential areas

Flooded areas → Ploughed agricultural land Unchanged

MultiTemp2017 > 28.06.2017

Presenter

Presentation Notes

An output could be image content information which is used for detecting anomalies or changes between new data and given reference data from a database. Figure left Resulting change map based on data taken before and one day after the tsunami. Each change of the previously classified categories is mapped into a new category resulting after the tsunami (e.g., flooded areas, debris) and is encoded with a different color. Categories within areas which remained unaffected by the tsunami are marked in reddish brown. Figure right Resulting rebound change map based on data taken one day after and three months after the tsunami. Each change of the previously classified categories is mapped into one of the allocated categories prior to the tsunami and is encoded with a different color. Categories within areas which remained unaffected by the rebound are marked in reddish brown.

Validation of the Proposed Approach

• We developed an image time series analysis concept for low and high level image content representation [6].

• This concept has already been validated with large datasets [5], [6]. • The selection of the first use case and the protected areas were made

jointly with partners from the H2020 ECOPOTENTIAL project.


[5] C.O. Dumitru, G. Schwarz, D. Espinoza-Molina, M. Datcu, H. Hummel, and Ch. Hummel, “Classification and Semantic Annotation of Extended Wadden Sea Features Using SAR Images and their Environmental Interpretation”, journal paper to be published. [6] C.O. Dumitru, G. Schwarz, and M. Datcu, “SAR Image Land Cover Datasets for Classification Benchmarking“, journal paper to be publish.

Presenter

Presentation Notes

We developed an image time series analysis concept for low and high level image content representation [6]. This concept has already been validated with large datasets [5], [6]. And finally, the selection of the first use case and the protected areas were made jointly with partners from the H2020 ECOPOTENTIAL project.

Thank you for your attention!

For further questions please contact: [email protected]


Presenter

Presentation Notes

Any questions from the audience?

mailto:[email protected]

Why and when do we need these different representations?

• Images and image patches: Visual analysis, data for subsequent processing

• Extracted features: Feature and texture analysis, data for subsequent processing

• Classification results: Higher level spatial groupings, data for subsequent analytics

• Semantic annotation: Physical relationships, data for subsequent analytics

• Knowledge graphs: Spatio-temporal ontologies and evolutions, final analytics

www.DLR.de • Chart 15 > Lecture > Author • Document > Date

Presenter

Presentation Notes

We are faced with many potential image representations. We defined a systematic ordering.

Proposed Image Handling

• Multi-scale patch cutting for change detection applications • Optimal patch size and number of levels depends on: selected application, instrument

type, image parameters (e.g., resolution, pixel spacing), and type of features to be used [1] • For Sentinel-1 the patch size is 128x128 pixels and for TerraSAR-X is 160x160 pixels


[1] C.O. Dumitru, S. Cui, and M. Datcu, “Validation of Cascaded Active Learning for TerraSAR-X Images”, in Proc. of IIM 2015, Bucharest, Romania, 2015.

before before before after after after

ploughed agriculture land mixed forest river

before before – after after low density residential area transport infrastructure

MultiTemp2017 > 28.06.2017

Presenter

Presentation Notes

Typical image examples extracted from pre-event and post-event TerraSAR-X tsunami datasets.

Documents

Image Representation Alternatives for the Analysis …...Image Representation Alternatives for the Analysis of Satellite Image Time Series Corneliu Octavian Dumitru, Gottfried Schwarz,