TGS Salt Identification Challenge - homepages.cae.wisc.eduhomepages.cae.wisc.edu/~ece539/project/f18/wangz_rpt.pdf · CS 539 Final Project: TGS Salt Identification Challenge Zheming

TGS Salt Identification Challenge

CS 539 Final Project

Instructor Yu Hen Hu

By

Zheming Wang

Dec 2018

CS 539 Final Project: TGS Salt Identification Challenge Zheming Wang

- 2 -

Index 1. Introduction ................................................................................................... - 3 -

2. Problem Insight .............................................................................................. - 3 -

2.1 data summary ........................................................................................... - 3 -

2.2 metric ....................................................................................................... - 4 -

2.3 Common Structure: U-net ......................................................................... - 4 -

3. Experiment on Euler ...................................................................................... - 5 -

3.1 Platform and Adjustment .......................................................................... - 5 -

3.1.1 Tool kits and Platform ........................................................................ - 5 -

3.1.2 Adjustment for Resumable Training ................................................... - 5 -

3.2 Revision of network Structure .................................................................. - 5 -

3.2.1 Baseline U-net Model: ........................................................................ - 5 -

3.2.2 Revised U-net Model:......................................................................... - 6 -

3.2.3 Res-Unet Model: ................................................................................ - 7 -

3.2.4 Some Failed Try: ................................................................................ - 7 -

3.2.5 The revised RU-net Model: ................................................................ - 8 -

3.3 Fine-tuning and Expert-voting .................................................................. - 8 -

3.3.1 Introduction of Lovasz-SoftMax Loss Function: ................................ - 8 -

3.3.2 RU-net-Lovasz model: ....................................................................... - 8 -

3.3.3 Cross Validation, Augmentation, and Voting: .................................... - 9 -

3.4 Final Result Comparison .......................................................................... - 9 -

4. Conclusion ................................................................................................... - 10 -

Reference ......................................................................................................... - 11 -


- 3 -

1. Introduction Many areas of Earth with large accumulations of oil and gas also have huge deposits of salt below the surface. But unfortunately, knowing where large salt deposits are precisely is very difficult. Professional seismic imaging still requires expert human interpretation of salt bodies. This leads to very subjective, highly variable renderings, as well as potentially dangerous situations for oil and gas company drillers. In this Project, we applied different convolutional networks into Kaggle’s TGS Salt identification challenge dataset to generate an automatic and accurate subsurface salt detector. We started with a basic U-net and then added Res-Net technique inside. After further data augmentation, parameters tuning, cross validation, and experts voting, we trained a model with about 84% public test score and ranking top 10% in this Kaggle Challenge. During the model training period on Euler, we found that Keras cannot save whole state of the training model, which becomes an obstacle for resuming the model after serialization. Hence, we wrote a patch for it and uploaded all code we used to the GitHub at https://github.com/wzmJimmy/NeuralNetwork_Coursework.

2. Problem Insight 2.1 data summary The dataset contains seismic images randomly chosen at various locations in the subsurface. The images are 101 x 101 pixels and each pixel is classified as either salt or sediment. In addition to the seismic images, the depth of the imaged location is provided for each image. The train and test sets have sizes of 4,000 and 18,000. The goal is to segment regions that contain salt.

Graph 1: The input image and output mask.


- 4 -

By basic knowledge of Seismology, we know that the segmentation tries to tell the salt region, whose texture is relative chaotic, from the surrounding sediments, whose texture is always striped or stratified, showed in the Graph 1, where the white region in output mask is sale deposit.

2.2 metric The metric used for this competition is defined as the mean average precision at different intersection over union (IoU) thresholds [1]. IoU is widely used in graph segmentation problems to show comparison between prediction and ground-truth. IoU of a proposed set of object pixels and a set of true object pixels is calculated as:

𝐼𝑜𝑈(𝐴, 𝐵) = 𝐴 ∩ 𝐵𝐴 ∪ 𝐵

(1)

Other than the basic IoU, this competition further a mean average precision pattern on different thresholds as follows, where 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛(𝑡) is an indicator of whether 𝐼𝑜𝑈 >= 𝑡 for each threshold:

Precısıon>>>>>>>>>>>> = 1

𝑛𝑡ℎ𝑟𝑒𝑠ℎA 𝑝𝑟𝑒𝑐𝑖𝑠𝑜𝑛(𝑡)

𝑛

𝑡=1 (2)

2.3 Common Structure: U-net

Graph 2: Basic U-net structure.


- 5 -

U-net is commonly used as a baseline model for graph segmentation after it first presented in thesis of Olaf Ronneberger et al. [2] in 2015. The usage of up-convolution layer makes the input and output of the same dimension. The combination of contract path and up-sampling help the successive layer learn to assemble a more precise output. The U-net is also proved to be fast in training. Hence, we follow the literature to use U-net as baseline.

3. Experiment on Euler 3.1 Platform and Adjustment 3.1.1 Tool kits and Platform The main tool kits we used is Keras with TensorFlow as backend in Python. We first ran some scripts on Kaggle following some kernels, then made some changes to the script and adjusted it to Euler. In most time, we used 4 GTX 1080 and 1 GTX Titan to train 5-fold cross validation model in parallel. And it took at most 10 hours to train a single model. 3.1.2 Adjustment for Resumable Training Keras is a high-level API for neural network training. It provides callback functions like history, early-stop, and model-checkpoint for statistic and training use. A remarkable point of Keras is that it can save a model in md5 after training with a single save command. However, when resuming models saved in Euler, we found that there was always some information lost. After searching on some blogs [3], we found that Keras does not save callbacks information and even resets them when a model training starts. We then tried to train model in a single task on Euler but found there are plenty of cases in which resumable training is a must such as single task time limit, and unpredictable crash down. Therefore, after checking the Keras source code [4], we wrote a patch for it which can train a model in specific time, save and resume the model smoothly, and save the best model automatically. The patch code is included in a single GitHub link: https://github.com/wzmJimmy/MyKerasPatch.

3.2 Revision of network Structure 3.2.1 Baseline U-net Model: First, we followed a U-net starter kernel in Kaggle [5] to build a baseline model with structure in Graph 3.


- 6 -

Graph 3: baseline U-net model.

In this baseline model, there are 23 convolutional layers and 4 max-pooling layers. The activation function is elu with he_normal initializer and the optimizer is Adam with binary cross-entropy loss function. There are dropout layers between normal convolutional layers to avoid over-fitting. An early-stop with patience equals 15 is also applied to avoid overfitting. The relative batch size and epoch number are 32 and 150. With GTX-1080 in Euler, it cost us 608s to early stop at the 50th epoch and get a test IoU score is 0.675. 3.2.2 Revised U-net Model:

Graph 4: revised U-net model with basic preprocessing.


- 7 -

In this model, we found the input image is of grayscale, so we only took one channel as input, which reduced the number of parameters in the first layer and improved the test score. The channel numbers in each layer are doubled to make the network deeper and more flexible. At the same time, an adaptive learning rate method was applied to enhance the performance. Moreover, a threshold selection function was used to decide the best threshold for prediction. As for the data preprocessing. we flipped the figure horizontally and used a stratified train-test splitting method to enlarge training size and make it representative. Finally, the model training took us about half an hour and got test score of 0.722. 3.2.3 Res-Unet Model:

Graph 5: U-net model with basic Res-Net.

Follow this top kernel [6], we added basic Res-Net structure into the U-net. There are 50 convolution layers inside this model. Inside the Res-Net structure, we used batch normalization layers to keep input in a reasonable range for the relu activation function. With 200 epochs and more than two hours for training, we got a model whose test score is 0.797. 3.2.4 Some Failed Try: Followed this kernel [7], we tried to add depth data as an artificial channel but did not help improving test score. We also tried to add cumulative sum channel in the input layer, which only has a similar performance to the original Res-Unet and started to overfitting at the very beginning part. Hence, we changed to a different direction.


- 8 -

3.2.5 The revised RU-net Model: With discussion in the same kernel [6], we found that the resizing operation at the beginning and end part may misleading the metric. So, we removed these operations and changed the convolutional layers correspondingly as showed in Graph 6.

Graph 6: Revised RU-net.

By this structure, we trained a model with test score of 0.800. Then we nailed down the model structure and tried different parameters and training patterns to improve the performance of this model.

3.3 Fine-tuning and Expert-voting 3.3.1 Introduction of Lovasz-SoftMax Loss Function: The Lovasz-SoftMax function is a tractable surrogate of common Cross-entropy loss function for optimization of IoU in neural network. It is proved to outperform traditionally used Cross-entropy loss function in both single image and total dataset IoU in paper of Maxim Berman et. al. [8]. This loss function has already been implemented by the author in GitHub [9] . 3.3.2 RU-net-Lovasz model: We tried to use this new loss function to further train our RU-net model. As is cited by Jiaxin [6], “Lovasz loss performs better in training but is easy to be trapped in local minimum and time-consuming.” We chose a two-stage training pattern. In the first stage, the commonly used Cross-entropy loss function is used for pretraining, while in the following stage, Lovasz loss is used for further improvement.


- 9 -

By this pattern, we trained the first stage for 100 epochs without early stop and second stage for 200 epochs with early stop. We obtain a model with test score of 0.817. 3.3.3 Cross Validation, Augmentation, and Voting: After realizing the RU-net-Lovasz model, we chose to further boost it by cross validation, data augmentation, and voting. We used the imgaug module [10] to generate abundant input image resisting noise and disturbance. By 5-fold cross validation and further parameter tuning, we got a best single model with test score of 0.833. Finally, we used a simple expert fusion method, majority voting, to further improve the model test score to 0.839.

3.4 Final Result Comparison Table 1: Final result comparison.

Model Description Model Performance

Public IoU Private IoU Rank U-net Baseline 0.700 0.675 81% RU-net U-net with Res-net structure 0.818 0.797 48% RU-net revised Remove Resampling 0.829 0.800 46% RU-net-Lovasz Add Lovasz loss function 0.846 0.817 33% 5-fold best cv Add imgaug and CV 0.854 0.833 14% Voting Voting over several best CV 0.858 0.839 10%

Table 1 shows the progress we made from baseline U-net model to the final RU-net-Lovasz model with majority voting over best cross validation. The model selection and hyper-parameter tuning each accounts for a half of the improvement. Actually, the fine-tuning process of the RU-net-Lovasz is not perfect. With the Graph 7, we can see that the validation IoU is continuing raising up, while the learning pattern is rugged and rough. They both showed that there are still potential for enhancement. However, the training process of these model is really time-consuming, where each model took us more than 10 hours to train. With limited time, we stopped at this point. But we will continue this process in the future.


- 10 -

Graph 7: Convergence process of the best single model.

4. Conclusion In this project, we explored different CNN techniques like U-net and Res-Net, and applied them to the Kaggle TGS salt identification dataset. After trying different structures, and integrated cross validation, data augmentation, and majority voting, we obtained a model with public IoU of 0.84 and rank at the top 10% in the leaderboard. During the training process, we learned how to make Keras model resumable and how to take advantage of multiple GPUs. Moreover, we realized that other the model choosing, data preprocessing and hyper-parameter tuning can also make a great difference. As for the further improvement, we will first try different well-trained neural networks like X-ception and SE-net as decoder. Then we will take deeper insight into the model and apply several feature engineering techniques. Finally, further parameter tuning including optimizer and learning rate choosing will be implemented.


- 11 -

Reference

[1] "Explanation of Scoring Metric," [Online]. Available: https://www.kaggle.com/pestipeti/ explanation-of-scoring-metric.

[2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," 2015.

[3] "Why reset model.history? / comprehensive model state saving," [Online]. Available: https:// github.com/keras-team/keras/issues/6697.

[4] "keras/keras/callbacks.py," [Online]. Available: https://github.com/keras-team/keras/blob/master/ keras/callbacks.py#L360.

[5] K. Åmdal-SævikKeras, "Keras U-Net starter - LB 0.277," [Online]. Available: https:// www.kaggle.com/keegil/keras-u-net-starter-lb-0-277.

[6] J. (Jiaxin), "U-net with simple ResNet Blocks v2," [Online]. Available: https://www.kaggle.com/ shaojiaxin/u-net-with-simple-resnet-blocks-v2-new-loss.

[7] B. G. d. Amaral, "unet-with-depth," [Online]. Available: https://www.kaggle.com/bguberfain/unet-with-depth.

[8] Maxim Berman, Amal Rannen Triki, Matthew B. Blaschko, "The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.," CVPR, 2018.

[9] m. Berman, "LovaszSoftmax," [Online]. Available: https://github.com/bermanmaxim/ LovaszSoftmax.

[10] "imgaug," [Online]. Available: https://github.com/aleju/imgaug.

Documents

TGS Salt Identification Challenge - homepages.cae.wisc.eduhomepages.cae.wisc.edu/~ece539/project/f18/wangz_rpt.pdf · CS 539 Final Project: TGS Salt Identification Challenge Zheming