[DLHacks 実装]Perceptual Adversarial Networks for Image-to-Image Transformation

Image-to-Image Translation with Conditional Adversarial Nets (Pix2Pix)

& Perceptual Adversarial Networks for

Image-to-Image Transformation (PAN)

2017/10/2 DLHacks Otsubo

Topic : image-to-image “translation”

Pix2Pix [CVPR2017] •  Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros

-  iGAN [ECCV 2016] -  interactive-deep-colorization [SIGGRAPH 2017] -  Context-Encoder [CVPR 2016] -  Image Quilting [SIGGRAPH 2001] -  Texture Synthesis by Non-parametric Sampling [ICCV 1999]

•  University of California •  178 citations

PAN [arXiv2017] •  Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao •  University of Technology Sydney, The University of Sydney,

Universite Paris-Est

Background

•  Many tasks are regarded as “translation” from input image to output image -  Diverse methods exist for them

Istheresingleframeworktoachievethem?

Overview

Pix2Pix •  General-purpose solution to image-to-image

translation using single framework -  Single framework: conditional GAN (cGAN)

PAN •  Pix2Pix - (per-pixel loss)

+ (perceptual adversarial loss)

Naive Implementation : U-Net (①)

①per-pixel loss (L1/L2)

Pix2Pix (①+②)

②adversarial loss

Pix2Pix’s loss (①+②)

PAN (②+③)

③perceptual adversarial loss

PAN’s loss (②+③)

L1 norm

m : constant

Example1 : Image De-Raining

•  Removing rain from single images via a deep detail network [Fu, CVPR2017]

•  ID-GAN (cGAN) [Zhang, arXiv2017] -  per-pixel loss -  adversarial loss -  pre-trained VGG’s

perceptual loss

Input Output (Ground Truth)

Example1 : Image De-Raining

•  Removing rain from single images via a deep detail network [Fu, CVPR2017]

•  ID-GAN (cGAN) [Zhang, arXiv2017] -  per-pixel loss -  adversarial loss -  pre-trained VGG’s

perceptual loss

(cf. PAN uses discriminator’s perceptual loss)

Example2 : Image Inpainting

•  Globally and Locally Consistent Image Completion [Iizuka, SIGGRAPH2017]

•  Context Encoders (cGAN) [Pathak, CVPR2016] -  per-pixel loss -  adversarial loss

Example3 : Semantic Segmentation

Cityscape / Pascal VOC •  DeepLabv3 [Chen, arXiv2017] •  PSPNet [Zhao, CVPR2017]

http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?cls=mean&challengeid=11&compid=6

Cell Tracking / CREMI •  Learned Watershed

[Wolf, ICCV2017] •  U-Net

[Ronneberger, MICCAI2015] http://www.codesolorzano.com/Challenges/CTC/Welcome.html

Result1 : Image De-Raining

(≒pix2pix)→

(≒pix2pix)

Result2 : Image Inpainting

Result3 : Semantic Segmentation

Discussion

vs. No perceptual loss (Pix2Pix) -  Perceptual loss enables D to detect more

discrepancy between True/False images vs. Pre-trained VGG perceptual loss (ID-GAN)

-  VGG features tend to focus on content -  PAN features tend to focus on discrepancy -  PAN’s loss leads to avoid adversarial

examples [Goodfellow, ICLR2015] (?)

Why is perceptual adversarial loss so efficient?

Minor Difference

•  Pix2Pix uses Patch-GAN -  Small size(70×70) patch-discriminator -  Final output of D is average of

patch-discriminator’s responses (convolutionally applied)

•  Implement 1.  Pix2Pix (Patch Discriminator) 2.  PAN (Patch Discriminator) 3.  PAN (Normal Discriminator) Wang et al. might compare 1 with 3.

Implementation

2017/10/17 DLHacks Otsubo

My Implementation

•  https://github.com/DLHacks/pix2pix_PAN

•  pix2pix - https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

•  PAN -  per-pixel loss à perceptual adversarial loss -  not same as paper’s original architecture -  num of parameters is same as pix2pix

My Experiments

•  Facade (label à picture) •  Map (picture à Google map) •  Cityscape (picture à label)

Result (Facade pix2pix)

Result (Facade PAN)

Result (Map pix2pix)

Result (Map PAN)

Result (Cityscape pix2pix)

Result (Cityscape PAN)

Result (PSNR[dB])

Discussion – Why pix2pix > PAN?

•  per-pixel loss is needed? •  patch discriminator is not suited for PAN? •  positive margin m

•  (bad pix2pix implementation in PAN’s paper…?)

[DLHacks 実装]Perceptual Adversarial Networks for Image-to-Image Transformation

Technology

Lecture 22: Adversarial Image, Adversarial Training, Variational Autoencoders… · 2018-11-08 · Adversarial Images Adversarial Training Autoencoder VAE GAN Conclusions Lecture

Generative adversarial text to image synthesis

Reality Transform Adversarial Generators for Image

Adversarial Attacks Beyond the Image Spaceopenaccess.thecvf.com/content_CVPR_2019/papers/... · Adversarial Attacks Beyond the Image Space Xiaohui Zeng1, Chenxi Liu2( ), Yu-Siang

Benchmarking Adversarial Robustness on Image Classification · Benchmarking Adversarial Robustness on Image Classiﬁcation Yinpeng Dong1, Qi-An Fu1, Xiao Yang1, Tianyu Pang1, Hang

Generative Adversarial Network-based Image Super ...openaccess.thecvf.com/content_ECCVW_2018/papers/11133/...Generative Adversarial Network-based Image Super-Resolutionusing Perceptual

[DL輪読会]Image-to-Image Translation with Conditional Adversarial Networks

Perceptual Adversarial Networks for Image-to-Image ... · PDF filetraining process. Through combining the perceptual adversarial loss and the generative adversarial loss, we presented

Generative adversarial network based regularized image ... · Generative adversarial network based regularized image reconstruction for PET Zhaoheng Xie 1, Reheman Baikejiang , Tiantian

Semantics-Enhanced Adversarial Nets for Text-to-Image Synthesisopenaccess.thecvf.com/content_ICCV_2019/papers/Tan... · 2019-10-23 · Semantics-enhanced Adversarial Nets for Text-to-Image

DLhacks paperreading_20150902

Image Colorization using Generative Adversarial Networks

[DLHacks 実装] The statistical recurrent unit

Image Processing Defense on Adversarial Attackcs229.stanford.edu/proj2017/final-reports/5241871.pdf · Image Processing Defense on Adversarial Attack Mark Liu(maliu2), Li Cai(licai0)

Discriminative Region Proposal Adversarial Networks for ......Our image-to-image translation model, called Discriminative Region Proposal Adversarial Networks (DRPAN), is composed

Image-to-Image Translation with Conditional Adversarial Networksの紹介

Non-Adversarial Image Synthesis With Generative Latent Nearest …openaccess.thecvf.com/content_CVPR_2019/papers/Hoshen... · 2019-06-10 · Non-Adversarial Image Synthesis with Generative

Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou

StarGAN: Unified Generative Adversarial Networks …openaccess.thecvf.com/content_cvpr_2018/papers/Choi...StarGAN: Uniﬁed Generative Adversarial Networks for Multi-Domain Image-to-Image

Unsupervised Semantic-Preserving Adversarial Hashing for Image Search · 2020. 9. 2. · DENG et al.: UNSUPERVISED SEMANTIC-PRESERVING ADVERSARIAL HASHING FOR IMAGE SEARCH 4033 reconstruct