Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Tuesday, 9 May 2017
Andrew Edelsten - NVIDIA Developer Technologies
ZOOM, ENHANCE, SYNTHESIZE! MAGIC UPSCALING AND MATERIAL SYNTHESIS USING DEEP LEARNING
2
DEEP LEARNING FOR ARTActive R&D but ready now
▪ Style transfer
▪ Generative networks creating images and voxels
▪ Adversarial networks (DCGAN) – still early but promising
▪ DL & ML based tools from NVIDIA and partners
▪ NVIDIA
▪ Artomatix
▪ Allegorithmic
▪ Autodesk
3
STYLE TRANSFERSomething Fun
Content Style▪ Doodle a masterpiece!
▪ Uses CNN to take the “style” from one image and apply it to another
▪ Sept 2015: A Neural Algorithm of Artistic Style by Gatys et al
▪ Dec 2015: neural-style (github)
▪ Mar 2016: neural-doodle (github)
▪ Mar 2016: texture-nets (github)
▪ Oct 2016: fast-neural-style (github)
▪ 2 May 2017 (last week!): Deep Image Analogy (arXiv)
▪ Also numerous services: Vinci, Prisma, Artisto, Ostagram
4HTTP://OSTAGRAM.RU/STATIC_PAGES/LENTA
5
STYLE TRANSFER
▪ Game remaster & texture enhancement
▪ Try Neural Style and use a real-world photo for the “style”
▪ For stylized or anime up-rez try https://github.com/nagadomi/waifu2x
▪ Experiment with art styles
▪ Dream or power-up sequences
▪ “Come Swim” by Kirsten Stewart - https://arxiv.org/pdf/1701.04928v1.pdf
Something Useful
6
GAMEWORKS: MATERIALS & TEXTURESUsing DL for Game Development & Content Creation
▪ Set of tools targeting the game industry using machine learning and deep learning
▪ Launched at Game Developer Conference in March, tools run as a web service
▪ Sign up for the Beta at: https://gwmt.nvidia.com
▪ Tools in this initial release:
▪ Photo to Material: 2shot
▪ Texture Multiplier
▪ Super-Resolution
7
PHOTO TO MATERIAL
▪ From two photos of a surface, generate a “material”
▪ Based on a SIGGRAPH 2015 paper by NVIDIA Research & Aalto University (Finland)
▪ “Two-Shot SVBRDF Capture for Stationary Materials”
▪ https://mediatech.aalto.fi/publications/graphics/TwoShotSVBRDF/
▪ Input is pixel aligned “flash” and “guide” photographs
▪ Use tripod and remote shutter or bracket
▪ Or align later
▪ Use for flat surfaces with repeating patterns
The 2Shot Tool
8
MATERIAL SYNTHESIS FROM TWO PHOTOS
Flash image Guide image
Diffuse
albedoSpecular Normals Glossiness Anisotropy
9
TEXTURE MULTIPLIER
▪ Put simply: texture in, new texture out
▪ Inspired by Gatys, Ecker & Bethge
▪ Texture Synthesis Using Convolutional Neural Networks
▪ https://arxiv.org/pdf/1505.07376.pdf
▪ Artomatix
▪ Similar product “Texture Mutation”
▪ https://artomatix.com/
Organic variations of textures
10
SUPER RESOLUTION
11
SUPER RESOLUTIONZoom.. ENHANCE!
Zoom in on the
license plate
OK!Sure!
Can you
enhance that?
12
SUPER RESOLUTIONThe task at hand
Upscale
(magic?)
W
H
Given alow-resolution image
n * W
n * H
Construct ahigh-resolution image
13
UPSCALE: CREATE MORE PIXELSAn ill-posed task?
Pixels of the upscaled image
Pixels of the given image? ? ?
? ? ? ? ? ?
? ? ?
? ? ? ? ? ?
? ? ?
? ? ? ? ? ?
14
TRADITIONAL APPROACH▪ Interpolation (bicubic, lanczos, etc.)
▪ Interpolation + Sharpening (and other filtration)
Filter-based sharpeningInterpolation
▪ Rough estimation of the data behavior too general
▪ Too many possibilities (8x8 grayscale has 256(8∗8) ≈ 10153 pixel combinations!)
15
A NEW APPROACHFirst: narrow the possible set
Photos
Textures
All possible imagesFocus on the domain of “natural images”
Natural images
16
A NEW APPROACH
Data from natural images is sparse, it’s compressible in some domain
Then “reconstruct” images (rather than create new ones)
Second: Place image in the domain, then reconstruct
+prior information
+constraints
ReconstructCompress
17
PATCH-BASED MAPPING: TRAINING
Model
params
Mapping
Training images
,
LR,HR pairs of patches
training
Low-resolution patch High-resolution patch
18
PATCH-BASED MAPPING
LR patch
HR patch
Encode Decode
𝒙𝑳
𝒙𝑯
High-level information about the patch
19
PATCH-BASED MAPPING: SPARSE CODING
LR patch
HR patch
Sparse code
Encode Decode
𝒙𝑳
𝒙𝑯
High-level information about the patch“Features”
20
PATCH FEATURES & RECONSTRUCTION
𝒙 = 𝑫𝒛 = 𝒅𝟏𝒛𝟏 +⋯+ 𝒅𝑲𝒛𝑲
= 0.8 * + 0.3 * + 0.5 *
𝑫
𝒅𝟑𝟔 𝒅𝟒𝟐 𝒅𝟔𝟑𝒙
Image patch can be reconstructed as a sparse linear combination of features
Features are learned from the dataset over time
𝒛
𝒙
𝑫 - dictionary
- patch
- sparse code
21
GENERALIZED PATCH-BASED MAPPING
MappingMapping
LR patch
HR patchHigh-level
representation of the LR patch
“Features”
High-level representation of
the HR patch
Mapping in feature space
22
GENERALIZED PATCH-BASED MAPPING
Mapping in feature space
MappingMapping
LR patch
HR patch
Trainable parameters
𝑊1 𝑊2 𝑊3
23
MAPPING OF THE WHOLE IMAGEUsing Convolutions
LR image
HR image
Mapping in feature space
MappingMapping
Convolutional operators
24
AUTO-ENCODERS
input output ≈ input
25
AUTO-ENCODER
input
features
Encode
output ≈ input
Decode
26
AUTO-ENCODER
𝑥 𝑦
Parameters
𝑊
Training
𝑊 = 𝑎𝑟𝑔𝑚𝑖𝑛
𝑖
𝐷𝑖𝑠𝑡(𝑥𝑖 , 𝐹𝑊 𝑥𝑖 )
𝑦 = 𝐹𝑊(𝑥)
Inference
𝑥𝑖 - training set
27
AUTO-ENCODER
input
Encode
information loss
▪ Our encoder is LOSSY by definition
28
SUPER-RESOLUTION AUTO-ENCODER
Training
𝑦 = 𝐹𝑊(𝑥)
Inference
𝑥𝑖 - training set
𝑊 = 𝑎𝑟𝑔𝑚𝑖𝑛
𝑖
𝐷𝑖𝑠𝑡(𝑥𝑖 , 𝐹𝑊 𝑥𝑖 )
𝑥 𝑦
Parameters
𝑊
29
𝑊 = 𝑎𝑟𝑔𝑚𝑖𝑛
𝑖
𝐷𝑖𝑠𝑡(𝑥𝑖 , 𝐹𝑊 𝐷(𝑥𝑖) )
SUPER RESOLUTION AE: TRAINING
y
𝑥𝑖 - training set
Ground-truth HR image
Downscaling
LR image
SR AE
Reconstructed HR image
𝑥
𝐹W
𝐷
ො𝑥
𝑊
30
SUPER RESOLUTION AE: INFERENCE
Given LR image
Constructed HR image
y
ො𝑥
𝑦 = 𝐹𝑊(ො𝑥)
SR AE
𝐹W
𝑊
31
SUPER-RESOLUTION: ILL-POSED TASK?
32
THE LOSS FUNCTION
33
THE LOSS FUNCTION
Distance function is a key element to obtaining good results.
Measuring the “distance” from a good result
𝑊 = 𝑎𝑟𝑔𝑚𝑖𝑛
𝑖
𝐷 𝑥𝑖 , 𝐹𝑊(𝑥𝑖 )
Choice of the loss function is an important decision
34
LOSS FUNCTION
1
𝑁𝑥 − 𝐹 𝑥 2
MSEMean Squared Error
35
LOSS FUNCTION: PSNR
1
𝑁𝑥 − 𝐹 𝑥 2
MSEMean Squared Error
PSNR Peak Signal-to-Noise Ratio
10 ∗ 𝑙𝑜𝑔10𝑀𝐴𝑋2
𝑀𝑆𝐸
36
LOSS FUNCTION: HFEN
1
𝑁𝑥 − 𝐹 𝑥 2
MSEMean Squared Error
PSNR Peak Signal-to-Noise Ratio
10 ∗ 𝑙𝑜𝑔10𝑀𝐴𝑋2
𝑀𝑆𝐸
𝐻𝑃(𝑥 − 𝐹 𝑥 ) 2
HFEN(see A)
High Frequency Error Norm High-Pass filter
Perceptual loss
Ref A: http://ieeexplore.ieee.org/document/5617283/
37
REGULAR LOSS
Result 4x Result 4x
38
REGULAR LOSS + PERCEPTUAL LOSS
Result 4x Result 4x
39
WARNING… THIS IS EXPERIMENTAL!
40
SUPER-RESOLUTION: GAN-BASED LOSS
Total loss = Regular (MSE+PSNR+HFEN) loss + GAN loss
Generator Discriminator
𝑥𝐹(𝑥)
𝐷(𝑦)𝑦
= −𝑙𝑛𝐷(𝐹 𝑥 )GAN loss
real
fake
Extended presentation from Game Developer Conference 2017
https://developer.nvidia.com/deep-learning-games
GameWorks: Materials & Textures
https://gwmt.nvidia.com
QUESTIONS?