Upload
mark-chang
View
10.223
Download
0
Embed Size (px)
Citation preview
Neural Art -‐-‐ 電腦作畫 by Mark Chang
A Neural Algorithm of Ar7s7c Style
• 作者: – Leon A. Gatys. – Alexander S. Ecker. – MaAhias Bethge
• 所屬單位: – Werner Reichardt Centre for Integra7ve Neuroscience and Ins7tute of Theore7cal Physics, University of Tubingen, Germany.
– Bernstein Center for Computa7onal Neuroscience, Tubingen, Germany.
如何作畫?
大腦 畫家
景物 畫風 畫作
電腦 類神經網路
大綱
• 人類視覺 • 電腦視覺 • 電腦作畫 • 作品展示
人類視覺
• 神經元 • 視覺傳遞途徑 • 錯覺
神經元
• Neuron • Ac7on Poten7al
Dendrite
Axon
Cell Body
Time
Voltage
Threshold
視覺傳遞途徑
Re7na
Visual Area V1
Visual Area V4 Inferior Temporal Gyrus (IT)
視覺傳遞途徑
Visual Area V1
Inferior Temporal Gyrus (IT)
Recep7ve Fields
Visual Area V4
錯覺
電腦視覺
• Neural Networks • Convolu7onal Neural Networks • VGG 19
Neural Networks
n
W1
W2
x1
x2
b Wb
y
nin = w1x1 + w2x2 + wb
nout
=1
1 + e�nin
Sigmoid
Rec7fied Linear
nout
=
⇢nin
if nin
> 0
0 otherwise
Neural Networks
x
y
n11
n12
n21
n22
b b
z1
z2
Input Layer
Hidden Layer
Output Layer
W12,y
W12,x
W11,y
W11,b W12,b
W11,x W21,11
W22,12
W21,12
W22,11
W21,b W22,b
Convolu7onal Neural Networks
• Convolu7onal Layer
depth
width width depth
weights weights
height
shared weight
Convolu7onal Neural Networks
• Stride • Padding
Stride = 1
Stride = 2
Padding = 0
Padding = 1
Convolu7onal Neural Networks
• Pooling Layer 1 3 2 4
5 7 6 8
0 0 4 4
6 6 0 0
4 5
3 2
no overlap
no padding no weights
depth = 1
7 8
6 4
Maximum Pooling
Average Pooling
Convolu7onal Neural Networks
Convolu7onal Layer
Convolu7onal Layer Pooling
Layer
Pooling Layer
Recep7ve Fields Recep7ve Fields
Input Layer
Convolu7onal Neural Networks
Input Layer
Convolu7onal Layer with
Recep7ve Fields:
Max-‐pooling Layer with
Width =3, Height = 3
Filter Responses
Filter Responses
Input Image
VGG 19
• Karen Simonyan & Andrew Zisserman. Very Deep Convolu7onal Networks for Large-‐scale Image Recogni7on.
• ImageNet Challenge 2014 • 19 (+5) layers – 16 Convolu7onal layers (width=3, height=3) – 5 Max-‐pooling layers (width=2, height=2) – 3 Fully-‐connected layers
VGG 19
depth=64 conv1_1 conv1_2
maxpool
depth=128 conv2_1 conv2_2
maxpool
depth=256 conv3_1 conv3_2 conv3_3 conv3_4
depth=512 conv4_1 conv4_2 conv4_3 conv4_4
depth=512 conv5_1 conv5_2 conv5_3 conv5_4
maxpool maxpool maxpool
size=4096 FC1 FC2
size=1000 sogmax
電腦作畫
• 內容產生 • 畫風產生 • 作品產生
內容產生
大腦 畫家 景物
畫布 計算 兩者 差異
神經反應
補上線條和顏色
內容產生
計算 兩者 差異
Filter Responses VGG19
修正差異
相片
畫布
修正後
Width*Height Depth
內容產生
Layer l’s Filter l Responses:
Lcontent
(P l, Cl) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
Layer l’s Filter Responses: Lcontent
(P l, Cl) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2Input Photo: L
content
(p, c, l) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
Lcontent
(p,x, l) =1
2
X
i,j
(X l
i,j
� P l
i,j
)2
@Lcontent
(p,x, l)
@X l
i,j
= X l
i,j
� P l
i,j
X l
X li,j
Input Canvas: x
Width*Height (j)
Depth (i)
Width*Height (j)
Depth (i)
內容產生
• Backward Propaga7on Layer l’s Filter l Responses: X l
Input Canvas:
x
VGG19
@Lcontent
@x=
@Lcontent
@X l
@X l
x
x x� ⌘@L
content
@xUpdate Canvas
Learning Rate
內容產生
內容產生
VGG19
conv1_2 conv2_2 conv3_4 conv4_4 conv5_2 conv5_1
畫風產生
• ”Style” is posi7on-‐independent
style extrac7on
畫風產生
VGG19 畫作
G
G
Filter Responses Gram Matrix
Width*Height
Depth
Depth
Depth
Posi7on-‐ dependent
Posi7on-‐ independent
畫風產生
1. .5
.5
.5
1.
1. .5 .25 1.
.5 .25 .5
.25 .25
1. .5 1.
Width*Height
Depth
k1 k2
k1
k2
Depth
Depth
Layer l’s Filter Responses Gram Matrix
F l1
F l2
F l3
F l4
F l1
F l2
F l3
F l4
G
Gli,j = F l
i · F lj
Gl4,1 = F l
4 · F l1
= 1⇥ 1 + 0⇥ 0.5 + 0⇥ 0 + ...
= 1
畫風產生
VGG19 Filter
Responses Gram Matrix
計算 兩者 差異
G
G
畫風
畫布
修正差異
修正後
畫風產生
Lstyle(a,x, l) =1
2
X
i,j
(X li,j �Al
i,j)2
@Lstyle(a,x, l)
@F li,j
= ((F l)T (X l �Al))j,i
Layer l’s Filter Responses
Layer l’s Gram Matrix
Layer l’s Gram Matrix
F li,j
Ali,j X l
i,j
Input Artwork:
Input Canvas: a
x
畫風產生
畫風產生
VGG19
Conv1_1 Conv1_1 Conv2_1
Conv1_1 Conv2_1 Conv3_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
作品產生
Ltotal
= ↵Lcontent
+ �Lstyle
a
Lcontent
(p, c, l) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
x
x x� ⌘@L
total
@xx
Filter Responses VGG19
Lcontent
(p,x)
Lstyle(a,x)
Gram Matrix
作品產生
VGG19 VGG19
Lcontent
(p,x) Lstyle(a,x)
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
Conv4_2
Ltotal
= ↵Lcontent
+ �Lstyle
作品產生
作品展示
• 內容 v.s. 畫風 • 不同起始狀態 • 不同VGG Layers • 素描、水彩 • 詩中有畫、畫中有詩
內容 v.s. 畫風
0.15 0.05
0.02 0.007
↵
�
不同起始狀態 noise 0.9 *noise + 0.1*photo photo
不同VGG Layers
Conv1_1 Conv2_1
Conv1_1 Conv2_1 Conv3_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
↵
�= 0.002
素描、水彩
詩中有畫、畫中有詩
延伸閱讀 • A Neural Algorithm of Ar7s7c Style – hAp://arxiv.org/abs/1508.06576
• Texture Synthesis Using Convolu7onal Neural Networks – hAp://arxiv.org/abs/1505.07376
• Convolu7onal Neural Network – hAp://cs231n.github.io/convolu7onal-‐networks/
• Neural Network Back Propaga7on – hAp://cpmarkchang.logdown.com/posts/277349-‐neural-‐network-‐backward-‐propaga7on
• 電腦賦詩: – hAp://www.slideshare.net/ckmarkohchang/computa7onal-‐poetry
程式碼
• Python Tensorflow – hAps://github.com/ckmarkoh/neuralart_tensorflow
• Python Theano – hAps://github.com/woonketwong/ar7fy
• Python Theano (ipython notebook) – hAps://github.com/Lasagne/Recipes/blob/master/examples/styletransfer/Art%20Style%20Transfer.ipynb
• Python deeppy – hAps://github.com/andersbll/neural_ar7s7c_style
圖片來源
• hAp://www.taipei-‐101.com.tw/upload/news/201502/2015021711505431705145.JPG
• hAps://github.com/andersbll/neural_ar7s7c_style/blob/master/images/starry_night.jpg?raw=true
特別感謝
• 臺大資工imlab