Classify MNIST using Convolutional Neural NetworkL0444.000300 Introduction of Artificial Intelligence

May 31, 2018

Yu-Jung HeoBioIntelligence Laboratory,

School of Computer Science and Engineering,Seoul National University

http://bi.snu.ac.kr

Contents Future schedule Review Convolutional Neural Network

Convolution Convolution Layer Pooling Layer Feature visualization

Hands on experience

2

Download ipython notebook:https://www.dropbox.com/s/0zg21futo4iwkte/practice8_cnn.ipynb?dl=0

Future Schedule May 31 (today)

Classify MNIST using Convolutional Neural Network June 5 June 7

Regularization technique June 12

Final exam June 17

Due date of final project

important to do final project!

3

Review (1/5) MNIST (Hand-written digits)

Label : 0~9 Image shape : 28x28 Gray scale #data : Training 60,000 (6,000 / class) / Test 10,000 (1,000 / class)

4

Review (2/5) Load data Convert class labels from scalars to one-hot vectors

Load batch Initialize variables

Each epoch (to max_epoch) Each batch (to max_batch)

Load batch Training model by minimizing loss function Calculate accuracy Model save

5

Review (3/5) Model

Define Input using placeholder

Model: MultiLayer Perceptron (MLP)

Output passed multilayer perceptron fed into input

6

Review (4/5) Cross-entropy

[model]

0

0.2

0.1

0.1

0.4

0.5

0.4

0.4

0.2

𝐼𝐼1 𝐼𝐼2 𝐼𝐼3

𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆(𝑦𝑦)𝑖𝑖 =exp(𝑦𝑦𝑖𝑖)∑𝑖𝑖 exp(𝑦𝑦𝑖𝑖)

7

Review (5/5) Cross-entropy

Prediction True label Correct?0.3 0.3 0.4 0 0 1 Yes0.3 0.4 0.3 0 1 0 Yes0.1 0.2 0.7 1 0 0 No

Prediction True label Correct?0.1 0.2 0.7 0 0 1 Yes0.1 0.7 0.2 0 1 0 Yes0.3 0.4 0.3 1 0 0 No

𝐻𝐻𝑦𝑦 𝑦𝑦𝑦 = −�𝑖𝑖

𝑦𝑦𝑖𝑖log(𝑦𝑦𝑦𝑖𝑖) Cross entropy loss: 1.38 vs. 0.64

8

Convolutional Neural Networks (CNN)

9

Convolution (1/3) weighted average of the function 𝑆𝑆(𝜏𝜏) at the moment 𝑆𝑆 where the weighting is

given by 𝑔𝑔(−𝜏𝜏) simply shifted by amount 𝑆𝑆 https://en.wikipedia.org/wiki/Convolution

10

Convolution (2/3) Try it by yourself!

-1 0.5 1 -10 1 -2 0

0.4 0 2 0.20.5 1 -1 0

0 1 01 -4 10 1 0

* ?

11

Convolution (3/3) Effects on convolution

https://docs.gimp.org/en/plug-in-convmatrix.html 12

Convolutional Neural Networks(1/4) Convolution neural network

Convolutional layer Pooling layer Fully Connected layer (FC layer)

13

Convolutional Neural Networks(2/4) Classify images using MultiLayer Perceptron Problem?

14

Convolutional Neural Networks(3/4) Too many variations!

15

Convolutional Neural Networks(4/4) Representation learning that generates high-order abstract feature map

by combining the information contained in the input image repeatedly using “convolution” and “subsampling”

Learn Local connection using convolution Topological and geometrical local patterns in image data

Image 1 Image 2

16

Convolutional Layer (1/4)

17

Convolutional Layer (2/4)

Shared weights!

18

Convolutional Layer (3/4)

19

Convolutional Layer (4/4)

20

Pooling layer Pooling

Invariance to translation and rotation Reduce the number of parameters Max-pooling, average pooling, L2-norm pooling, …

21

Feature visualization of CNN

22

Hands on experience!

Open your ipython notebook!

23

Any questions?

24