85
An Introduction to Convolutional Neural Networks Alessandro Giusti Dalle Molle Institute for Artificial Intelligence Lugano, Switzerland

An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

An Introduction toConvolutional Neural Networks

Alessandro GiustiDalle Molle Institute for Artificial Intelligence

Lugano, Switzerland

Page 2: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Sources & Resources

- Andrej Karpathy, CS231nhttp://cs231n.github.io/convolutional-networks/

- Ian Goodfellow et al., Deep Learninghttp://www.deeplearningbook.org/

- F. Chollet, Deep Learning with Pythonhttps://www.manning.com/books/deep-learning-with-python

- Jonathan Hui, CNN Tutorialhttps://jhui.github.io/2017/03/16/CNN-Convolutional-neural-network/

- Tim Demetters, Understanding Convolutionshttp://timdettmers.com/2015/03/26/convolution-deep-learning/

Some images in this presentation are extracted from the sources listed above

Page 3: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

What is a convolution?

Page 4: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

What is a convolution?

Page 5: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

How does a convolution look like?

1 3x3 kernel1 input map 1 output map

Page 6: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

What about multiple maps?

1 input map 1 output map1 3x3 kernel

Page 7: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

1 input map 2 output maps2 3x3 kernels

Page 8: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

3 input maps 2 output maps3x2 3x3 kernels

Page 9: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

3 input maps 2 output maps3x2 3x3 kernels

Page 10: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

3 input maps 2 output maps3x2 3x3 kernels

Page 11: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

3 input maps 2 output maps3x2 3x3 kernels

Quiz: how many parameters does this layer have?

Page 12: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

3 input maps 2 output maps3x2 3x3 kernels

= 54 ...

Page 13: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

3 input maps 2 output maps3x2 3x3 kernels

= 54 ... + 2 biases

Page 14: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

3 input maps 2 output maps3x2 3x3 kernels

= 54 ... + 2 biases= 56 trainable parameters (weights)

Page 15: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Details: padding

How many 3x3 patchesare fully contained in a5x5 map?

Page 16: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Details: padding

9: the output map is 3x3

Page 17: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Details: padding

• This is known as “valid” padding mode (default)

• An alternative pads the input map with zeros to yield a same-sized map

Page 18: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Details: striding

• Stride 1x1 is most frequently used: shift 1 pixel at a time patches are heavily overlapping

• Stride 2x2 skips one patch horizontally and vertically

Page 19: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Why convolutional layers?

• Sparse connectivity

• Parameter sharing

• Translation invariance

Page 20: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Sparse connectivity

Fully connected

3x1 convolutional

Page 21: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Sparse connectivity

Fully connected

3x1 convolutional

Page 22: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Receptive fields

Fully connected

3x1 convolutional

Page 23: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Receptive fields

3x1 convolutional

3x1 convolutional

Deeper neurons depend on wider patches of the input

Page 24: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Parameter sharing

Fully connected

3x1 convolutional

5x5 = 25 weights(+ 5 bias)

3 weights!(+ 1 bias)

Quiz: how many parameters does this layer have?

Page 25: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Translational invariance

Page 26: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Max pooling layers

Quiz: how many parameters does this layer have?

... on many maps?

Page 27: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Max pooling downsamples activation maps

Page 28: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

ExerciseInput:

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

map

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

38x38

38x38

Page 29: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Receptive fieldsInput

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

map

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

Page 30: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Receptive fieldsInput

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

map

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

?

?

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

?

How large is the receptive field of the black neuron?

Page 31: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Receptive fieldsInput

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

Co

nv

3x3

map

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

?

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

Co

nv

3x3

MP

2x2

22x22

How large is the receptive field of the black neuron?

13x13

Page 32: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Why convnets work

Convnets learn a hierarchy of translation-invariant spatial pattern detectors

Page 33: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

What are layers looking for?Data from a convnet trained on

ImageNet

Page 34: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Shallow layers respond to fine, low-level patterns

Page 35: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Intermediate layers ...

Page 36: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Deep layers respond to complex, high-level patterns

Page 37: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Detail: backprop with max pooling

The gradient is only routed through the input pixel that contributes to the output value; e.g.:

Gradient of • with respect to • = 0

Page 38: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

A typical architectureAs we move to deeper layers:• spatial resolution is reduced• the number of maps increasesWe search for higher-level patterns, and don’t care too much about their exact location.There are more high-level patterns than low-level details!

Page 39: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

A typical architecture

Extract high-level features from pixel data Classify

Page 40: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

We will manipulate 4D tensors

Images are represented in 4D tensors:Tensorflow convention:(samples, height, width, channels)

Page 41: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

The software stack

Page 42: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

What is Keras?

• A model-level library, providing high-level building blocks for developing deep-learning models.

• Doesn’t handle low-level operations such as tensor manipulation and differentiation.

• Relies on backends (such as Tensorflow)

• Allows full access to the backend

Page 43: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Why Keras?

Pros:

• Higher level fewer lines of code

• Modular backend not tied to tensorflow

• Way to go if you focus on applications

Cons:

• Not as flexible

• Need more flexibility? Access the backend directly!

Page 44: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

More about ConvNets

Alessandro GiustiDalle Molle Institute for Artificial Intelligence

Lugano, Switzerland

Page 45: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

1. UNDERFITTING & OVERFITTING ON OUR ROCK PAPER SCISSORS NET

Rock Paper Scissors

Page 46: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 47: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 48: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 49: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

1, 2, 4, 8

1

24

8 1

2

4

8

Page 50: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 51: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 52: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

8, 16

1632

1632

Page 53: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 54: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

16

32

Page 55: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

32,64

Page 56: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

64

128

12864

Page 57: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 58: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield
Page 59: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

The ML Pipeline (Chollet)

Page 60: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

2. VISUALIZATION TECHNIQUESRock Paper Scissors

Page 61: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Visualizing the weights of the net

...

We want to see this

Page 62: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Visualizing the weights of the net

11x11x3 filters (visualized in RGB) in the first convolutional layers

Page 63: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Visualizing the activations of intermediate layers

...

We want to see this

Page 64: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Visualizing the activations of intermediate layers

Page 65: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Visualizing the input that maximally activates some neurons

...

We want to compute (and see) the input that maximally activates this guy

Page 66: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Step 1

...

Compute the gradient of this with respect to the input

Page 67: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Step 2

...

Nudge the input accordingly: our guy will increase its activation

Page 68: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Goto step 1

...

Page 69: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Shallow layers respond to fine, low-level patterns

Page 70: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Intermediate layers ...

Page 71: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Deep layers respond to complex, high-level patterns

Page 72: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

USING PRETRAINED WEIGHTSStand on the shoulder of giants

Page 73: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Using pretrained weights

Step 1 Step 2 Step 3

Page 74: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Option 1

Co

nv

MP

Co

nv

MP

Co

nv

MPInput

Flat

ten

Den

se

Den

se

Ou

tpu

ts

Page 75: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Option 1

Co

nv

MP

Co

nv

MP

Co

nv

MPInput

Flat

ten

Den

se

Den

se

Ou

tpu

ts

Page 76: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Option 1

Co

nv

MP

Co

nv

MP

Co

nv

MPInput

Flat

ten

Save these featuresfor the whole training and testing datasets.

Then, train a new classifier that uses these features as input

Page 77: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Option 2

Co

nv

MP

Co

nv

MP

Co

nv

MPInput

Flat

ten

Den

se

Den

se

Ou

tpu

ts

Freeze Train only

Page 78: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Option 3

Co

nv

MP

Co

nv

MP

Co

nv

MPInput

Flat

ten

Den

se

Den

se

Ou

tpu

ts

Freeze Train onlyFinetune

Page 79: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

A MILE-HIGH OVERVIEW OF FULLY CONVOLUTIONAL NETWORKS FOR SEGMENTATION

Page 80: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Overall idea

Page 81: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

“Convolutionalization” of a dense layer

Page 82: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

SOME POSSIBLE PROJECTS

Page 83: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Deep Learning on vibration data for detecting fence violations

Page 84: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Deep Learning on wearable sensor data for robot control

Page 85: An Introduction to Convolutional Neural Networks...Details: padding •This is known as “valid” padding mode (default) •An alternative pads the input map with zeros to yield

Learning to predict errors in weather forecasts