Deep Learning in Computer Vision

Bikash SantraSenior Research Fellow

Electronics and Communication Sciences Unit

Indian Statistical Institute, Kolkata, India

What Computers “See”

Slide Credit: Ava Soleimany, MIT

Images Are Numbers

Slide Credit: Ava Soleimany, MIT

Images Are Numbers

Slide Credit: Ava Soleimany, MIT

Images Are Numbers

What is Image Processing?

1. Photometric Transformations

a) Sharpening

b) Smoothing

c) Contrast enhancement

d) Stretching

2. Geometric Transformations

1. Scaling

2. Rotations

3. Translation

3. Image Compression

4. And many more

Slide Credit: Ava Soleimany, MIT

What is Computer Vision?

1. Emulates human vision

2. Goal is to understand

images and its contents

Slide Credit: Ava Soleimany, MIT

Tasks in Computer Vision

Feature

Extraction

Slide Credit: Ava Soleimany, MIT

High Level Feature Detection

Slide Credit: Ava Soleimany, MIT

Manual Feature Extraction

Slide Credit: Ava Soleimany, MIT

Manual Feature Extraction

Slide Credit: Alexander Amini, MIT

Question 1

Slide Credit: Ava Soleimany, MIT

Question 2

Yes, we can! Using deep learning

Neural NetworksDeep

Neural Networks: Architectures

Slide Credit: Fei-Fei Li et al., Stanford University

Slide Credit: Alexander Amini, MIT

Example Problem

Slide Credit: Alexander Amini, MIT

Example Problem: Will I pass this class?

Slide Credit: Alexander Amini, MIT

Example Problem: Will I pass this class?

Slide Credit: Alexander Amini, MIT

Example Problem: Will I pass this class?

Slide Credit: Alexander Amini, MIT

Example Problem: Will I pass this class?

Slide Credit: Alexander Amini, MIT

Quantifying Loss

Slide Credit: Alexander Amini, MIT

Empirical Loss

Slide Credit: Alexander Amini, MIT

Binary Cross Entropy Loss

Slide Credit: Alexander Amini, MIT

Mean Squared Error Loss

Slide Credit: Alexander Amini, MIT

Mean Squared Error Loss

Slide Credit: Alexander Amini, MIT

Loss Optimization

Solved using

gradient descent

Convolutional Neural Network

Slide Credit: Abin - Roozgard

Introduction

Convolutional neural networks

Signal processing, Image processing

improvement over the multilayer perceptron

performance, accuracy and some degree of

invariance to distortions in the input images

Slide Credit: Ava Soleimany, MIT

FCNN for Image Processing

Slide Credit: Ava Soleimany, MIT

Using Spatial Structure

Slide Credit: Ava Soleimany, MIT

Using Spatial Structure

Slide Credit: Ava Soleimany, MIT

Applying Filters to Extract Features

Slide Credit: Ava Soleimany, MIT

Feature Extraction with Convolution

Slide Credit: Fei-Fei Li et al., Stanford

Convolutional Neural Networks

CNN is evolved basically to deal with images.

Slide Credit: Ava Soleimany, MIT

Components of CNN

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Image Source: http://cs231n.github.io/convolutional-networks/

Convolution

Operation

Demo

Slide Credit: Fei-Fei Li et al., Stanford

Convolution Layer

Slide Credit: Ava Soleimany, MIT

Introducing Non-Linearity

Slide Credit: Fei-Fei Li et al., Stanford

Pooling Layer

Slide Credit: Fei-Fei Li et al., Stanford

Pooling Layer

Slide Credit: Ava Soleimany, MIT

CNNs for Classification: Feature Learning

Slide Credit: Ava Soleimany, MIT

CNNs for Classification: Class Probabilities

Slide Credit: Ava Soleimany, MIT

CNNs: Training with Backpropagation

Deep Learning Resources

Name Language Link Note

Pylearn2 Python http://deeplearning.net/software/pylearn2/ A machine learning library built on Theano

Theano Python http://deeplearning.net/software/theano/ A python deep learning library

Keras Python https://keras.io/ A python deep learning library

Caffe C++ http://caffe.berkeleyvision.org/ A deep learning framework by Berkeley

Torch Lua http://torch.ch/ An open source machine learning framework

Overfeat Lua http://cilvr.nyu.edu/doku.php?id=code:start A convolutional network image processor

Deeplearning4j Java http://deeplearning4j.org/ A commercial grade deep learning library

Word2vec C https://code.google.com/p/word2vec/ Word embedding framework

GloVe C http://nlp.stanford.edu/projects/glove/ Word embedding framework

Doc2vec Chttps://radimrehurek.com/gensim/models/d

oc2vec.htmlLanguage model for paragraphs and documents

StanfordNLP Java http://nlp.stanford.edu/ A deep learning-based NLP package

TensorFlow Python http://www.tensorflow.org A deep learning based python library

PyTorch Python https://pytorch.org/ A deep learning based python library

Thank you