Embed Size (px)
Citation preview
René Donner Deep Learning
Overview
3
The (amazing) things Deep Learning can do
How does it work?
How can you start with DL?
René Donner Deep Learning
Roughly …
4
Deep learning finds patternsin data corresponding tohigh-level, abstract concepts
What can it do?
René Donner Deep Learning
What it can be used for
6
Image recognition
Text understanding, translation
Voice recognition
Playing video games
Driving cars
…
René Donner Deep Learning
Image recognition
7
René Donner Deep Learning
Scene labeling
8
http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html
René Donner Deep Learning
Text recognition
9
http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python/
Large-Scale Deep Learning for Intelligent Computer Systems, Jeff Dean, Google, BayLearn 2015
René Donner Deep Learning
Text understanding
10
2013 Glove: Global Vectors for Word Representation, Jeffrey Pennington, Richard Socher and Christopher D. Manning
René Donner Deep Learning
Word embeddings
11
René Donner Deep Learning
Information extraction / Reasoning
12
MetaMind
René Donner Deep Learning
Some well know research groups
13
Stanford / BaiduAndrew Ng
NYU / FacebookYann LeCun
UToronto / GoogleGeoffrey Hinton
René Donner Deep Learning
NVIDIA
14
Images: NVIDIA website
How does it work?
René Donner Deep Learning
Difference to classic ML
16
http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
René Donner Deep Learning
Deep learning
17
http://theanalyticsstore.ie/deep-learning/
René Donner Deep Learning
Visualization
18
1. Layer
higher Layers
Emergence of Object-Selective Features in Unsupervised Feature Learning, Adam Coates, NIPS 2012
René Donner Deep Learning 19
http://theanalyticsstore.ie/deep-learning/ http://stats.stackexchange.com/questions/114385/what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma
René Donner Deep Learning 20
https://medium.com/machine-learning-world/how-to-debug-neural-networks-manual-dc2a200f10f2
René Donner Deep Learning
Optimization
21
Stochastic gradient descent
Automatic differentiation
blog.datumbox.com
René Donner Deep Learning
Local minima
22
Less problematic than thought - saddle points
https://ganguli-gang.stanford.edu/figures/14.Saddlepoint.jpg
René Donner Deep Learning
Deep learning
23
Low level features of color images
https://www.coursera.org/course/neuralnets
René Donner Deep Learning
Deep learning
24
http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf
René Donner Deep Learning
Inception topologies
25
ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
René Donner Deep Learning
Network Aims
26
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
Classification accuracy
Inference speed (e.g. for video)
Size (mobile devices)
Energy per prediction (battery)
René Donner Deep Learning
Model Zoos
27
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
Readily trained networks
Transfer learning - adapt to your task
ONNX exchange format
René Donner Deep Learning
MNIST - Code Demo
28
René Donner Deep Learning
MNIST
29
http://deeplearning4j.org/rbm-mnist-tutorial.html
René Donner Deep Learning
Deep learning - why does it work?
30
Can cope with huge amounts of data
Learns small invariances
Overcomplete, sparse, representations
Learn Embedding
Lots of data
Recent advance: it is actually computable!
René Donner Deep Learning
Deep learning - pros
31
Not-domain specific
Supervised / Semi-supervised / Unsupervised
Classification / regression in last layer
Simple math
Hip
René Donner Deep Learning
Deep learning - cons
32
Lots of meta-parameters
Needs a lot of data
Very compute intensive
Hip
Getting started with DL
René Donner Deep Learning
Frameworks
34
Many different DL toolboxes
Efficiency important (GPU)
Attention to numerical issues
René Donner Deep Learning
Frameworks
35
Caffehttp://caffe.berkeleyvision.org/Plain text filesFastest CNN, GPU
Kerashttps://github.com/fchollet/kerasPython, on top of Theano
TensorFlowhttp://tensorflow.org/Python, by Google
MXNethttps://github.com/dmlc/mxnetPython, R, Julia
Slid
e fro
m c
affe
tuto
rial
René Donner Deep Learning
Tensorflow
36
General gradient descent library
René Donner Deep Learning
Tutorials
37
Stanford tutorialhttps://deeplearning.stanford.edu/wiki/index.php/UFLDL_TutorialMatlab code snippets
videolectures.nethttp://videolectures.net/deeplearning2015_montreal/
courserahttps://www.coursera.org/course/neuralnets
René Donner Deep Learning
Practical hints
38
Bengio ArxivPractical Recommendations for Gradient-Based Training of Deep Architectureshttp://arxiv.org/abs/1206.5533http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
Kaggle http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challengehttp://benanne.github.io/2014/04/05/galaxy-zoo.html
Relevant conferences NIPS (https://sites.google.com/site/deeplearningworkshopnips2013/accepted-papers)CVPR, ICMLMany interesting papers on arxiv.org
René Donner Deep Learning
Current research topics
39
Parallelization
What is deep learning, actually?
Alternative, faster, simpler methods
Multi-domain, transfer learning
Generative Adversarial Networks (GANs)
