Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextﬂow.com René Donner...

Co-Founder & CTOrene.donner@contextflow.com

René Donner

Deep Learning – an Overview

René Donner Deep Learning

Overview

The (amazing) things Deep Learning can do

How does it work?

How can you start with DL?

Roughly …

Deep learning finds patternsin data corresponding tohigh-level, abstract concepts

What can it do?

What it can be used for

Image recognition

Text understanding, translation

Voice recognition

Playing video games

Driving cars

Image recognition

Scene labeling

http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html

Text recognition

http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python/

Large-Scale Deep Learning for Intelligent Computer Systems, Jeff Dean, Google, BayLearn 2015

Text understanding

2013 Glove: Global Vectors for Word Representation, Jeffrey Pennington, Richard Socher and Christopher D. Manning

Word embeddings

Information extraction / Reasoning

MetaMind

Some well know research groups

Stanford / BaiduAndrew Ng

NYU / FacebookYann LeCun

UToronto / GoogleGeoffrey Hinton

NVIDIA

Images: NVIDIA website

How does it work?

Difference to classic ML

http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html

Deep learning

http://theanalyticsstore.ie/deep-learning/

Visualization

1. Layer

higher Layers

Emergence of Object-Selective Features in Unsupervised Feature Learning, Adam Coates, NIPS 2012

René Donner Deep Learning 19

http://theanalyticsstore.ie/deep-learning/ http://stats.stackexchange.com/questions/114385/what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma

René Donner Deep Learning 20

https://medium.com/machine-learning-world/how-to-debug-neural-networks-manual-dc2a200f10f2

Optimization

Stochastic gradient descent

Automatic differentiation

blog.datumbox.com

Local minima

Less problematic than thought - saddle points

https://ganguli-gang.stanford.edu/figures/14.Saddlepoint.jpg

Deep learning

Low level features of color images

https://www.coursera.org/course/neuralnets

Deep learning

http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf

Inception topologies

ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky

“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015

Network Aims

Classification accuracy

Inference speed (e.g. for video)

Size (mobile devices)

Energy per prediction (battery)

Model Zoos

Readily trained networks

Transfer learning - adapt to your task

ONNX exchange format

MNIST - Code Demo

http://deeplearning4j.org/rbm-mnist-tutorial.html

Deep learning - why does it work?

Can cope with huge amounts of data

Learns small invariances

Overcomplete, sparse, representations

Learn Embedding

Lots of data

Recent advance: it is actually computable!

Deep learning - pros

Not-domain specific

Supervised / Semi-supervised / Unsupervised

Classification / regression in last layer

Simple math

Deep learning - cons

Lots of meta-parameters

Needs a lot of data

Very compute intensive

Getting started with DL

Frameworks

Many different DL toolboxes

Efficiency important (GPU)

Attention to numerical issues

Frameworks

Caffehttp://caffe.berkeleyvision.org/Plain text filesFastest CNN, GPU

Kerashttps://github.com/fchollet/kerasPython, on top of Theano

TensorFlowhttp://tensorflow.org/Python, by Google

MXNethttps://github.com/dmlc/mxnetPython, R, Julia

Tensorflow

General gradient descent library

Tutorials

Stanford tutorialhttps://deeplearning.stanford.edu/wiki/index.php/UFLDL_TutorialMatlab code snippets

videolectures.nethttp://videolectures.net/deeplearning2015_montreal/

courserahttps://www.coursera.org/course/neuralnets

Practical hints

Bengio ArxivPractical Recommendations for Gradient-Based Training of Deep Architectureshttp://arxiv.org/abs/1206.5533http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html

Kaggle http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challengehttp://benanne.github.io/2014/04/05/galaxy-zoo.html

Relevant conferences NIPS (https://sites.google.com/site/deeplearningworkshopnips2013/accepted-papers)CVPR, ICMLMany interesting papers on arxiv.org

Current research topics

Parallelization

What is deep learning, actually?

Alternative, faster, simpler methods

Multi-domain, transfer learning

Generative Adversarial Networks (GANs)

Co-Founder & CTOrene.donner@contextflow.com

René Donner

Deep Learning – an Overview

Deep Learning – an Overview · Co-Founder & CTO rene.donner@contextﬂow.com René Donner...

Documents

Donner - Deep Learning - Overview and practical aspects

1 Human Activity Recognition with Deep Learning: Overview

Deep learning in neural networks: An overview...∂E = ∂ ∂ ∆ =,..., ∂ ∂:= −; ∈,) ′ (); ∈in ,)

A survey of traditional and deep learning-based feature ... · deep learning in computer vision, the reader is referred to [86] and for a general deep learning overview to [79]. Deep

An Overview of Multi-Task Learning in Deep Neural Networks

Deep Learning in Neural Networks: An Overviewpeople.idsia.ch/~juergen/DeepLearning2July2014.pdf · Deep Learning in Neural Networks: An Overview ... numerous neural network experts

Deep Learning in Practice - lri.frgcharpia/deeppractice/2020/... · Introduction Overview Context I Deep learning: impressive results in the machine learning literature I yet di cult

Deep Reinforcement Learning with VizDoomFirst-Person …3 Deep Reinforcement Learning Overview General reinforcement learning goal is to learn optimal policy for an agent, which maximizes

Paper overview: "Deep Residual Learning for Image Recognition"

Deep Learning in Medical Imaging: General Overview...571 Deep Learning in Medical Imaging kjronline.org Korean J Radiol 18(4), Jul/Aug 2017 Deep learning is a part of ML and a special

Deep learning for medical image processing: overview

Deep Learning for Medical Image Processing: Overview ... · Deep Learning for Medical Image Processing: Overview, Challenges and Future Muhammad Imran Razzak, Saeeda Naz and Ahmad

Deep Learningcs229.stanford.edu/notes-spring2019/cs229-notes-deep_learning.pdf · Deep Learning We now begin our study of deep learning. In this set of notes, we give an overview

Lecture 3: Overview of Deep Learning System · Lecture 3: Overview of Deep Learning System CSE599W: Spring 2018. The Deep Learning Systems Juggle We won’t focus on a specific one,

Deep Learning for Arti cial General Intelligence · Overview 1 Introduction Abstract Relevance to AGI Universal Intelligence De ning Deep Learning 2 Learning Unsupervised Learning

Overview of High-Performance Deep Learning (HiDL) Projecthibd.cse.ohio-state.edu/static/media/talks/slide/hidl... · 2018-11-15 · Overview of High-Performance Deep Learning (HiDL)

Introductory Overview Lecture The Deep Learning Revolution ...rsalakhu/talks/JSM_part2.pdfThe Deep Learning Revolution Part II: Optimization, Regularization Russ Salakhutdinov Machine

Overview of Deep Learning Stack at NERSC · Overview of Deep Learning Stack at NERSC Wahid Bhimji, Mustafa Mustafa User Training Jan/2019. Deep learning Stack 2. ... Deep Learning

Lecture 3: Overview of Deep Learning Systemdlsys.cs.washington.edu/pdf/lecture3.pdf · 2018-12-26 · Lecture 3: Overview of Deep Learning System CSE599W: Spring 2018. The Deep Learning

Deep Learning I - Korea Universitydsba.korea.ac.kr/wp/wp-content/seminar/Deep learning/Deep Learning... · Deep Learning I Deep learning is research on learning models with multilayer