Deep Learning Explained - BGUmcv172/wiki.files/2017-06...Deep Learning Explained Dolev Pomeranz, Chief Architect Trax @ BGU 2017

Deep Learning ExplainedDolev Pomeranz, Chief Architect

Trax @ BGU

2017

We capture retail today. So you can see the opportunity of tomorrowIntellectual Property Information © 2016 Trax Image Recognition. All Rights Reserved


2

Intro to Trax



3

Traxs’ Business Application

3

Manual Audit

Slow and ExpensiveInconsistent untraceable

Fast and CheapConsistentTraceable

Trax Automatic Recognition Audit

‘Big Data’ for retail

AVAILABILITY SHARE OFSHELF

PRICING PROMOTIONALACTIVATIONS

COMPETITVEINSIGHTS

PLANOGRAMCOMPLIANCE

SHELF STANDARDS

../../../../../../../Desktop/Photo Recognition/Trax/%EB%8F%99%EC%98%81%EC%83%81/2013-Auditors manual counting Audit-20minutes.mp4

../../../../../../../Desktop/Photo Recognition/Trax/%EB%8F%99%EC%98%81%EC%83%81/Photo Recognition Audit-46 seconds.mp4

../../../../../../../Desktop/Photo Recognition/Trax/%EB%8F%99%EC%98%81%EC%83%81/Video Recognition Audit-20 seconds.mp4



4

Trax unlocks ‘Big Data’ for the retail industry

Scale of coverage Scale of the Data

• Visits• Scenes

• Images• Products


v


5

Welcome to Trax Universe



6

Trax’s Visual Challenges

# Classes

Fine-Grained Classification

Crowded Scene

Dynamic Classes



7

Trax AI Retail InfrastructureImages Actionable Insights

Trax AI Retail Infrastructure



8

Computational

power

Data Science

Models

Data Science

Engine

Deep Fine-Grained

Recognition Engine

Deep

Learning

Computational

power

Retail ‘Big Data’

Inside the Trax AI Retail Infrastructure



9

AI – Quest for learning



10

What can we learn

CV NLP SR RL



11

The AI revolution



12

The AI revolution


AI & Machine learning


What is learning?

𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 → 𝐺𝑒𝑛𝑒𝑟𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛


The power of Math

Source http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html

→𝐺𝑒𝑛𝑒𝑟𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛 𝐿𝑜𝑐𝑎𝑙𝑖𝑡𝑦 → 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒


v


16



17

Introduction to ‘Neural networks’


Biologically inspired?



The real Neuron



Simple and Complex cells (1981 Nobel – Hubel, Wiesel)



Place and Grid cells (2014 Nobel - O'Keefe, Moser, Moser)



Concept cell – Luke Skywalker cell


History of Neural Networks

• 1940’s – Complex behavior from a network of simple units [Hebb]

• 1950’s – Perceptron [Rosenblatt]

– Can implement NAND (universal gate)


Neural Networks

• The Neuron

𝑧 ∈ ℝ ֜ 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔

𝑧 = 𝑓 𝑊 ∙ 𝑋 + 𝑏


Neural Networks

Activation functions

𝑧 =1

1 + 𝑒− 𝑊∙𝑋+𝑏

𝑆𝑖𝑔𝑚𝑜𝑖𝑑

𝑧 = tanh 𝑊 ∙ 𝑋 + 𝑏

𝑇𝑎𝑛ℎ

𝑧 = max 0,𝑊 ∙ 𝑋 + 𝑏

𝑅𝑒𝑙𝑢


Neural Networks

• MLP - multilayer perceptron (fully connected)



27

Deep learning explained



28

Why should we explain?

Source: https://www.eff.org/ai/metrics



29





30

State of CV and AI just before DL

Andrej Karpathy blog

http://karpathy.github.io/ 2012/10/22etats/-fo-retupmoc-noisiv/



31



Deep Learning



32




33



Main properties

Deep & Huge

Architecture

Single

Internet & MobileAugmentation

Backpropagation

SGD

GPU

Transfer learning

Model

Data

Training



35

Data



36

Data – Internet & Mobile

Pope Francis Pope Benedict



37

Data – Underappreciated



38

Data – Augmentation + Synthesis

https://www.cs.tau.ac.il/~wolf/papers/repcounticcv.pdfhttps://sites.google.com/site/mrsdproject201415teamg/document

/softwarehttps://github.com/udacity/self-driving-car-sim



39

Data – Transfer Learning

https://medium.com/merantix/applying-deep-learning-to-real-world-problems-

ba2d86ac5837



40

Model



41

Model – Architecture

Convolutional layer



42


Representation learning



43


Pooling Layer



44

Model – Deep & Huge

Remember MLP



45


LeNet [1989]



46


AlexNet [2012]

(SuperVision)



47


GoogLeNet [2014]

(Inception)



48


http://www.topbots.com/a-brief-history-of-neural-network-architectures/



49

Model – Single VS Pipeline

• Traditional

• Deep



50

Training



51

Training – Backpropagation

How to minimize?

𝑓 𝑥 = 𝑦

argmin𝑥

𝑓 𝑥



52


Method 1 – Random search



53


Method 1 – Random search



54


Method 2 – Approx. Gradient Decent

𝑑𝑓 𝑥

𝑑𝑥= lim

ℎ→0

𝑓 𝑥 + ℎ − 𝑓 𝑥

ℎ



55



𝑑𝑓 𝑥

𝑑𝑥≈𝑓 𝑥 + ℎ − 𝑓 𝑥 − ℎ

2ℎ

Approximating the gradient – numerical analysis

𝐸𝑟𝑟𝑜𝑟:

𝑂 ℎ2



56



𝐿 =1

𝑁

𝑖=1

𝑁

𝐿𝑖 𝑥𝑖 , 𝑦𝑖 ,𝑊 + 𝜆𝑅 𝑊



57



𝛻𝑊𝐿 =𝑑𝐿 𝑤1𝑑𝑤1

, ⋯ ,𝑑𝐿 𝑤𝑚𝑑𝑤𝑚𝐺𝑟𝑎𝑑𝑖𝑒𝑛𝑡



58



Back to Neural nets

AlexNet → 𝑚 = 65,000,000



59



Remember the huge data set

→ 𝑁 = 14,000,000



60


Running time

𝐼𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑆𝑝𝑒𝑒𝑑 = 𝐹𝑜𝑟𝑤𝑎𝑟𝑑 𝑝𝑎𝑠𝑠 𝑠𝑝𝑒𝑒𝑑 ∗ 𝑁𝑒𝑤 𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑠 ∗ 𝐷𝑎𝑡𝑎 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒



61


Summary so far

Random Approx. GD

Code Simple Simple

New weights

passes

Fast 𝑂(1) Slow 𝑂(𝑚)

Iteration speed Slow 𝑂(𝑁) Slow 𝑂(𝑚 ∗ 𝑁)

Gradient None Approx.



62


Can we do better?



63


Calculus recap

The sums, products, and compositions of analytic functions are analytic

Any analytic function is infinitely differentiable

https://en.wikipedia.org/wiki/Function_composition



64


Neural network is a composition of analytical functions

= tanh 𝑊2,1 ∙ tanh 𝑊1,1 ∙ 𝑋 + 𝑏1,1 , tanh 𝑊1,2 ∙ 𝑋 + 𝑏1,2 , ⋯ + 𝑏2,1



65


Gradients using the Chain rule

𝑐

𝜕𝐿

𝜕𝑐

𝜕𝐿

𝜕𝑎=𝜕𝐿

𝜕𝑐

𝜕𝑐

𝜕𝑎

𝑎

𝑏

𝜕𝐿

𝜕𝑏=𝜕𝐿

𝜕𝑐

𝜕𝑐

𝜕𝑏

Assume known

Then:

And:



66


• Analytic functions– Exploiting structure

• Chain rule – Gradient based only on following neighbor neurons

• Backward pass– Propagating the gradient, caching calculations



67


Dynamic programming



68


𝐵𝑎𝑐𝑘𝑝𝑟𝑜𝑝𝑎𝑔𝑎𝑡𝑖𝑜𝑛 = 𝐶ℎ𝑎𝑖𝑛 𝑅𝑢𝑙𝑒 + 𝐷𝑦𝑛𝑎𝑚𝑖𝑐 𝑝𝑟𝑜𝑔𝑟𝑎𝑚𝑚𝑖𝑛𝑔

𝑚𝑎𝑡ℎ 𝑐𝑜𝑚𝑝𝑢𝑡𝑒𝑟 𝑠𝑐𝑖𝑒𝑛𝑐𝑒



69

Training – SGD

Great improvement but still slow

Remember the huge data set

→ 𝑁 = 14,000,000



70

Training – SGD

Simple solution − Stochastic Gradient Decent

• Each epoch permutated the data– stochastic

• Each iteration take a small constant subset (i.e., mini-batch)– gradient approximation



71

Training – SGD



72

Training – SGD

Algorithms comparison

Random Approx. GD Analytic SGD

Code Simple Simple Complex

New weights

passes

Fast 𝑂(1) Slow 𝑂(𝑚) Fast 𝑂(1)

Iteration speed Slow 𝑂(𝑁) Slow 𝑂(𝑚 ∗ 𝑁) Fast 𝑂(1)

Gradient None Approx. Exact



73

Training – Intermediate summary

Can we do even better?



74


Simple network classifying MNIST digits

https://cloud.google.com/blog/big-data/ 2017/01nrael/-wolfrosnet-dna-

peed-gninrael-tuohtiw-a-dhp



75


Working on mini-batches of 100 images





76


Working on mini-batches of 100 images





77


Using algebra matrix notations





78

Training – GPU

Someone already knows how to multiply matrices fast!



79

Training – GPU



80

Training – GPU

One GPU to rule them all



81

Training – Summary

𝐼𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑆𝑝𝑒𝑒𝑑 = 𝐹𝑜𝑟𝑤𝑎𝑟𝑑 𝑝𝑎𝑠𝑠 𝑠𝑝𝑒𝑒𝑑 ∗ 𝑁𝑒𝑤 𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑠 ∗ 𝐷𝑎𝑡𝑎 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒

BackpropagationGPU SGD



82

Recent research



83

One Model To Learn Them All

[16/6/2017] L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, J. Uszkoreit



84

See, Hear, and Read: Deep Aligned Representations

[3/6/2017] Y. Aytar, C. Vondrick, A. Torralba



85

Distill – Research Debt

http://distill.pub/2017/research-debt/



86

Open source


v


87

Open source libraries


v


88


Intellectual Property Information © 2016 Trax Image Recognition. All Rights Reserved.

This document and the information contained herein is confidential. You must hold this document and any oral information provided

in connection with this document in strict confidence; Any reproduction, retransmission, republication, translation, or other use of, all

or part of this document is expressly prohibited, unless prior written permission has been granted by Trax.

This document is provided for information purposes only for the exclusive use of the recipients to whom it is addressed and the

contents hereof are subject to change without notice. Whilst the information contained herein has been prepared in good faith, it is

not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law,

including implied warranties and conditions of merchantability, fitness for a particular purpose and/or in relation to the accuracy,

completeness or reliability of the information contained in this document or any oral information provided in connection herewith, or

any data it generates. Trax accepts no responsibility, obligation or liability (whether direct or indirect, in contract, tort or otherwise) in

relation to any of such information. Trax expressly disclaim any and all liability may be based on this document and any errors

therein or omissions therefrom; and no contractual obligations are formed either directly or indirectly by this document.

Trax, the Trax logo and other all other Trax trademarks, logos and service marks used in this document are the trademarks or

service marks of Trax and its affiliates. All other marks contained herein are the property of their respective owners. Trax has

intellectual property rights relating to technology that is described in this document.

We capture retail today,

So you can see the opportunity of tomorrow

Documents

Deep Learning Explained - BGUmcv172/wiki.files/2017-06...Deep Learning Explained Dolev Pomeranz, Chief Architect Trax @ BGU 2017