31

Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

  • Upload
    mlconf

  • View
    81

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017
Page 2: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Dynamic Deep Learning A paradigm shift in AI research and Tools

Soumith Chintala Facebook AI Research

Page 3: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Active Research & Future AI

Tools for AI keeping up with change

Overview

Page 4: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Today's AI DenseCap by Justin Johnson & group https://github.com/jcjohnson/densecap

Page 5: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Today's AI

DeepMask by Pedro Pinhero & group

Page 6: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Today's AIMachine Translation

Page 7: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Today's AI

Text Classification (sentiment analysis etc.) Text Embeddings

Graph embeddings Machine Translation

Ads ranking

Page 8: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Data

BatchNorm

ReLU

Conv2d

Model

ObjectiveTrain Model

Today's AI Future AI Tools for AI

Today's AI

Page 9: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Data

BatchNorm

ReLU

Conv2d

Model

ObjectiveTrain Model

BatchNorm

ReLU

Conv2dDeploy & Use New

Data Prediction

Today's AI Future AI Tools for AI

Today's AI

Page 10: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Data

BatchNorm

ReLU

Conv2d ObjectiveTrain Model

BatchNorm

ReLU

Conv2dDeploy & Use New

Data Prediction

Today's AI Future AI Tools for AI

Static datasets + Static model structure

Today's AI

Page 11: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Data

BatchNorm

ReLU

Conv2d ObjectiveTrain Model

BatchNorm

ReLU

Conv2dDeploy & Use New

Data Prediction

Today's AI Future AI Tools for AI

Static datasets + Static model structure

Offline Learning

Today's AI

Page 12: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AISelf-driving Cars

Page 13: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AIAgents trained in many environments

Cars Video gamesInternet

Page 14: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AIDynamic Neural Networks

self-adding new memory or layers changing evaluation path based on inputs

Page 15: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AI

Live data

BatchNorm

ReLU

Conv2d Prediction

Continued Online Learning

Page 16: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AI

Sample-1

BatchNorm

ReLU

Conv2d

Prediction

Data-dependent change in model structure

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm ReLU

Conv2d

Page 17: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AI

Sample-2

BatchNorm

ReLU

Conv2d

Prediction

Data-dependent change in model structure

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm ReLU

Conv2d

Page 18: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AI

Sample

BatchNorm

ReLU

Conv2d

Prediction

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

Change in model-capacity at runtime

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

Page 19: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Current AI Research / Future AI

Sample

BatchNorm

ReLU

Conv2d

Prediction

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

Change in model-capacity at runtime

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

BatchNorm

ReLU

Conv2d

Page 20: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

A next-gen framework for AI • Interop with many dynamic environments

- Connecting to car sensors should be as easy as training on a dataset - Connect to environments such as OpenAI Universe

•Dynamic Neural Networks - Change behavior and structure of neural network at runtime

•Minimal Abstractions - more complex AI systems means harder to debug without a simple API

Page 21: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Tools for AI research and deploymentMany machine learning tools and deep learning frameworks

Page 22: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Tools for AI research and deploymentStatic graph frameworks Dynamic graph frameworks

Page 23: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

Today's AI Future AI Tools for AI

Dynamic graph Frameworks•Model is constructed on the fly at runtime •Change behavior, structure of model • Imperative style of programming

Page 24: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

PyTorch Autogradfrom torch.autograd import Variable

Page 25: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

from torch.autograd import Variable

x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))

PyTorch Autograd

Page 26: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

from torch.autograd import Variable

x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))

i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t())

MMMM

PyTorch Autograd

Page 27: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

from torch.autograd import Variable

x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))

i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h

MMMM

PyTorch Autograd

Page 28: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

from torch.autograd import Variable

x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))

i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h

Add

MMMM

PyTorch Autograd

Page 29: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

from torch.autograd import Variable

x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))

i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh()

Add

MMMM

Tanh

PyTorch Autograd

Page 30: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

from torch.autograd import Variable

x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))

i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh()

next_h.backward(torch.ones(1, 20))

Add

MMMM

Tanh

PyTorch Autograd

Page 31: Soumith Chintala, AI research engineer, Facebook, at MLconf NYC 2017

http://pytorch.org Released Jan 18th 20,000+ downloads 200+ community repos 3000+ user posts

With ❤ from