39
Deep Dive into deeplearn.js Kai Sasaki Treasure Data Inc

Deep dive into deeplearn.js

Embed Size (px)

Citation preview

Page 1: Deep dive into deeplearn.js

Deep Dive into deeplearn.jsKai Sasaki

Treasure Data Inc

Page 2: Deep dive into deeplearn.js

About me

Kai Sasaki - 佐々木 海 (@Lewuathe)

Software Engineer at Treasure Data Apache Hivemall Committer Hadoop, Spark, deeplearnjs Contributor

Page 3: Deep dive into deeplearn.js

What is deeplearn.js?

deeplearn.js is a deep learning library running on your browser accelerated by WebGL. Mainly developed by Google PAIR project.

https://research.googleblog.com/2017/08/harness-power-of-machine-learning-in.html

Page 4: Deep dive into deeplearn.js

Demo

https://deeplearnjs.org/ - SqueezeNet - Teachable Machine

Page 5: Deep dive into deeplearn.js

Similar Products

WebDNN - https://mil-tokyo.github.io/webdnn/ Keras.js - https://github.com/transcranial/keras-js

Page 6: Deep dive into deeplearn.js

Technology

- Written in TypeScript - Available in TypeScript and ES6 project - Accelerated by WebGL - Tested on Chrome and Firefox - Can run both training and inference

https://www.npmjs.com/~types

Page 7: Deep dive into deeplearn.js

Architecture

Graph: Representing computation data flow much like graph in TensorFlow etc Op: A graph node that represents a computation on tensor NDArray: Real data copied from/to CPU and GPU backed by Typed Array

Page 8: Deep dive into deeplearn.js

Computation Graph

Page 9: Deep dive into deeplearn.js

Graph

Node

Node

Node Tensor

Tensor

Tensor

Page 10: Deep dive into deeplearn.js

Graph

Node

Node

Node

Node

Node Tensor

Tensor

Tensor

Page 11: Deep dive into deeplearn.js

Graph

Node

Node

Node

Node

Node

Node

Tensor

Tensor

Tensor

Tensor

Tensor

add

multiply

conv2d

reluetc

Page 12: Deep dive into deeplearn.js

Operation

Op

Op

Op

Op

Op

Op

NDArray

NDArray

NDArray

NDArray

NDArray

Page 13: Deep dive into deeplearn.js

Kernel Implementation

Page 14: Deep dive into deeplearn.js

math

Web Browser

WebGL

Page 15: Deep dive into deeplearn.js

math

NDArrayMathCPUNDArrayMathGPU

Web Browser

WebGL

Page 16: Deep dive into deeplearn.js

math

math

NDArrayMathCPUNDArrayMathGPU

Web Browser

WebGL

kernelNDArray

Page 17: Deep dive into deeplearn.js

math

math

NDArrayMathCPUNDArrayMathGPU

Web Browser

WebGL

kernelNDArray

Conv2d softmax etc…

Graph

matmul

Page 18: Deep dive into deeplearn.js

Kernel Exampleimport {Array1D, NDArrayMathGPU, Scalar} from 'deeplearn';

const math = new NDArrayMathGPU();const a = Array1D.new([1, 2, 3]);const b = Scalar.new(2);

const result = math.add(a, b);

// Option 1: With async/await.// Caveat: in non-Chrome browsers you need to put this in an async function.console.log(await result.data()); // Float32Array([3, 4, 5])

// Option 2: With a Promise.result.data().then(data => console.log(data));

// Option 3: Synchronous download of data.// This is simpler, but blocks the UI until the GPU is done.console.log(result.dataSync());

Page 19: Deep dive into deeplearn.js

TensorFlow like Exampleconst graph = new Graph();// Make a new input in the graph, called 'x', with shape [] (a Scalar).const x: Tensor = graph.placeholder('x', []);// Make new variables in the graph, 'a', 'b', 'c' with shape [] and random// initial values.const a: Tensor = graph.variable('a', Scalar.new(Math.random()));const b: Tensor = graph.variable('b', Scalar.new(Math.random()));const c: Tensor = graph.variable('c', Scalar.new(Math.random()));

//…

// At this point the graph is set up, but has not yet been evaluated.// **deeplearn.js** needs a Session object to evaluate a graph.const math = new NDArrayMathGPU();const session = new Session(graph, math);

Page 20: Deep dive into deeplearn.js

TensorFlow Integration

Page 21: Deep dive into deeplearn.js

Port TensorFlow model

deeplearn.js can import TensorFlow model from checkpoint. Need to port the weights from checkpoint files.

https://deeplearnjs.org/demos/mnist/mnist.html

Page 22: Deep dive into deeplearn.js

Porting weights

$ python scripts/dump_checkpoints/dump_checkpoint_vars.py \ —model_type=tensorflow \ —output_dir=demos/mnist/ \ —checkpoint_file=/tmp/tensorflow/mnist/logs/fully_connected_feed/model.ckpt-1999

# Will save a lot of files # (one file per variable and manifest.json)

Page 23: Deep dive into deeplearn.js

Porting weights

import {CheckpointLoader} from 'deeplearn';

const varLoader = new CheckpointLoader('.');varLoader.getAllVariables().then(async vars => { const math = new NDArrayMathGPU(); const hidden1W = vars[‘hidden1/weights’] as Array2D; const hidden1b = vars[‘hidden1/biases’] as Array1D; //…

}

Page 24: Deep dive into deeplearn.js

GPU Acceleration

Page 25: Deep dive into deeplearn.js

WebGL

WebGL (Web Graphics Library) is a JavaScript API for rendering interactive 3D and 2D graphics within any compatible web browser without the use of plug-ins.

- Mozilla Developer Network

Page 26: Deep dive into deeplearn.js

WebGL

- Provides the standard API - Implemented by major web browsers such as Chrome, Firefox, Safari and Opera - Hardware Agnostic

Page 27: Deep dive into deeplearn.js

WebGL Graphic Pipeline

https://www.tutorialspoint.com/webgl/webgl_graphics_pipeline.htm

Page 28: Deep dive into deeplearn.js

WebGL Graphic Pipeline

http://math.hws.edu/graphicsbook/c6/s4.html

Page 29: Deep dive into deeplearn.js

WebGL

1. deeplearnjs compiles GLSL shader 2. Send input NDArray as texture into GPU 3. Run a fragment shader 4. Fetch result NDArray from GPU

Page 30: Deep dive into deeplearn.js

1. Compile Shader

float unaryOperation(float x) {return (x < 0.0) ? 0.0 : x;

}

void main() {float x = getAAtCoords(); // Sample a value from a texturefloat y = unaryOperation(x);

setOutput(y);}

Page 31: Deep dive into deeplearn.js

2. Upload NDArray to GPU

gl.bindTexture(gl.TEXTURE_2D, texture);gl.texSubImage2D( gl.TEXTURE_2D, 0, 0, 0, width, height, textureFormat, getTextureType(gl), data);

// Texture format is RGBA// Texture type is gl.UNSIGNED_BYTE or gl.FLOAT

gl.bindTexture(gl.TEXTURE_2D, null);

Page 32: Deep dive into deeplearn.js

3. Run shader program

gl.bindBuffer(gl.ARRAY_BUFFER, buffer);

// Map buffer data to texture coordinationgl.vertexAttribPointer( loc, arrayEntriesPerItem, gl.FLOAT, false, itemStrideInBytes, itemOffsetInBytes);

// Draw somethinggl.drawElements(gl.TRIANGLES, 6, gl.UNSIGNED_SHORT, 0);

Page 33: Deep dive into deeplearn.js

4. Download the result

getBufferSubDataAsyncExtension.getBufferSubDataAsync( gl2.PIXEL_PACK_BUFFER, 0, downloadTarget);

// Download as Typed Array -> NDArray// Typed Array is backed by raw buffer memory (ArrayBuffer) // accessed through views (e.g. Float32Array)

https://www.khronos.org/registry/webgl/extensions/WEBGL_get_buffer_sub_data_async/

Page 34: Deep dive into deeplearn.js

Future Works

Page 35: Deep dive into deeplearn.js

Automatic TensorFlow to deeplearn.js

Direct importing from TensorFlow GraphDef format makes model importing easier.

https://deeplearnjs.org/docs/roadmap.html

Page 36: Deep dive into deeplearn.js

Eager Mode

Eager execution mode enables us to run computation without Graph definition.

Similar to TensorFlow eager mode.

Page 37: Deep dive into deeplearn.js

Decoupling NDArray from storage mechanism

NDArray is now tightly coupled with storage layer (GPU texture)

Tracking NDArray to avoid memory leak often becomes hard work.

Page 38: Deep dive into deeplearn.js

Great Demos

One of the reason we use deeplearn.js is that it enables us to create interesting application by using deep learning easily.

deeplearn.js is now enhancing demos to show the potential of deep learning on Web Browser.

Page 39: Deep dive into deeplearn.js

Patches Welcome!