Kaz Sato, Evangelist, Google at MLconf ATL 2016

Machine Intelligence at Google ScaleML APIs, TensorFlow and Cloud ML

+Kazunori Sato@kazunori_279

Kaz Sato

Staff Developer AdvocateTech Lead for Data & AnalyticsCloud Platform, Google Inc.

What we’ll cover

What is Neural Network and Deep Learning

Machine Learning use cases at Google services

Externalizing the power with ML APIs

TensorFlow: the open source library for ML

TensorFlow in the Wild

Distributed training and prediction with Cloud ML

What is Neural Network and Deep Learning

Neural Network is a function that can learn

xn

> b?

w1

wn

x2

x1

Inspired by the behavior of biological neurons

How do you classify them?

weights

bias(threshold)

Programmers need to specify the parameters

Let’s see how neural network solves the problem

The computer tries to find the best parameters

A neuron classifies a data point into two kinds

Gradient Descent: adjusting the params gradually to reduce errors

From: Andrew Ng

How do you classify them?

What we see What the computer “sees”

28 x 28 gray scale image =

784 numbers

input vector (pixel data)

output vector (probability)

How do you classify them?

More neurons = More features to extract

Hidden Layers:

mapping inputs to

a feature space,

classifying with

a hyperplaneFrom: Neural Networks, Manifolds, and Topology, colah's blog

How about this?

More hidden layers = More hierarchies of features

How about this?

We need to go deeper neural network

From: Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations, Honglak Lee et al.

From: mNeuron: A Matlab Plugin to Visualize Neurons from Deep Models, Donglai Wei et. al.

Machine Learning use casesat Google services

25

signalfor Search ranking,

out of hundreds

improvementto ranking quality

in 2+ years

#3 #1

Search

machine learning for search engines

RankBrain: a deep neural network for search ranking

WaveNet by Google DeepMind

28

[glacier]

Google Photos

28

29

Smart reply in Inbox by Gmail

10%of all responses sent on mobile

http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-worlds-most.html

Saved Data Center cooling energy for 40%Improved Power Usage Effectiveness (PUE) for 15%

32

AndroidAppsGmailMapsPhotosSpeechSearchTranslationYouTubeand many others ...

Used across products:

2012 2013 2014 2015

Deep Learning usage at Google

Externalizing the powerwith ML APIs

TensorFlow Cloud Machine Learning ML API

Easy-to-Use, for non-ML engineers

Customizable, for Data Scientists

Machine Learning products from Google

Image analysis with pre-trained models

No Machine Learning skill required

REST API: receives an image and returns a JSON

General Availability

Cloud Vision API

Confidential & ProprietaryGoogle Cloud Platform 36

FacesFaces, facial landmarks, emotions

OCRRead and extract text, with support for > 10 languages

LabelDetect entities from furniture to transportation

LogosIdentify product logos

Landmarks & Image PropertiesDetect landmarks & dominant color of image

Safe SearchDetect explicit content - adult, violent, medical and spoof

3737

Demo

Pre-trained models. No ML skill required

REST API: receives audio and returns texts

Supports 80+ languages

Streaming or non-streaming

Public Beta - cloud.google.com/speech

Cloud Speech API

Confidential & ProprietaryGoogle Cloud Platform 39

Features

Automatic Speech Recognition (ASR) powered by deep learning neural networking to power your applications like voice search or speech transcription.

Recognizes over 80 languages and variants with an extensive vocabulary.

Returns partial recognition results immediately, as they become available.

Filter inappropriate content in text results.

Audio input can be captured by an application’s microphone or sent from a pre-recorded audio file. Multiple audio file formats are supported, including FLAC, AMR, PCMU and linear-16.

Handles noisy audio from many environments without requiring additional noise cancellation.

Audio files can be uploaded in the request and, in future releases, integrated with Google Cloud Storage.

Automatic Speech Recognition Global Vocabulary Inappropriate Content Filtering

Streaming Recognition

Real-time or Buffered Audio Support Noisy Audio Handling Integrated API

4040

Demo

Pre-trained models. No ML skill required

REST API: receives text and returns analysis results

Supports English, Spanish and Japanese

Public Beta - cloud.google.com/natural-language

Cloud Natural Language API

Confidential & ProprietaryGoogle Cloud Platform 42

Features

Extract sentence, identify parts of speech and create dependency parse trees for each sentence.

Identify entities and label by types such as person, organization, location, events, products and media.

Understand the overall sentiment of a block of text.

Syntax Analysis Entity Recognition

Sentiment Analysis

4343

Demo

TensorFlow:An open source library forMachine Intelligence

Google's open source library for

machine intelligence

tensorflow.org launched in Nov 2015

Used by many production ML projects

What is TensorFlow?

Sharing our tools with researchers and developers around the world

repositoryfor “machine learning”

category on GitHub

#1

Released in Nov. 2015

From: http://deliprao.com/archives/168

Before

Hire Data Scientists↓

Understand the math model↓

Impl with programming code↓

Train with single GPU↓

Build a GPU cluster↓

Train with the GPU cluster↓

Build a prediction serveror Impl mobile/IoT prediction

After

Easy network design and impl

↓

Train with single machine

↓

Train on the cloud

↓

Prediction on the cloud

or mobile/IoT devices

many peoplestuck here

# define the networkimport tensorflow as tfx = tf.placeholder(tf.float32, [None, 784])W = tf.Variable(tf.zeros([784, 10]))b = tf.Variable(tf.zeros([10]))y = tf.nn.softmax(tf.matmul(x, W) + b)

# define a training stepy_ = tf.placeholder(tf.float32, [None, 10])xent = -tf.reduce_sum(y_*tf.log(y))step = tf.train.GradientDescentOptimizer(0.01).minimize(xent)

TensorBoard: visualization tool

Portable and ScalableTraining on:

Mac/Windows

GPU server

GPU cluster / Cloud

Prediction on:

Android and iOS

RasPi and TPU

Distributed Training with TensorFlow

TensorFlow in the Wild(or democratization of deep learning)

TensorFlow

powered

Fried Chicken

Nugget Server

From: http://www.rt-net.jp/karaage1/

TensorFlow powered Cucumber Sorter

From: http://workpiles.com/2016/02/tensorflow-cnn-cucumber/

Autonomous Driving

of RasPi carwith Inception 3on TensorFlow

From: https://github.com/zxzhijia/GoPiGo-Driven-by-Tensorflow

TV popstar classifierwith 95% accuracy

From: http://memo.sugyan.com/entry/2016/06/14/220624

TensorFlow＋

RasPi forsorting garbages

From: https://techcrunch.com/2016/09/13/auto-trash-sorts-garbage-automatically-at-the-techcrunch-disrupt-hackathon/

TensorFlow +

Drones

for counting trucks

From: http://www.brainpad.co.jp/news/2016/09/02/3454

From: http://otoro.net/Generative Arts with TensorFlow

Distributed Training and Prediction with Cloud ML

From: Andrew Ng

The Bigger, The Better

The Challenge: Computing Power

DNN requires large training datasets

Large models doesn't fit into a GPU

Requires try-and-errors to find the best design, configs and params

↓Need to spend a few days or

weeks to finish a training

GPUs run at nanosecondsGPU cluster needs microsec network

Enterprise

Google Cloud is

The Datacenter as a Computer

Jupiter network

10 GbE x 100 K = 1 Pbps

Consolidates servers with

microsec latency

Borg

No VMs, pure containers

10K - 20K nodes per Cell

DC-scale job scheduling

CPUs, mem, disks and IO

Distributed Training with TensorFlow

by data parallelism

split data,

share model

● CPU/GPU scheduling

● Communications

○ Local, RPC, RDMA

○ 32/16/8 bit quantization

● Cost-based optimization

● Fault tolerance

Distributed Systems for Large Neural Network

What's the scalability of Google Brain?

"Large Scale Distributed Systems for Training Neural

Networks", NIPS 2015

○ Inception / ImageNet: 40x with 50 GPUs

○ RankBrain: 300x with 500 nodes

Fully managed distributed training and prediction

Supports custom TensorFlow graphs

Integrated with Cloud Dataflow and Cloud Datalab

Limited Preview - cloud.google.com/ml

Cloud Machine Learning (Cloud ML)

7272

Ready to use Machine Learning models

Use your own data to train models

Cloud Vision API

Cloud Speech API

Cloud Translate API

Cloud Machine Learning

Develop - Model - Test

Google BigQuery

Stay Tuned….

Cloud Storage

Cloud Datalab

NEW

Alpha

GA BetaGA

AlphaGA

GA

Tensor Processing Unit

ASIC for TensorFlow

Designed by Google

10x better perf / watt

latency and efficiency

bit quantization

TPU on Production

RankBrain

AlphaGo

Google Photos

Speech

and more

Thank you!

Links & Resources

Large Scale Distributed Systems for Training Neural Networks, Jeff Dean and Oriol Vinals

Cloud Vision API: cloud.google.com/vision

Cloud Speech API: cloud.google.com/speech

TensorFlow: tensorflow.org

Cloud Machine Learning: cloud.google.com/ml

Cloud Machine Learning: demo video