BINARY DATA ADVENTURES IN BROWSER JAVASCRIPT

OR HILTCH / CTO @ STREAMRAIL

INTRO• Background • Evolution in the browser • The Binary Arsenal in JS (type system, inputs, outputs) • Solving Cool, Practical Problems

001101100111001001010101001000011

101010111010000010101001010101001

010100101110101010010101010100101

0101010111110010101000101011101001

01010101111010000001010101000011

010110010101010101111000010101010

TRADITIONALLY• Outside of JS:

Low level stuff: Compression (gzip/deflate), Bitfields (ACLs), Networks (CRC32), Graphics (algorithms, positioning), ...

• Inside of JS: Solving weird issues

$http.post('/someUrl', {score:42}). success(function(data, status, headers, config) { // not today :-( }). error(function(data, status, headers, config) { // data=Object {‘error’: ‘Float64 is not int’} }

$http.post('/someUrl', {score:42}). success(function(data, status, headers, config) { // not today :-( }). error(function(data, status, headers, config) { // data=Object {‘error’: ‘Float64 is not int’} }

typeof 3.14159 === typeof 3

In JS, all numbers are of type Float64 But all bitwise ops are Int32!

Bitwise OR: number | 0 Bitwise NOT: ~~number Shift: number >> 0

parseInt: parseInt(number, 10) floor: Math.floor(number) modulo: number - number % 1

BROWSERS, EVOLVED• Type system: Blob, ArrayBuffer, TypedArray, DataView • Inputs:

Input methods: XHR 2, File API, Canvas, WebSockets, WebRTC, ... • Outputs:

MSE, Canvas, WebSockets, WebRTC, WebGL, ...

WHAT MAKES TYPED ARRAYS FAST

• Passed to native interfaces completely AS IS (direct memory access) • Native “endianness” - watch out! • DataView adapter

Byte order 0x12345678 Hex inside a Uint32Array of 4

Little-endian: 78 56 34 12

12 34 56 78Big-endian:

STREAMING VIDEO WITH JS

• No plugins (Netflix: silverlight, Facebook: flash) • MSE: Pure JS DASH/HLS (YouTube) • DRM Support (EME) • MSE + WebRTC: Bittorrent, in the browser! (WebTorrent)

‘USE ASM’sys/libkern/strlen.c size_t strlen(const char *str) { const char *s; for (s = str; *s; ++s); return(s - str); }

asm.js function strlen(ptr) { ptr = ptr|0; var curr = 0; curr = ptr; while (MEM8[curr]|0 != 0) { curr = (curr + 1)|0; } return (curr - ptr)|0; }

a subset of JS for handling numbers faster

• OdinMonkey - AOT in FF (fallback to IonMonkey JIT) • Support by all major browsers (to an extent) • Typed Array used as “memory” • All add/sub are 32 bit (number | 0) • DI: function(stdlib, foreign, heap)

‘USE ASM’

KRIPKEN TO THE RESCUEC/C++ > LLVM > Emscripten > asm.js!

• Huge code bases (SQLite, BananaBread,J2ME VM, …)- close to impossible by hand

• Enjoy Clang/LLVM optimizations- decades of work done to optimize compiled code

• OpenGL > WebGL “for free” - directly map a lot of OpenGL ES 2.0 command to WebGL - not only for graphics rendering, also for GPU offloading

• Use Docker to run Emscripten - lots of software (emscripten, emscripten-fastcomp, emscripten-

fastcomp-clang, llvm clang) - build your own or use one from DockerHub

KRIPKEN TO THE RESCUE

emscripten in real life

• Obtain bytes of video (and audio) by downloading them

• Decode the video (and audio) using native decoders in the browser/OS

• Animate the frames to create the video experience

VIDEO TAG EMULATION

• Obtain bytes of video (and audio) by downloading them > XHR 2.0, response type “arraybuffer”

• Decode the video (and audio) using native decoders in the browser/OS > Cross compile a native decoder to JS using Emscripten

• Animate the frames to create the video experience > For each decoded pixel, create a 2D Texture, transform it to RGBA using a fragment shader, and render it with WebGL using canvas (“experimental-webgl”).

VIDEO TAG EMULATION

YUV -> RGB• Luma + (2 * Chroma) vs

3 * (Chroma + Luma)

• Different needs, different inputs

• Color space conversion heavily computational (floating point coefficients matrix multiplication)

WEBGL RASTERIZATION

function yuv2canvas(buffer, width, height) { var lumaSize = width * height; var chromaSize = lumaSize >> 2;

webGLCanvas.YTexture .fill(buffer.subarray(0, lumaSize));

webGLCanvas.UTexture .fill(buffer.subarray(lumaSize, lumaSize + chromaSize));

webGLCanvas.VTexture .fill(buffer.subarray(lumaSize + chromaSize, lumaSize + 2 * chromaSize));

webGLCanvas.drawScene(); }

4:2:0

=

+

precision highp float; varying highp vec2 vTextureCoord; uniform sampler2D YTexture; uniform sampler2D UTexture; uniform sampler2D VTexture; const mat4 YUV2RGB = mat4 ( 1.1643828125, 0 , 1.59602734375, -.87078515625, 1.1643828125, -0.39176171875, -0.81296875 , .52959375, 1.1643828125, 2.017234375 , 0 , -1.081390625, 0 , 0 , 0 , 1 );

void main(void) { gl_FragColor = vec4(texture2D(YTexture, vTextureCoord).x, texture2D(UTexture, vTextureCoord).x, texture2D(VTexture, vTextureCoord).x,

1) * YUV2RGB; }

<SCRIPT TYPE="X-SHADER/X-FRAGMENT">

LIVE DEMO

THANK YOU!

OR HILTCH / CTO @ STREAMRAIL