Unconventional webapps with gwt:elemental & html5

Unconventional webapps with GWT/Elemental, WebRTC &

WebGLAlberto Mancini

alberto@jooink.com

Blog: http://jooink.blogspot.com

Online Demo: http://www.jooink.com/experiments/DevFest2012

GWT 2.5 Elemental

Elemental is a new library for fast, lightweight, and "to the metal" web programming in GWT.

Elemental includes every HTML5 feature, ... DOM access ... WebGL, WebAudio, WebSockets, WebRTC, Web Intents, Shadow DOM, the File API, and more. ... we generate Java code directly from the WebIDL files used by JavaScript engines. (like DART)

Unconventional

... insomma qualcosa che non ci sembrava possibile fare con il solo browser, tipo accedere a devices attaccati al 'client', una webcam per esempio.

L'idea e' farsi una foto ...

WebRTC

WebRTC (Web Real-Time Communication) is an API definition being drafted by W3C and IETF.

The goal of WebRTC is to enable applications such as voice calling, video chat and P2P file sharing without plugins.

Specification & Up to date info: http://www.webrtc.org

Support: Chrome 22+, ... well work-in-progress at least for firefox.

what we need: getUserMedia (chrome !!)

WebRTC (with Elemental)Video in a canvas

final VideoElement videoElement = Browser.getDocument().createVideoElement(); final CanvasElement canvas =

Browser.getDocument().createCanvasElement(); final CanvasRenderingContext2D ctx2d = (CanvasRenderingContext2D)canvas.getContext("2d"); ... //repeatedly, e.g. in a Timer or RepeatingCommand ctx2d.drawImage(videoElement, 0, 0);

Take snapshot as easy as: Image img = new Image(canvas.toDataURL("png"));

WebRTC (with Elemental) binding <video/> to userMedia (search for facelogin on googlecode)

public void bindVideoToUserMedia(final VideoElement video, final EventListener l) { final Mappable map = (Mappable) JsMappable.createObject(); map.setAt("video", true); Browser.getWindow().getNavigator().webkitGetUserMedia(map, new NavigatorUserMediaSuccessCallback() {

public boolean onNavigatorUserMediaSuccessCallback(LocalMediaStream stream) {

setVideoSrc(video, stream); video.play(); ... }, new NavigatorUserMediaErrorCallback() {...}); }

private static native void setVideoSrc( VideoElement v, LocalMediaStream s) /*-{ v.src = window.webkitURL.createObjectURL(s); }-*/;

WebGL

WebGL (Web Graphics Library) is a JavaScript API for rendering interactive 3D graphics and 2D graphics within any compatible web browser without the use of plug-ins.

Specification: http://www.khronos.org/webgl/

Support: http://caniuse.com/webgl

Chrome 22+, FireFox 15+, Safari 6+, Opera 12+ (partial)

WebGL (with Elemental)Video in a canvas (3d rendering context) ... WebGLRenderingContext ctx3d = (WebGLRenderingContext)canvas.getContext("experimental-webgl");

Drawing is a bit harder: create a texture, create a background rectangle, use the video as a texture ctx3d.texImage2D(WebGLRenderingContext.TEXTURE_2D, 0, WebGLRenderingContext.RGBA, WebGLRenderingContext.RGBA, WebGLRenderingContext.UNSIGNED_BYTE, videoElement);

WebGL (with Elemental)shaders

Vertex Shader

precision mediump float;attribute vec2 a_position;attribute vec2 a_texCoord;uniform vec2 u_resolution;varying vec2 v_texCoord;void main() { vec2 zeroToOne = a_position / u_resolution; vec2 zeroToTwo = zeroToOne * 2.0; vec2 clipSpace = zeroToTwo - 1.0; gl_Position = vec4(clipSpace * vec2(1, -1), 1, 1); v_texCoord = a_texCoord;}

Fragment Shader

precision mediump float;uniform sampler2D u_image;varying vec2 v_texCoord;

void main() { gl_FragColor = texture2D(u_image, v_texCoord);}

from html5rocks.com's WebGL Fundamentals ... probably :o

WebGL (with Elemental)shaders

Fragment Shader...// Sepia tone// Convert to greyscale using NTSC weightings float grey = dot(texture2D(u_image, v_texCoord).rgb, vec3(0.299, 0.587, 0.114)); gl_FragColor = vec4(grey * vec3(1.2, 1.0, 0.8), 1.0);...// Negative vec4 texColour = texture2D(u_image, v_texCoord); gl_FragColor = vec4(1.0 - texColour.rgb, 1.0);

Fragment Shader

precision mediump float;uniform sampler2D u_image;uniform sampler2D SECOND_u_image;uniform float alpha;varying vec2 v_texCoord; void main(void) { vec4 prev = texture2D(SECOND_u_image, vec2(v_texCoord.s,1.0-v_texCoord.t)); vec4 cur = texture2D(u_image, v_texCoord); gl_FragColor = alpha*prev + (1.0-alpha)*cur;}

from http://r3dux.org/2011/06/glsl-image-processing/

WebGL (with Elemental)3D

WebGL e' una libreria 2D !!

Trasformazioni prospettiche (proiezioni), frustum, modelview vanno implementate.

VertexShader: applichera' le trasformazioni (matrici) che noi gli forniremo, vertice per vertice

FragmentShader: usera', pixel-per-pixel, le informazioni calcolate vertice-per-vertice (ed interpolate dal 'sistema') per assegnare il colore

WebGL (with Elemental)3D

double a = (right + left) / (right - left); double b = (top + bottom) / (top - bottom); double c = (farPlane + nearPlane) / (farPlane - nearPlane); double d = (2 * farPlane * nearPlane) / (farPlane - nearPlane); double x = (2 * nearPlane) / (right - left); double y = (2 * nearPlane) / (top - bottom); double[] perspectiveMatrix = new double[]{ x, 0, a, 0, 0, y, b, 0, 0, 0, c, d, 0, 0, -1, 0 };

....ctx3d.uniformMatrix4fv(pMatrixUniform, false, Utilities.createArrayOfFloat32(perspectiveMatrix));...private native static Float32Array createFloat32Array(JsArrayOfNumber a) /*-{ return new Float32Array(a);}-*/;

WebGL (with Elemental)3D/Wavefront-OBJ

Loading Wavefront-obj files ...

parser in java facile da trovare in rete

ClientBundle DataResource perfetto per il loading asincrono dei modelli

Javascript per calcolare le normali se non sono nel modello

Unconventional

Tutto quello che abbiamo visto fino ad ora si poteva fare, probabilmente con lo stesso sforzo, direttamente in javascript.

Ma usare GWT ci da o no una marcia in piu' ?

Nota: non stiamo dicendo che quello che segue non si potesse fare in javascript, ndr: Turing completeness, se ce ne fosse bisogno !

GWT

Compiler: Java -> Javascript

Javascript as a target language

NyARToolkit

ARToolKit is a computer tracking library for creation of strong augmented reality applications that overlay virtual imagery on the real world.NyARToolKit is an ARToolkit class library released for virtual machines, particularly those that host Java, C# and Android.

Up to date infohttp://nyatla.jp/nyartoolkit/wp/?page_id=198

Support: Java ... "Write once, run anywhere" (WORA), or sometimes write once, run everywhere (WORE) ... in my browser ?!?!?!

NyARToolkit con GWT

http://jooink.blogpsot.com

super-source tag per InputStream & altre parti della JRE non disponibili in GWT

byte[] raster;UInt8ClampedArray image = ImageData.getData();private static native byte[] toArrayOfBytes(Uint8ClampedArray a) /*-{ return a; }-*/;

Documented in JAPANESE !!!

How It Works

video

cam WebRTC <video/> canvas ImageData

byte[]

Elemental/jsni

GWT(NyARToolkit)mv matrix

WebGL + model

Demo http://www.jooink.com/experiments/DevFest2012