Upload
jooink
View
1.997
Download
2
Embed Size (px)
DESCRIPTION
WebRTC, and most of HTML5 tecnologies as File API, WebGL, WebWorkers, ...,answer to a common problem, i.e. to make 'in browser' apps able to do operations impossible before, as access to the video and audio stream by a javascript api.
Citation preview
WebRTC... GWT & Browser Performance
Alberto Mancini, Francesca TosiJooinK.com
WebRTCPlug-in free realtime communication …
WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs.
source: webrtc.org
WebRTC
Just another JS API
or
WebRTC is a new front in the long war for an open and unencumbered webBrendan Eich (Mozilla CTO and inventor of JavaScript)
?
Browser PiercingOnce upon a time ...
LDP: Firewall Piercing mini-HOWTOFrançois-René Rideauv0.97, 24 November 2001
Browser Piercing
a Javascript Application cannot
➔ access the filesystem and then File API
➔ open full-duplex socket connection and then WebSockets
Browser Piercing
a Javascript Application cannot
➔ use graphics accelerator and then WebGL
➔ span multiple threads and then WebWorkers
Browser Piercing
a Javascript Application cannot
➔ acquire audio and video and then WebRTC
➔ communicate P2P and then WebRTC
WebRTC across platforms● Chrome (mobile too)● Firefox (mobile too)● Opera● Native Java and Obj-C
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints, successCallback,
errorCallback);
WebRTC JS-APIConstraints
● Controls the contents of the MediaStream● Media type, resolution, frame rate
var constraints = {video: true};
video: { mandatory: { ... }, optional [{... }]}
WebRTC JS-APIConstraints
● Controls the contents of the MediaStream● Media type, resolution, frame rate
var constraints = {video: true};
video: { mandatory: { chromeMediaSource: ‘screen’ }, optional [{... }]}
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-APIerrorCallback
function errorCallback(error) { console.log("error: ", error);}
;-)
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,errorCallback);
WebRTC JS-APIsuccessCallback
function successCallback(stream) { ...}
stream: MediaStream; a flux of audio- or video-related data.
WebRTC JS-APIsuccessCallback
function successCallback(stream) { var video = ... video.src =
window.URL.createObjectURL(stream);
}
(W3C File API)
WebRTC JS-APIfull sample
var constraints = {video: true};
function successCallback(stream) { var video = document.querySelector("video"); video.src = window.URL.createObjectURL(stream);}
function errorCallback(error) { console.log("navigator.getUserMedia error: ", error);}
navigator.getUserMedia(constraints, successCallback, errorCallback);
WebRTC JS-APIDo not forget
● to play the video
● to polyfill getUserMedia (moz*,webkit*,ms*)
WebRTC JS-APIRTCPeerConnection
→getUserMedia
+RTCPeerConnection
←
WebRTC JS-APIRTCDataChannel
Bidirectional communication of arbitrary data
● low latency● API similar to WebSockets
RTCPeerConnection samplepc = new RTCPeerConnection(null);pc.onaddstream = gotRemoteStream;pc.addStream(localStream);pc.createOffer(gotOffer);
function gotOffer(desc) { pc.setLocalDescription(desc); sendOffer(desc);}
function gotAnswer(desc) { pc.setRemoteDescription(desc);}
function gotRemoteStream(e) { attachMediaStream(remoteVideo, e.stream);}
RTCPeerConnection
RTCPeerConnection
.
RTCPeerConnection
.
STUN
.
TURN
.
Yes BUTVideoChat already exists and we are too late to build skype without skype ...
● Video chat:○ SimpleWebRTC○ easyRTC○ webRTC.io
● Peer-to-peer data:○ PeerJS○ Sharefest
Yes BUT
● getUserMedia: gives us real time stream from the cam
● we can draw the <video/> into a <canvas/>
● from the canvas we can extract pixel data getImageData()
and then elaborate.
Computing (GWT 4 us)goals- Almost native performance- Manage complex algorithms and applications
Computing (GWT 4 us)goals- Almost native performance- Manage complex algorithms and applications
our approach- javascript as a target language (GWT) - hand written optimized pieces of code
JooinK’s WebRTC sample
Sample
http://picshare.jooink.com
Demo …
PicShareTecnologies
● WebGL for 3D graphics● WebRTC for MediaSteams● ImgUR for pictures store● AppEngine as a glue
FOSS
gwt-nyartoolkit.googlecode.com
gwt-webgl.googlecode.com
The strategy: compile a java-lib with GWT into javascript
What’s nextChallanges
● real-time on mobile apps● real world recognition: computer vision
(BoofCV)
What’s nextOur approach
● use TypedArrays everywhere ● offload computation to the graphic
accelerator through WebGL● help the JIT compiler/optimizer by asm.js
That’s all folks!!!
Alberto [email protected]
Markers