Enabling Voice Applications with WebRTC and ORTC in Microsoft Edge

ENABLING VOICE APPLICATIONS WITH WEBRTC & ORTC IN MICROSOFT EDGE

MARK ROBERTS SOFTWARE ENGINEER

MROBERTS@TWILIO.COM

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

PSTN

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

Ready Not Ready

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

Ready Not Ready

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

Ready Not Ready

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

Ready Not Ready

+ =

<script src="twilio.js"></script>

© 2009 - 2014 Twilio, Inc. All rights reserved.

Four Basic Tasks

Four basic tasks for a voice application:

© 2009 - 2014 Twilio, Inc. All rights reserved.

Four Basic Tasks

Four basic tasks for a voice application: 1. Capture audio from a microphone

© 2009 - 2014 Twilio, Inc. All rights reserved.

Four Basic Tasks

Four basic tasks for a voice application: 1. Capture audio from a microphone 2. Send audio to an endpoint

© 2009 - 2014 Twilio, Inc. All rights reserved.

Four Basic Tasks

Four basic tasks for a voice application: 1. Capture audio from a microphone 2. Send audio to an endpoint 3. Receive audio from an endpoint

© 2009 - 2014 Twilio, Inc. All rights reserved.

Four Basic Tasks

Four basic tasks for a voice application: 1. Capture audio from a microphone 2. Send audio to an endpoint 3. Receive audio from an endpoint 4.Playback received audio

FIRST, THERE WAS FLASH…

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

<script src="twilio.js"></script>

ALONG CAME WEBRTC…

© 2009 - 2014 Twilio, Inc. All rights reserved.

WebRT-what⁇

© 2009 - 2014 Twilio, Inc. All rights reserved.

WebRT-what⁇

• Simple JavaScript APIs • RTCPeerConnection • RTCIceCandidate • RTCSessionDescription

• First-class browser support

• Built on open standards(STUN, TURN, ICE, DTLS, RTP)

© 2009 - 2014 Twilio, Inc. All rights reserved.

Four Basic Tasks

Four basic tasks for a voice application: 1. Capture audio from a microphone 2. Send audio to an endpoint 3. Receive audio from an endpoint 4.Playback received audio

© 2009 - 2014 Twilio, Inc. All rights reserved.

Four Basic Tasks

Four basic tasks for a voice application: 1. Capture audio from a microphone 2. Send audio to an endpoint 3. Receive audio from an endpoint 4.Playback received audio

WebRTC Endpoint WebRTC Endpoint

WebRTC Endpoint WebRTC Endpoint

establish connectivity

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

WebRTC Endpoint

establish connectivity

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

WebRTC Endpoint

establish connectivity

secure connectivity

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

WebRTC Endpoint

establish connectivity

secure connectivity

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

WebRTC Endpoint

establish connectivity

secure connectivity

send and receive media

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

Secure Real-time Transport Protocol (SRTP)

WebRTC Endpoint

establish connectivity

secure connectivity

send and receive media

© 2009 - 2014 Twilio, Inc. All rights reserved.

WebRT-what⁇

• Simple JavaScript APIs • RTCPeerConnection • RTCIceCandidate • RTCSessionDescription

• First-class browser support

• Built on open standards(STUN, TURN, ICE, DTLS, RTP)

© 2009 - 2014 Twilio, Inc. All rights reserved.

WebRT-what⁇

• Simple JavaScript APIs • RTCPeerConnection • RTCIceCandidate • RTCSessionDescription

• First-class browser support

• Built on open standards(STUN, TURN, ICE, DTLS, RTP)

• Interoperability

• No plugins required!

• Bring-your-own signaling

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

Secure Real-time Transport Protocol (SRTP)

WebRTC Endpoint

establish connectivity

secure connectivity

send and receive media

WebRTC Endpoint WebRTC Endpoint

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

<script src="twilio.js"></script>

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

<script src="twilio.js"></script>

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

<script src="twilio.js"></script>

EDGE & ORTC ANNOUNCED…

© 2009 - 2014 Twilio, Inc. All rights reserved.

Object RTC (ORTC)

© 2009 - 2014 Twilio, Inc. All rights reserved.

Object RTC (ORTC)

• Built on the same open standards as WebRTC (STUN, TURN, ICE, DTLS, RTP)

• Bring-your-own signaling

© 2009 - 2014 Twilio, Inc. All rights reserved.

Object RTC (ORTC)

• Built on the same open standards as WebRTC (STUN, TURN, ICE, DTLS, RTP)

• Bring-your-own signaling

• Does not use Session Description Protocol (SDP)

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

Secure Real-time Transport Protocol (SRTP)

WebRTC Endpoint

establish connectivity

secure connectivity

send and receive media

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

Secure Real-time Transport Protocol (SRTP)

SDP

WebRTC Endpoint

© 2009 - 2014 Twilio, Inc. All rights reserved.

Session Description Protocol (SDP)

v=0 o=- 7419095041521713450 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS BcUNfqxMTfmJ2aPDEipCTK4BnvabR7xKeTil m=audio 35898 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126 c=IN IP4 54.244.51.11 a=rtcp:39742 IN IP4 54.244.51.11 a=sendrecv a=rtcp-mux a=rtpmap:111 opus/48000/2 a=fmtp:111 minptime=10; useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000

a=fingerprint:sha-256 F1:A4:DB:A3:AA:CC:1C:B5:2F:91:A0:2F:E8:FD:0B:80:F7:02:1F:9E:23:49:1C:06:1F:4F:AB:96:44:F1:CB:D6 a=setup:actpass m=video 51034 UDP/TLS/RTP/SAVPF 100 101 116 117 96 c=IN IP4 54.244.51.11 a=rtcp:27538 IN IP4 54.244.51.11 a=ice-ufrag:mKHEfIR34arV95jd a=ice-pwd:YdwDVrwNPLwIsFF2sJtHZDSk a=fingerprint:sha-256 F1:A4:DB:A3:AA:CC:1C:B5:2F:91:A0:2F:E8:FD:0B:80:F7:02:1F:9E:23:49:1C:06:1F:4F:AB:96:44:F1:CB:D6 a=setup:actpass a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=sendrecv a=rtcp-mux a=rtpmap:100 VP8/90000 a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack

© 2009 - 2014 Twilio, Inc. All rights reserved.

Session Description Protocol (SDP)

v=0 o=- 7419095041521713450 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS BcUNfqxMTfmJ2aPDEipCTK4BnvabR7xKeTil m=audio 35898 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126 c=IN IP4 54.244.51.11 a=rtcp:39742 IN IP4 54.244.51.11 a=sendrecv a=rtcp-mux a=rtpmap:111 opus/48000/2 a=fmtp:111 minptime=10; useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000

© 2009 - 2014 Twilio, Inc. All rights reserved.

Session Description Protocol (SDP)

v=0 o=- 7419095041521713450 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS BcUNfqxMTfmJ2aPDEipCTK4BnvabR7xKeTil m=audio 35898 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126 c=IN IP4 54.244.51.11 a=rtcp:39742 IN IP4 54.244.51.11 a=sendrecv a=rtcp-mux a=rtpmap:111 opus/48000/2 a=fmtp:111 minptime=10; useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000

© 2009 - 2014 Twilio, Inc. All rights reserved.

Object RTC (ORTC)

• Built on the same open standards as WebRTC (STUN, TURN, ICE, DTLS, RTP)

• Bring-your-own signaling

• Does not use Session Description Protocol (SDP)

• Lower-level JavaScript APIs

WebRTC Endpoint

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

RTCIceTransport

RTCIceGatherer

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

RTCIceTransport

RTCIceGatherer

RTCDtlsTransport

WebRTC Endpoint

Interactive Connectivity

Establishment (ICE)

Datagram Transport Layer Security (DTLS)

Secure Real-time Transport Protocol (SRTP)

RTCIceTransport

RTCIceGatherer

RTCDtlsTransport

RTCRtpReceiver

RTCRtpSender

© 2009 - 2014 Twilio, Inc. All rights reserved.

Object RTC (ORTC)

• Built on the same open standards as WebRTC (STUN, TURN, ICE, DTLS, RTP)

• Bring-your-own signaling

• Does not use Session Description Protocol (SDP)

• Lower-level JavaScript APIs • RTCIceGatherer • RTCIceTransport • RTCDtlsTransport • RTCRtpSender • RTCRtpReceiver

© 2009 - 2014 Twilio, Inc. All rights reserved.

Object RTC (ORTC)

• Built on the same open standards as WebRTC (STUN, TURN, ICE, DTLS, RTP)

• Bring-your-own signaling

• Does not use Session Description Protocol (SDP)

• Lower-level JavaScript APIs • RTCIceGatherer • RTCIceTransport • RTCDtlsTransport • RTCRtpSender • RTCRtpReceiver

• Power and flexibility over “plain” WebRTC

© 2009 - 2014 Twilio, Inc. All rights reserved.

Bridging the Gap

ORTC

• RTCIceGatherer

• RTCIceTransport

• RTCDtlsTransport

• RTCRtpSender

• RTCRtpReceiver

WebRTC

• RTCPeerConnection

• RTCIceCandidate

• RTCSessionDescription

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

<script src="twilio.js"></script>

© 2009 - 2014 Twilio, Inc. All rights reserved.

Browser Support

ORTC

<script src="twilio.js"></script>

© 2009 - 2014 Twilio, Inc. All rights reserved.

Review

© 2009 - 2014 Twilio, Inc. All rights reserved.

Review

• Flash is no longer the only real-time communications solution for browsers.

© 2009 - 2014 Twilio, Inc. All rights reserved.

Review

• Flash is no longer the only real-time communications solution for browsers.

• WebRTC and ORTC enable interoperable, plugin-free communication.

© 2009 - 2014 Twilio, Inc. All rights reserved.

Review

• Flash is no longer the only real-time communications solution for browsers.

• WebRTC and ORTC enable interoperable, plugin-free communication.

• Real-time communications is complex, but developing voice applications does not have to be complicated.

© 2009 - 2014 Twilio, Inc. All rights reserved.

Voice Applications

Ready Not Ready

<script src="twilio.js"></script>

twilio.js

THANK YOU

MARK ROBERTS SOFTWARE ENGINEER

MROBERTS@TWILIO.COM