Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives

Implementing a WebRTC endpoint in GStreamer:

challenges, problems and perspectives

8-9 October 2015Dublin, Ireland

Conference 2015

Miguel París

Jose A. [email protected]

[email protected]

2

Who we areJosé Antonio Santos Cadenas

● Software Engineer

● Telematic Systems Master's

● Developer at Naevatec (Madrid, Spain)

● Kurento Media Server (KMS) manager

● Integrating with client APIs

● [email protected]

Miguel París

● Software Engineer

● Telematic Systems Master's

● Researcher at Universidad Rey Juan Carlos (Madrid, Spain)

● Kurento real-time manager

● [email protected]

● Twitter: @mparisdiaz

mailto:[email protected]

What is WebRTC

3

● WebRTC is a new movement bringing RTC to the WWW

● It is based on standards

● Supported by Chrome & Firefox

● Windows Edge added partial support recently

● Safari is rumored to be on the way of providing it

● FOSS implementations for Smartphone platforms (iOS and Android)

● More tan 1B devices currently support WebRTC

DevelopingRTC applications

for the WWW

Before WebRTC After WebRTC

Begin End

• Unified APIs• Standards• FOSS• Multiplatform

WebRTC standards

4

● Standard APIs: Standardized by the WebRTC WG at W3C

Media Capture APIs● Standardize APIs for media capture

PeerConnection APIs (WebRTC 1.0 almost finished)● Standardize API for P2P media communications● Media = Audio + Video + Data● DataChannel API: low latency

WebRTC stats● RTP stats for inbound and outbound directions● encode/decode stats● DataChannel stats

Implementing WebRTC in GStreamer

5

● Deep interest in the GStreamer community

● Several implementations

OpenWebRTC (Ericsson)● Design for working as part of client applications

Kurento Media Server (Kurento.org)● Design for working as part of media server infrastructures

Main differences between them: OpenWebRTC captures media from camera and mic. KMS doesn't OpenWebRTC renders to display and speakers. KMS doesn't OpenWebRTC encodes/decodes audio/video. KMS only if needed.

6

<GstPipeline>pipeline0

[>]

KmsWebrtcEndpointkmswebrtcendpoint1

[>]

GstRTPRtxQueuertprtxqueue1

[>]

GstRtpVP8Payrtpvp8pay1

[>]

GstRtpOPUSPayrtpopuspay1

[>]

KmsWebrtcSessionkmswebrtcsession1

[>]

KmsRtcpDemuxkmsrtcpdemux1

[>] GstRtpSsrcDemuxrtpssrcdemux5

[>]

KmsWebrtcTransportSinkNicekmswebrtctransportsinknice1

[>]

GstNiceSinknicesink1

[>]

GstDtlsSrtpEncdtlssrtpenc1

[>]

GstFunnelfunnel

[>]

GstSrtpEncsrtp-encoder

[>]

GstDtlsEncdtls-encoder

[>]

KmsWebrtcTransportSrcNicekmswebrtctransportsrcnice1

[>]

GstDtlsSrtpDecdtlssrtpdec1

[>]

GstSrtpDecsrtp-decoder

[>]

GstDtlsDecdtls-decoder

[>]

GstDtlsSrtpDemuxdtls-srtp-demux

[>]

GstNiceSrcnicesrc1

[>]

GstRtpBinrtpbin1

[>]

GstRtpSsrcDemuxrtpssrcdemux4

[>]

GstRtpSessionrtpsession3

[>]


[>]


[>]

KmsWebrtcEndpointkmswebrtcendpoint0

[>]

GstRtpVP8Depayrtpvp8depay0

[>]

KmsAgnosticBin2kmsagnosticbin2-1

[>]

GstQueuequeue3

[>]

KmsParseTreeBinkmsparsetreebin1

[>]

KmsVp8Parsekmsvp8parse0

[>]

GstFakeSinkfakesink3

[>]

GstTeetee3[>]


[>]

GstTeetee2[>]

GstRTPOpusDepayrtpopusdepay0

[>]

KmsAgnosticBin2kmsagnosticbin2-0

[>]

GstQueuequeue1

[>]

KmsParseTreeBinkmsparsetreebin0

[>]

GstOpusParseopusparse0

[>]


[>]

GstTeetee1[>]


[>]

GstTeetee0[>]

GstRTPRtxQueuertprtxqueue0

[>]GstRtpVP8Payrtpvp8pay0

[>]

GstRtpOPUSPayrtpopuspay0

[>]

KmsWebrtcSessionkmswebrtcsession0

[>]

KmsRtcpDemuxkmsrtcpdemux0

[>]


[>]

KmsWebrtcTransportSinkNicekmswebrtctransportsinknice0

[>]

GstNiceSinknicesink0

[>]

GstDtlsSrtpEncdtlssrtpenc0

[>]

GstFunnelfunnel

[>]

GstSrtpEncsrtp-encoder

[>]

GstDtlsEncdtls-encoder

[>]

KmsWebrtcTransportSrcNicekmswebrtctransportsrcnice0

[>]

GstDtlsSrtpDecdtlssrtpdec0

[>]

GstSrtpDecsrtp-decoder

[>]

GstDtlsDecdtls-decoder

[>]

GstDtlsSrtpDemuxdtls-srtp-demux

[>]

GstNiceSrcnicesrc0

[>]

GstRtpBinrtpbin0

[>]

GstRtpJitterBufferrtpjitterbuffer1

[>] GstRtpPtDemuxrtpptdemux1

[>]

GstRtpJitterBufferrtpjitterbuffer0

[>] GstRtpPtDemuxrtpptdemux0

[>]


[>]


[>]


[>]


[>]

LegendElement-States: [~] void-pending, [0] null, [-] ready, [=] paused, [>] playingPad-Activation: [-] none, [>] push, [<] pullPad-Flags: [b]locked, [f]lushing, [b]locking; upper-case is setPad-Task: [T] has started task, [t] has paused task

proxypad40[>][bfb]

sink[>][bfb]

sink_audio[>][bfb]

proxypad42[>][bfb]

sink[>][bfb]

sink_video[>][bfb]

sink[>][bfb]

src[>][bfb]

send_rtp_sink_1[>][bfb]

proxypad33[>][bfb]

src[>][bfb]

src[>][bfb]


proxypad31[>][bfb]

send_rtp_src_0[>][bfb]

sink[>][bfb]

rtp_src[>][bfb]

rtcp_src[>][bfb] rtcp_sink

[>][bfb]

sink[>][bfb]

src_1[>][bfb] recv_rtp_sink_1

[>][bfb]

rtcp_src_1[>][bfb] recv_rtcp_sink_1

[>][bfb]

src_421259003[>][bfb] recv_rtp_sink_0

[>][bfb]

rtcp_src_421259003[>][bfb] recv_rtcp_sink_0

[>][bfb]

proxypad44[>][bfb]

proxypad45[>][bfb]

proxypad46[>][bfb]

proxypad47[>][bfb]

sink[>][bfb]

proxypad34[>][bfb]

rtp_sink_0[>][bfb]

rtp_sink_0[>][bfb]

src[>][bfb]

proxypad36[>][bfb]

rtcp_sink_0[>][bfb]

rtcp_sink_0[>][bfb]

proxypad37[>][bfb]

rtp_sink_1[>][bfb]

rtp_sink_1[>][bfb]

proxypad39[>][bfb]

rtcp_sink_1[>][bfb]

rtcp_sink_1[>][bfb]

proxypad29[>][bfb]

funnelpad5[>][bfb]

src[>][bfb]

funnelpad6[>][bfb]

funnelpad7[>][bfb]

funnelpad8[>][bfb]

funnelpad9[>][bfb]

rtp_src_0[>][bfb]

rtcp_src_0[>][bfb]

rtp_src_1[>][bfb]

rtcp_src_1[>][bfb]

src[>][bfb][T]

proxypad28[>][bfb]

sink[>][bfb]

sink[>][bfb]

rtp_src[>][bfb]

proxypad26[>][bfb]

proxypad27[>][bfb]

rtcp_src[>][bfb]

rtp_sink[>][bfb]

rtp_src[>][bfb]

rtcp_sink[>][bfb]

rtcp_src[>][bfb]

sink[>][bfb]

rtp_src[>][bfb]

dtls_src[>][bfb]

src[>][bfb][T]

send_rtp_sink[>][bfb]


recv_rtp_sink[>][bfb]

recv_rtcp_sink[>][bfb]



proxypad30[>][bfb]

proxypad32[>][bfb]


proxypad35[>][bfb]

send_rtcp_src_0[>][bfb]

proxypad38[>][bfb]


sink[>][bfb]

rtcp_sink[>][bfb]

send_rtp_src[>][bfb]

send_rtcp_src[>][bfb]

recv_rtp_src[>][bfb]

sync_src[>][bfb]

sink[>][bfb]

rtcp_sink[>][bfb]




sync_src[>][bfb]

proxypad14[>][bfb]

sink[>][bfb]

sink_audio[>][bfb]

audio_src_0[>][bfb]

proxypad15[>][bfb]

sink[>][bfb]

sink_video[>][bfb]

proxypad24[>][bfb]

proxypad25[>][bfb]

video_src_0[>][bfb]

sink[>][bfb]

src[>][bfb]

sink[>][bfb]

proxypad23[>][bfb]

src_0[>][bfb]

sink[>][bfb]

proxypad43[>][bfb]sink

[>][bfb]src

[>][bfb][T]

sink[>][bfb]

src[>][bfb]

sink[>][bfb]

src_0[>][bfb]

sink[>][bfb]

src_2[>][bfb]

sink[>][bfb]

src_0[>][bfb]

src_1[>][bfb]

sink[>][bfb]

src[>][bfb]

sink[>][bfb]

proxypad19[>][bfb]

src_0[>][bfb]

sink[>][bfb]

proxypad41[>][bfb]sink

[>][bfb]src

[>][bfb][T]

sink[>][bfb]

src[>][bfb]

sink[>][bfb]

src_0[>][bfb]

sink[>][bfb]

src_2[>][bfb]

sink[>][bfb]

src_0[>][bfb]

src_1[>][bfb]

sink[>][bfb]

src[>][bfb]


proxypad7[>][bfb]

src[>][bfb]

src[>][bfb]


proxypad5[>][bfb]


sink[>][bfb]

rtp_src[>][bfb]

rtcp_src[>][bfb]

rtcp_sink[>][bfb]

sink[>][bfb]

src_1442068093[>][bfb]

recv_rtp_sink_0[>][bfb]

rtcp_src_1442068093[>][bfb]

recv_rtcp_sink_0[>][bfb]

src_836061664[>][bfb]

recv_rtp_sink_1[>][bfb]

rtcp_src_836061664[>][bfb]

recv_rtcp_sink_1[>][bfb]

proxypad16[>][bfb]

proxypad17[>][bfb]

proxypad20[>][bfb]

proxypad21[>][bfb]

sink[>][bfb]

proxypad8[>][bfb]

rtp_sink_0[>][bfb]

rtp_sink_0[>][bfb]

src[>][bfb]

proxypad10[>][bfb]

rtcp_sink_0[>][bfb]

rtcp_sink_0[>][bfb]

proxypad11[>][bfb]

rtp_sink_1[>][bfb]

rtp_sink_1[>][bfb]

proxypad13[>][bfb]

rtcp_sink_1[>][bfb]

rtcp_sink_1[>][bfb]

proxypad3[>][bfb]

funnelpad0[>][bfb]

src[>][bfb]

funnelpad1[>][bfb]

funnelpad2[>][bfb]

funnelpad3[>][bfb]

funnelpad4[>][bfb]

rtp_src_0[>][bfb]

rtcp_src_0[>][bfb]

rtp_src_1[>][bfb]

rtcp_src_1[>][bfb]

src[>][bfb][T]

proxypad2[>][bfb]

sink[>][bfb]

sink[>][bfb] rtp_src

[>][bfb]

proxypad0[>][bfb]

proxypad1[>][bfb]

rtcp_src[>][bfb]

rtp_sink[>][bfb]

rtp_src[>][bfb]

rtcp_sink[>][bfb]

rtcp_src[>][bfb]

sink[>][bfb]

rtp_src[>][bfb]

dtls_src[>][bfb]

src[>][bfb][T]






recv_rtcp_sink[>][bfb] proxypad4

[>][bfb]

proxypad6[>][bfb]


proxypad9[>][bfb]


proxypad12[>][bfb]


proxypad18[>][bfb]

recv_rtp_src_0_1442068093_111[>][bfb]

proxypad22[>][bfb]

recv_rtp_src_1_836061664_100[>][bfb]

sink[>][bfb] src

[>][bfb][T]sink_rtcp[>][bfb]

sink[>][bfb]

src_100[>][bfb]

sink[>][bfb] src

[>][bfb][T]sink_rtcp[>][bfb]

sink[>][bfb]

src_111[>][bfb]

sink[>][bfb]

src_836061664[>][bfb]

rtcp_sink[>][bfb]

rtcp_src_836061664[>][bfb]




sync_src[>][bfb]

sink[>][bfb]

src_1442068093[>][bfb]

rtcp_sink[>][bfb]

rtcp_src_1442068093[>][bfb]




sync_src[>][bfb]

Kurento WebRtcEpThe pipeline

Interactive Connectivity EstablishmentICE I

7

● Using libnice (NiceSrc and NiceSink)

Involved RFCs:

RFC 5245: Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols

RFC 5389: Session Traversal Utilities for NAT (STUN)

RFC 5766: Traversal Using Relays around NAT (TURN): Relay Extensions to Session Traversal Utilities for NAT (STUN)

Interactive Connectivity EstablishmentICE II

8

● Our experience with libnice

0.1.7 version seem to be stable, but lack many required features

0.1.10 version incorporate relevant features but seem not to provide the appropriate stability yet

● Leaks of TCP sockets● Connection states● Connectivity problems (Fails 1/1000)

Fixed in 0.1.13 ? (we have to check it)

ICE III

9

● What next needed in ICE for WebRTC?

IPv6 shall be mandatory in WebRTC. We need to check IPv6 support in libnice draft-ietf-rtcweb-transports-09 (mandates IPv6 support)

HTTP CONNECT protocol for TURN might be important in corporate networks (connectivity)

draft-hutton-httpbis-connect-protocol-00 STUN consent freshness may become mandatory for

WebRTC. This may require some adjustments on libnice for supporting consent negotiation (security)

Draft-ietf-rtcweb-stun-consent-freshness-16 ICE stated published through events

ICE state should be reported as specified in the WebRTC 1.0 WG draft

DTLS / SRTP I

10

● Using dtlssrtp(dec|enc) bins

dtls(dec|enc) manages DTLS handshake

srtp(dec|enc) decrypts/encrypts RTP packets● Involved RFCs:

RFC 5764: Datagram Transport Security (DTLS) Extension to Establish Keys for the Secure Real-time Protocol (SRTP)

RFC 3711: The Secure Real-time Transport Protocol (SRTP)

DTLS / SRTP II

11

● What next for WebRTC?

New cryptographic framework for group communications

This is under definition at the Privacy Enhanced RTP Conferencing (PERC) IETF WG

Only early drafts here draft-jones-perc-private-media-framework-00 draft-jones-perc-private-media-reqts-00 draft-mattsson-perc-srtp-cloud-00 draft-westerlund-perc-webrtc-use-case-00

PERC shall provide in the next few months (improve efficiency) Enabling unicast group communications without requiring infrastructure to

decipher New mechanisms enabling unicast group communications without

requiring infrastructure to cipher once per RTC session. These may require re-factoring of current DTLS stack

DTLS state published through events.

Full connectivity requires ICE connectivity + successful DTLS handshake. DTLS is being incorporated by the W3C in the WebRTC 1.0 spec as event

generator dealing with connection health

Congestion control I

12

● This is an essential feature for providing satisfactory QoE for WebRTC users

How does it work?

Bandwidth estimator Based on packet delay (useful RTP ExtHdr: abs-send-time) Based on packet loss

REMB (Receiver Estimated Maximum Bitrate) GStreamer does not provide any kind of features for this off the self

How Kurento estimates bandwidth?

on-sending-rtcp

Get packet-loss, bytes-sent, etc. from RtpSession stats

Advantages: simple

Drawbacks Reactive and not predictive Requires packet loss to detect BW constraints Getting RtpSession stats quite inefficient

Congestion control II

13

● For having satisfactory congestion control we should look to the following drafts

draft-alvestrand-rmcat-remb-03: REMB mechanism implemented by Chrome

draft-ietf-rmcat-cc-requirements-09: Congestion Control Requirements for Interactive Real-Time Media

draft-ietf-avtcore-rtp-circuit-breakers-10: Multimedia Congestion Control: Circuit Breakers for Unicast RTP Sessions

● In summary, congestion control should:

be predictive

be efficient

be fair

"break the circuit" (drop the BW) in case of evidence of huge congestion This is to avoid breaking other types of traffic.

DataChannels

14

● Kurento has DataChannels support

● Protocols: DTLS-SCTP

● Drafts

draft-ietf-rtcweb-data-channel

draft-ietf-rtcweb-data-protocol

draft-ietf-mmusic-data-channel-sdpneg Impl. based on sctp(dec|enc) provided by Ericsson

What have Kurento done?

Fixed problems when using Gstreamer 1.4

Fixed a lot of bugs

Implementation of the DataChannel protocol AppSrc, AppSink

Thank you

Suggestions, comments and complains:[email protected]@gmail.com

http://www.kurento.orghttp://www.github.com/[email protected]: @kurentoms

http://www.nubomedia.eu

http://www.fi-ware.org

http://ec.europa.eu

mailto:[email protected]

Internet

Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives