www.kurento.org

Presentation for the GSMA WebRTC interest group on April 7th, 2014

The future of multimedia communications and services:

Kurento and it’s role

Luis Lopezlulop@kurento.org

http://www.kurento.org

www.kurento.org 2

Who am I?

• Luis Lopez– Associate Professor at ETSIT/URJC

• http://www.urjc.es• http://www.etsit.urjc.es

– Director of the FUN-LAB research group

• Future Networks Laboratory• http://www.gsyc.es

– Leader of the Kurento.org initiative• http://www.kurento.org

– Software engineer, software developer and Open Source Software Advocate

www.kurento.org 3

What does Kurento mean?

www.kurento.org 4

Kurento’s History

2004•Kurento is born as a tool for transferring research results to industry

2012•Kurento goes open source LGPL v2.1. Professional services provided by Naevatec (startup)

June’13 •1M€ of investment for providing multimedia capabilities to FI-WARE

Dec13•Kurento Featured at WebRTC Expo (Paris)

Feb’14•3.3M€ of investment for making Kurento an elastic cloud platform (NUBOMEDIA)

April’14•Kurento featured at IMS World Conference (Barcelone)

July’14•Kurento featured at DevCon5 (New York)

Sep’14•500K€ of investment for maintenance of Kurento FI-WARE APIs

www.kurento.org 5

Kurento team• 18 people: 15 developers, 1 sysadmin, 2 coordination

www.kurento.org 6

Real-time communications: our visionN

umbe

r of u

sers

General purposeRTC multimedia Services

Specific purpose RTC multimedia services

Commodity, no business opportunities here

Who cares about the immense minority?Many business opportunities here

www.kurento.org 7

The opportunity: from services to platforms and APIs

www.kurento.org 8

Who shares this vision: The WebRTC movement

Num

ber o

f use

rs

General purposeRTC multimedia Services

Specific purpose RTC multimedia services

Commodity, no business opportunities here

Who cares about the immense minority?Many business opportunities here

Disruption directionof WebRTC

www.kurento.org 9

Before and after WebRTC: the perspective of the application developer

Developing the client side

Before WebRTC After WebRTC

Begin End

• Unified APIs• Standards• FOSS• Multiplatform

www.kurento.org 10

WWW TECHNOLOGY & STANDARDS

Why developers are important for the WebRTC movement?

The effect of WWW developers on different technological areas

BankingCRM TV

CommerceDocument

Skype?Traditional phone service?

www.kurento.org 11

Should telcos take the hint?

WebRTC

WWW development community RCS/IMS development community

WebRTC makes multimedia RTC to be just another feature of your WWW application

www.kurento.org 12

What’s the relationship of WebRTC with Kurento?

Developing the client side

Developing the infrastructure

side

Before WebRTC After WebRTC Next natural step…

Begin End Begin End

Begin End

• Unified APIs• Standards• FOSS• Multiplatform

• Unified APIs• Standards• FOSS• Multiplatform

• Unified APIs• Standards• FOSS• Multiplatform

www.kurento.org 13

But is the infrastructure important?What RTC applications will be demanded by the market on the next few years?

www.kurento.org 14

Kurento vision for Future RTC multimedia services

Media ishere Media got

there

Media gotthere

AnalyzeTransform

Store

Transport

EnrichAugment

Adapt

SensorsContext

EventsMedia is

here

Traditional infrastructures

Future Internet infrastructures

www.kurento.org 15

Enrich, analyze, adapt, augment: What’s the problem?

Complexity

www.kurento.org 16

Future Internet

Multimedia Infrastructure

Simple Development

APIs

The Kurento equation for Future Internet infrastructures

www.kurento.org 17

What’s Kurento?

• Kurento.org– Developer platform

• Infrastructures + APIs• Kurento is not a service

– Future Internet• Generalized multimedia

– Audio/Video/Sensor

• Advanced media processing– Computer Vision, Augmented reality

• Interoperability– WebRTC, RTP, HTTP, all codecs, etc.

– Open Source Software• LGPL v2.1

www.kurento.org 18

Kurento Media Server: The nucleus of Kurento

• KMS is a middleware for media streams– Receives the stream– Process the stream– Issues the stream

SendReceiveAnalyze

AugmentEnrich

TransformTranscode

RecordProcess

Replicate

MediaSource

MediaSink

KMS

www.kurento.org 19

Media API: The API for accessing KMS capabilities

SendReceiveAnalyze

AugmentEnrich

TransformTranscode

RecordProcess

Replicate

MediaSource

MediaSink

KMS

JavaMedia API

JavaScriptMedia API

Otherlanguages

Applications define the processingof streams getting through KMS

www.kurento.org 20

Media API: media elements and media pipelines

Media Element

• Provides a specific media functionality

› Send/receive media

› Process media

› Transform media

• Exchange media through

› Sources

› Sinks

Media pipeline

• Chain of media elements implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC

www.kurento.org 21

The Lego Game: Developers create applications connecting media elements

Application 2

Application 3

Application 1

Toolbox of media elements

www.kurento.org 22

Media API trivial exampleMediaPipeline mp = contentSession.getMediaPipelineFactory().create();

PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build();

HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();

playerEndpoint.connect(httpEndpoint);

httpEndpoint.getUrl(); //URL where the media is made available

Media Pipeline

HttpGetEndpoint

Media fromfile or URI

HTTP mediastreaming Si

nk

SRC

PlayerEndpoint

www.kurento.org 23

The signaling: adapting the WWW model to the multimedia services

Process WWW request- DDBB access- Authentication- XML processing- Etc.

HTTP request: I want this resource

HTTP response:The resource

Process media request- Media API- DDBB access- Authentication- XML processing- Etc.

Signaling request: I want this media

Signaling response:The media is here

This is how you create WWW applications

(Servlets, ASP, PHP, Rails, etc.)

This is how you createKurento enabled

applications:Multimedia RTC is just

another feature of your application

Developerscreate this

www.kurento.org 24

The Content Handler: reacting to signaling events

KMS

Sink

SRC

Sink

SRC

Sink SRC

Sink

Media API

REST API (Open API protocol) The Content HandlerEquivalent to a Servlet/ASP/PHP scriptExposes APIs for specifying:- when receiving “this request”…- … execute “this logic”Developer can use the media API

Code building the media pipeline and executing the application logic the

developer wants

www.kurento.org 25

Kurento Application Server, the container of Handlers

• Is an extension of Java EE technologies.• Compatible with all Java EE Servlet

containers• Hold Handlers

– Your specific application code• Receives signaling requests:

– I want “this media” in this way …• Dispatches request to the appropriate

handler– @annotations based mapping

• Generate an answer showing “where and how to find the media”– URL where media is located– Negotiated SDP

Java EE compatible container

HTTP Servlet

SIP Servlet

Web service

s

Kurento REST API

Specific handler implementations

Media API DD.BB.

Kurento Application Server (KAS)

Other java APIs.

www.kurento.org 26

Putting it altogether: the Architecture

Kurento Media Server (KMS)

ReceiveVideo

AugmentedReality

SendVideo

Computer Vision

Video Playingand Recording

Java EE compatible container

HTTP Servlet

SIP Servlet

Web service

s

Kurento REST API

Specific handler implementations

Signaling and WWW traffic

Med

ia

Med

ia

Signaling and WWW traffic

Media API DD.BB.

Kurento Application Server (KAS)

Other java APIs.

www.kurento.org 27

Application execution flowClientCode

ApplicationServer (KAS)

MediaServer (KMS)

I want this media in this way …(JSON-RPC request)

Commands requestingthe creation of a pipeline

What you want is here …(JSON-RPC answer)

Media negotiation

phase

Media exchange

phase

1

2

Specific applicationlogic at the server-side

(Content Handler)

Mediapipelinecreation

Media exchange between client and server

www.kurento.org 28

Kurento in the context of IMSKMS = MRF

(Media Resource Function)

KAS = AS(Application Server)

S-CSCF(Serving- Call Session

Control Function

MediaControl

MediaEvents

Context andother information

services

UE(User Equipment)

Application logic & orchestration

Media capabilities (pipeline)

Media elements can generate events through deep analysis of media• IVR• Sentiment analysis• Face recognition• Object tracking• Etc.

Application logic can use context and other external information services for controlling media elements or dynamically modifying media pipelines

www.kurento.org 29

Kurento use case: WebRTC video conference with “secret” participant

• The “connection” primitive acts from source to destination– You can connect how many sinks as you to a source– Not-connected sources or sinks don’t break anything.

• In this example– User 1 and 2 maintain a video conference

• webRtcEndpoint1.connect(webRtcEndpoint2), webRtcEndpoint2.connect(webRtcEndpoint1)

– User 3 can “see” that video conference, but he cannot participate• webRtcEndpoint1.connect(webRtcEndpoint3), webRtcEndpoint2.connect(webRtcEndpoint3’)

Media Pipeline

WebRTC audiostreaming

Sink

SRC

WebRTC videostreaming

SRC

Sink

SRC

Sink

WebRTC audiostreaming

WebRtcEndpoint 1

WebRtcEndpoint 3

WebRtcEndpoint 2SR

CSi

nk

WebRTC audiostreaming

WebRtcEndpoint 3’

User 1 User 2

User 3

www.kurento.org 30

Kurento use case: WebRTC bridge/SBC between WebRTC and RTP domains

• Kurento transparently adapts codecs and formats. Just connect:– webRtcEndpoint.connect(rtpEndpoint)– rtpEndpoint.connect(webRtcEndpoint)

• Support for the most common audio and video codecs– VP8, H.264, H.263, OPUS, AMR, Speex, G.711, etc.

Media Pipeline

WebRTC audiostreaming

Sink

SRC

SRC

Sink

Media Pipeline

Sink

SRC

SRC

Sink

WebRTC videostreaming

RTP audiostreaming

RTP videostreaming

www.kurento.org 31

Sink

SRC

SRC Sink

SinkSRC

Kurento use case: interoperable MCU

SRC

Sink

• Group communications– Dispatcher

• Media router• N to M

– Mixer• Composite

mixed media• N to 1

www.kurento.org 32

Kurento use case: Video conference recorded and distributed (CDN)

• Kurento makes possible to interconnect different types of endpoints in your application

– RecorderEndpoint• Records media in the desired format

– HttpEndpoint• Exposes the media flow through HTTP pseudostreaming (HTML5 <video> tag)

Sink

SRC

Media Pipeline

Sink

SRC

SinkSi

nkSR

C

Sink

Mediato

file or repository

CDN or Web

browser

www.kurento.org 33

Kurento use case: video-conference with augmented reality

• The infrastructure exposes advances processing capabilities• You can connect them following the same scheme

– playerEndpoint.connect(jackVaderFilter)– jackVaderFilter.connect(httpEndpoint)

• Use RTC endpoints if you want real-time media processing– WebRtcEndpoint, RtpEndpoint

Media Pipeline

Mediafrom

file or URI

Sink

SRC

SRC

Sink

JackVaderFilter

Result: https://www.youtube.com/watch?v=yJAQs23eoXw

www.kurento.org 34

Kurento use case: WebRTC video game

Result: https://www.youtube.com/watch?v=PYCw9-4oWWA

Sink

SRC

Sink

Sink

SRC

Sink

SRC

Sink

SRC

Sink

SRC

WebRtcEndpoint

MirrorFilter PointerDetectorFilter

ChromaFilter FaceOverlayFilter

Media Pipeline

• Beyond video-conferencing: receiving events from the media stream– Immersive and personalized advertising– Games and entertainment

Events (buttons)

Game or advertisement logic

Change faceChange background

www.kurento.org 35

Media PipelineMedia Pipeline

IoT and context aware applications: hot IVR plugin

• Pipelines are dynamic, you can hot modify them using context or sensor information– Ex. when device is moving faster than 40 Kh, add IVR system

• “Record this call”, “Redirect to my secretary”, “Play file one”, Etc.• Voice commands can modify the pipeline themselves

– “Record this call” add a RecorderEndpoint– “Play file 1” add a player

– Ex. When security sensor fires, activate face-recognition/person-detection, etc

Sink

SRC

SRC

Sink

Sink

SRC

Sink

SRC

SRC

Sink

IvrFilter

VoiceEventsDynamic pipeline modification Depending on context or other events

www.kurento.org 36

Media Pipeline

Smart-cities applications• Advanced computer vision filters for

– Crowd detection– Car plate recognition– Intrusion detection– Traffic control– Etc. CrowdEvents

Sink

SRC

Sink

SRC

CrowdDetector

RtspPlayer

www.kurento.org 37

Other scenarios• There are opportunities on many verticals

– eHealth– Logistics– Security– Social networks– Jounalisms

• Creation of additional Media Elements is simple– Integrate your own IVRs– Integrate your own Computer Vision capabilities– Integrate your own repositories– Etc.

• Potential applications are unlimited– Just combine the media elements

www.kurento.org 38

Thank you

Suggestions, comments and complains:lulop@kurento.org

http://www.kurento.orghttp://www.github.com/kurentoTwitter: @Kurentoms