Delivering Real-Time

Communications with MobilePresented by Robin Raymond [http://about.me/robinraymond]

Chief Architect

Hookflash.com / OpenPeer.org

WebRTC, Mobility, Cloud, and More...

IIT REAL-TIME COMMUNICATIONSConference & Expo Sept 30 - Oct 2, 2014 Chicago

2014-10-01 9:00am WEST: Alumni Lounge

Why should I care about mobility?Digital time spent on mobile apps exceeds the desktop:

Mobile matters

Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital

Why should I care about mobility?Mobile is where RTC is used most:

Mobile matters

Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital

What's different about mobile RTC?

Mobility differences

A mobile device is always with us…

but what makes mobile RTC so different than desktop RTC?

What's different about mobile RTC?

Mobility differences

Slower

CPUsLimited

Battery LifeMobile + Wireless

Connectivity

Small differences. Huge impact.

Limited

Storage /

Memory

Screen

Sizes /

Rotation

CPUs Limitations

CPU limitations

Slower CPU and encoding/decoding

The CPU is limited. Encoding, decoding,

capturing and rendering audio/video consume the

CPU's resources.

CPU usage also impacts battery life.

CPU limitations

Solution #1: Some assembly requiredFor heavily used encoding, decoding, rendering, scaling and

other routines, some assembly code should be expected.

CPU limitations

Expect to have to do this

optimization assembly work or find

a library that does it for you.

Solution #2: Hardware encoders

Some mobile devices have hardware encoders/decoders

either in the CPU or on the camera.

But there are problems:

● Encoders/decoders are not ubiquitous

● Not always real-time friendly (buffered)

● Not always 100% compatible with other encoders/decoders

● May only support a limited number of simultaneous streams

● Other applications may have the hardware busy

CPU limitations

Solution #3: On-demand high resolutionSend lower resolution/bitrate streams for media not in focus (thus

conferencing signalling needs to be made mobile friendly)!

CPU limitations

In-focus. Sent with high

resolution/frame rate.

Not-in-focus sent with low

resolution / low bitrate.

Battery Life

Battery life

Is there anyone out there?

Mobile RTC clients need to listen for incoming RTC

connection requests from a remote clients

(typically via a server):

Battery life

Maintaining a Connection Consumes Battery Life

Wifi/WWan antennas need a fair amount of power to run.

Battery life

Keeping this connection active to a

server consumes battery life.

Solution #1 - Specialized "push friendly" servers

are required

Be sure to use push friendly protocol services for mobile or

your communication will break down.

Battery life

When a RTC client application goes to the background the

connection to the server will typically be killed. The server

will assume the client is gone.

Solution #1 - Register for push

Most mobile devices offer a "push" service. Register for a

push instead of keeping a server connection alive.

Battery life

Temporary connection

Push service

Solution #1 - Send push upon incoming request

Incoming connections cause a push to happen.

Battery life

Server needs to "hold" incoming connection

request while push is delivered.

Push service

Solution #2 - Application as a service

Ideally have the server ping the client to see if it's alive so the

client application can go to "sleep" otherwise.

Battery life

Registering the connection as a "VoIP/RTC" connection and/or

registering the application as a background service will allow

the connection to be processed when data is received.

It's not as efficient as push but it is better.

Backgrounding and Messaging

When an application goes to sleep in the

background to save battery it can't accept

receiving directed messages anymore.

Battery life

Hi Bob!Zzzz...

Backgrounding and Messaging

Protocols that "retry" sending message until they

get delivered do not work when applications are

sleeping.

Battery life

Hi Bob!

Hi Bob!

Hi Bob!*FAILURE*

Zzzz...

Solution #1: Server Store + Forward

Servers need to store messages until the

receiving application wakes up.

Battery life

Zzzz...

*AWAKE*

*CONNECTED*

Some time later...

Solution #2: Peer-to-peer Synchronize

Messages are stored locally and synchronized

once applications are both awake and connected.

Battery life

Zzzz...

*AWAKE*

*CONNECTED*

Some time later...

Mobility + Wireless Connectivity

Mobility + wireless connectivity

Mobile 4G/3G and IPv4 Address Exhaustion

Mobility + wireless connectivity

Mobile

Pro

vid

er

Firew

all

IP=2.3.4.5

IP=2.3.4.5

IP=2.3.4.5

With millions of new mobile devices, mobile providers are sharing a single IPv4 for

many devices. There aren't enough IPv4 addresses for each mobile device online.

Mobile 4G/3G and IPv4 Address Exhaustion

Mobility + wireless connectivity

Mobile

Pro

vid

er

Firew

all

10.0.0.80

5000

7542

32812

2.3.4.5

For efficient usage of IPv4, typically symmetric NAT firewalls are used. Every local

port gets mapped to a different firewall port per connection destination.

99.98.97.96

15.16.17.18

Solution #1: Mobile 4G/3G requires TURN relay

Mobility + wireless connectivity

Mobile

Pro

vid

er

Firew

all

10.0.0.80

5000

7542

32812

2.3.4.5

Symmetric NAT firewalls aren't friendly for direct media connections and TURN

relays are often required for wwan mobile for IPv4.

99.98.97.96

Media Relay

TURN=5.6.7.8

Mobility + wireless

connectivity

Solution #2: Mobile 4G/3G and IPv6

Mobile

Pro

vid

er

Filt

er

FE80::0202:B3F

F:FE1E:8329

5000 5000

An inexhaustible supply of IPv6 addresses exist so applications should take

advantage of mobile devices and providers that support IPv6!

Mobile provider firewalls don't need to share IPs or do NAT port mappings. They

only act as a filter to block bad traffic thus TURN relays can be avoided.

2607:f0d0:1

002:51::4

FEC and Wireless Packet Loss

FEC = Forward Error Correction

A/V packets can include forward/backward predictive

information so lost packets can be reconstructed.

Mobility + wireless connectivity

But wireless loss is often bursty

What do we do in this situation?

Due to wireless buffering overflow, mobile packet loss often

happens in bursts. Single frame FEC does not work well.

Mobility + wireless connectivity

Solution #1 - Scalable Video Codecs

For video, a base layer video frame can be transmitted with

enhancement video resolutions/frames. Loss of enhancement

frames are non critical.

Mobility + wireless connectivity

Loss of video enhancements

frames will decrease quality but

the overall video transmission is

maintained.

Solution #1 - Scalable Video Codecs

NACK (Negative Acknowledgement) / RTX (Retransmission) /

other schemes are used to protect the base layer.

Mobility + wireless connectivity

Lost base layer packets are

protected with NACK / RTX and

enhancements frames reference

protected base layer frame.

Burst audio loss requires a burst capable audio

codec and/or multi-packet RTP FEC modes.

In some conditions, retransmitting the same

packet multiple times increases the probability of

burst packet loss but not always.

Solution #2 - Audio Burst FEC

Mobility + wireless connectivity

Burst audio loss is an ongoing field of study and

there are proposals but this is not a heavily

standardized area.See also:

http://research.google.com/pubs/pub41611.html

https://github.com/vr000m/payload-1d2d-parity-fec/blob/master/1d2d/draft-singh-payload-1d2d-parity-scheme.xml

http://tools.ietf.org/html/rfc5956

http://tools.ietf.org/html/draft-lennox-payload-ulp-ssrc-mux

Solution #2 - Audio Burst FEC

Mobility + wireless connectivity

Limited storage / memory

Limited Storage / Memory

Recording and Storing Audio/Video

Locally storing raw audio/video frames on a

mobile device is not practical due to the limited

storage space.

Limited storage / memory

Recording and Storing Audio/Video

Encoding video locally on the mobile device

consumes the mobile CPU, meaning less CPU is

available for RTC.

Limited storage / memory

Solution #1: Server Encoding

Relay all media through a server and have the

server do the heavy encoding work.

Limited storage / memory

Increased latency

due to relay and

increased media cost

for hosting.

Solution #2: Local Encoding

Audio/Video is encoded locally and optionally

upload to a server.

Limited storage / memory

Optional to upload audio/video on

a server.

To not impact RTC bandwidth,

upload after the video recording is

complete and not in real-time.

Solution #2: Local Encoding OffloadedUsing mobile hardware encoders makes more sense here

since playback formats are often compatible within the device.

Limited storage / memory

Hardware video encoders work well since they

need not be fully real-time and the encoding

format need not be compatible with the remote

device.

Screen sizes and rotation

Screen Sizes / Rotation

Camera rotation / aspect ratio

Camera rotation and multi angle camera (e.g. back vs front) selection affects the

capture frame rate, aspect ratio, and scalings scalings.

Screen sizes and rotation

Don't waste bits by needless encoding and

sending frame rates and sizes beyond the

remote party's capability to render.

Dynamic Sizing of Render

Local and remote mobile parties can both rotate. The capture side need dynamic

updates of the render's current output capabilities. The sender should maximize

output by not needlessly encoding beyond the capabilities of the render.

Screen sizes and rotation

NOTIFY RENDER SIZE=X,Y

NOTIFY RENDER SIZE=Y,X

Signalling protocol

needs to update

sender constantly of

changing conditions.

Letterboxing / Pillarboxing vs Stretching

The only thing worse than black letterbox / pillarbox bars are stretched images.

Don't stretch images!

Screen sizes and rotation

LetterboxPillarbox

Letterboxing / Pillarboxing vs Cropping

Cropping can work as a solution to fill unused areas but must be done within care

and in known use cases.

Screen sizes and rotation

Image loss = OK

Image loss = NOT

OK

● Specialized signalling helps save battery via push

● Messaging is difficult without store and forward or synchronization

● Heavily encoding / decoding optimization is needed

● Hardware encoders / decoders aren't ubiquitous across devices

● WWAN often requires TURN or IPv6 to work

● Scalable Video Codecs with base layer protection help with burst loss

● Continuous signalling updates of render capability saves CPU and

bandwidth

● Record on a server or passively encode with remaining CPU or via

onboard hardware encoders

● Choose image letterbox/pillarbox or clipping, and not stretching

Mobile Wrap Up

Wrap up

Thanks to the IIT RTC Conference and Hookflash for

allowing me to present.

http://hookflash.com/

http://openpeer.org/

C++ Open Source ORTC Library (mobile friendly)

http://ortc.org/

Thank you