Yapi.js, An Adaptive Streaming Web Player

INTRODUCE YAPI.JS

(AN ADAPTIVE STREAMING WEB PLAYER)

PART2

KKBOXVideo陳建辰 jessechen

AGENDA

• yapi.js, video element and MSE

• Introduce yapi sample control panel

• Test cases

BRIEF SUMMARY OF LAST PRESENTATION

http://goo.gl/cQoq2K

HOW TO USE YAPI.JS

var yapi = new yapi.MediaPlayer().initialize();

yapi.attachView(<video></video>);

yapi.load(MANIFEST_URL);

YAPI.JS / VIDEO ELEMENT / MSE

VIDEO ELEMENT

var videoNode = document.createElement(‘video’);

videoNode.src = VIDEO_URL;

With html5 video element, you can play single video source easily

more info here: https://goo.gl/Fyi3Z3

MSE

“ MSE (Media Source Extension) extends HTMLMediaElement to allow JavaScript to generate media streams for playback.

Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams. “

var video = document.createElement(‘video’);

video.src = VIDEO_URL

MEDIASOURCE IS A ‘SOURCE’

set ‘src’ attribute of video element to an url pointed to media source

new window.MediaSource();window.URL.createObjectURL(ms);

SOURCE BUFFER

sourceBufferVideo = ms.addSourceBuffer(VIDEO_CODEC); sourceBufferAudio = ms.addSourceBuffer(AUDIO_CODEC);

// get stream buffer via network

sourceBufferVideo.appendBuffer(buffer);

// sourcebuffer provides buffer info after append complete

BUFFER INFO

var buffered = sourceBuffer.buffered; buffered.length; // how many discontinuous buffered time range

buffered.start(0); // first buffer start time

buffered.end(0); // first buffer end time

get buffer information from buffered attribute

ADAPTIVE STREAMING// assume segment of time 0s~5s is needed // yapi would decide which bitrate to download loadSegment(segment_1_240p_url); // segment file doesn’t have meta data, we have to append it first sourceBuffer.appendBuffer(init_240p_buffer); // then loaded segment sourceBuffer.appendBuffer(segment_1_240p_buffer); // if yapi decide to load 480p for 5~10s (adaptive algorithm) // then yapi repeat same process, but with 480p // then playback would be 240p during 0~5s and 480p for 5~10s

GRAPH

yapi append stream buffer to sourcebuffer ‘adaptively’

MediaSource

Media Element media element plays

while available

MSE EXTENDS MEDIA ELEMENT

• MSE focus on providing stream buffer to media element

• playback behavior still hold by media element

MEDIA EVENTS

http://www.w3.org/2010/05/video/mediaevents.html

Dispatched by HTML5 media element, notifying playback process

USEFUL EVENTS FOR YAPI.JS

loadstart: Indicate loading media begins, this fires when setting src attribute

loadedmetadata: while element has basic info of playback, e.g duration

timeupdate: while current time of playback is ticking

seeking/seeked: while conducting seek

ended: playback ends

play/playing: playback resume from other status to playing

yapi

YAPI PROXY EVENTS

Media Elementeventevent with same name

ADDITIONAL EVENTSloadedmanifest: after manifest is loaded/parsed

bitratechanged: when bitrate is changed

enableabr: when adaptive activation changed

buffering: when playback pending

cuechanged: webvtt subtitle cue changed (in and out)

BEHAVIOR WITH MEDIA SOURCE

SEEK BEHAVIOR 1

• mediaElement.currentTime = TIME_SEEK_TOwill conduct seek

• use yapi.seek(TIME_SEEK_TO) instead

SEEK BEHAVIOR 2

• seek is totally done by mediaElement, so yapi.js listen to seeking and seeked event to know seek starts and ends

• seeking event dispatched by yapi would have seekFrom and seekTo as parameters

PENDING DETECTION

• No useful event to indicate pending situation waiting stalled emptied

• pending detection is done on every timeupdate event, yapi.js would check media source buffered length

PLAYBACK END

• mediaElement dispatch ended event natively when playback goes to end

• but not while attaching media source to it

• mediaSource.endOfStream() must be invoked beforehand

yapi

YAPI ENCAPSULATE MEDIA ELEMENT

Media Elementexternal api

SUMMARY

• media source provide stream buffer for mediaElementmediaElement.src = createObjectURL(ms)

• listen to events dispatched by yapi, instead of mediaElement

• use api exposed by yapi (call api of mediaElement not recommended) e.g seek

M3U8 ON SAFARI

• attach m3u8 to mediaElement on safari by simplymediaElement.src = URL_TO_M3U8

INTRODUCE YAPI SAMPLE UI

MediaCtrlPanel… and its children

Volume UI

HOW’D YOU ASSEMBLE?

ProgressBar slider

slider

client

yapi

SLIDER CLASS

• slider constructor takes a node and orientation as mandatory parameters

• it focus on 1. locate pointer with given param (event or percentage)2. return location with given param3. notify listeners events occurred

TAKE A LOOK AT PROGRESS BAR

• played bar

• buffered bar

• seek time indicator

• preview

AND VOLUME UI

• value bar

BEHAVIOR FLOWscenario 1: progress changed by app. e.g yapi.seek(TIME) invoked

MediaCtrlPanel ProgressBar slideryapi

scenario 2: progress changed by user interaction, e.g click on slider

MediaCtrlPanel ProgressBar slideryapi

SCENARIO 1: FROM APP

MediaCtrlPanel.onSeeking(event)yapi.seek(time)

MediaCtrlPanel listens to ‘seeking’ event emitted from yapi, then calculates percentage of current progress

(currentTime / totalDuration)

SCENARIO 1: FROM APP

MediaCtrlPanel. ProgressBar.locatePointer(percentage)yapi.seek(time)

SCENARIO 1: FROM APP

MediaCtrlPanel. ProgressBar.locatePointer(percentage)

slider.locatePointer(percentage)

yapi.seek(time)

progressBar update played bar

SCENARIO 1: FROM APP

MediaCtrlPanel. ProgressBar.locatePointer(percentage)

slider.locatePointer(percentage)

yapi.seek(time)

slider.locatePointer would invoke method locatePointerInternal,with1. it’s argument 2. notToDispatchEvent as true (I know it’s tricky)

SCENARIO 2: FROM SLIDER

assume user conduct seek by click on slider, b/c slider.addEventListener(‘click', locatePointerInternal);

so it will dispatch POINTER_LOCATED event

slider

(Remember notToDispatchEvent param?)

SCENARIO 2: FROM SLIDER

progressBar listens to ‘POINTER_LOCATED’ event andupdate played bar

ProgressBar slider

SCENARIO 2: FROM SLIDER

MediaCtrlPanel also listens to ‘POINTER_LOCATED’ eventand it will invoke yapi.seek(calculatedTime)

(percentage of time is within POINTER_LOCATED event obj)

MediaCtrlPanel ProgressBarslideryapi

A LESSON

“ take care of flow direction, or it might be an infinite loop “

MediaCtrlPanel ProgressBar slideryapi

yapi MediaCtrlPanel ProgressBar slider

LET’S TALK ABOUT EVENT

LISTEN TO EVENT

consider on these 2 phrases

“ progressBar listens to ‘POINTER_LOCATED’ event “

“ MediaCtrlPanel also listens to ‘POINTER_LOCATED’ event “

BAD IMPLEMENTATIONbut it works!

progressBar.addEventListener();mediaCtrlPanel.addEventListener();

slider.addEventListener(POINTER_LOCATED, callback);

ADD LISTENERit would call method with same name of eventBus

slider eventBus.addEventListener(POINTER_LOCATED, callback)

while invoking this, eventBus will manipulate an object holding map of event name and callback.

in the case, POINTER_LOCATED would be key, and an array as value with callback pushed into it.

DISPATCH EVENTPOINTER_LOCATED is an event from slider

it will find array of key: POINTER_LOCATED, and invoke every element (callback) of it.

slider eventBus.dispatchEvent(POINTER_LOCATED)

WHAT’S THE BAD PART?• mediaCtrlPanl needs a reference of slider

MediaCtrlPanel ProgressBar sliderconflicts with

mediaCtrlPanel.addEventListener();

slider.addEventListener();

• for every dispatcher, an eventBus instance is neededslider eventBus.addEventListener() eventBus.dispatchEvent()

assume progressBar would dispatch a ‘PROGRESS_BAR_HIDE’ event, then every module which listens this event needs progressBar reference,

and progressBar needs a private eventBus for mapping

progressBar? another eventBus

TAKE A LOOK AT DOM EVENT

while click on <td> <table> would also get it

“by default”

http://www.w3.org/TR/DOM-Level-3-Events/

HOW COME?

<td>.addEventListener(‘click’, callback)<table>.addEventListener(‘click’, callback)

TAKE A LOOK AT DOM EVENT

http://www.w3.org/TR/DOM-Level-3-Events/

a singleton eventBus holds by every node

while add listener

it knows which node listens

while dispatchingit invoke callback with node as context

THEREFORE

MediaCtrlPanel… and its children

Volume UI

ProgressBar slider

slider

eventBus

that’s why DI needed

SUMMARY

• find common part and reuse it

• remember: behavior has direction

• event indicates states of application, handle it well

client

INITIALIZE CTRL PANEL

MediaCtrlPanelProgressBar

Volume UI

• in client app, invoke new MediaCtrlPanel()

• call setup method, it loads template html

• then apply component functions and initialize progress-bar and volume-ui

INITIALIZATION DETAIL• in client app, invoke new MediaCtrlPanel()

• in panel.setup(), loads template html

• after html loaded, apply component functions

also initialize progress-bar and volume-ui

IF A MODULE HANDLING THIS• that module called ‘yapiDOM’ with a method ‘render’

• takes 2 parameters:1. constructor of a component, 2. selected node this component would mount on

• while `render()` invoked, yapiDOM will instantiate given constructor and run setup method of component(load template and apply component functions )

IT WOULD BE LIKE…

very similar to React !

ATTACH VARIABLE

• attach yapi player to media ctrl panel, and listens to player event

IF YAPIDOM ALSO HANDLES THIS

• yapiDOM has another method ‘createElement’

• takes 2 parameters:1. constructor of a component, 2. object with map of key/variable

• while `createElement()` invoked, yapiDOM will instantiate given constructor and run setup method of component

SETUP COMPONENT

1. load html (js sync / html async)

2. apply component functions

3. hold reference of given map object

4. return created component node

IT WOULD BE LIKE…

just like React !

SUB COMPONENT

• create target component inside setup method of media ctrl panel

SUMMARYComponent

1. have a conventional ‘setup’ method with yapiDOM

2. setting these while setup:• template• behavior• reference

yapiDOM

1. createElement method:run instance method ‘setup’ of given component class

2. render method:append node to target node

attach singleton eventBus here

UNIT TEST JASMINE / KARMA

JASMINE =MOCHA +

CHAI + SINON

framework

assertion

spy

COMPARISONsetup of running on browser

jasmine

mocha

COMPARISON

https://github.com/ddhp/test-framework-comparison

RUN ON COMMAND LINE

• can be run on phantomJS, a headless browser

• easier integrate with CI

TASK RUNNER

• grunt-mocha takes html file as source

• for grunt-jasmine, set up source, vendor, spec javascript files in task config

WHICH I PREFER

• flexibility comes with complexity

• mocha can do everything, so it’s important that you make choice and being consistent

• jasmine is capable enough for client-side unit test

TDD / BDD

• mocha.setup() can decide which strategy to use (link)

• it’s just the difference between syntax

TIPS OF WRITING TEST• it doesn’t matter which strategy to adopt, just get your hands

dirty

• get code coverage 100%

• add relevant test case while adding feature (b/c there must be a reason you do this)

• also while fixing bug (prevent from occurs again)

INTRODUCE KARMA

CONSIDER THESE FEATURES

• watch on modification

• preprocess

• run on different browsers simultaneously

• code coverage report

KARMA IS…

• a test runner

• would spawn a server by default on port 9876 which serves test cases

CONFIG KARMA• list needed plugins

• decide which framework and browsers to run on

• setup preprocessors

• list reporters

• list required helpers, vendors, sources and specs in files

TIPS

• setup different config for developing and CIdeveloping: runs on different browsers CI: runs on phantomjs with coverage report

• if grunt-watch is exploited, it’s better to disable watch feature of karma i.e each karma run is a single run, and re-run it when file changed

THANK YOU

w3 mse spec, media event sampleMDN media element doc, HTMLMediaElement docdash.jsAngular.js Directive docReact.js Getting Start tutorial

ref: