Mobile Meow
By YouTube's Greg Schechter and Matt Ward
Bring YouTube to the Mobile Web
Video
Video
Video
Flash
<video>
<video>
IE
<video>
Chrome
<video>
Firefox
<video>
Opera
<video>
Safari
Video
Playback Stats
Flash Flash
Flash Native <video>
Flash Flash==Flash
Native
○ For fallbacks, use rtsp:// protocol (serving .3pg) if the device won't support HTML5
○ Use custom protocol / URL scheme to launch your own Android, iOS native app (no Windows Phone yet)
■ youtube://video_id
Native
<video>
The Different Browsers
Safari Android
Safari Android Chrome Opera Mini Silk
IE
The Different Browsers
Firefox Opera
Safari Android
Firefox
Chrome
Opera
Opera Mini Silk
IE
The Different Browsers
SAMPLE
Fragmentation
By brownpau at http://www.flickr.com/photos/brownpau/533267369/
<video>
How do we start
<video src="funny_cat_video">
The Source
<video src="funny_cat_video.webm/mp4/ogg">
The Source
Mobile Formats Support
Chrome Safari Firefox Opera IE Android
H.264
WebM Android 4+ Android 4+ Android 4+
HLS Android 3+
Platform Versions and Distribution
The Source
<video> <source src="funny_cat_video.mp4"> <source src="funny_cat_video.webm"></video>
HLS = HTTP Live Streaming
<video src="playlist.m3u8">
HLS
index.m3u8
format-0.m3u8
format-0.m3u8
format-0.m3u8
video-0-1.tsvideo-0-2.ts
...
video-1-1.tsvideo-1-2.ts
...
video-2-1.tsvideo-2-2.ts
...
HLS
Big Features○ Live and Adaptive Streaming○ Built in CDN failover○ Metadata + Cue Points
Apple App Requirements○ > 10 minutes in duration or 5 mb in 5 minutes○ at least one 64 kbps or lower stream required
Tools and more information on Apple's website:Apple's Using HTTP Live Streaming
Power
Power Consumption: H264 vs WebM
**fullscreen flash**windowed flash
Research done by Yossi Oren For more information visit http://iss.oy.ne.ro/
Basic video tag
Safari○ Renders a thumbnail poster and play button (flaky)○ Background is set to black and cannot be overwritten○ If a parent element has display:none set at any point the video fails to
playback properly
Opera○ Gives video dimentions black background if the information is
available○ Has issues with stretching WebM
IE and Android○ Play button/film icon for all video tags regardless if browser can play
the format○ Background is set to black/grey and cannot be overwritten
Poster Attribute
Use the poster attribute to get a thumbnail Safari, Chrome, Android, and Opera
○ Preserve aspect ratio of poster image
IE and Firefox○ Stretch image to size of video tag
IE○ Keeps poster as paused state
All Others○ Replace video with the paused frame
Chrome○ Default click to toggle play/pause○ Remove with an empty onclick handler
Safari○ Will launch any supported video with or without an onclick handler
Android and IE○ Shows click interactions but does not play with empty onclick
Basic video tag
Pro tips:○ Create your own cued state to convey you can play the video○ Set the background color to black for continuity across platforms
Basic video tag
Controls
<video controls>
Controls Chrome Safari Firefox Opera IE Android
Play/pause Button
Volume Controls
Only in Fullscreen
Mute toggle only
Don't work in Android 4
Seek bar
Fullscreen button
Button Gesture Button
Playback Type
Both Fullscreen only
Inline only Determined by hardware
Fullscreen only
Both
Fullscreen
requestFullScreen
requestFullScreen
● Desktop Only○ Firefox, Safari, and Chrome○ Vendor Prefixed
Fullscreenvar elem = document.getElementById("my-element");document.onwebkitfullscreenchange = function() { // The fullscreen element: // document.webkitFullScreenElement; console.log ("We went fullscreen!");};elem.webkitRequestFullScreen();
What about Mobile?
Isn't it already fullscreen?
Yes (sort of)
Fullscreen
● Desktop○ requestFullScreen
● Mobile
Fullscreen
● Desktop○ requestFullScreen
● Mobile○ Open New Tab
Fullscreen
● Desktop○ requestFullScreen
● Mobile○ Open New Tab○ webkitEnterFullscreen
● Webkit only
● Video element only
● metadata must be loaded
Fullscreen
● Desktop○ requestFullScreen
● Mobile○ Open New Tab○ webkitEnterFullscreen
● Webkit only
● Video element only
● metadata must be loaded
● Pro Tip: Remember size context changes so use viewport to scale icons and controls
@viewport { width: device-width; }
Custom Controls
● Preserve your brand● Unified experience across platforms and
browsers● Allows us to expand the set of controls and add our own
○ annotations○ playlist○ captions○ more
Custom Controls Pro Tips● User expect to be able to drag the progress
bar○ Need to remender to prevent scroll on touchstart
● Volume can't be set everywhere and users are accustomed to using device controls○ So don't build controls for it
Custom Controls Pro Tips● Fullscreen
○ The browsing context is always fullscreen so fake it● Don't overload clicking for both play/pause
and show/hiding controls● Don't trigger content with hover
Custom Controls Pro Tips● Fingers are fat
○ Average finger is 11mm so make targets at least 40px with 10px padding
○ Use SVG's so icons can be scaled and shared with desktop application
○ Do what you can in css
autoplay
<video autoplay>
autoplay
In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.
autoplay
In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.
What about everyone else?
Autoplay
Chrome Safari Firefox Opera IE Android
Attribute
Scripted Buggy
autoplay
autoplay
<video onclick="this.play()">
autoplayfunction someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call.} function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); myVideoElement.play();}
autoplayfunction someClickEvent(evt) { // In a user initiated thread. myVideoElement.load(); getVideoData(); // Triggers an ajax call.} function onGetVideoDataReturned(data) { // Not in a user initiated thread. setVideoElementSrc(data); myVideoElement.load(); // For Android window.setTimeout(function() { myVideoElement.play(); }, 0);}
autoplay
setMediaPlaybackRequiresUserAction
Embeds
<script>
Embeds
<script>○ We need our content to be sandboxed○ More than just a video tag
Embeds
<script>○ We need our content to be sandboxed○ More than just a video tag
<object>
Embeds
<script>○ We need our content to be sandboxed○ More than just a video tag
<object>○ Can load content with the data
attribute○ But no way to interact with it via
JavaScript
Embeds
<iframe>
Embeds
<iframe>○ Allows our content to be sandboxed○ JavaScript API communication
<iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen></iframe>
Embeds
Daily impressions of Embed
April 2011 April 2012
iframe
object
Total
Pro tip: Plan for the future (if you can) <iframe type="text/html" width="640" height="385" frameborder="0" src="http://www.youtube.com/embed/VIDEO_ID" allowfullscreen></iframe>
Embeds
Embeds Pro Tips:
html { /** Hack to fix iPhone resizing. */ overflow: hidden;}
Embeds Pro Tips:
body { /** Dymanic Resizing **/ background-color: #000; height: 100%; width: 100%; /** Remove highlight when use clicks **/ -webkit-tap-highlight-color: rgba(0, 0, 0, 0);}
API
API Communication
API Communication● Poll the URL fragment?
○ http://youtube.com/embed/video_id#fragment
API Communication● Poll the URL fragment?
○ http://youtube.com/embed/video_id#fragment● Messages are one dimensional.● Polling eats up CPU and is not instant.● Both directions of communication use the same
fragment.
API Communication● Better idea: PostMessage API
○ someWindow.postMessage(message, targetOrigin);
API Communication● Better idea: PostMessage API.
○ someWindow.postMessage(message, targetOrigin);● Uses JSON for native encoding and decoding of data.● No polling.● Native event listeners.● Communications are sandboxed per-window.● Calls are asynchronous.● Great for Mobile, but desktop support is missing in IE7
Remember all communication is asynchronous
Testing
● Device lab○ accurate○ costly○ space inefficient○ boring
Testing: The Old Fashioned Way
Testing: With Software● Hardware Emulators / Simulators
○ available for major systems○ approximation varies○ still boring
● Android Emulator○ bulky, slow, no video codecs
● iOs Simulator○ both tablet and phone
Testing: With Software● Browser Simulators
○ Poorer approximation● Opera Mobile Simulator
○ missing video tag support● Fennec (FF) Simulator
○ poor touch control mapping○ sends desktop user agent
Testing: Automation● Selenium!
○ use with simulated or real devices○ not boring
Testing: Automation● Android Webdriver
○ Still no video support ● iOS Webdriver
○ Need to register as ios dev○ Intermittent issues with playback
● IE○ No webdriver APIs for mobile
Testing: Automation● Chrome, Opera
○ Driver APIs built-in○ Remote debugging
● FFMobile○ No webdriver APIs
Testing: Automation Strategy● Test API methods in mobile context first
○ cheap○ can approximate video playback by video.
currentTime
Testing: Automation Strategy● More sophisticated tests to follow
○ screenreader● Screen cap processing
○ check for distortion○ playback accuracy
Testing: Automation Strategy● Screen cap processing
Testing: Hardware Assistance
The Future
Cats in Space by WF&TD
Or
PointerEventshttp://msdn.microsoft.com/en-us/library/ie/hh673557(v=vs.85).aspx
Track
<video> <track src="cats_meow" kind="subtitles"></video>
More to learn● HTML5 Video Spec
○ http://dev.w3.org/html5/spec-author-view/video.html● Browser Blogs
○ http://dev.opera.com/○ http://hacks.mozilla.org/○ http://blogs.msdn.com/b/ie/○ http://peter.sh/
● Documentation○ https://developer.mozilla.org/en-US/
● Other○ http://developer.apple.
com/library/safari/#documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html
Questions?
By cloudzilla http://www.flickr.com/photos/cloudzilla/378829651/
[email protected]@google.com
can haz question?
Recommended
