Lecture 8aCSE3030: GUI Technology1 Multimedia Processing and Visual Basic CSE3030: GUI Technology Lecture 8a, Rev. Sept. Dan Eaves

Lecture 8a CSE3030: GUI Technology 1

Multimedia Processing and

Visual Basic

CSE3030: GUI Technology

Lecture 8a, Rev. Sept.

Dan Eaves


Preliminary

• As with static images, we have two different problem domains with moving ones, as well as an extra one:– Desktop video, typically from CD or DVD

– Web video, typically downloaded using some form of streaming

– Desktop or Web animation, typically using Shockwave/ Flash

• We’ll focus on – The compression techniques, esp. the different codecs available:

(codec = COmpress DECompress)

– Integration issues: Getting multimedia into VB.NET projects


CD-ROM as Delivery Medium

• CD-ROM capacity minutes of sound– 44,000 (samples per second) * 2 (bytes per

sample) * 2 (channels for stereo sound) * 60 (seconds per minute) * 70 (minutes) 740MB.

– Let’s say we want to store a 800x600 pixel video, 24 frames per second, 24 bit colour:

• 800*600*3*24 34MB per second.

• 750/34 = 21 seconds of uncompressed video per CD-ROM. Ouch!


DVD as Delivery Medium

• DVD Capacity is a multiple of 4.7 GB– (Movie DVDs have 9.4 GB; most for home use

have 4.7.)– 4,700/34 = 134 seconds = 2 minutes, 14 seconds– 9.400/34 = 268 seconds = 4 minutes, 28 seconds

• No matter which media we use, one hell of a lot of compression is going to be necessary!!!!!!


What Do You, As a GUI Developer, Need to Know about Digital Video?

• We’ve already done half of it:– You have to know why there’s a problem, and why

(see previous slide)

• You have to know a bit about how you deal with the problem, because:– There’s a demand for at least some video in many

systems– There’s frequently a need to integrate your systems

into video-based multimedia systems– As far as “solutions” go, one size doesn’t suit all.


About Players

• The leading “players” are:– Windows Media Player– Apple Quicktime– RealNetwork’s RealPlayer– Netscape’s WinAmp

• All started as CD and MP3 players• All have evolved into much more• We’ll look at the Windows Media Player as an

example – Because it can be readily integrated into VB.NET systems – others limited to C++/C#


What’s the Windows Media Player?Answer #1

• A free product– Important questions:

• Supplied independently? – (Not according to Microsoft!)

• Runs standalone?– (Not according to Microsoft!)

• The EC has found that WMP is a rerun of the “IE is a part of Windows” ploy that did in Netscape – and fined Microsoft 500MB

– Supports audio and video, streaming video, DVD movies, Web stuff, with play lists, indexes of CDs, etc., etc.

– Supports integrated local music collections, including album art– Tightly integrated into WWW and IE.

• Album details when playing CDs• Videos when available on Web sites• Links to music and video stores: Digital Media Mall (gak!)• For a good review of WMP 10, go here.

http://www.winsupersite.com/reviews/wmp10.asp



• A product designed to be extended– New media formats– New codecs– New functionality– New tightly integrated Web links– Care and feeding of different portable music playing

hardware (but not iPod)• Example

– As supplied, MP3 ripping only at a few settings– A plug-in ($US10 or 15) provides full functionality



• A product designed to be used as a plug-in itself– The Media Player can easily be added to your

programs

– (Hummmm… a plug-in that supports plug-ins to itself..)

– Current release: 10, release of 28/11/2005

– Has a matching Software Development Kit (SDK)


The Media Player’s Competitors Are

• Apple with QuickTime– Started out only on Apples, now runs on all platforms

– Part of Apple’s attempt to take over the general multimedia market to complement its traditional total domination of the multimedia creation segment

– Current release: 7.0.4

• RealNetworks with RealPlayer Ten– Started out supporting proprietary streaming media

– Now supports lots of different media formats, via plug-ins

– Evolution: Specialized and Web specific General media player

– Current release: 10.5 (?!)


The Media Player’s Competitors Are (2)

• Nullsoft’s (Netscape) WinAmp 5.12


All 4 Can be Used as VB Plug-Ins, but…

• QuickTime– Can be integrated into Web browser, but prefers Netscape– COM rather than NET (like Media Player itself)– Provides a Windows SDK, but only for C/C++– The native QT format, .MOV, ignored by Microsoft

• RealPlayer One– COM rather than NET (like Media Player itself)– Core engine (Helix) placed in public domain, being used to

develop plug-ins for various environments– Specialised for streaming media in the RM (Real Media)

formats. (Helix open source project has access to these via binary code provided by RealNetworks)

– Partnership with Sun, intended for the Linux/Unix market


HELIX – The Core of the RealOne PlayerBeing extended by independent developers as universal toolkit and plug-in

From: https://www.helixcommunity.org/


QuickTime as a Bolt-on

• Native support for integration in Unix/Linux (e.g., Java) applications

• QuickTime 6 Windows SDK for Windows applications – COM model, designed for C/C++

• Native format: MOV aka QT


MPEG4 (Based on Apple’s Native QuickTime Format


QuickTime Believes In (Note: These are formats, not codecs)

• Video formats: AVI, DV, Motion JPEG, MPEG-1, MPEG-2, MPEG-4

• Audio formats: AIFF/AIFC, Audio CD, MP3, Sound Designer II, System 7 Sound, uLaw (AU), WAV, MPEG-4

• Still-image formats: BMP, GIF, JPEG/JFIF, MacPaint, PICT, PNG, Photoshop (with layers), SGI, Targa, FlashPix, TIFF

• Animation formats: 3DMF, Animated GIF, FLC/FLI, Flash, PICS • Other formats: KAR (Karaoke), MIDI, QuickDraw GX, QuickTime

Image File.


RealOne Player Believes In (Note: These are formats, not codecs)

• A2B .mes• Active Stream Format .asf• Audio File .au• Blue Matter Files . bmo, .bmr, .bmt• GIF File Format .gif, .png• IBM EMMS Files . emm• Liquid Audio .lqt• MJuice Files .mjf• MPEG Files .mp3, .mpeg, .mpa, .mp2, .mpv, .mx3• MPEG URL (MIME Audio File).m3u• Macromedia Flash .swf• QuickTime Files .avi, .aiff• RAM Meta File .ram, .rmm• RealAudio, RealMedia ra, .rm, .rmx, .rmj, .rms• RealOne Music .mnd• RealPix .rp• RealText .rt• WAVE .wav• Windows Media Audio .wma


Windows Media Player Believes In

• Windows Media file (asf): asf, .asx, .wm, .wmx, .wmp, and .wpl

• Windows Media Audio file (wma): .wma and .wax

• Windows Media Video file (wmv): .wmv, and .wvx

• DVD Video: .vob

• Music CD Playback: .cda

• Windows video file (avi): .avi

• Windows audio file (wav): .wav

• Movie file (mpeg): .mpeg, .mpg, .mpe, .m1v, .mp2, .mpv2, .mp2v*, and .mpa

• MP3 audio file (mp3): .mp3 and .m3u extensions.

• MIDI file (midi): .mid, .midi, and .rmi

• But not .mov, the Apple QuickTime format! Or MPEG4, yet.


Why All the Different Formats?

• Both formats and, especially, codecs need to be asymmetrical:– Spend lots of time putting data into the format, but

– Spend little time going from file to screen

– (We have lots of time available to compress – not real time, but little time to decompress – is real time)

– (Unless, of course are doing real time streaming!)

• Raw file sizes are so large, and processing demands so great, that:– No obvious solutions, right answers

– Many different tricks possible

• Formats predate contemporary desktop processing power and storage capacity, as well as development of DVD

• Perhaps MPEG4 will become universal standard.


Streaming

• Getting video over the web is like drinking from a fire hose: Too much data too fast…

• Two types of streaming– Progressive download –

• delay start of playing and pray you keep ahead

– True streaming• A lost chunk of data won’t be missed, so

• Abandon TCP/IP and HTTP – no receipt acknowledgement

• Compress drastically

• Adjust compression to current transmission rate (e.g., cheat and set up something like a session/state)

• Lost frames, uncoordinated sound, are the only sins…


The Real Streaming Formats

• RealNetworks has developed a number of proprietary formats for video and audio

• These are highly lossy, so quality is very low.

• They are also very compact, so Web broadcasting/ singlecasting becomes possible.– (You pays your money and you takes your choice!)

• Note that streaming in real time presents different problems from streaming on demand: the first has to use symmetrical compression while the second can be asymmetrical


Streaming is a GUI Technology???

• Remember that “.NET” in VB.NET?– .NET is designed to write programs which operate on the Web

both serverside and clientside.

– While 3030 is limited to desktop/LAN programming• Your job may well not be

• With widespread adoption of (cheap) broadband Web access, streaming will become important, again…

• Both Microsoft, RealPlayer and Apple have their own streaming technologies.


Codecs

• Symmetrical or asymmetrical? The second is better!

• The kind of scenes– “Talking head” with static background is best

– “Talking head” with background of trees in wind is the worst

– We want as little difference between frames as possible

• Primary compression technique– Store only differences

• Between parts of a frame

• Across successive frames

– Store complete frames (key frames) only occasionally (every 7th?)


You Need to Know

• There are lots of different codecs

• They specialise in different types of scenes

• The most important factor in successful compression is in the nature of the scene being compressed, not the codec chosen.

• You will have to disappoint your customers regularly regarding what’s possible in multimedia

Documents

Lecture 8aCSE3030: GUI Technology1 Multimedia Processing and Visual Basic CSE3030: GUI Technology Lecture 8a, Rev. Sept. Dan Eaves