MPEG 4

MPEG 4: The ultimate low bit rate format

MPEG 4: The ultimate low bit rate format

www.chiariglione.org/mpeg/

Overview: ISO/IEC 14496 Coding of audio-visual objects Overview: ISO/IEC 14496 Coding of audio-visual objects • Low bit rate multimedia system -

typically less than MPEG 2 video• Object based - each element is coded

separately• Open ended system that can continue

to develop in the future• Scalable & Interactive

• Version 1: October 1998

• Low bit rate multimedia system - typically less than MPEG 2 video

• Object based - each element is coded separately

• Open ended system that can continue to develop in the future

• Scalable & Interactive

• Version 1: October 1998

VersionsVersions

Version 2

Version 1

ProfilesProfiles

QualityQuality

ComplexityComplexity

DVD

Video CD

Mobiles

MPEG-1

MPEG-2

HDTV

Digital cinema

Advanced Simple Profile

Simple Profile

MPEG 4

Current uses:Current uses:

• 3G mobile phones,• Portable devices, PDAs, video iPod• Interactive television / IPTV• New interactive multimedia formats• Web pages• Interactive music format• Security systems

• 3G mobile phones,• Portable devices, PDAs, video iPod• Interactive television / IPTV• New interactive multimedia formats• Web pages• Interactive music format• Security systems

Basics:Basics:

• Object based system: using Natural and/or Synthetic objects

• Makes use of local processing power to recreate sounds and images

• This makes it one of the most efficient compression systems

• Object based system: using Natural and/or Synthetic objects

• Makes use of local processing power to recreate sounds and images

• This makes it one of the most efficient compression systems

Basics: Object TypesBasics: Object Types

• Photos - JPE, GIF, PNG,• Video - MPEG-2, Divx, AVI, H.264,

QuickTime• Speech - CELP, HVXC, Text to Speech• Music - AAC, MP3, surround• Synthetic music• Graphics - Java code• Text• Animated objects, e.g., talking heads

• Photos - JPE, GIF, PNG,• Video - MPEG-2, Divx, AVI, H.264,

QuickTime• Speech - CELP, HVXC, Text to Speech• Music - AAC, MP3, surround• Synthetic music• Graphics - Java code• Text• Animated objects, e.g., talking heads

Basics:Basics:

• The selected objects are put together into a 2D or 3D scene

• In 3D the viewer can change the shape of the image and view it from other positions in the 3D space. Similar to VRML

• Each object is compressed using the best and most optimum method for that type of data

• The selected objects are put together into a 2D or 3D scene

• In 3D the viewer can change the shape of the image and view it from other positions in the 3D space. Similar to VRML

• Each object is compressed using the best and most optimum method for that type of data

Basics: Virtual StudioBasics: Virtual Studio

• Virtual production techniques are increasingly used in TV production

• Well known chroma-key method using a blue/green screen background

• Actors are overlaid on to a ‘virtual studio’ background image

• The composition of the screen image and the sound can now take place in the decoder at home

• Virtual production techniques are increasingly used in TV production

• Well known chroma-key method using a blue/green screen background

• Actors are overlaid on to a ‘virtual studio’ background image

• The composition of the screen image and the sound can now take place in the decoder at home

BIFS:BIFS:

• Binary Format for Scene descriptor• New description language like HTML

but written in binary not in English • Has scalable levels for audio and video

which can be set by access rights or by interrogation of the receiver to set the best ‘Quality of Service’ (QoS)

• Binary Format for Scene descriptor• New description language like HTML

but written in binary not in English • Has scalable levels for audio and video

which can be set by access rights or by interrogation of the receiver to set the best ‘Quality of Service’ (QoS)

Synchronised StreamingSynchronised Streaming

• Each element can be time stamped to synchronise with other objects in the frame

• Flexi Time: The viewer can vary the time for playback

• There are 3 types, set by the producer, minimum, maximum and optimal

• Audio can be set to change pitch or stay fixed

• Each element can be time stamped to synchronise with other objects in the frame

• Flexi Time: The viewer can vary the time for playback

• There are 3 types, set by the producer, minimum, maximum and optimal

• Audio can be set to change pitch or stay fixed

Compression: SpeechCompression: Speech

• HVXC - Harmonic Vector Excitation Coding• CELP - Code Excited Linear Prediction• 2 - 24 K bit/sec• Synthesised speech: • Text to speech synthesis, 200-1200 bit/sec• Very low delay, 20 ms, for video phone use

MP3 takes too long to encode/decode

• HVXC - Harmonic Vector Excitation Coding• CELP - Code Excited Linear Prediction• 2 - 24 K bit/sec• Synthesised speech: • Text to speech synthesis, 200-1200 bit/sec• Very low delay, 20 ms, for video phone use

MP3 takes too long to encode/decode

Compression: Natural AudioCompression: Natural Audio

• MPEG - AAC (Advanced Audio Coding)

• MP3, AAC, 5.1 surround

• 6 - 380 K bit/sec

• MPEG - AAC (Advanced Audio Coding)

• MP3, AAC, 5.1 surround

• 6 - 380 K bit/sec

Parametric audio codingParametric audio coding• Synthesised audio: Spectral noise re-synthesised • This process separates unique audio sounds

from predictable noise shapes which can then be re-synthesised locally

• Signal is represented by three objects:– Transients: localized in time,– Sinusoids: localized in frequency,– Noise: no strict localization…

• Synthesised audio: Spectral noise re-synthesised • This process separates unique audio sounds

from predictable noise shapes which can then be re-synthesised locally

• Signal is represented by three objects:– Transients: localized in time,– Sinusoids: localized in frequency,– Noise: no strict localization…

Parametric audio codingParametric audio coding

Transients: CastanetsTransients: Castanets


Sinusoids: HarpsichordSinusoids: Harpsichord


Noise: Heavy metalNoise: Heavy metal

Compression: Structured Audio Compression: Structured Audio

SAOL - Structured Audio Orchestra Language (pronounced sail)

• Down loadable sound fonts• Wavetable synth + GM2 type spec.• Any kind of virtual instruments• Virtual effects algorithms and mixers• MIDI data rates e.g. 300 bit/sec

SAOL - Structured Audio Orchestra Language (pronounced sail)

• Down loadable sound fonts• Wavetable synth + GM2 type spec.• Any kind of virtual instruments• Virtual effects algorithms and mixers• MIDI data rates e.g. 300 bit/sec

Interactive Audio Interactive Audio

• Download and remix tracks• Flash interface and compressed

audio loops

• Download and remix tracks• Flash interface and compressed

audio loops

www.yourspins.com

http://www.yourspins.com/

http://www.yourspins.com/

Compression: VideoCompression: Video

• Overall MPEG-4 supports a wide range of standards from very small, poor quality, pictures up to HDTV

• MPEG 1 & 2• MPEG 4 - part 2• MPEG 4 - part 10, 'Advanced Video Coding’

AVC H.264 - high quality video codec• Developed jointly by MPEG and the ITU

• Overall MPEG-4 supports a wide range of standards from very small, poor quality, pictures up to HDTV

• MPEG 1 & 2• MPEG 4 - part 2• MPEG 4 - part 10, 'Advanced Video Coding’

AVC H.264 - high quality video codec• Developed jointly by MPEG and the ITU

Compression: Video - H.264Compression: Video - H.264

• Half to one quarter of the normal bit rate of MPEG-2

• Scalable from 3G to HD• More advanced B frame operation where

the frame can link to any frame in the video sequence

• Smaller 4X4 grids of pixels• 4 Profiles and 16 Levels• Bit rates from 64 Kbps - 240 Mbps

• Half to one quarter of the normal bit rate of MPEG-2

• Scalable from 3G to HD• More advanced B frame operation where

the frame can link to any frame in the video sequence

• Smaller 4X4 grids of pixels• 4 Profiles and 16 Levels• Bit rates from 64 Kbps - 240 Mbps

Compression: VideoCompression: Video

• Mixture of pixel based and vector graphics

• Video is no longer a rectangular shape, it can be any shape

• Synthetic images with bit rates from 5 kbit/sec - 10 Mbit/sec

• Supports the mapping of video textures onto moving objects and meshes

• Mixture of pixel based and vector graphics

• Video is no longer a rectangular shape, it can be any shape

• Synthetic images with bit rates from 5 kbit/sec - 10 Mbit/sec

• Supports the mapping of video textures onto moving objects and meshes

2D mesh model of a fish2D mesh model of a fish

By deforming the mesh the fish can be animated

By deforming the mesh the fish can be animated

Animated Objects:Animated Objects:

• The Animation Framework eXtension, AFX• Face animation: The face models are not

part of MPEG-4 only the movement codes for the expressions, eye movement, etc.

• Body animation works in the same way and can be used in games

• The Animation Framework eXtension, AFX• Face animation: The face models are not

part of MPEG-4 only the movement codes for the expressions, eye movement, etc.

• Body animation works in the same way and can be used in games

Future Options:Future Options:

• MPEG-4 is still being developed and all new parts will work with the old formats

• Studio quality versions for HDTV• Digital cinema 45 - 240 Mbit/sec H.264• Home video cameras with MPEG-4

output straight to the web from the hard drive

• MPEG-4 is still being developed and all new parts will work with the old formats

• Studio quality versions for HDTV• Digital cinema 45 - 240 Mbit/sec H.264• Home video cameras with MPEG-4

output straight to the web from the hard drive


• Integrated Service Digital Broadcast (ISDB)• Newspaper + TV + data• Integration with MPEG 7 databases• Games with 3D texture mapping

• Integrated Service Digital Broadcast (ISDB)• Newspaper + TV + data• Integration with MPEG 7 databases• Games with 3D texture mapping


• TeleVision Modelling Language (TVML)• Computer generated TV programs +

presenters - Max Headroom??

• TeleVision Modelling Language (TVML)• Computer generated TV programs +

presenters - Max Headroom??


• Information booths• Talking objects - fridge, cars, toaster?• Security cameras over the web• Interactive manuals and training materials• New downloadable interactive music format,

SAOL

• Information booths• Talking objects - fridge, cars, toaster?• Security cameras over the web• Interactive manuals and training materials• New downloadable interactive music format,

SAOL

MPEG 7MPEG 7

MPEG 7MPEG 7Multimedia Content Descriptor Standard

Database system to automatically define, organise and search for text, pictures, sound FX, graphics, video clips, songs, music, etc.

• On-line Music library• Automatic identification of music• Uses XML to store metadata

Multimedia Content Descriptor Standard

Database system to automatically define, organise and search for text, pictures, sound FX, graphics, video clips, songs, music, etc.

• On-line Music library• Automatic identification of music• Uses XML to store metadata

MPEG 7MPEG 7Proposed uses:• Live broadcast monitoring, radio output• Digital libraries e.g., image catalogue,

musical dictionary, bio-medical imaging, sound FX, film, video and radio archives

• Cultural services history museums, art galleries, etc.

Proposed uses:• Live broadcast monitoring, radio output• Digital libraries e.g., image catalogue,

musical dictionary, bio-medical imaging, sound FX, film, video and radio archives

• Cultural services history museums, art galleries, etc.

MPEG 7MPEG 7• Home entertainment e.g., systems for the

management of personal multimedia collections, e.g. music, home video, searching a game, karaoke

• E-Commerce e.g., personalised advertising, on-line catalogues, directories of e-shops

• Education e.g., repositories of multimedia courses, multimedia search for support material

• Home entertainment e.g., systems for the management of personal multimedia collections, e.g. music, home video, searching a game, karaoke

• E-Commerce e.g., personalised advertising, on-line catalogues, directories of e-shops

• Education e.g., repositories of multimedia courses, multimedia search for support material

MPEG 7MPEG 7

• Investigation services e.g., human characteristics recognition, forensics

• Journalism e.g. searching speeches of a certain politician using their name, voice or face

• Multimedia directory services e.g. Yellow Pages, tourist information, geographical information systems

• Investigation services e.g., human characteristics recognition, forensics

• Journalism e.g. searching speeches of a certain politician using their name, voice or face

• Multimedia directory services e.g. Yellow Pages, tourist information, geographical information systems

MPEG 7MPEG 7

• Multimedia editing e.g., personalised electronic news service, media authoring

• Social e.g. on-line dating services

• Surveillance e.g., traffic control

• Multimedia editing e.g., personalised electronic news service, media authoring

• Social e.g. on-line dating services

• Surveillance e.g., traffic control

http://www.eptascape.com/products/demoflv.htm

http://www.eptascape.com/products/demoflv.htm

MPEG 21MPEG 21

MPEG 21MPEG 21

• An infrastructure for the delivery and consumption of multimedia content

• Users are seen as either creators, consumers, rights holders, content providers, or distributors

• An infrastructure for the delivery and consumption of multimedia content

• Users are seen as either creators, consumers, rights holders, content providers, or distributors

MPEG 21MPEG 21

• Every media element is defined as a ‘Digital Item’

• Metadata defines what media we can use, what we can do with it and who owns it

• Designed to work with MPEG 4 files and MPEG 7 database

• Every media element is defined as a ‘Digital Item’

• Metadata defines what media we can use, what we can do with it and who owns it

• Designed to work with MPEG 4 files and MPEG 7 database

Technology

MPEG 4