Corralling the Chaos of within Multiple File Formats the good ‘ol days versus today •File wrapper format “standards” that are too loose and interpreted in too many different

Corralling the Chaos

of

Ancillary Data

within

Multiple File Formats

Sara Kudrle

Miranda Technologies

A BELDEN Brand

Wild West of Files

Anatomy Lesson

Corral The Chaos

Wrap It Up

The Poor Cousin

Wild West of Files

Ingest and Quality Control

Playout

and Graphics

Publishing for

On Demand

Editing and Content Prep

Live Production

Media Management and Storage

Automation

Linear Channels

Pre- Produced

Content and Ads

WAN Non Linear Delivery WAN

Typical Television Broadcast Facility and Workflow

Broadcast Facilities Have Been Stream / SDI Centric

Ingest and

Quality Control

Playout and Graphics

Non Linear Publishing


Live

Production


Automation

Linear Channels

Pre- Produced Content and

Ads

WAN

Linear Feeds

Tape (Linear Medium)

Linear Streams

Isolated Islands of IT / File Based Infrastructure

Broadcast Industry Specific SDI Infrastructure

Non Linear Delivery

WAN

Today Facilities Becoming Increasingly “File Based”

Ingest and

Quality Control

Playout

and Graphics

Non Linear Publishing


Live Production


Automation


Live Production

Linear Channels

Pre- Produced Content and Ads

WAN

Tape (Linear Medium) Non Linear

Delivery WAN

Media Files

Media Files

Linear Feeds

Linear Streams

Isolated Islands of

IT / File Based Infrastructure



IT / File Base Infrastructure

Contrasting the good ‘ol days versus today

• File wrapper format

“standards” that are too

loose and interpreted in too

many different ways (e.g.

MXF)

WAN Files

• Missing / misplaced meta

data

SDI

• “Iron Clad” tape formats

• “Cast in stone” device

interconnect

(SMPTE SDI) with precise

• No provisions for descriptive

meta data

Then (the good ‘ol days)

Today (the wild west)

The Early Days of File Based

Same Vendor Provides Ingest and Playout

SDI

SDI

SDI

Third party

media server

Homogeneous Environment

ONE Common File Format

INGEST PLAYOUT

STORE

Today’s Reality – The Wild West of Files

SDI

Third party

media server

INGEST

PLAYOUT

STORE

Networked VTRS

WAN File

Delivery

Service

IMX

LXF

?XF Ethernet /

IP Network

MXF

Flv A

Most File Formats are Not Interoperable

IMX LXF ?XF MXF

Flv X X X X

Wild West of Files

Anatomy Lesson

Corral The Chaos

Wrap It Up

The Poor Cousin

Wild West of Files

Anatomy of a Media File

Wrapper

Codec

Essence

Anatomy of a Media File

Ties Essence together & describes it • Structural metadata such as SOM, EOM,

Duration, types of media included in file, etc.

• Descriptive metadata such as creation date, usage

rules, key words

• Time References and Index

• Video (typically compressed)

• Multiple Audio (often compressed)

• Ancillary Data (e.g. Captions, AFD, …)

“Essence”:The actual media which includes:

Wrapper:

Media

File

Two Storage Approaches

Wrapper

Descriptive Metadata

Timecode / index

Video Essence

Audio Essence(s) Audio Essence(s)

Ancillary Data Essence

Method 1:

Embedded Essence

Examples:

• MXF XDCAM

• GXF

• LXF

One File

Structural Metadata

Method 2:

Reference Files / External Essence

Wrapper


Timecode / index

Video Essence



Structural Metadata

Main

File

Multiple

“Referenced”

Files

Examples:

• Omneon MOV

• MXF P2

• MXF AS-02

Two Storage Approaches

Wrapper


Timecode / index

Video Essence



Method 1:

Embedded Essence

Examples:

• MXF XDCAM

• GXF

• LXF

One File

Structural Metadata

Method 2:

Reference Files / External Essence

Wrapper


Timecode / index

Video Essence



Structural Metadata

Main

File

Multiple

“Referenced”

Files

Examples:

• Omneon MOV

• MXF P2

• MXF AS-02

Benefit:

one file includes

everything, easier

media management

Benefit:

Additional essences can be

more quickly added after

the fact (e.g multiple

languages audio and

captions)

Wild West of Files

Anatomy Lesson

Corral The Chaos

Wrap It Up

The Poor Cousin

Wild West of Files

Popular Wrapper Formats

Vendor Specific

Wrappers

Associated with

Camera and VTR

Formats

Vendor Specific

Server Formats

Generic

Industry

Wrappers

.MOV

P2 MXF

(Panasonic)

Wrappers

Originating in

PC/Multi-Media

MXF OP1a

MXF-AS-XX

GXF

(Grass Valley)

LXF

(Harris)

IMX/XDCAM MXF

(Sony)

.AVI

MPEG TS File

Television

Distribution

format

© 2012 SMPTE · The 2012 Annual Technical Conference & Exhibition · www.smpte2012.org

Key Features of Popular Media Wrappers

MPEG-2 TS

MOV / MP4 AVI

GXF &

LXF Generic

MXF

IMX / XDCAM

MXF P2 MXF

Avid MXF

(OP Atom)

MXF AS-02

Editing

NLE Support

Random Access

Edit-while-ingest

Playout

Transcoder Support

Native Server Support

ANC carriage

Additions/Changes without full remux

Archiving

Descriptive Metadata Carriage

Future-proofness

Popular Wrapper Formats



MPEG-

2 TS MOV / MP4 AVI

GXF &

LXF Generic

MXF

IMX / XDCAM

MXF P2 MXF

Avid MXF

(OP Atom)

MXF AS-02

Editing

NLE Support

Random Access

Edit-while-ingest

Playout

Transcoder Support


ANC carriage


Archiving


Future-proofness

Key Features / Functionality



MPEG-

2 TS MOV / MP4 AVI

GXF &

LXF Generic

MXF

IMX / XDCAM

MXF P2 MXF

Avid MXF

(OP Atom)

MXF AS-02

Editing

NLE Support

Random Access

Edit-while-ingest

Playout

Transcoder Support


ANC carriage


Archiving


Future-proofness

Not

Supported

(Bad)

Strongly

Supported

(Very Good)

Partially

Supported



MPEG-

2 TS MOV / MP4 AVI

GXF &

LXF Generic

MXF

IMX / XDCAM

MXF P2

MXF

Avid MXF

(OP Atom)

MXF AS-02

Editing

NLE Support

Random Access

Edit-while-ingest

Playout

Transcoder Support


ANC carriage


Archiving


Future-proofness

Observation 1: There are a lot of wrapper formats

and they are obviously very different

Observation 2: No ONE wrapper format supports all

the identified functionality



MPEG-

2 TS MOV / MP4 AVI

GXF &

LXF Generic

MXF

IMX / XDCAM

MXF P2

MXF

Avid MXF

(OP Atom)

MXF AS-02

Editing

NLE Support

Random Access

Edit-while-ingest

Playout

Transcoder Support


ANC carriage


Archiving


Future-proofness

Observation 3: There are a couple

clear “bad choices”


MPEG-2 TS

MOV / MP4 AVI

GXF &

LXF Generic

MXF

IMX / XDCAM

MXF P2

MXF

Avid MXF

(OP Atom)

MXF AS-02

Editing

NLE Support

Random Access

Edit-while-ingest

Playout

Transcoder Support


ANC carriage


Archiving


Future-proofness

Observation 4: At first glance MXF appears promising but in

reality has turned out to have major

interoperability issues




MPEG-

2 TS MOV / MP4 AVI

GXF &

LXF Generic

MXF

IMX / XDCAM

MXF P2

MXF

Avid MXF

(OP Atom)

MXF AS-02

Editing

NLE Support

Random Access

Edit-while-ingest

Playout

Transcoder Support


ANC carriage


Archiving


Future-proofness

Observation 5:

Hard-core constraints (Tape fmt Heritage),

Clearly Documented, No Ambiguity

Reasonable bit rate (HD at 50 Mbps),

Wide NLE and Server Support

IMX / XDCAM MXF has turned out to

be a very popular Generic File Format

Wrapper Divergence!


Why has MXF Failed and What Are Alternatives?

More

Specific

More

constraints

Better

Generic MXF

LXF Leitch/Harris

GXF Grass Valley

IMX/XDCAM

MXF Sony

Vendor

Specific

Variants

Generic

Variants

• Far too loose &

flexible

• Subject to broad

interpretation

• No hard reference

• Vendor specific

• Some published

some proprietary

• Some overly

constrained

• Harder to Extend

AS-02 Versioning

AMWA

Application

Specifications

AS-03 Program

Delivery AS-10

Production AS-12 Ad

Delivery

AS-11 Contribution

• Constraints on

MXF for different

Applications

• Industry

Standard ?


AMWA MXF AS-XX Application Specifications

• Defined by AMWA (Advanced Media Workflow Association)

• AS-XX MXF Specifications Define a set of rules / constraints on MXF

to suit a particular use

• Defines MXF use for different situations / applications

• Shims constrain further to meet needs of a user

MXF

Constrained

Parameters,

and Values

Available Codecs

Constraints on

Codecs and

Bit-rate ranges

AS-XX

Application Specification

User A SHIM User B SHIM


AS-XX Format Summary

Number

Application

Domain Essences CODECs

HD Bit

Rates (Mbps)

Meta

Data

Audio

Chans

AS-02 Versioning External Not Specified Depends

on

Codec

Not

Specified

AS-03 Program

Delivery Embedded MPEG2, H.264

LGOP 5-50

Mb/s 2-16

AS-10 Production

(e.g. Sony

XDCAM MXF)

Embedded MPEG-2 MP@HL,

MPEG-2 422P@HL

MPEG-2 MP@H14

LGOP

Depends

on

Codec

2-8

AS-11 Contribution Embedded

?

AVC Intra 100 Mbps 2-64

AS-12 Ad Delivery Embedded MPEG2, H.264

LGOP

5-50

Mb/s

2-16


Different AS-XX Specifications for Different Apps

AS-12 Ad Delivery

Regional Playout

Facility

Central Broadcaster / Playout Content Creation

AS-02 Versioning

AS-10 End to End ( Acquisition – Production – Playout)

AS-11 Contribution

Aquisition Post Contribution Playout

Preparation Distribution

AS-03 Program Delivery

Packaging of multiple

versions of same program

Multiple Languages,

Multiple Editorial Versions

Delivery of finished

programs. Single File, single

audio language, constrained

codecs


Commercials. Identical to

AS-03 but with Advertising

specific Meta Data

Enable Interoperation and an end to end

workflow without transcoding.

Specifies Long GOP MPEG2 codec at a decent

bit rate

Designed as extension to SMPTE RDD 9 and

is used by IMX & XDCAM-HD

Vendor-neutral file format

for the delivery of finished program material.

Similar to AS-03 but features higher bit rate

AVC-Intra video and uncompressed Audio


Different AS-XX Specifications for Different Apps

AS-12 Ad Delivery

Regional Playout

Facility

Central Broadcaster / Playout Content Creation

AS-02 Versioning

AS-10 Production using Long GOP Codec (e.g. XDCAM)

AS-11 Contribution

Aquisition Post Contribution Playout

Preparation Distribution

AS-03 Program Delivery


programs. Single audio

language, constrained

codecs

= User A SHIM User B SHIM Observation :

Even within versions, AMWA has

specified that “An Application

Specification can be further constrained

to meet the needs of a user through use

of a SHIM”

In other words, they are

different


Why has MXF Failed and What Are Alternatives?

• Constraints on

MXF for different

Application

• Industry

Standard ? More

Specific

Better

MXF

LXF Leitch/Harris

GXF Grass Valley

IMX/XDCAM

MXF Sony

Vendor

Specific

Variants

Generic

Variants

AS-02 Versioning

AMWA Application

Specifications

AS-03 Program

Delivery AS-10

Production AS-12 Ad

Delivery

AS-11 Contribution

Wild West of Files

Anatomy Lesson

Corral The Chaos

Wrap It Up

The Poor Cousin

Wild West of Files

Ancillary Data

Four main ways to carry Ancillary data in files

In the VBI portion of video (IMX only)

In the MPEG GOP Header (limited)

In the wrapper (limited)

As a s436m payload

ANC Data In Legacy File Formats Was Inconsistent

and Incomplete

HANC

Audio

T/C

VANC AFD, Dolby, CC, VCHIP,

SCTE Triggers, Private Data

Active

Video

File Wrapper

Video Essence

Audio Essence (s)

Some ANC in Codec Header

Some ANC in Wrapper Meta

SDI

Time Code

And

/ Or

Some,

Not All

Some ANC in VBI (SD only)

Legacy File

Formats

Ancillary Data

Dolby

Metadata

SCTE-104

Ad Triggers

Broadcast

Flag

OP47

Teletext Private

Data

$

XDS

V-Chip

WSS

AFD

Timecode

Closed

Captions

Dolby

Metadata

SCTE-104

Ad Triggers

Broadcast

Flag

OP47

Teletext Private

Data

$

VBI (IMX Only) XDS

V-Chip

WSS

AFD

Timecode

Closed

Captions

Ancillary Data

Dolby

Metadata

SCTE-104

Ad Triggers

Broadcast

Flag

OP47

Teletext Private

Data

$

VBI (IMX Only)

Codec Header

XDS

V-Chip

WSS

AFD

Timecode

Closed

Captions

Ancillary Data

Dolby

Metadata

SCTE-104

Ad Triggers

Broadcast

Flag

OP47

Teletext Private

Data

$

VBI (IMX Only)

Codec Header

XDS

V-Chip

WSS

AFD

Timecode

Closed

Captions

Wrapper

Metadata

Ancillary Data

Dolby

Metadata

SCTE-104

Ad Triggers

Broadcast

Flag

OP47

Teletext Private

Data

$

VBI (IMX Only)

Codec Header

XDS

V-Chip

WSS

AFD

Timecode

Closed

Captions

Wrapper

Metadata Key ANC

Data not

Captured

Ancillary Data

ANC Data In Legacy File Formats Was Inconsistent

and Incomplete

HANC

Audio

T/C

VANC Various ANC Data

Active

Video

Original Source Video Legacy File

Formats

File Wrapper

Video Essence

Audio Essence (s)

Time Code

Playback Video

Ingest

HANC

Audio

T/C

VANC Various ANC Data

Active

Video

Playback

Missing

ANC Data

Can’t Easily

Add or Modify

ANC Data to File

Original

ANC Data Subset of

ANC Data

Wild West of Files

Anatomy Lesson

Corral The Chaos

Wrap It Up

The Poor Cousin

Wild West of Files

SMPTE To the Rescue

SMPTE 436M:

MXF Mappings for

VBI Lines and

Ancillary Data Packets

Codec Header

Wrapper

Metadata

SMPTE 436m

Key ANC

Data NOW

Captured

XDS

V-Chip

WSS

AFD

Timecode

Closed

Captions Broadcast

Flag

OP47

Teletext Dolby

Metadata

Private

Data

SCTE-104

Ad Triggers $

Ancillary Data

SMPTE 436 Dramatically Improves ANC Carriage

HANC

Audio

T/C

VANC AFD, Dolby, CC, VCHIP,

SCTE Triggers, Special Data

Active

Video

SMPTE 436 ANC Essence

SDI

MXF File with

436M Essence

File Wrapper

Video Essence

Audio Essence (s)

Time Code

ALL

ANC

Wrapper

Video Essence

Audio Essence

SMPTE 436 ANC Essence CC

AFD

DOLBY

SCTE

Propri

Frame Accurate Index DID

ANC

Data

Line

Num H

C

S

MXF File with

436M Essence

CC

AFD

DOLBY

SCTE

Propri

CC

AFD

DOLBY

SCTE

Propri

CC

AFD

DOLBY

SCTE

Propri

Typical SMPTE

436 Packet


HANC

Audio

T/C

VANC DID tagged streams

Active

Video

VANC

Audio

T/C

VANC DID tagged streams

Active

Video SMPTE 436 ANC Essence

MXF File with

436M Essence

File Wrapper

Video Essence

Audio Essence (s)

Time Code

Original Source Video Playback Video

Ingest Playback

Add and

Modify ANC Easily

Original

ANC Data

Original

ANC Data

Augmented

ANC Data

Original

ANC Data

Augmented

ANC Data


Ancillary Data Using SMPTE 436M

Standardized by SMPTE in 2006

Can carry any DID/SDID (i.e. future-proof)

Carried as a separate essence payload in MXF

Can be embedded or referenced

Easy to re-wrap

Easy to Add / Modify

Supported in MXF and AS-N Application Specifications

Now being added to SMPTE RDD9 for AS-10 / IMX/XD-CAM

SMPTE-436M

Ancillary Data

Carriage in

Files

It solves the biggest file interoperability

problems we have experienced

The most important thing to happen

to File Based since the folder was invented

What Can I Do?

S436m

Get Involved!

Ask for it!

Comments or Questions?

[email protected]

The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials. 2011. Title of Presentation, Meeting name and location.: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at [email protected] or 914-761-1100 (3 Barker Ave., White Plains, NY 10601).

SMPTE Meeting Presentation

Corralling the Chaos of Ancillary Data within Multiple File Formats

Corralling the Chaos of Ancillary Data within Multiple File Formats

Sara Kudrle, Sr. Software Engineer Miranda Technologies, 125 Crown Point Ct. Grass Valley, CA 95945, [email protected].

Written for presentation at the SMPTE 2012 Technical Conference, Hollywood, CA.

Abstract. The baseband Serial Digital Interface (SDI) workflows and formats were “iron clad” and well defined. These were the "good old days" when devices interconnected with ease thanks to the rigor and breadth of SMPTE standards for SDI. Nowadays, with the great flexibility of file based media workflows and the multitude of formats needed for different applications, the industry is dealing with incompatible wrappers and inconsistent or non-extendable Ancillary (ANC) data carriage. This paper will look at these evolving workflows and the resulting "Wild West" of files. More specifically, the paper explores the challenges faced with the handling of ANC data such as Active Format Descriptor (AFD) , closed captions, ad insertion triggers, audio metadata, etc., within various file formats. The paper then describes the unified and extensible approach offered by SMPTE 436M for the carriage of ANC data within Material Exchange Format (MXF) wrapped files. Could SMPTE 436M be the champion the industry needs to restore order to the Wild West and corral some of the chaos?

Keywords. SMPTE 436M, AFD, ANC, Ancillary Data, ad insertion, closed captions, metadata, MXF, standards, SDI, file based workflows, wrappers.

Copyright © 2012 Society of Motion Picture and Television Engineers. All rights reserved.

2

Introduction As broadcast facilities move towards file based workflows, they are leaving behind the well defined linear workflows that they are accustomed to. They are instead entering the Wild West of Files with various formats and buckets of bits that are easy to define and re-define leading to interoperability problems. Gone are the days of linear workflows starting with linear feeds and linear mediums, with tapes on the input side and linear streams on the output side. These are being replaced by media files.

At some point we started seeing isolated islands of infrastructure based on files. These were generally handled by one vendor from start to end. This homogenous environment made file based transactions relatively painless since the definition of the bits was in the hands of one entity. Nowadays these islands have multiplied and are occupied by many vendors with many definitions of the same bits. We are receiving preproduced content and ads as media files; many different types of media files. What ensues is eventually chaos as the definitions of ancillary data in particular clash.

How can we corral the chaos? We need one definition to follow; a standard, if you will. SMPTE 436M is such a standard.

The good old days. Let’s go back and look at the good old days. These were the days of SDI and hardware based definitions. These were the days of “Iron Clad” tape formats. The “Cast in Stone” device interconnect (SMPTE SDI) with precise specifications on Video, Audio and Ancillary Data. Having everything defined in the hardware made interoperability with other devices such as third party devices, manageable and simple. They just connected! However, there were no provisions for descriptive meta data.

Figure 1. Use the Figure Caption Style for a caption below each figure, outside of the graphics

box. The graphic itself is in the Figure Style.

Then started the days of file based workflows. Suddenly the media was defined in bits in software. These bits could be easily defined and redefined by third parties wanting to make use of spare bits. Due to a homogenous environment a vendor could get away with it since their devices understood the definitions of the bits and handled them appropriately. For in the early days of file-based playout systems, most often users would continue using their existing SDI workflow and ingest from a tape of satellite feed, transforming the data into files, and then storing in a central storage before sending to a playout device. This even included server to server transfers. Often all of these devices from ingest to playout were supplied by the same vendor making the workflow appear smooth and simple. But that solution has been displaced.


3

The reality of today. Today we have multiple Codec formats and multiple file wrapper formats. We have metadata that can be misplaced or placed in various locations depending on format. We have metadata that can be missing or end up missing after passing through various equipment. We have file wrapper format “standards” that are flexible (perhaps too flexible) and allow inpretation in many different ways. One that comes to mind is MXF (Material eXchange Format): a fantastic idea but someone forgot to shut the barn door, letting formats run wild.

At the same time, we have content being ingested from different sources using these various, different formats. For instance, Networked VTRs using IMX and third party media servers using LXF and Satelite feeds putting SDI in a flavor of MXF and then files coming in over the Wide Area Network (WAN). All of these are being stored together in centralized servers and then shuffled to the same playout server with varying degrees of success, depending on their particular format flavor. In some cases these file are not interoperable and must first pass through another device to convert them.

Today’s Reality – The Wild West of Files

SDI

Third partymedia server

INGEST

PLAYOUT

STORE

Networked VTRS

WANFileDeliveryService

IMX

LXF

?XFEthernet /IP Network

MXF Flv A



As the amount of content received or transferred as files increases, becoming the normal mode of operation, we will need to rely on files being interoperable in order for workflows to continue to be smooth and functional. Currently, the workflows often depend on the destination system to accurately and nimbly interpret the correct format as these files race towards them. Welcome to the “Wild West of Files!”

What Does a Media File Look Like? Basically a media file is made up of a wrapper and forms of essence. The wrapper provides a description of the essence and ties it all together. Meanwhile, the essence is the actual media or the content of the file. Let’s look at these more closely, starting with the wrapper.

The wrapper, which describes the media and ties it together includes the structural metadata such as SOM, EOM, duration, types of media included in the file, also known as essence descriptors, and other structural information. The wrapper can also include descriptive


4

metadata such as creation date, usage rules, key words and other information that describes the contents. The wrapper usually has time references and index as well.

The “Essence” is the actual media or the actual contents of the file such as the video which is typically compressed and the audio. There can be multiple audio essences in a file and these are often compressed. Another type of essence would be Ancillary Data such as captions, Active Format Description (AFD), Dolby, Ad Insertion Triggers, Interactive Triggers, and any other data that would enhance or be useful to the other media essences contained in the file.

Storage?

There are two different ways that these are generally stored. With the first approach, the wrapper “wraps” around the various forms of essence and keeps them embedded all in one file. This is useful for keeping information together which makes media management simpler. An example of this method is MXF XDCAM, GXF and LXF.



The second method keeps the wrapper which can also be called a reference file separate from the essence which are kept in external files. There could be one or several external files depending on the various forms of essence required for a particular media. By keeping them separate, adding, deleting or modifying the essence containers is simpler since they are easily accessible. This makes it easy to add extra language essences or changing the captions without having to reprocess the whole file. Examples of this method include Omneon MOV, MXF P2 and MXF AS-02.




5

What about a Hybrid approach? Another complication can be hybrid files which comprise of both storage methods at once. These files embed some of the essence withing the wrapper while others are referenced as external essence files.

Sounds like File Format Fiasco! Since there are so many different forms of essence and so many variants of codecs and wrapper formats along with differing storage techniques, one of the most important and difficult challenges in a file-based world is choosing the correct set of variables for a particular operation. Another challenge is making certain that all of these different variables play nice together and do not further complicate the situation by causing interoperability problems.

How to choose a wrapper? With all the formats available, how are we supposed to choose? Let’s look at a few of the wrapper formats and what they are used for within the three main areas of interest: editing, playout and archive. The first one that comes to mind is the MPEG Transport Stream file which is one of the oldest formats still in use and is actually a television distribution bitstream captured to a file. This format may be future proof, but it does not support editing nor the carriage of descriptive metadata needed for archive.

Another set of old formats, MOV and AVI, originated as personal computer formats over 20 years ago. Although these formats provide some editing support they do not handle editing while ingesting and they only have limited support for ancillary data. AVI does not support necessary characteristics for playout such as native server support, allowing changes without requiring a full remux, and ANC carriage.

Now the next two formats, XDCAM and P2 MXF, are closely associated to VTR and camera manufacturers


MPEG-‐2 TS

MOV / MP4 AVI

GXF&LXF

Generic MXF

IMX / XDCAM MXF

P2MXF

Avid MXF

(OP Atom)

MXFAS-‐02

EditingNLE SupportRandom AccessEdit-‐while-‐ingestPlayoutTranscoder SupportNative Server SupportANC carriageAdditions/Changeswithout full remuxArchivingDescriptive Metadata CarriageFuture-‐proofness

KEY:


6

Not Supported Partially Supported Strongly Supported Figure 1. Use the Figure Caption Style for a caption below each figure, outside of the graphics


Next are the generic industry wrappers such as MXF OP1a and AS-02. The AS-02 format does well with archive and playout but is lacking Non Linear Editing support.

And, of course, there are the vendor specific house formats, such as GXF and LXF and Sony XDCAM MXF, that can be difficult to decipher especially when they contain areas of undocumented metadata.

One thing that becomes obvious looking at all of these formats is that not one of them does everything well. Some are strong in certain areas and weak in others. They each have their strengths based on which functionality they were designed to serve. Now, if we were to choose one wrapper to fit everywhere, it is clear that a few of the mentioned formats are bad choices. For intstance, MPEG2 TS with its lack of support for editing and AVI with its lack of support for playout clearly would not be first choices in covering all areas. Other choices such as generic MXF types have a certain appeal but in reality are plagued by interoperability problems. One of the generic MXF formats has become a surprisingly popular format. That format would be IMX/XDCAM MXF. It works well for editing and has strong transcoder and native server support for playout and is future proofed for archiving. If only it had a little more support for Ancillary data and descriptive metadata carriage, it would be perfect!

MXF keeps popping up. What is MXF? MXF (Material eXchange Format) is an open file format wrapper designed to improve interoperability among content creation components, video servers and other devices in a file based work-flow. MXF was designed to make the interchange of Essence (Video, Audio and Data), Descriptive Metadata and Structural Metadata be simple and easy. It is a file transfer format which is not compression scheme specific and also metadata aware. One important distinction of MXF over other wrappers is its ability to support descriptive metadata which is becoming increasingly important as we archive the plethora of essences out there and wish to retrieve them later. MXF has become the de facto standard and should allow devices to work together despite the manufacturer, which should improve the smoothness of file based workflows. However different use cases that have arisen have given rise to MXF variants.

MXF Variants? MXF covers a lot of ground in making file transfers seamless, however, its generic definition does not suit all cases. First of all, the generic variant is generally regarded as too loose and flexible. There are no hard references and it is subject to broad interpretation.

This has caused Application Specifications within MXF to crop up. There is MXF AS-02 which is specifically designed for versioning and there is AS-03 for program delivery, AS-10 for production, AS-11 for contribution and AS-12 for ad delivery. These specifications put constraints on the MXF format.

Then, there are some vendors whom have decided they need to create their own version of MXF to meet their unique needs, such as Grass Valley with GXF and Leitch (aka Harris) with LXF. However, these constrain the format even more and make it hard to extend. Some of


7

these are proprietary and are not even published. Furthermore, Application Specifications can be further confined to the particular needs of a user by the use of SHIMs. Essentiall, two versions of the same specification can be different.

What about ANC Data? As we become more and more file based and file based delivery and playout becomes required by broadcast stations, the ANC Data becomes increasingly important. This auxillary data provides us with the necessary information to compliment the essence and delivery it in its entirety and expected format reliably. A video without its closed captions for the hearing impaired would not bring much value to a deaf person. If the video is missing its ad insertion triggers, the station will not be making much income. If the Active Format Descriptor (AFD) is wrong or missing the odds of it being displayed correctly decrease substantially. Without the correct audio metadata, the sound will not be played correctly.

Legacy file formats do not meet all the needs. The ANC data of legacy formats was inconsistent and incomplete. Some of the data could be carried in the wrapper metadata and some of the data could be carried in the codec header or in the VBI for SD.

The most common types of data needed for playout are timecode, AFD, Closed Captions, XDS, V-Chips, and WSS. All of these can be carried in the VBI, but this is only possible with IMX files. In order to get past this limitations some codecs have started encoding this data as part of the MPEG user data. This approach is not practical since the data is not always supported and therefore the files are not interoperable. Another way to carry the data is with the wrapper structural data, but this approach is limited and not very useful. In the end, during playout, it is easy to end up with missing, incomplete or incorrect ANC data. Plus key ancillary data such as Broadcast Flag, OP47 Teletext, Dolby Metadata, SCTE Ad Triggers and other private data are not captured at all.

Has ANC Data been neglected? Yes! Ancillary data in the file-based domain is akin to audio in the SDI realm. Both were considered the poor second cousin, meaning that not much thought was given to them. They were an afterthought and therefore one could say that ANC data has been neglected. When setting up new HDTV environments, the video codecs are payed close attention to, but often the audio and now especially the ANC data are easily overlooked and considered not as important and are forgotten until later.

What is SMPTE 436M? SMPTE 436M is a standard that defines a space for Ancillary data within an MXF file. It defines mappings for VBI Lines and Ancillary data packets. SMPTE 436M works with MXF to provide a more solidly defined space in which to carry ANC data. In order for systems to work together with different types of data, the data has to be explicity defined and more reminiscent of the rigid hardware standards that facilitated SDI interchanges.

This standard works to define what types of data goes where and makes it possible for multiple devices from multiple vendors to make sense of the data they are passing between them. This standard defines the sequence order of the ANC elements, such as audio and data, and the corresponding picture elements.


8

Why is it useful? SMPTE 436M gives a definite set of rules to follow which will make interoperability more predictable and reliable between various devices. If the area where the Ancillary data is clearly defined and defined in a rigid way that will not change regardless of the essence that it contains nor due to the functionality that it is used for then it will have dependable results.

SMPTE 436M appears to be a functional solution for the problem of ancillary data. It allows the carriage of all basic ancillary data, such as captions and AFD and timecode. It also allows for the carriage of key data left out of legacy file systems such as OP47 Teletext, Dolby Metadata, SCTE-104 Ad Triggers and much more. Finally there is a space for ANY ancillary data to be placed which should help future-proof this standard for a while.

With SMPTE 436M all ancillary data has a defined space in the file format and no longer has to be scattered across sections. This drastically improves the carriage of the data and adds to it being reliably preserved, discovered and decoded correctly. SMPTE 436M describes the line number where each piece of data will reside. SMPTE 436M captures all ancillary data and allows the augmented data to be added and modified easily.

Can the Wild West of Files be Tamed? With the help of guidelines from SMPTE 436M, the Wild West of Files can indeed be tamed. The Wild West of Files can have their formats corralled into a usable subset that will help with interoperability. The files, using SMPTE 436M, can carry ANC data throughout the workflow and have it maintained and usable.

By having solid definitions for ancillary data reminiscent of the “iron clad” SDI days, the interchange of media between various devices could become seamless and invisible to the user. The users in a sense could become format agnostic and not have to worry where data is maintained and instead concentrate on the creative aspects or the functional aspects of their jobs.

What can be done to further the cause? Make sure that your vendors are aware of SMTPE 436M. Make sure that your vendors are making their products work within the specifications of SMPTE 436M to allow interoperability between devices in your workflow and allowing the carriage of ANC data from one device to another without any hiccups. The SMPTE 436M standard has been around for many years, becoming a standard in 2006. It is currently undergoing a retrofit within SMPTE to make it up to date and take into account 3D. If you have any input, join the SMPTE Standards Community and get involved in this standard. While the SMPTE 436M standard is being reworked, careful attention is being paid to the use of language in order to avoid the standard being interpreted in multiple ways. A great standard works when it is well-known and understood and its interpreted correctly each time allowing it to always works the same way. That is the goal of SMPTE 436M!

Conclusion The explosion of file formats and wrappers along with the need for more ancillary data at the same time that broadcasters are requiring files at ingest is making the importance of file interoperability paramount to success. In order for files to interchange reliably and correctly broadcasters need to follow some clear cut rules and definitions. Up until the use of SMPTE 436 and MXF, files have been defined in various documented and undocumented ways leading to unknown sets of ancillary data being lost, incorrectly decoded, or just misplaced. It is a wild


9

west free for all out in File Land. But SMPTE 436M can corral the ancillary data into a manageable solution so that File Land can stop being the Wild West of Files and instead become an Ancillary Data Utopia.

Format for Reference and Bibliography Entries Bibliography entries are in alphabetical order; references, in numerical order according to their citations in text. In bibliographies, the last name is listed first; if there is no author, the entries begin with "Anon." and are alphabetized by title. In references, the listing is as in the examples below. If more than three authors are listed, use the first name followed by et al. The following are examples of format for various types of publications or non published documents. Use standard abbreviation for journal articles when known; otherwise, spell out.

Journal Articles M. C. Gruszka, “Bumps on the All-Digital Road,” TV Technology, 12:12-39, June 1994.

Articles Published in Proceedings M. Shibata, “A Temporal Segmentation Method for Video Sequence,” Proc. SPIE Visual Comm. and Image Proc. '92, pp. 1194-1205 1992.

Books L. S. Birks, Electron Probe Microanalysis, Wiley: New York, 1982.

Chapter in a Book P. E. Wilcock, “Analog Components,” in Video Pictures of the Future, SMPTE: Scarsdale, N.Y., 1983.

Unpublished Paper W. E. Glenn, J. Marcinka, and R. Dhein, “Subband Coding Compression System for Program Production,” presented at the 136th SMPTE Technical Conference, Los Angeles, Oct. 1994.

Dissertation or Thesis T. L. Gilbert, “Rate of Decay of Auditory Sensation,” Ph.D. dissertation, Dept. of Chemistry, University of Massachusetts, 1984.

Standards SMPTE 259M, “Television -10-Bit 4:2:2 Component and 4fsc NTSC Composite Digital Signals—Serial Digital lnterface," SMPTE Mot. Imag. J., 102:174-179, Feb. 1993.


Documents

Corralling the Chaos of within Multiple File Formats the good ‘ol days versus today •File wrapper format “standards” that are too loose and interpreted in too many different