Temporal Video Bound-aries

Computer Science EngineeringLee Sang Seon

WhyTemporal Video Boundaries

Techniqueis useful in the

Video content analysis?

Index Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries Conclusion Q & A

Introduction Brief definition of Temporal Video Boundary

technique→ Examine the temporal boundary problem at

different levels of video content structure analysis

Why we need Temporal Video Boundary technique?

Show example

Example : Oscar awards

Insufficient metadata

opening

ending

Example : Oscar awards

Detailed metadata

opening

ending

actor

winners

awards

ending

Basic notions - modali-ties Video contains three types of modalities (i) Visual (ii) Audio (iii) Textual

Each modality has three levels(i) low-level (ii) mid -level (iii) high-level→ levels describe the amount of details described in each modality in terms of granularity and ab-straction

Basic notions - modali-ties For each modality and for each level there if

a set of attributes. These can be formalized as vectors:

Basic notions - modali-ties Adding to this, given a set of vectors

→ their average value denote the vector

Basic notions - method Local method→ the difference is computed between con-

secutive frames

Global method→ the difference if computed over a series of

frames

Micro-Boundaries Definition

Boundaries associated to the smallest video units for which a given attribute is constant or slowly varying

The attribute can be any feature in the visual, audio, or text domain

Example

Make family histogram

Data structure that represents the color in-formation of a family of frames.

Set of frames that exhibits uniform features

= Frame histogram

Histogram difference measures Histogram difference using L1 metrics

Bin-wise histogram intersection

Total number of color bins used

Histogram of previous frame

Histogram of current frame

Merging of family his-tograms

Multiple ways to compare and merge families - contiguity & memory

1. Contiguous with zero memory → A new frame histogram is compared with

previous frame histogram

2. Contiguous with limited memory→ A new frame histogram is compared with

previous family histogram

Multiple ways to compare and merge families - contiguity & memory

3. Non contiguous with unlimited memory → A new frame histogram is compared with all

previous family histograms within the same video.

4. Hybrid→ First a new frame histogram is compared using

the contiguous frames and then generated fam-ily histograms are merged using non contigu-ous case.

Compare different Histogram difference measures

Macro-Boundaries Definition

Boundaries between collections of video micro-segments that are clearly identifiable organic parts of an event defining a structural (action) or thematic (story) unit

Video : collection of stories that may or may not be interconnected

→ Macro-Boundaries detection= Segmenting stories

textual cues

audio cuesvisual cues

Two types of uniform segment detection Unimodal segment detection

A video segment exhibits same characteristic over a period of time

Multimodal segment detection A video segment exhibits a certain characteris-

tic taking into account attributes from different modalities

Single Modality Segmen-taion

Partition a continuous bit-stream of audio data into non-

overlapping segments

Classification

Seven mid-level audio cate-gories

Using low-level audio features

Audio segmen-tation & classifi-

cationText transcript

Extracted from either the closed captions or speech-to-

text conversion

Segmented and categorized with respect to a predefined

topic list

Frequency-of-word-occurrence metric is used

Multimodal Segmentaion

Pre-merging Steps

Uniform seg-ment detec-

tion

Intra-modal segment clus-

tering

Attribute template de-termination

Dominant at-tribute de-

termination

Template ap-plication

Descent Methods

Goal :Create macro-bound-

aries that are more ac-curate than the bound-aries produced by indi-

vidual modalities.

Descent MethodsText seg-

ment

Audio segment

Video segment

Single descent Method

Single descent with intersecting

union

Single descent with intersec-

tion

Single descent with secondary

voting attributes

Single descent with conditional

union

Mega-Boundaries Definition

Boundaries between collections of macro-seg-ments that exhibit different structural and fea-ture consistency (e.g. different genres)

Example Commercial detection method

Trigger & Verifiers Model

Features that can aid in determining the location of the commercial break

Triggers

Features that can determine the boundaries of the commercial break

Veri-fiers

Black framesTime interval be-tween detected black frames as

triggers

Used as verifiers

Letterbox change

High cut rate(= low cut distance)

Bayesian Belief Network Modelstart

Genetic Algorithms

ConclusionType of bound-

aries Methods Example

Micro-boundaries Frame & Family histogram comparing and merging

Visual scene segmenta-tion

Macro-boundaries Single modality segmenta-tion

&Multimodal segmentation

Multimodal story segmen-tation

Mega-boundaries Trigger & Verifier Commercial detection

Whenever metadata is availableor unavailable,

we can segment video by using this technique that

categorized three types

&Thank you!

Q & A