1

Synchronization

TTM4142, 2007

Harald Øverby/Leif Arne Rønningen

2

Outline

• What is synchronization• Relationships• Logical Data Units (LDU)• Reference model for synchronization• Specification

3

Definition of synchronization

• SynchronizationThe act or result of synchronizing; the state of being synchronous

• SynchronizeTo happen at the same time; to represent or arrange (events) to indicate coincidence or coexistence; to make synchronous in operation; to make (motion or picture sound) exactly simultaneous with the action

Source: Merriam-Webster dictionary

4

Definition of synchronization

• Synchronization creates relationships between independent objects– Time dependent / time independent objects

• Three types of relationships– Content

– Spatial

– Temporal

5

Relationships

6

Content relationships

• Connection between media objects and data• Example

– Between data in a table and a picture

• Update in data triggers update in other views• Consistency

7

Spatial relationships

• Layout relationships• Define space of a media object

– Objects arranged in a two dimensional area

– Output device

• Content assigned to frames– Positioning: fixed, relative

• Example– Powerpoint slide

– Web page

8

Temporal relationships

• Time relationships between objects– Important for time dependent media objects

• Examples– Timing relationships between audio object and video object

• Further on: focus on temporal relationships

9

Intra object synchronization

• Time relation between various presentation units of one time-dependent media object

• Example: bouncing ball

10

Inter object synchronization

• Time relation between two or more media objects• Examples: Audio, Pictures (P), Video, Animation

11

Logical data unit (LDU)

• LDU: Information unit of a time-dependent media object

• Granularity of LDU– Video: Frame, object, pixel

12

Logical data unit (LDU)

• Closed LDU– Predictable duration

– Ex: Stored media objects

• Open LDU– Duration not known before the presentation starts

– Ex: Input from live source

13

Logical data unit (LDU)

• Physical units as LDU or blocks of samples as LDU

14

Logical data unit (LDU)

• User-defined vs capturing defined

15

Logical data unit (LDU)

16

Synchronization in multimedia systems• Television systems

– Sound and video are multiplexed

– Delay affects both sound and video

– Synchronization always maintained

• Focus on time-independent media– LDU to be put in relationship are independent media objects

• Quality of Service– Hard guarantees

– Soft guarantees

17

Synchronization examples

• Lip synchronization– Coupling between audio and video

18

Lip synchronization experiment

• Skew between video and audio• +/- 40 ms intervals• Positive skew: audio ahead of video• Short clips

19

Example questionnaire

20

Experiments - results

21

Experiments - results

22

Results summary

• Asymmetry– Positive skew more disturbing

• Difficult to detect errors when object far away

• Statistical errors• Long movie

– +/- 80 ms : no distraction

– -240 ms, +160 ms : disturbing

23

QoS: Skew values

24

Reference Models for Multimedia Synchronization• Support to understand requirements• Many proposals• Focus on the following four-layer model

– Specification Layer

– Object Layer - presentation object synch

– Stream Layer - inter-stream synch of media streams

– Media Layer - intra-stream synch of continious media

25

Reference model for synchronization

26

Media Layer

• Application handles only one continuous media stream as an LDU sequence

• Device-independent interface, file and device access– Read (device-handle, LDU)– Write (device-handle, LDU)– etc

• Application runs one process for each stream• Guarantees single LDU processing

27

Stream Layer• Handles continuous media streams and media stream groups• Guarantees intra-stream synch• Group presents streams in parallel• Gurarantees inter-stream synch of streams in a group• Handles resource reservation and scheduling of LDU

processing• Typical operations

– start (stream), stop (stream)– create_group (list_of_streams)– start (group), stop (group)– setcuepoint (stream/group, at, event)

28

Object Layer1

• Deals with all kinds of media• Hides the difference between discrete and continuous

media• Accepts a synchronization specification as input• Media-oriented QoS (such as allowed skew, allowed

jitter)• Plans and coordinates presentation execution• Initiates presentation of time-dependent and time-

independent media objects

1 In DMP the term could have been Scene Layer

29

Specification Layer

• Does not provide a specific interface, is open• Tools for generating synch specification

– Edit

– Format

• Maps user interface QoS specifications to the Object Layer

30

Synchronization Specification, main categories• Interval-based

– Temporal relationships between the presentation of media objects

• Axis-based– Presentation objects placed in common axes

• Control-flow-based– Presentation flow synchronized at certain points

• Event-based– Events trigger presentation actions

31

Synchronization in a distributed environment• More complex than a local environment

– Distributed storage of media

– Distributed storage of synchronization information

– Communications delay and delay jitter

– Multi-party communication patterns

32

Transport of synchronization specification• Additional synchronization channel• Multiplexed channel

33

Location of synchronization operations

34

Multiple communications

35

Multi-step Synchronization

• Synchronization• during object acquirement• of retrieval• during delivery of the LDUs to the network• during the transport• at the sink• within the output device

36

Synchronization SpecificationComplex specification comprise:

• multiple object synchronization

• user interactions

Requirements for a specification method:

• Support of object consistency and for maintenance of specifications

• Media objects should be kept as logical units in the specification

• All types of synchronization relations should be easily describable

• Support for the integration of time-dependent and time-independent

media objects

• Quality of service requirements must be describable

• Support of hierarchical levels of a synchronization

37

Interval-based Specification

38

Synchronization Based on a Global TimerMediaobjects

39

Synchronization Based on Virtual Axes

40

Synchronization Based on Reference Points

41

Petri Net Specification

42

43

Event-based Synchronization