Chapter 4 - Building Blocks Edt-010311_033357

8/6/2019 Chapter 4 - Building Blocks Edt-010311_033357

1/53

CMM5013 Multimedia AuthoringCMM5013 Multimedia Authoringand Programmingand Programming

Chapter 4:Chapter 4:

Multimedia BuildingMultimedia Building

BlocksBlocks


2/53

Part 3 - Sound

What is sound?

Waveforms and attributes of sound

Capturing digital audio

Sampling

Soundcard technology

MIDI (Musical Instrument Digital Interface)


3/53

Sound

Sound is a complex relationship involving avibrating object (sound source), a transmissionmedium (usually air), a receiver (ear) and aperceptor (brain)

As the sound vibrates it bumps into molecules ofthe surrounding medium causing pressurewaves to travel away from the source in alldirections

As pressure waves get further from the sourcethey become weaker as their energy dissipates


4/53

Waveforms

Sound waves are manifest as waveforms

Periodic waveform=waveform that repeats itself atregular intervals

Noise=Waveforms that do not exhibit regularity Cycle=unit of regularity

Hertz (or Hz) after Heinrich Hertz (a pioneer inthe field of acoustics)

One cycle = 1 Hz kHz or kiloHertz

(1 kHz = 1000 Hz)


5/53

Waveforms


6/53

Example waveforms

PianoPiano

Pan flutePan flute

Snare drumSnare drum


7/53

Capture and playbackof digital audio

Air pressureAir pressurevariationsvariations

Captured viaCaptured viamicrophonemicrophone

Air pressureAir pressurevariationsvariations

ADCADC

Signal isSignal isconverted intoconverted into

binarybinary01010011010101001101

01101011110110101111

AnalogueAnalogueto Digitalto DigitalConverterConverter

DACDAC

ConvertsConverts

back intoback intovoltagevoltage

Digital toDigital toAnalogueAnalogueConverterConverter


8/53

The attributes of sound

Sound is describedin terms of severalcharacteristics: Pitch (frequency of

the waveform inHz)

Amplitude (orloudness)

Timbre (ortone quality)

In addition, allsounds have aduration and

successive
http://upload.wikimedia.org/wikipedia/commons/c/cc/9577_Guitarz1970_Clean_E9_Guitar_Chord_(Mike_Tribulas).ogghttp://upload.wikimedia.org/wikipedia/commons/c/cc/9577_Guitarz1970_Clean_E9_Guitar_Chord_(Mike_Tribulas).ogg


9/53


10/53

Digital samplingSampling frequency

:D e fin itio n T h e re co rd in g o f va lu e a t d iscre te in te rv a ls in t:D e fin itio n T h e re co rd in g o f va lu e a t d iscre te in te rva ls in ti


11/53

The Nyquist theorem

The Nyquist theorem states: In order to be able to reconstruct a signal,

the sampling frequency must be at leasttwice the frequency of the signal being

sampled Named after Harold Nyquist of Bell Telephone Labs

(1928)

The highest frequency that can be

produced in a given digital audio system(i.e. half the sampling rate) is called theNyquist frequency


12/53

Digital samplingSampling frequency


13/53

Sample resolution

The resolution of a sample is thenumber of bits it uses to store a givenamplitude value, e.g.

8 bits (256 different values) 16 bits (65536 different values)

A higher resolution will give higher qualitybut will require more memory (or disk

storage)


14/53

Quantisation

Definition: The restriction of any continuouslyvarying signal to a finite set of discretevalues.

Samples are usually represented as integers.

If the input signal has a voltage correspondingto a value of between 53 and 54, the ADCmay round it off to 53

Due to this rounding that must occur, thevalue of a sample is generally slightlydifferent from the original signal This is known as quantisation error and is

unavoidable

Increasing the sample resolution can reduce


15/53

Quantisation example


16/53

Calculating the sizeof digital audio

The size is the raw uncompressed memorythat the digital audio occupies

The formula is as follows:

The answer will be in bytes Where:

sampling rate is in Hz duration is in seconds resolution is in bits number of channels = 1 for mono, 2 for

stereo, etc

8

channelsofnumberresolutiondurationrate


17/53

Calculating the data rateof digital audio

Data rate = the rate at which the data mustbe supplied to the audio hardware in orderto be played back at the correct speed

The formula is as follows:

The answer will be in bytes per second Where:

sampling rate is in Hz resolution is in bits number of channels = 1 for mono, 2 for

stereo, etc.

8

channelsofnumberresolutionrate


18/53

CD-DA (CD-Digital Audio)

Compact disks are recorded using 2channels (stereo) using 16 bits ofinformation at 44.1 KHz

It therefore requires just over 10MB to storea single minute of CD audio

Compact disks can store about 74 minutesof digital audio

but only around 650 MB of data!

A Sony MiniDisc has a data capacity ofabout 140MB


19/53

Digital audio editingsoftware

One of the most powerful andprofessional PC-based packages is atool called Sound Forge

http://www.sonicfoundry.com/http://www.sonicfoundry.com/


20/53

The purpose of a soundcard

The purpose of a soundcard is to provide some or all of the

following functions:

Play sounds

Provide a MIDI device interface

Synthesise musical sounds and sound effects

Provide a game port interface (joystick, game pad, etc...)

Record sounds from an input (microphone, CD player, etc...)

Mix sounds coming from many sources (microphone, CD

player, digital audio output) and send them to an output

(speakers, headphones)

Control a CD-ROM drive with a drive controller (mostly in

older soundcards)


21/53

MIDI (Musical Instrument DigitalInterface)

MIDI is a standard for specifying a musicalperformance

Rather than send raw digital audio, it sendsinstructions to musical instruments telling them

what note to play, at what volume, using whatsound, etc.

The synthesiser that receives the MIDI events isresponsible for generating the actual sounds

The size of MIDI files is small Proportional to the number of MIDI events

Although they cannot contain any digital audio!


22/53

MIDI sequencers

A MIDI sequencerallows musicians toedit and createmusical

compositions like aword processor Cut and paste

Insert / delete


23/53

Summary

There are two main types of digital audio Sampled audio

Captured by sampling an analogue waveform at a setrate

MIDI data Instructions on how to perform some musical

composition

Sampled audio requires more storage space thanMIDI information

Modern soundcards can capture and playback bothsampled audio and MIDI information


24/53

Part 4 - Video

Analogue video

What is digital video?

Calculating the size of digital video

Compression techniques Digital video formats

Video capture hardware

Digital video editing Consumer desktop video


25/53


26/53

PAL video

The PAL video image iscomposed of 625lines

The actual picture is

contained in 576lines The rest is taken up

by Teletextinformation

768 576 = 1.333 =4:3

25

( )

625

lines

625

lines

768 pixels768 pixels

576

lines

576

lines


27/53

What is digital video?

Digital video is the digitisation of analoguevideo signals into numerical format

Conversion from analogue to digital format

requires the use on an ADC (Analogueto Digital Converter) A Digital to Analogue Converter (DAC) can be

used to output digital video on analogue

equipment Sound can be captured separately

Digital video can have zero or more channels ofaudio

NICAM stereo broadcasts have 2 channels

DVD has 6 channels of sound (called 5.1)

C l l ti th i f di it l


28/53

Calculating the size of digitalvideo

Where:

width = image width in pixels

height = image height in pixels

colourdepth is measured in bits per pixel fps = number of frames per second

8

fpsdepthcolourheightwidth)(size

=bytes


29/53


30/53

Compression techniques

Since the size of raw digital video is soprohibitively large we need some meansto compress the information

Lossy compression techniques causesome information to be lost from theoriginal image

You can never recreate the source image from thecompressed version

Lossless compression techniques do notlose information

You can always recreate an exact replica of thesource information


31/53

Compression techniques

The two main forms of compression are:

Compression of repeating information Take a newsreader as an example. Most of the

screen does not change (the background,desk, etc).

Only need to store parts of the image thatactually change

Removal of low-visibility artefacts Things the eye cannot easily identify can be

removed or highly compressed andsynthesised upon playback


32/53

Digital video formats

Microsoft AVI

Files with a .avi extension

Apple QuickTime

Files with a .mov or .qt extension MPEG / MJPEG

Files with a .mpg extension


33/53

Microsoft AVI

Audio Video Interleave format

Interleaving is a technique used to embed two ormore things into the same stream of information

In every chunk of information you will find some

video data and some audio data

001001010010010101010011110101...001001010010010101010011110101...

001001010010010101010011110101...001001010010010101010011110101...

......

001001010010010101010011110101...001001010010010101010011110101...

001001010010010101010011110101...001001010010010101010011110101...

VideoVideoinformationinformation

AudioAudioinformationinformation

8, 168, 16or 24or 24

bitsbits


34/53

MPEG video

Named after the Moving Picture ExpertsGroup who devised the compression andfile formats

There are a number of MPEG formats: MPEG-2 is used for digital TV broadcasts and

DVDs

MPEG-1 is a format used for low quality video(generally displayed on computers)

MPEG-1 Layer 3 is the popular encodingmechanism for MP3 audio files

MPEG-4 is a new format for multimediapresentations

Can require separate hardware to decode

higher quality MPEG video data


35/53

MPEG compression example

A simple scene showing a car movingA simple scene showing a car movingacross a desert landscapeacross a desert landscape

Only the differenceOnly the difference

between the currentbetween the currentand next frame needsand next frame needsto be storedto be stored

This is called intraframe codingThis is called intraframe coding


36/53

QuickTime

Developed by Apple,Inc.

Primarily for playbackwithout any

hardware assistance Can achieve

compression ratiosof 25 to 200:1

The QuickTime formatcan also store audio,graphics, 3D andtext making it moremuch versatile formultimedia


37/53

Video capture hardware

The hardware hasvarious input andoutput connectors Composite video

in/out S-Video in/out

Audio in/out

Special chips providethe processingpower tocompress/decompress the videoinformation

CODEC (Compressor /


38/53

Digital video editing

Analogue tape editing is a linear process To find the section you want, you may have to

forward or rewind the video tape

To move a section to another place in the

sequence you have to either re-record thesection onto another tape or physically cutand splice the video tape

Digital video editing can be non-linear You can move sections around inside the

computer and play those sections back inany order

Non-destructive editing and Edit Decision Lists

(EDL)


39/53

Consumer desktop video


40/53

Summary

Today we have seen how analogue videoformats are composed and how digitalvideo can be used to store theseelectronically

Digital video demands huge file sizes even before sound is added on!

Compression techniques help to reduce the

file sizes to more manageable levels


41/53

Part 5 - Animation

Depiction of objects as they vary overtime

Traditionally, based on individual drawing

or photographing the frames in asequence

Computer animation also results in asequence of images, but these are

created by software.


42/53

Animation

Such animation is made from a series of stills and

relies on something calledpersistence of vision. Persistence of vision is the phenomenon were an

object on the eyes retina remains for a brieftime after viewing.

This means that a series of still images which varyslightly, if shown rapidly will give the illusion ofmovement.

If each of the eight pictures below were shown atthe same point in rapid succession, the result

would be a rotating arrow.


43/53

Animation

The pictures shown below are stills of therotating Ford logo from the Ford MotorCompany site.


44/53

Animation

TV gives the illusion of continuous movement byshowing the stills at the rate of 30 frames persecond.

Feature films are filmed at a rate of 24 frames per

second but shown at a rate of 48 frames persecond.

Today, most animation is performed by computer.Examples are Bugs, Toy Story, Jurasic Park and

the BBCs Walking with Dinosaurs. The computer will produce a wire frame of the

scene, then apply textures and light effectsbefore moving onto the production of the nextframe, each of which may take hours or evendays to produce.


45/53

Cel Animation

Cel Animation is the technique used to producethe old Tom & Jerry cartoons and the newcomputer generated Disney cartoons.

It draws its name from the celluloid films used in

old hand drawn animations. The celluloid films permitted layering, where the

background to a sequence may be drawn on oneor more films and then the films containing the

animated characters place on top. Such animation starts with the production of

keyframes, which are thefirst and last frames ofan action scene.


46/53

Cel Animation

The frames in between the keyframes arethen produced using a process known astweening.

Tweening is where the number of frameswhich must appear between keyframes iscalculated and the frames drawn.

Tweening may be performed by computer if

the frames are not too far apart.


47/53

Computer Animation

Computer animation works in much thesame way as Cel Animation.

It even uses the same terms, such as

keyframes, layers and tweening. Although theoretically limited by the scan

rate of the monitor being used, the framerate of any animation is typically dictated

by the memory and processing power ofthe computer as well as the channel it isbeing shown across (if any).


48/53

Computer Animation

To produce smooth animation, a minimum rate of15 frames per second must be sustained.

Any lower and the animation will appear jerky.

The frame rate depends mostly on the power of

the processor and the bandwidths between mainmemory (66/100MHz) and the processor and theprocessor and the graphics card (VL, PCI, AGPx1x2 x4).

It also depends of course on the performance ofthe graphics card and the speed of its onboardmemory and integrated acceleration techniques.


49/53

Computer Animation

The frame rate does of course also depend on thecharacteristics of the animation frames.

The more frames, the greater the demands on thehardware.

The pixel depth of each frame also impacts greatlyon performance.

As does the resolution of each frame of theanimation.

If you do not have control of the playbackhardware, it may be necessary to work to andrecommend a minimum spec machine e.g. MPC2etc.

This will inevitably mean keeping frame rate and

pixel depths low and the size of the animation( ixels hi h and wide) relativel small.


50/53

Morphing

Morphing is a special effect in motionpictures and animations that changes (ormorphs) one image into another through aseamless transition.

This is done by first creating or scanning thefirst and last images.

Then key points are specified, i.e. points on

the original image which should becomepoints on the final image.

The computer then produces the frames inbetween, with each successive frame

becoming progressively more morphed


51/53

Morphing

The creator can usually specify the number ofintermediate frames to be produced as well thenumber of key points.

Both the number of the key points and number of

frames will impact on the time taken to producethe sequence of frames.


52/53

Morphing


53/53

Exercise

1)What is sound? What is the differentbetween sound and waveform? Pleaseelaborate more

2)What is the different between PAL andNTSC?

3)What is morphing?

4)

Documents

Chapter 4 - Building Blocks Edt-010311_033357