Der Titel / the Titelwiki.ktxsoftware.com/slides_2015/gametech13.pdf · wrapped on the scope, I poked around the sync generator to find an appropriate frequency or a tone. So those

© author(s) of these slides including research results from the KOM research network and TU Darmstadt; otherwise it is specified at the respective slide

23-Jan-16

Prof. Dr.-Ing. Ralf Steinmetz

KOM - Multimedia Communications Lab

Template all v.3.4

PPT-for-all___v.3.4_office2010___2012.09.10.pptx

Game Technology

Lecture 13 – 23.01.2016

Game Sound

Dipl-Inf. Robert Konrad

Dr.-Ing. Florian Mehm

KOM – Multimedia Communications Lab 3

Intro – Red Dead Redemption (2010)


Organization

Date Lecture Topic

24.10.2015 1 Input and Output

2 The Game Loop

3 Software Rendering

4 Advanced Software Rendering

28.11.2015 5 Basic Hardware Rendering

6 Bumps and Animations

7 Physically Based Rendering

8 Physics 1

19.12.2015 9 Physics 2

10 Procedural Content Generation

11 Artificial Intelligence

12 Multiplayer

23.1.2016 13 Audio

14 Compression and Streaming

15 Scripting (Optional) + Guest talk


Remaining exercises

Due until February 13

Exercise 10 (PCG) already out

Reduced compared to last year (less time to learn for the exam)

Exercise results uploaded to „points“ branch

Solutions up to exercise 7 uploaded

Lecture recordings for this block

Audio: Today

Video: Tomorrow

Organization


Saturday, February 20th, 2016

90 Minutes

11:30 – 13:00

Please be there earlier so we can assign seats

S101/A1

Allowed: Dictionary, Non-programmable calculator

Not math-heavy, formulae for cross and dot product are provided

Exam


Document with example questions

Most relevant topics for each lecture

Example problems

No solutions, but some/similar problems might be in the exam

Will be online latest middle of next week

No C++ exam questions

Bonus Points

>50%: 0.3; >70%: 0.7; >90%: 1.0

The exam has to be passed without the bonus points – bonus is added only

after the exam has been passed regularly

Will upload points for exercises 10-12 and bonus point info before lecture

Exam

KOM – Multimedia Communications Lab 8http://www.freesound.org/people/lukaso/sounds/69690/, , http://www.arcade-

museum.com/game_detail.php?game_id=13334,,

https://en.wikipedia.org/wiki/Slot_machine#/media/File:Liberty_bell.jpg

Early History

Contact (1934)First Pinball machine to

use electric sound

Liberty Bell (1895)Rang a bell when players

won

http://www.freesound.org/people/lukaso/sounds/69690/

http://www.arcade-museum.com/game_detail.php?game_id=13334

https://en.wikipedia.org/wiki/Slot_machine#/media/File:Liberty_bell.jpg


“The thrust motors from your rocket ship, the rocket turning signals, the

firing of your missiles and explosions to fill the air with the sights and

sounds of combat as you battle against the saucers for the highest

score.”

The 70’s


Computer Space (1972)

https://www.youtube.com/watch?v=b3BQsCCwo8w

KOM – Multimedia Communications Lab 11https://edfries.wordpress.com/2015/03/13/fixing-computer-space-3/

Computer Space Sound


Pong (1972)

https://www.youtube.com/watch?v=e4VRgY3tkh0


“Now the issue of sound ...People have talked about the sound and I’ve

seen articles written about how intelligently the sound was done and how

appropriate the sound was. The truth is, I was running out of parts on the

board. Nolan wanted the roar of a crowd of thousands—the approving

roar of cheering people when you made a point. Ted Dabney told me to

make a boo and a hiss when you lost a point, because for every winner

there’s a loser. I said, “Screw it, I don’t know how to make any one of

those sounds. I don’t have enough parts anyhow.” Since I had the wire

wrapped on the scope, I poked around the sync generator to find an

appropriate frequency or a tone. So those sounds were done in half a

day. They were the sounds that were already in the machine.”

Al Acorn, Designer of Pong

Pong (1972)


Adaptive (Faster playback the more dangerous the aliens become)

Earliest Music – Space Invaders (1978)

https://www.youtube.com/watch?v=437Ld_rKM2s


Looping Music – Rally X (1980)

https://www.youtube.com/watch?v=LJxbkhFPgb8


Carnival (1980) - Juventino Rosas - Over the Waves Waltz

Looping Music

https://www.youtube.com/watch?v=U7X0iCGCv0Y


“Television Interface Adaptor”

2 channels

4bit waveform selector

Tuning different depending on PAL or NTSC

No complete diatonic scale available

Very difficult to compose music

E.g. Tapeworm (1982)

https://www.youtube.com/watch?v=rVFvlJ98Fu4

Home Consoles – Atari VCS (1977)

https://www.youtube.com/watch?v=rVFvlJ98Fu4


5 channels

2 pulse-wave (one with sweep)

1 triangle wave

1 noise

1 sampler (e.g. voice, effects)

Nintendo Entertainment System (1983)


Home Computers

https://www.youtube.com/watch?v=a324ykKV-7Y


IBM PC (1981)

Beeper

PCjr (1983)

3+1 Sound Chip

Hired Sierra On-Line to produce games

First game: King’s Quest (1984)

https://www.youtube.com/watch?v=ymLvOunSuoQ

Home Computers


Sound Interface Device (SID)

3+1 chip

Wide range for each channel

Ballblazer (1984)

Procedurally generated music, composed from small pieces of music

Commodore 64 (1982)


Chris Hülsbeck

The Great Giana Sisters (1987)

https://www.youtube.com/watch?v=Ne-09Bs_bRo


TurboGrfx 16 (1987)

Sega Genesis/Mega Drive (1988)

16 bit era


16 bit DSP

8 stereo channels

MIDI

Super NES


AdLib (1986)

9 channel FM synthesis

MIDI

Sound Blaster (1989)

Copy of AdLib

+ digital sampling

Roland MT-32 (1987)

Deal with Sierra

King’s Quest IV (1988)

Home Computers in the 16 bit era


”Paula” sound chip

4 channel 8 bit stereo

Composition software developed in-house

Ultimate SoundTracker (Karsten Obarski)

Later versions established MOD format

Included digital samples to be used

Turrican (1990)

https://www.youtube.com/watch?v=G9cznwHK0Ok&list=PLECABBE43061EA8B8

Amiga (from 1985)


PC

Sega Saturn (1994)

Sony Playstation (1994)

Sega Dreamcast (1998)

CD-ROM

The 7th Guest (1992)


Often on-board sound

3D sound standard

Some game(rs) rely on sound

E.g. Counterstrike

In development practice

Often neglected until later in the project

Outsourced

Few dedicated sound programmers

Today


Sound waves

Air compression

Longitudinal Waves

~343 m/s

20 to 17000 Hz

Sound Waves


Converts electrical signals to sound waves

Using an acoustic membrane

Loudspeakers


Two identical audio sensors

Measure actual wave forms

Using the eardrum

Ears


Sensitivity depends on frequency

Sound Perception

Human Voice


Depends on perceived

Arrival time

Loudness

Timbre

Tone quality

More exact around us horizontally

Less acurate above/below

Sound Localization


Measure time differences between ears

Loudness differences between ears

Because of the head

Depends highly on frequency

Partly used for frequencies > 800 Hz

Exclusively used for frequencies > 1600 Hz

Sound Localization Left/Right


Analyzes frequency differences caused by the ear forms

Also to a lesser degree head, shoulders,…

Sadly somewhat individual

Analyzes changes due to head movements

Sound Localization Front/Back


Distance

Increased distance Decreased amplitude (amplitude *= 1 / distance)

(and slightly decreased frequency)

Direction

Interpolate between speakers

Better quality add more speakers

3D Sound with speakers


Can also be simulated using headphones

Brain analyzes sounds to infer directions

Try it at home (Virtual Barber Shop Hair Cut)

https://www.youtube.com/watch?v=8IXm6SuUigI

Long time uninteresting

Now back into focus VR

Immersion

Already wearing a device on the head

3D sound essential

3D Sound with Headphones – Binaural Audio

https://www.youtube.com/watch?v=8IXm6SuUigI


Use mannequin heads that simulate human head and eardrums

Not personalized

Binaural Audio Capturing


Need to capture Head-Related Transfer

Function

For each ear, captures how sound is perceived

from a specific direction

Compute at runtime how a sound should be

heard

Additionally: Room simulation

Simulate reflections, reverb

Helps with realism and sound localization

Binaural Audio Synthesis


Programmable Sound Generator (PSG)

Specialized circuits for driving oscillators

E.g. sine wave, …

Subtractive Synthesis

Start with a waveform, subtract by running it through filters (e.g. low-pass filter)

Many chips in the 8-bit-era: 3 + 1

E.g. 3 square waves

1 noise channel

Sound Synthesis


Sine

Pleasing, pure (no harmonics), used

for flutes, lasers, …

Square/Pulse waves

Hollow

Triangle

More like sine waves

Sawtooth

Used for bass

Noise

Commonly used waves


http://studio.substack.net return function (t) {

function sin (x) { return Math.sin(2

* Math.PI * t * x) }

function square (x) { return

(Math.floor(x) % 2 == 0) ? -1 : 1; }

function saw (x) { return (-x % 1 -

0.5) * 2; }

function triangle (x) { return 2 *

Math.abs(2 * (x - Math.floor(x +

0.5))) - 1; }

//return sin(441);

//return square(t * 200) / 20;

//return saw(t * 240) / 10;

//return triangle(t * 440) / 3;

}

Try it out

http://studio.substack.net/


Frequency Modulation

Modulator (usually sin) drives the Carrier

Marks the 16-bit-era (Super NES, Mega Drive, AdLib)

FM Synthesis


http://greweb.me/2013/08/FM-audio-api/

Try it out!

http://greweb.me/2013/08/FM-audio-api/


Established in 1983

Commands to start/stop sound, volume, …

Standard

Made composer’s lives easier

Less dependent on specific models and drivers

See non-standardized tracker formats

1991 General MIDI Standard

Streamline how instruments are modelled

Musical Instrument Digital Interface (MIDI)


Use digital samples to synthesize sound

Change pitch as required

More realistic

At the cost of memory

Linear arithmetic synthesizers

The attack transient is the most important information for the human ear

Combine digitally sampled attack with simple synthesis

Basis for tracker music

Amiga, Gravis Ultrasound, (MIDI)

Wavetable Synthesis


Specialized or general hardware for signal processing

Echo Delays of 50ms and more

Reverb <50ms, for reflections of indoor surfaces

Time stretching Alter the speed without the pitch

Compression Reduce the dynamic range of sound

Equalization Attenuate or amplify frequency bands

Filtering Low-pass, high-pass, …

https://upload.wikimedia.org/wikipedia/commons/4/42/Echo_samples.ogg,

https://en.wikipedia.org/wiki/File:Reverberation_effect.ogg

Digital Signal Processing


Similar to wavetable synthesis

However, very small samples (1-50 ms)

Played at varying pitches, frequencies, …

https://en.wikipedia.org/wiki/Granular_synthesis

Granular Synthesis

https://en.wikipedia.org/wiki/File:Granulizer.ogg


Sample the sound at regular intervals

Encode binary

Pulse-Coded Modulation (PCM)


Small ring buffer

Discretely sampled waveform

Pointer to last sample written

Pointer to next sample to read

Computer -> Speaker


Ringbuffer

Last read

Last written


Sampled audio values, arranged in chunks

For example (iOS)

256 samples in each chunk

1 sample per channel (i.e. 2 for stereo)

Sampling rate 44.1 kHz

1 second of sound in stereo

44100 * 2 = 88200 samples

88200 / 256 = 345 chunks

What is in the ring buffer?


Superpositioning

Adding waves

Again physically accurate

Actual danger of superposition effects

Avoid mixing identical sounds

Sound Mixing


Music

Wing Commander 2 (1991)


Long files

Played/Streamed in the background

Mostly not influenced by gameplay

Music


Sound Effects


Short files

Triggered by gameplay

Modified according to position, environment,…

Diegetic

Directly related to the game world

E.g. footstep sounds, ...

Non-Diegetic

Not directly part of the game world

E.g. interface sounds, background music, ...

Sound Effects

KOM – Multimedia Communications Lab 58http://www.garycmartin.com/mouth_shapes.html

No Sound Effects


Important for realistic characters

Watch out for repetitions...

Speech


Originally based on square waves,…

http://www.bfxr.net

Sound effects back then

http://www.bfxr.net/


Recorded samples

Little to no hardware support

Number of simultaneously mixed samples usually limited

Sound effects now


Frequency increases/decreases when

sound source/receiver moves

For increasing/decreasing distance

B: destination

S: source

c: speed of sound

Doppler Effect

𝑓𝐵 = 𝑓𝑆 ∙𝑐 + 𝑣𝐵𝑐 − 𝑣𝑠

𝑓𝐵 = 𝑓𝑆 ∙𝑐 − 𝑣𝐵𝑐 + 𝑣𝑠


Highly dependent on surface structure and wavelength

Large surface, small wavelength

Direct reflection

Rough surface

Scatters sound

Model occlusion using raycasts

Alter sound based on blocking geometry and amount of blockage

Sound reflections, Occlusion

KOM – Multimedia Communications Lab 64www.youtube.com/watch?v=qS9XyJlJ-6Q&list=PL-FaA4k7UdNZUROsuHh7IeZMKdxiawBWv&index=14

Music

Anna Bon - Op. 3 No. 2 - Sonata for 2 flutes & continuo in D major - Fuga


Pitch

Frequency

c d e f g a h c

Duration

Duration

Loudness

~Amplitude

Tone Color

Wave form

Instrument

Music


Early games used simple wave forms

Square waves

Triangle waves

Sawtooth waves

Plus noise

NES

Game Boy

Master System

Early 80s Music


Example from NES

https://www.youtube.com/watch?v=psojgjb1F6A


CDs


Plays music

No application control

Apart from start/stop

No file loading while music plays

CDs


Play back large sound files manually

More flexible than CD audio

But not as flexible as previous methods

Today audio compression is widely used

WAV, MP3,…


Orchestras


Sequencer


Basically works like old tracker programs

But more and bigger samples, more effects,…

But almost always export only to wav

Sequencer


Dynamic Music

Wing Commander (1990)

KOM – Multimedia Communications Lab 75https://www.youtube.com/watch?v=YFekhE70vEw, https://www.youtube.com/watch?v=UtckgYN_T-Q

Overlap between music and sound effects

The Legend of Zelda:Skyward Sword (2011)

Portal 2 (2011)

https://www.youtube.com/watch?v=YFekhE70vEw

https://www.youtube.com/watch?v=UtckgYN_T-Q


Vertical Adaptation

Level 1

Level 2

Level 3

Effect 1: Low-Pass Filter

Effect 2: Hall

Normal

More Tension

Maximal Tension

Unconscious 1

Unconscious 2

KOM – Multimedia Communications Lab 77https://www.youtube.com/watch?v=ir1Z25IATDk

Example

Super Mario Sunshine, 2002


Horizontal Adaptation

Loop_Exploring

Exploring

Loop_Battle

Loop_City

Enter Battle

Find City


Different compositional challenges

Vertical: Compose fitting harmonies, rhythms, …

Horizontal: Compose good transitions, maximize connectivity

Fits different gameplay

Vertical: Different states players can enter and leave over a longer time

E.g. Riding a mount vs. Walking

Stages in a boss fight

Horizontal: Larger states, more differences

Overworld vs. Underworld

Boss fight to winning or losing

Different repetitiveness

Horizontal and Vertical Adaptation


iMuse

Monkey Island 2 (1991)


iMuse Special Endings

Bar 1 Bar 2 Bar 3 Bar 4

Ending 1 Ending 2

Ending 3


iMuse Special Endings

https://www.youtube.com/watch?v=AjtxK_WT784


iMuse Transitions

Location 1 Location 1 Location 1

Location 2 Location 2 Location 2

Transition 1 Transition 2


iMuse Transitions

https://www.youtube.com/watch?v=7N41TEcjcvM


Layering/Horizontal Adaptation

Animations synchronized with sound cues

More iMuse Techniques

Baseline

Instrument 1

Instrument 2

Bar 1 Bar 2


More iMuse Techniques

https://www.youtube.com/watch?v=-XuClagw6IQ


Simple APIs

PlaySound()

PlaySoundLooping()

More advanced APIs

Soundbanks – Define which sounds should load together

Realtime effects (DSP)

Streaming

3D sounds, environment

Authoring tool for sound designer

Debugger

Game Engine Integration


Directional Sources

Approximate by cone

E.g. human speakers

Omnidirectional sources

Represent by spheres

Ambient sounds

Background sounds

Types of Sound Sources


Sound Designer works with Authoring Tool

Can work mostly independent

Debugger

Example: WWise

Game WWise

EventsSwitches

StatesRTPCs


Single named event, such as Event_Footstep_Left

Can cue playback of...

A single sound

A different dynamic music

An effect

...

The result can be modified by switches or states

Events


Events


Can switch between alternative

sounds

Used to modify triggers

afterwards

Examples

Different footstep sounds

Firing modes

...

Event_Footstep_Left

Switch_Footsteps_Metal

Event_Footstep_Left

Switches


Model different states the

game/the avatar is in

Examples

Normal

Underwater

Unconscious

...

States


Real-time parameters controls

Game sends value (float)

Maps effects/sounds to different values using curves

Examples

Fatigue

Engine sounds (Parameter: RPM)

RTPCs


RTPCs


Horizontal and Vertical possible

Define transitions

• Fading

• Transition Segments

Adaptive Music


Stingers

Short musical segments that are triggered in the game

Select when they are played (at specific points in the music, …)

Adaptive Music


Sound and Music often essential for games

Increase immersion

Help with transferring information to the player

Badly executed can take away from the experience

Sounds not fitting the game world

Repetitive music

Plan early

Often, addings sounds as an afterthought leads to more work

E.g. when new code for sound integration is needed

Summary

Documents

Der Titel / the Titelwiki.ktxsoftware.com/slides_2015/gametech13.pdf · wrapped on the scope, I poked around the sync generator to find an appropriate frequency or a tone. So those