View
226
Download
0
Embed Size (px)
Citation preview
BPC: Art and Computation – Fall 2006BPC: Art and Computation – Fall 2006
Digital media I: AudioDigital media I: Audio
Glenn Bresnahan Robert Putnam
BPC: Art and Computation – Fall 2006 2
OutlineOutline
Part I (Glenn)
– What is sound?– How do we hear?
Part II (Robert)
– Qualities of sound– Sound reproduction
• analog v. digital
– Sound in VR
BPC: Art and Computation – Fall 2006 3
Waves revisitedWaves revisited
BPC: Art and Computation – Fall 2006 4
Waves – (non)artistic renderingWaves – (non)artistic rendering
BPC: Art and Computation – Fall 2006 5
Wave propertiesWave properties
How might we describe waves?
BPC: Art and Computation – Fall 2006 6
Wave propertiesWave properties
How might we describe waves?– Height– Time between waves– Speed of the wave
BPC: Art and Computation – Fall 2006 7
Shoals and tidesShoals and tides
BPC: Art and Computation – Fall 2006 8
Tide tablesTide tables
BPC: Art and Computation – Fall 2006 9
Cause of tidesCause of tides
Gravity from moon and sun
365 days
27.3 days (29.5 days)
New moon
Full moon
1 day
BPC: Art and Computation – Fall 2006 10
Phases of the moonPhases of the moon
Moon phases:New moonWaxing
crescentFirst quarterWaxing gibbousFull moonWaning gibbousLast quarterNew moon
BPC: Art and Computation – Fall 2006 11
Moon phasesMoon phases
Moon Phase
0%
25%
50%
75%
100%
Date
Per
cen
tag
e V
isib
le (
ph
ase)
BPC: Art and Computation – Fall 2006 12
Sunrise and sunsetSunrise and sunset
Sunrise - Sunset Times
0.00
4.00
8.00
12.00
16.00
20.00
24.00
Date
Ho
ur
of
Day
Sunrise Time
Sunset Time
BPC: Art and Computation – Fall 2006 13
Sunrise and sunsetSunrise and sunset
Daylight Hours
8.00
9.00
10.00
11.00
12.00
13.00
14.00
15.00
16.00
Date
Ho
urs
of
Da
ylig
ht solstice
solstice
BPC: Art and Computation – Fall 2006 14
Waves – sine wavesWaves – sine waves
Sine wave is the fundamental wave
Sine Wave
-1.5
-1
-0.5
0
0.5
1
1.5
BPC: Art and Computation – Fall 2006 15
Waves – propertiesWaves – properties
Sine Wave
-1.5
-1
-0.5
0
0.5
1
1.5
Amplitude
Wavelength (distance)
BPC: Art and Computation – Fall 2006 16
Waves in motion – propertiesWaves in motion – properties
Sine Wave
-1.5
-1
-0.5
0
0.5
1
1.5
Period (time for one cycle)
TimeTime 11 22Frequency cycles per time interval
BPC: Art and Computation – Fall 2006 17
What is sound?What is sound?
Examples
BPC: Art and Computation – Fall 2006 18
What is sound – vibrationWhat is sound – vibration
Striking an object will cause it to vibrateThe vibration is a sine wave Objects have a natural vibration frequency
– Resonance frequency– Frequency depends on type of material,
thickness, length/size, tension– May have multiple vibrating frequencies
The pitch depends on the frequency Loudness (amplitude) depends on size of the
object
BPC: Art and Computation – Fall 2006 19
What is sound – vibrations moves airWhat is sound – vibrations moves air
string
Energy(pluck)
vibration
WaveWaveAir pressure level
BPC: Art and Computation – Fall 2006 20
Properties of soundProperties of sound
Pitch is perception of frequency Frequency is measured in cycles per second
(cps)– Hertz (Hz) = cycles per second– The A above middle C is 440 Hz.– Humans hear appox. 20-20,000 Hz
Sound travels at approx. 1100 feet/second– Speed depends on pressure and temperature– Approx. 750 miles/hour– Approx. 1 mile every 4.8 seconds
Perceived loudness depends on pressure level– Sound pressure is measured in (micro)pascals (20uPa)– Loudness is usually expressed in decibels (dB)
BPC: Art and Computation – Fall 2006 21
Real WavesReal Waves
BPC: Art and Computation – Fall 2006 22
Properties of soundProperties of sound
Real sounds are far more complex than simple sine waves– Objects produce vibrations at multiple
frequencies– Sound waves interact with other objects
• Waves bounce (reflect) off surface – Reverberation/echo
• Wave are absorbed by materials
– Sound waves interact with each other
BPC: Art and Computation – Fall 2006 23
Combinations of wavesCombinations of waves
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
1 26 51 76 101 126
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
1 26 51 76 101 126
BPC: Art and Computation – Fall 2006 24
Properties of sound – real soundsProperties of sound – real sounds
BPC: Art and Computation – Fall 2006 25
Electrification of soundElectrification of sound
Microphones– Convert pressure levels into electrical signals
(voltages) Guitar pickups
– Converts string vibration to voltages• The pickup contains a magnet and a coil• The vibrating metal strings alter the magnetic field
and induce a voltage in the coil
Loud speakers convert an electrical signal back into air pressure
BPC: Art and Computation – Fall 2006 26
How do we hear?How do we hear?
Sound waves move through the air from the sound source to the ear
BPC: Art and Computation – Fall 2006 27
Anatomy of the earAnatomy of the ear
BPC: Art and Computation – Fall 2006 28
Anatomy of the ear - outerAnatomy of the ear - outer
Divided into three principal sections– Outer ear– Middle ear– Inner ear
Outer ear– External ear, aka pinna– Ear canal– Outer ear funnels the ear have to the
eardrum
BPC: Art and Computation – Fall 2006 29
Anatomy of the ear - middleAnatomy of the ear - middle
Middle ear– Eardrum– Set of 3 ear bones
• the 3 bones are rigid
– Act as a mechanical amplifier
– The 3rd bone, stapes, induces a vibration into the inner ear, i.e. the cochlea
BPC: Art and Computation – Fall 2006 30
Anatomy of the ear - innerAnatomy of the ear - inner
Inner ear / cochleaWhere the real work
is doneCochlea is a spiral
tube and filled with fluid
Stapes causes a wave to pass through the fluid
BPC: Art and Computation – Fall 2006 31
Anatomy of the ear - innerAnatomy of the ear - inner
Cochlea is a spiral tube lined with hair cells on a membrane (~15K HCs)
Hairs vary in length and thickness along the tube
Hairs resonate at different frequencies– High freq on near end,
low at far end
BPC: Art and Computation – Fall 2006 32
Anatomy of the ear - innerAnatomy of the ear - inner
BPC: Art and Computation – Fall 2006 33
Anatomy of the ear - innerAnatomy of the ear - inner
Hair cells are connected to the auditory nerve cells
The vibrations excite the nerve cells and cause them to fire (electrical signal)
A series of nerve cells pass the signal to brain
BPC: Art and Computation – Fall 2006 34
Binaural hearing - why two ears?Binaural hearing - why two ears?
Two ears, so we can identify locations of sounds– Time difference– Intensity difference– Sound color difference (caused by
movement of sound around head and shoulders)
BPC: Art and Computation – Fall 2006 35
Sound localization – pinnaSound localization – pinna
front
back Sound waves interact with
the asymmetric Pinna The effect on the sound
varies with the direction Up/down, back/front waves
result in different sounds entering ear canal
BPC: Art and Computation – Fall 2006 36
Digital media I: part IIDigital media I: part II
Other qualities of sound: pitch, timbre, “noise”, envelope
Sound reproduction: analog v. digitalSound in VR
BPC: Art and Computation – Fall 2006 37
What is pitch?What is pitch?
Our perception of the highness or lowness of a tone.
Closely related to frequency When frequency doubles, pitch rises by an
“octave” Examples But, what happens when there’s more
than one frequency in a sound?
BPC: Art and Computation – Fall 2006 38
Review: modes of vibration of a Review: modes of vibration of a stringstring
Fundamental
[e.g., 110 Hz]
2nd harmonic
[e.g., 220 Hz]
3rd harmonic
[e.g., 330 Hz]Examples
BPC: Art and Computation – Fall 2006 39
TimbreTimbre
Sound color, or “timbre” is a quality of sound that derives from the particular combination of frequencies (a.k.a., “harmonics” or “partials”) in a tone.
Two sounds can contain the same harmonics but sound very different because their individual harmonics are of different amplitudes.
Examples
BPC: Art and Computation – Fall 2006 40
Timbre, continuedTimbre, continued
Easy to demonstrate timbre with human voice
Hum.Slowly open mouth.Hear how the sound color changes
from “dark” to “bright”Example
BPC: Art and Computation – Fall 2006 41
Timbre, continued.Timbre, continued.
Timbre changes as a wind instrument is played louder or softer.
Example
BPC: Art and Computation – Fall 2006 42
Unpitched soundsUnpitched sounds
Can use human voice to demonstrate another distinction: pitched versus unpitched sounds
Make “s” soundNo identifiable “pitch”Related to concept of “noise”Examples
BPC: Art and Computation – Fall 2006 43
ExamplesExamples
Pitched sounds– Birdsong– Flutes– Stringed instruments– Etc.
Examples of unpitched sounds– Certain percussion instruments (cymbals, ratchets, etc.)– Wind, rain, footsteps in snow
Listen now. What do you hear? Frequencies, amplitudes. Pitched, unpitched. External versus internal sources.
BPC: Art and Computation – Fall 2006 44
Time variation of soundsTime variation of sounds
Most naturally occurring sounds are not static; i.e., they vary over time– Amplitude– Pitch– Timbre
Examples
BPC: Art and Computation – Fall 2006 45
Sound recording technologiesSound recording technologies
AnalogDigital
BPC: Art and Computation – Fall 2006 46
Analog recordingAnalog recording
Analog: “device or system that represents changing values as continuously variable physical quantities.”
Example: clock with hour, minute and second hands
Question: what values are changing when we hear sound?
BPC: Art and Computation – Fall 2006 47
Analog recording technologiesAnalog recording technologies
Phonautograph
BPC: Art and Computation – Fall 2006 48
Analog recording technologiesAnalog recording technologies
Mechanical: Gramophone, LP record, etc.
BPC: Art and Computation – Fall 2006 49
Analog recording technologiesAnalog recording technologies
Magnetic: Wire, tape recorder.
BPC: Art and Computation – Fall 2006 50
Analog recording technologiesAnalog recording technologies
Optical: movie soundtrack.
BPC: Art and Computation – Fall 2006 51
Analog sound reproductionAnalog sound reproduction
Amplification
Loudspeaker demo
BPC: Art and Computation – Fall 2006 52
Digital recordingDigital recording
Digital: “device or system that represents changing values as discontinuous, or ‘discrete,’ values.”
Example: clock with number readout.
BPC: Art and Computation – Fall 2006 53
Digital recordingDigital recording
With digital recording, we do not store a continuous record of the rise and fall of air pressure. We make measurements of the air pressure (or the voltage produced by a microphone) thousands of times per second and store these measurements as numbers.
BPC: Art and Computation – Fall 2006 54
Digital recordingDigital recording
Analog: continuous waveform
Digital: discrete values
BPC: Art and Computation – Fall 2006 55
Some buzzwordsSome buzzwords
ADC: analog-to-digital converterDAC: digital-to-analog converterSampling rate: samples/secondWord size: how much storage for
each sampleQuantization: [see next slide]
BPC: Art and Computation – Fall 2006 56
QuantizationQuantization
Selecting sample value from finite set of numbers.
16 bits = 65,536 choices
20 bits = 1,048,576 choices
Source: http://advisor.matrasi-tls.fr/digital_sampling_index.html
BPC: Art and Computation – Fall 2006 57
CD audioCD audio
44,100 samples per second.16-bit samples (65536 different
possible values)Frequency range: 0-22KHz.
BPC: Art and Computation – Fall 2006 58
Some benefits of digital audioSome benefits of digital audio
Easy to edit (visual interface)No noise with additional generationsFlexible signal processing (no
special hardware)Examples (reverb, pitch shift, noise
reduction, etc.)
BPC: Art and Computation – Fall 2006 59
An aside: MP3 audioAn aside: MP3 audio
CD takes up a lot of space: 3 minute song = 44100*2*2*3*60 = 31752000 B.
MP3 compression results in a factor of 5-10 savings in storage, but lower fidelity (e.g., noisier).
BPC: Art and Computation – Fall 2006 60
Another aside: MIDIAnother aside: MIDI
Musical Instrument Digital InterfaceA communications scheme for
computers, synthesizers, sequencers, etc.
Suited for popular musicStores Note-On, Note-Off, Velocity,
etc. (i.e., not waveforms)Example
BPC: Art and Computation – Fall 2006 61
DAFFIE audioDAFFIE audio
Soundserver– Plays sound files associated with
objects– Mixes many simultaneous sounds– Internet telephony support
BPC: Art and Computation – Fall 2006 62
DAFFIE localizationDAFFIE localization
All sounds are assigned a location in virtual space.
Typically, sounds are associated with visible objects, but “ambient” sound (e.g., wind, nature) is supported too.
Direction and distance are indicated by variations in loudness among the loudspeakers – done automatically by soundserver.
BPC: Art and Computation – Fall 2006 63
Uses of sound in VRUses of sound in VR
Communication (via telephony)Sound effectsMusic selectionsAmbient audioLive audio (via telephony)Previous projects have involved
controlling synthesizers or musical instruments remotely.
BPC: Art and Computation – Fall 2006 64
Recording sounds for DAFFIERecording sounds for DAFFIE
Field recording versus studio recording
Record, record, recordRemember that sounds are
combined “in real time” by the soundserver (so no need to put everything in a single file).
Stereo is good.
BPC: Art and Computation – Fall 2006 65
Recording sounds for DAFFIE, part Recording sounds for DAFFIE, part II.II.Remember that you can change
sounds in various ways– Change pitch/tempo– Use filters to change spectrum– Cut and paste
BPC: Art and Computation – Fall 2006 66
DAFFIE demoDAFFIE demo
Demonstrate– Proximity triggering– (Variable) Distance attenuation– Sound localization