OpenFrameworks Lections

4. Interactive audio

Denis Perevalovperevalovds@gmail.com

See in-depth details in my book

“Mastering openFrameworks”

Book’s examples are free, see

masteringof.wordpress.com

What is Digital Sound

What is the sound at allSound in the broad sense - Elastic waves propagating longitudinally in the environment and pose in her mechanical vibrations;in the narrow sense - The subjective perception of these oscillations special sense organs of animals or humans.Like any wave, the sound is characterized by amplitude and frequency.(Wikipedia)

http://blog.modernmechanix.com/mags/qf/c/PopularScience/9-1950/med_sound.jpg

Representation of sound in digital form

The real sound is captured by a microphone, then subjected to an analog-digital conversion.

It is characterized by temporal resolution - sampling, [Procedure - Sampling]amplitude resolution - capacity. [Procedure - quantization]

Amplitude

http://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Digital.signal.svg/567px-Digital.signal.svg.png

Time

Sampling frequency

8,000 Hz - telephone, enough for speech.

11,025 Hz - games, samples for electronic music.

22 050 Hz - the same as 11,025 Hz.

44 100 Hz - many synths and sample libraries. Audio CD.

48 000 Hz - the recording studio, live instruments and vocals. DVD.96,000 Hz - DVD-Audio (MLP 5.1).

192,000 Hz - DVD-Audio (MLP 2.0).

CapacityLength - the number of bits used to represent the signal samples in the quantization (in our case - the quantization of the amplitude).

8-bit samples of electronic music.

12-bit studio sound effects.

16 bitscomputer game players, samples, Audio CD.

18-bit studio sound effects

24-bit live sound, vocals,DVD-Audio

32 bitsrepresentation of floating point numbers, so accuracy is not lostfor sounds, so is used for internal processing of sound.

64-bit and floating point audio processing.

Representation of sound in memory

Example1 second 16-bit audio with a frequency of 44100 Hz can be represented as a vector

X = (x_1 x_2, ..., ..., x_44100),where 0 <= x_i <= 2 ^ 1.16 = 65535.

Representation of sounds such a way - with the help of the vector - calledPCM(Pulse Code Modulation).It is the most common.It is analogous to pixel representation of images.

The fundamental difference between sound and image

Since the images are very convenient to operate at the level of pixels. In particular,

1. two images, we believe the same if their values pixels close.

2. You can change the images based on the values of neighboring pixels (for example, the operation of smoothing).

For audio in PCM format for both possibilities not applicable

The fundamental difference between sound and image1. To the 1 st octave 440.00 Hz

2. it also shifted the phase

3. mi 2 nd octave 659.26 Hz

1. + 3.

2. + 3.

(Audacity was used for sound generation)

The last two sound sound the same. And their function amplitude - much different. Thus, chelovecheckoe ear perceives the sound spectrum, ie the composition of its frequency, not amplitude representation of sound.

That it is easy / difficult to do "straight" with the sound in PCMEasy:Changing and rearranging the individual samples, without regard to its neighbors- Rearrange the pieces, - Change the volume slices,- Do the reverse - the coup of sound from end to beginning,- To mix several sounds- Mix and change the stereo channels- Do simple compression,- Adding just an echo.Samplers, portastudii and studio program make it masterly.

Hard: Accounting neighboring samples- Compare two sounds at the similarities,- Suppress the low and high frequencies, - Add reverb.

This usually is not done right in the PCM, and by the spectral representation of sound (window Fourier transform).

Storage formats of sound

WAVwav = header + bytes of PCMKeeps sound without quality loss (Similar to the images - bmp)

MP3These losses are well suited for storing music.(Similar to the images - jpg)

AMRThese losses, suitable for storage of speech. Used in mobile telephony (2011).(Similar to the images - png)

Ways to generate a digital audio

Ways to generate a digital audio

That is, how to build PCM-representation of a sound or music:

1. SamplingUsed for the production of the music. Devices - samplers

2. (Subtractive) Synthesis Used mainly for modern electronic music.Devices - keyboards.

3. FM-synthesis

4. Additive synthesis

5. Granular Synthesis

6. S & S - Sample & Synthesis - Sampling, analysis and subsequent synthesis -today one of the best technologies play "live" instruments.

SamplingRecording: "Live Sound" - a microphone - ADC - PCM-format.

Playback: PCM-format - DAC - speaker.

Additional options: you can change the playback speed, then increase the tone and speed of the sample.Modern algorithms also enable you to change the tone of the sample without changing its speed, and vice versa.

http://josephdbferguson.com/uploads/akai_mpc1000.jpg

Sampler Akai MPC1000

(Subtractive) SynthesisIn precomputer time:a few simple waves (rectangular, sinusoidal, triangular) processed a set of filters (bass, treble, cut the desired frequency). The resultant was going to the speakers.

Now:done digitally.There are difficulties - should carefully consider the problems associated with the digital representation of sound ("aliasing").

http://www.jarrography.free.fr/synths/images/moog_minimoog.jpg

SynthesizerMinimoog

Sample playbackin openFrameworks

The project "soundscape"

User poke the mouse in different parts of the screen and begins to be heard a soundhttp://www.freesound.org/samplesViewSingle.php?id=221

// Declare variablesofSoundPlayer sample;// Sample playerofPoint p; // point and the radius - to draw a circlefloat rad;

void testApp:: setup () { sample.loadSound ("sound.wav"); // Load sample from the folder bin / datasample.setVolume (0.5f);// Volume [0, 1]sample.setMultiPlay (true);// Allow you to run multiple samplesofSetFrameRate (60) // speed drawing frame ofSetBackgroundAuto (false); // turn off background eraseofBackground (255,255,255);}

The project "soundscape"

void testApp:: update () {ofSoundUpdate ();// Update the status of the sound system}

void testApp:: draw () {

// If the sound is played, draw a transparent circleofEnableAlphaBlending ();if (sample.getIsPlaying ()) {// Random colorofSetColor (ofRandom (0, 255), ofRandom (0, 255), ofRandom (0, 255), 20);ofCircle (px, py, rad); }ofDisableAlphaBlending ();}

The project "soundscape"

// Clicked the mousevoid testApp:: mousePressed (int x, int y, int button) {

float h = ofGetHeight (); // screen height

// Calculate the desired playback speed of the sample,// In this case 1.0 - is the original sample ratefloat speed = (h - y) / h * 3.0;if (speed> 0) {sample.play ();// Start of a new samplesample.setSpeed (speed); // Set the playback speed

// Remember the point and the radius of the circle to drawp = ofPoint (x, y);rad = (3 - speed);rad = 20 * rad * rad;}}

The project "soundscape"

Additive synthesis

Additive synthesis based on the construction of sound by summing a set of harmonics (ie, sine waves of different frequencies) with variable volume.

Any sound can be represented with arbitrary accuracy as the sum of a large number of harmonics with varying volume. But in practice, work with a large number of harmonics requires large computational resources. Although, at present, there are several hardware and software additive synthesizers.

Project scenario "Additive Synthesizer"

A user on a white background with his hands in front of the camera. Therenharmonics. The screen is divided intonvertical strips, each considered to be the number of pixels, the brightness is less than a certain threshold. This number determines the volume of the corresponding harmonics.

Use n = 20 sinusoidal harmonics with frequencies100 Hz200 Hz...2000 Hz

Harmonics are played with looped samples, which simply changes the volume.

Synth Code 1 / 4

// Declare variables

// Video-grabber for "capture" the video framesofVideoGrabber grabber;int w;// Width of the frameint h;// Height of the frame

const int n = 20;// Number of harmonicsofSoundPlayer sample [n];// Samples of harmonicsfloat volume [n]; // Volume of harmonicsint N [n];// Number of pixels in the play harmonica

ofSoundPlayer sampleLoop; // Sample a drum loop

Synth Code 2 / 4

// Initializevoid testApp:: setup () {

w = 320;h = 240;grabber.initGrabber (w, h); // Connect the camera

// Grab samples harmonicsfor (int i = 0; i <n; i + +) {int freq = (i +1) * 100;sample [i]. loadSound (ofToString (freq) + ". wav"); // Files are called 100.wav, ...sample [i]. setVolume (0.0);// Volumesample [i]. setLoop (true);// Looping Soundsample [i]. play ();// Start sound}

}

Synth Code 3 / 4// Update the statevoid testApp:: update () {grabber.grabFrame (); // grab a frameif (grabber.isFrameNew ()) {// if you come to a new framefor (int i = 0; i <n; i + +) {volume [i] = 0; N [i] = 0;} // Reset the harmonicunsigned char * input = grabber.getPixels (); // pixels of the input imagefor (int y = 0; y <h; y + +) {for (int x = 0; x <w; x + +) {// Input pixel (x, y):int r = input [3 * (x + w * y) + 0];int g = input [3 * (x + w * y) + 1];int b = input [3 * (x + w * y) + 2];int result = (r + g + b> 400)? 0: 1;// Thresholdint i = (x * n / w);// In which to write the result of harmonic volume [i] + = result;N [i] + +;}}// Set the new volume of harmonicsfor (int i = 0; i <n; i + +) {if (N [i]> 0) {volume [i] / = N [i];} // Normalize the volume [0, 1]sample [i]. setVolume (volume [i] / n); // Volume.// Divide by n, otherwise it will be distortion of the output sound}} OfSoundUpdate ();// Update the status of the sound system}

Synth Code 4 / 4

// Drawvoid testApp:: draw () {ofBackground (255,255,255); // Set the background colorfloat w = ofGetWidth ();// Screen height and widthfloat h = ofGetHeight ();

ofSetColor (255, 255, 255); // Else draw a picture frame is incorrectgrabber.draw (0, 0, w, h);// Output frame

// Draw the harmonicsofEnableAlphaBlending (); // Enable transparencyofSetColor (0, 0, 255, 80);// Blue color with opacity of 80for (int i = 0; i <n; i + +) {float harmH = volume [i] * h;// Height of the bar harmonics iofRect (i * w / n, h - harmH, w / n, harmH);}ofDisableAlphaBlending ();// Disable transparency}

Performance on the “additive synthesizer”

http://www.youtube.com/watch?v=y70Oxk1RAOM

Sound synthesisin openFrameworks

Introduction

Sound synthesis on openFrameworks performed at the lowest level, "byte".

Therefore it is suitable esperimentalnyh projects with sound.

In complex projects, it is more convenient to use a special type of library SndObj (See enlargement oxfSndObj) or some other program like PureData or Max / MSP,which binds openFrameworks protocol OSC.

Program Structure

For sound synthesis conventional structure of the program improves audioRequested (). It is called the sound driver, when you need to fill in another piece of sound buffer sound.

Program Structure

in testApp.h, class testApp add:void audioRequested (float * input, int bufferSize, int nChannels);

in the setup () to add:ofSoundStreamSetup (2,0, this, 22050, 256, 4);

/ 2 output channels,// 0 input channels,// 22050 - sampling rate, samples per second// 256 - buffer size// 4 - how to use buffers. Affects the delay. // Buffer size and number of buffers - set the balance between delay and the resulting sound, "Glitter," which occurs when the computer is not fast enough.

Program Structure

in testApp.cpp add:

void testApp::audioRequested (Float * output, // output bufferint bufferSize, // buffer sizeint nChannels // number of channels){// Example of sound "white noise" to two channelsfor (int i = 0; i <bufferSize; i + +) {output [i * nChannels] = ofRandomf (); // [-1,1]output [i * nChannels + 1] = ofRandomf ();} }

Example

See an example audioOutputExample in OpenFrameworks.

Mouse moves1. up and down - changing the tone of the sound.2. left and right - changing panorama.Mouse click - generated noise.

Example of synthesis: RubberGravity

The rubber squares tensile generate sound.

http://www.youtube.com/watch?v=Pz6PO4H1LT0

Homework

Using the example audioOutputExample,add to the example of the swinging pendulum of sound generation. Namely:Let the position of the pendulum in the Y sets the pitch, and the position of the pendulum on X - panning.