Web Audio APILet your WebGL program dance

Announcement• Final project proposal due in one week.• Choice of demo date?• Plan for the 12/23 class.

2

Final Projects• Individual or Team of 2.• Voting during the demo.• Do NOT use THREE.js• Themes for this semester: Interactive Art in

WebGL– Animating objects with changing colors,

positions, shapes, …etc.– Interacts with mouse input or audio input (music).

3

Possible Interactions• Input:

– Mouse, keyboard– Audio– Video or camera?

• Output – changing the following– Size, number, position, color, …etc.– Viewpoint– Lighting

4

For Further Reading• Web Audio API:

– Good introduction at: http://www.html5rocks.com/en/tutorials/webaudio/intro/

– More information on the frequency analyzer: http://srchea.com/experimenting-with-web-audio-api-three-js-webgl

• Watch out for deprecated functions:– e.g., createJavaScriptNode() replaced by

createScriptProcessor()

5

WEB AUDIO6

Loading the Music: Option #1

• Option #1: <audio> element in HTML• The following code produces an audio

element with playback controls.

7

<button id="xButton">Rotate X</button><button id="yButton">Rotate Y</button><button id="zButton">Rotate Z</button><br/><button id="pButton" ...>Pause</button><button id="dButton" ...>Depth Test</button><br/><audio id="myAudio" src="Sleep Away.mp3" controls></audio>

A Simple Player

// Experimenting with HTML5 audio var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio);

audioSrc.connect(context.destination);

audio.play();

Adding Frequency Analyzer

var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); var sourceJs = context.createScriptProcessor(2048);

analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.6; analyser.fftSize = 512;

// Connect the MediaElementSource with the analyser audioSrc.connect(analyser); analyser.connect(sourceJs); sourceJs.buffer = audioSrc.buffer; sourceJs.connect(context.destination); audioSrc.connect(context.destination);

sourceJs.onaudioprocess = function(e) { // frequencyBinCount: how many values from the analyser frequencyData = new Uint8Array( analyser.frequencyBinCount); analyser.getByteFrequencyData( frequencyData );};

Visualizationfunction render() {

...

// update data in frequencyData analyser.getByteFrequencyData(frequencyData);

// render frame based on values in frequencyData gl.uniform1f( volumeLoc, frequencyData[160] / 255 ); gl.drawArrays( gl.TRIANGLES, 0, numVertices );

requestAnimFrame( render );}

Loading the Music: Option #2

• Option #2: XMLHttpRequest() in JS

12

loadSound("your_song.mp3");function loadSound(url) { var request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; // When loaded decode the data request.onload = function() {

context.decodeAudioData( request.response, ... ...}

request.send();}

Lab Time!• Download cube3_music.zip• Change frequencyData[160] to other entry.

– How does it respond to the music?• Uncomment console.log(frequencyData)

– What do you see in the console?• How about visualizing the whole array of

frequencyData[0..255]?

13