Kung-Fu PandaA Musical

Apurva GuptaMS-HCI

Brief

The aim of the sonification was to create a musical representation for the movie – Kung Fu Panda.

The most direct way was to map each pixel with a sound and let the composition tell the story. To overcome the monotony, brightness of the pixel was explored and played around with using volume gain. Based on how high the value of RGB is, the sound was assigned accordingly.

The Asian context of the movie influenced the choice of music used. For recording, the sound was created by striking a half-filled, full filled and empty metal container. These sounds were mapped to value of Green color.

To add texture to the composition, the total of 160 measures were sub-divided based on the scenes in the movies and leading cast in that scene. Beat compositions were taken and assigned to each character of the movie. Based on its appearance, would the composition play.

Computational Choices

• To achieve the musical composition, the data structure used was 2 dimensional lists to store the RGB values of the pixel. 1-D arrays was also used to store audio files

• An extensive use of “for loops” and “conditionals” was made. Nested for loops was used to place an appropriately mapped audio onto the track. Conditionals were used to compare the values of RGB and place the media file accordingly.

• Fitmedia, MakeBeat, SetEffect were used in abundance to place audio onto track, create beats or set volume gain or pan effect.

Process

Explore Convert Create

EXPLOREVarious ideas were explored and various computational methods were tried. Finally the idea to sonify a movie barcode was chosen and elaborated upon.

The movie Kung Fu panda was chosen because it was one of my most favorite movies and I am always excited to work around it in any form.

Also, I was well acquainted with the storyline and characters of the movie.

Story-Line

Movie BarcodeBased on the scenes of the movie

CONVERTThe 1280X480 pixel barcode was scaled down to 160X60 pixels. From the new image the pixel values for the middle row “image[x][30]” was extracted and saved in an array.

The array was 2 dimensional which has 160 elements with 3 sub-elements each. The array was obtained using the PIL (Python Image Library) Library of Python.

barcode = [[83, 71, 55], [94, 64, 38],[133, 74, 40],[172, 101, 71],[115, 67, 44],[61, 49, 33],[63, 63, 55],[71, 60, 58],[70, 59, 53],[79, 87, 66],[42, 66, 40],[28, 64, 62],[1, 68, 74],[76, 84, 71],[100, 71,

37],[105, 92, 60],[125, 116, 83],[85, 85, 51],[95, 98, 67],[99, 99, 73],[66, 58, 35],[90, 84, 60],[68, 75, 42],[101, 123, 84],[142, 163, 107],[100, 112, 76],[93, 93, 67],[86, 80, 46],[91, 80, 48],[107, 95, 81],[50, 39, 45],[13, 5, 18],[7, 11, 14],[7, 0, 7],[12, 13, 15],[25, 31, 17],[65, 61, 36],[55, 56, 25],[31, 39, 18],[38, 31, 25],[40, 23, 13],[43, 30, 22],[43, 30, 24],[41, 24, 16],[42, 22, 13],[39, 29, 28],[28, 37, 52],[14, 39,

69],[22, 43, 60],[26, 44, 64],[14, 31, 51],[24, 35, 57],[37, 44, 63],[21, 24, 39],[10, 10, 20],[14, 13, 19],[11, 8, 25],[15, 13, 26],[25, 27, 42],[14, 23, 52],[5, 19, 56],[10, 17, 36],[70, 59, 41],[126, 101, 47],[143, 113,

61],[85, 76, 37],[81, 34, 14],[114, 46, 27],[71, 44, 23],[43, 36, 28],[69, 37, 22],[94, 56, 7],[29, 42, 35],[38, 41, 32],[73, 47, 20],[56, 38, 24],[8, 35, 62],[14, 47, 78],[30, 52, 73],[9, 54, 93],[59, 48, 42],[63, 45,

33],[54, 32, 18],[55, 41, 32],[22, 23, 25],[14, 28, 39],[9, 27, 39],[31, 45, 54],[26, 38, 52],[25, 21, 22],[70, 45, 38],[61, 27, 25],[59, 32, 39],[53, 41, 41],[116, 115, 87],[111, 115, 64],[71, 87, 76],[76, 63, 46],[127,

90, 63],[136, 98, 62],[140, 104, 70],[126, 87, 56],[124, 88, 52],[124, 97, 52],[64, 66, 61],[59, 64, 58],[98, 103, 99],[91, 96, 92],[72, 78, 76],[75, 81, 79],[107, 116, 113],[88, 97, 94],[88, 65, 31],[92, 55, 28],[64, 55, 22],[51, 65, 30],[51, 46, 24],[39, 47, 24],[25, 47, 24],[50, 46, 34],[42, 39, 22],[13, 16, 21],[16, 19, 26],[38,

34, 25],[33, 29, 18],[52, 55, 60],[38, 42, 41],[52, 50, 29],[52, 57, 34],[2, 19, 29],[13, 31, 35],[15, 24, 19],[24, 42, 62],[22, 69, 79],[10, 53, 43],[20, 30, 39],[54, 42, 18],[90, 71, 39],[99, 74, 34],[92, 70, 29],[74, 61, 27],[85, 76, 47],[96, 79, 49],[96, 72, 38],[84, 64, 40],[86, 76, 66],[108, 88, 55],[88, 63, 32],[147, 113,

68],[157, 116, 54],[101, 79, 40],[99, 82, 38],[52, 63, 31],[102, 68, 30],[107, 30, 4],[51, 24, 3],[71, 41, 30],[52, 34, 10],[35, 2, 0],[91, 88, 99]]

1280 X 480160 X 60

CREATEThis section enumerates how the sonification of the pixels was achieved

Mapping Audio

• Step 1:Find maximum and minimum values for R, G, B color values

• Step 2:Segment into 3 sets based on their range (R, G,B each)

• Step 3:Map 9 audio files 3 each for R, G, B based on the sets

The selection of audio files was such that the 3 chosen audio files for each color belonged to the same family of sounds.

Red: GongGreen: Metal Strike (Recordings)Blue: Drums

Mapping Volume Level

• Step 1:Find maximum and minimum values for R, G, B color values

• Step 2:Scale min and max values of each color to -10 and 12

• Step 3:Using Linear equation to find respective value of volume gain for each color

Red

Green

Blue

Volume Mapping

Adding Texture

For this, an 8X4 matrix was made and scenes were mapped to the lead cast in the scene. Each character was allotted a beat composition which was played according to the characters appearance in the scene.

MEASURE PO FURIOUS FIVE TAI LUNG SHIFU/OOGWAY

BEGIN 1-20

TOURNAMENT 21-40

OOGWAY’S WISDOM 41-60

ENTERS TAI LUNG 61-80

TRAINING 81-100

BRIDGE 101-120

FIGHT 121-140

END 141-160

*

* : Ideally should be mapped with repetition of bright pixels vs dark pixels

Tracks for Oogway / Shifu

Tracks for Po

Tracks for Furious Five

Insights/Learnings/Problems

• The expression of ideas to music is difficult

• I could work my way through the code but the piece didn’t sound as pleasing as expected.

• Its hard to adhere to naming conventions and comments while “engrossed” in writing the code but it is the ABSOLUTE necessity to save yourself from wasting time (and frustration!!) later.

• Its fun to be able to morph colors into numbers and numbers into sounds.

• Indentation: The flexibility of the language can sometime make you pull your hair and bang into the screen.