ACM SIGCSE 2003: Multimedia Construction Projects Mark Guzdial College of Computing Georgia Institute of Technology [email protected] mark.guzdial

ACM SIGCSE 2003: Multimedia Construction Projects

Mark Guzdial

College of Computing

Georgia Institute of [email protected]

http://www.cc.gatech.edu/~mark.guzdial

Plan

7-7:15: Introduction What’s on your CD Why media computation

7:15-7:45: Picture manipulations in Python 7:45-8:15: Sound manipulations in Python 8:15-8:30: Break 8:30-8:45: How to do this in Java and Squeak 8:45-9:45: You Play 9:45-10: Wrap-up

What’s on your CD

Materials from our Introduction to Media Computation course Jython Environment for Students (JES) Book Course slides MediaTools: Squeak-based media exploration tools

Material for this workshop Java and Squeak API slides Java source code Squeak computer music essays

Computer science is more important than Calculus In 1961, Alan Perlis argued

that computer science is more important in a liberal education than calculus Explicitly, he argued that all

students should learn to program.

Calculus is about rates, and that’s important to many.

Computer science is about process, which is important to everyone

How close are we to being able to teach everyone CS?

Not very CS1 is one of the most despised courses for non-majors

At many departments, CS retention rates are lower than the rest of campus At Georgia Tech: 65% for 1995 cohort, vs. 73% for

Engineeering

Drop-out rates near 50% at many institutions Female enrollment in CS has been dropping nationally

Why?

Several recent studies and books claim that CS instruction tends to dissuade anyone but white males “Tedious,”

“taught without application relevance,”“boring,” “lacking creativity,” “asocial”

The potential of computing

How can we realize the potential impact of computing if future professionals in other disciplines despise computer science?

To realize that potential, we need practitioners in other disciplines to understand computing. At least some of those practitioners need to understand

it as a creative, exciting venture—as we do!

The best uses for computing technologies will come from others

Thomas Edison vs. D.W. Griffith If we want computing technologies to become

useful, they have to get out of our hands. It can’t be just through applications

That presumes that the current technologists know how everyone else wants to do things and should do things

Suggestion: D.W. Griffith knew things that Edison didn’t.

An attempt at relevant, creative computing: Introduction to Media Computation

A course for non-CS and non-Engineering majors International Affairs, Literature, Public Policy,

Architecture, Management, Biology, etc.

120 students this semester,planning 400-600 in the Fall 2/3 female in this semester’s CS1315

Focus: Learning programming within the context of media manipulation and creation

Motivating the Computing

As professionals, these students will often the use the computer as a communications medium.

All media are going digital,and digital media are manipulated with software.

Knowing how to program, then, is a communications skill.

Python as the programming language Huge and contentious issue Use in commercial contexts legitimatizes the choice

Industrial Light & Magic, Google, Nextel, etc.

Minimal syntax Looks like other programming languages

Potential for knowledge transfer for students “Executable pseudocode” for this workshop

Actually using Jython (http://www.jython.org) for Java class libraries

JES - Jython Environment for Students

Simple Python in JES

>>> print 34 + 5690>>> print 34.1/46.50.7333333333333334>>> print 22 * 33726>>> print 14 - 15-1>>> print "Hello"Hello>>> print "Hello" + "Mark"HelloMark

Command Area Editing

Up/down arrows walk through command history You can edit the line at the bottom

Just put the cursor at the end of the line before hitting Return/Enter.

Basic Picture Functions

makePicture(filename) creates and returns a picture object, from the JPEG file at the filename

show(picture) displays a picture in a window We’ll learn functions for manipulating pictures

later, like getColor, setColor, and repaint

Basic Sound Functions

makeSound(filename) creates and returns a sound object, from the WAV file at the filename

play(sound) makes the sound play (but doesn’t wait until it’s done)

blockingPlay(sound) waits for the sound to finish We’ll learn more later like getSample and

setSample

Demonstrating simple JES>>> print pickAFile()/Users/guzdial/mediasources/barbara.jpg>>> print makePicture(pickAFile())Picture, filename /Users/guzdial/mediasources/barbara.jpg height 294 width 222>>> print pickAFile()/Users/guzdial/mediasources/hello.wav>>> print makeSound(pickAFile())Sound of length 54757>>> print play(makeSound(pickAFile()))None>>> myfilename = pickAFile()>>> print myfilename/Users/guzdial/mediasources/barbara.jpg>>> mypicture = makePicture(myfilename)>>> print mypicturePicture, filename /Users/guzdial/mediasources/barbara.jpg height 294 width 222>>> show(mypicture)

Writing a recipe: Making our own functions To make a function, use the

command def Then, the name of the function,

and the names of the input values between parentheses (“(input1)”)

End the line with a colon (“:”) The body of the recipe is indented

(Hint: Use two spaces) Your function does NOT

exist for JES until you load it

Functions for picking and playing/showing files

def pickAndShow(): myfile = pickAFile() mypict = makePicture(myfile) show(mypict)

def pickAndPlay(): myfile = pickAFile() mysound = makeSound(myfile) play(mysound)

Functions that take input

def playNamed(myfile): mysound = makeSound(myfile) play(mysound)

def showNamed(myfile): mypict = makePicture(myfile) show(mypict)

CS1315:Introduction to Media Computation

Picture encoding and manipulation

Pictures and Pixels

A picture is a matrix of pixels A picture has two dimensions: Width and Height

Pixels are picture elements Each pixel object knows its color using RGB

encoding It also knows where it is in its picture

RGB

In RGB, each color has three component colors:

Amount of redness Amount of greenness Amount of blueness

Each does appear as a separate dot on most devices, but our eye blends them.

In most computer-based models of RGB, a single byte (8 bits) is used for each

So a complete RGB color is 24 bits, 8 bits of each

Encoding RGB

Each component color (red, green, and blue) is encoded as a single byte

Colors go from (0,0,0) to (255,255,255) If all three components are

the same, the color is in greyscale

(50,50,50) at (2,2) (0,0,0) (at position (1,2) in

example) is black (255,255,255) is white

Manipulating pixels

>>> pixel=getPixel(picture,1,1)>>> print pixelPixel, color=color r=168 g=131 b=105>>> pixels=getPixels(picture)>>> print pixels[0]Pixel, color=color r=168 g=131 b=105

getPixel(picture,x,y) gets a single pixel.

getPixels(picture) gets all of them in an array. (Square brackets is a standard array reference notation—which we’ll generally not use.)

What can we do with a pixel?

• getRed, getGreen, and getBlue are functions that take a pixel as input and return a value between 0 and 255

• setRed, setGreen, and setBlue are functions that take a pixel as input and a value between 0 and 255

We can also get, set, and make Colors getColor takes a pixel as input and returns a Color object with the

color at that pixel setColor takes a pixel as input and a Color, then sets the pixel to that

color makeColor takes red, green, and blue values (in that order) between

0 and 255, and returns a Color object pickAColor lets you use a color chooser and returns the chosen

color We also have functions that can makeLighter and makeDarker an

input color

We can change pixels directly…>>> file="/Users/guzdial/mediasources/barbara.jpg">>> pict=makePicture(file)>>> show(pict)>>> setColor(getPixel(pict,10,100),yellow)>>> setColor(getPixel(pict,11,100),yellow)>>> setColor(getPixel(pict,12,100),yellow)>>> setColor(getPixel(pict,13,100),yellow)>>> repaint(pict)

But that’s really dull and boring…That’s the subject of the next lecture

If you make something you like…

writePictureTo(picture,”filename”) Writes the picture out as a JPEG

Be sure to end your filename as “.jpg”!

If you don’t specify a full path,will be saved in the same directory as JES.

How do you find out what RGB values you have? And where?

Use the MediaTools!

Use a loop!Our first picture recipe

def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Used like this:>>> file="/Users/guzdial/mediasources/barbara.jpg">>> picture=makePicture(file)>>> show(picture)>>> decreaseRed(picture)>>> repaint(picture)

getPixels returns a sequence of pixels Each pixel knows its color and its original picture Change the pixel, you change the picture So the loop below assigns the index variable p to

each pixel in the picture picture, one at a time.

def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Let’s walk that through slowly…def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Here we get a picture object in as input and call it picture

picture

Now, get the pixelsdef decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

We get all the pixels from the picture, then make p be the name of each one one at a time

picture

Pixel, color r=168 g=131 b=105


Pixel, color r=168 g=132b=106

…

p

getPixels()

Get the red value from pixeldef decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

We get the red value of pixel p and name it value

picture



…

pvalue = 168


Now change the pixeldef decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Set the red value of pixel p to 0.5 (50%) of value

picture



…

pvalue = 168


Then move on to the next pixeldef decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Move on to the next pixel and name it p

picture



…

pvalue = 168


Get its red valuedef decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Change value to the new red value at the new p

picture




…

value = 160p

Get its red valuedef decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

Change value to the new red value at the new p

picture



…

p


And change this red value

def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Change the red value at

pixel p to 50% of valuepicture




…

value = 160p

And eventually, we do all pixels

We go from this… to this!

Increasing Red

def increaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*1.2)

What happened here?!?

Remember that the limit for redness is 255.

If you go beyond 255, all kinds of weird things can happen: Wrap around

Clearing Blue

def clearBlue(picture): for p in getPixels(picture): setBlue(p,0)

Combining into a sunset function

How do we turn this beach scene into a sunset?

What happens at sunset? At first, I tried increasing the

red, but that made things like red specks in the sand REALLY prominent.

That can’t be how it really works

New Theory: As the sun sets, less blue and green is visible, which makes things look more red.

A Sunset-generation Functiondef makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7)

Creating a negative

Let’s think it through R,G,B go from 0 to 255 Let’s say Red is 10. That’s very light red.

What’s the opposite? LOTS of Red! The negative of that would be 245: 255-10

So, for each pixel, if we negate each color component in creating a new color, we negate the whole picture.

Recipe for creating a negative

def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor( 255-red, 255-green, 255-blue) setColor(px,negColor)

Converting to greyscale

We know that if red=green=blue, we get grey But what value do we set all three to?

What we need is a value representing the darkness of the color, the luminance There are lots of ways of getting it, but one way that works reasonably well is

dirt simple—simply take the average:

Converting to greyscaledef greyScale(picture): for p in getPixels(picture): intensity = (getRed(p)+getGreen(p)+getBlue(p))/3 setColor(p,makeColor(intensity,intensity,intensity))

But that’s not really the best greyscale

In reality, we don’t perceive red, green, and blue as equal in their amount of luminance: How bright (or non-bright) something is. We tend to see blue as “darker” and red as “brighter” Even if, physically, the same amount of light is coming

off of each

Photoshop’s greyscale is very nice: Very similar to the way that our eye sees it B&W TV’s are also pretty good

Building a better greyscale

We’ll weight red, green, and blue based on how light we perceive them to be, based on laboratory experiments.

def greyScaleNew(picture): for px in getPixels(picture): newRed = getRed(px) * 0.299 newGreen = getGreen(px) * 0.587 newBlue = getBlue(px) * 0.114 luminance = newRed+newGreen+newBlue setColor(px,makeColor(luminance,luminance,luminance))

Comparing the two greyscales:Average on left, weighted on right

Let’s try making Barbara a redhead!

We could just try increasing the redness, but as we’ve seen, that has problems. Overriding some red spots And that’s more than just her hair

If only we could increase the redness only of the brown areas of Barb’s head…

Making Barb a redhead

def turnRed(): brown = makeColor(57,16,8) file = r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbara.jpg" picture=makePicture(file) for px in getPixels(picture): color = getColor(px) if distance(color,brown)<50.0:

redness=getRed(px)*1.5 setRed(px,redness) show(picture) return(picture)

Original:

Tuning our color replacement

If you want to get more of Barb’s hair, just increasing the threshold doesn’t work Wood behind becomes within the threshold value

How could we do it better? Lower our threshold, but then miss some of the hair Work only within a range…

Introducing the function range

Range returns a sequence between its first two inputs, possibly using a third input as the increment

>>> print range(1,4)[1, 2, 3]>>> print range(-1,3)[-1, 0, 1, 2]>>> print range(1,10,2)[1, 3, 5, 7, 9]

Replacing colorsin a range

def turnRedInRange(): brown = makeColor(57,16,8) file=r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbara.jpg" picture=makePicture(file) for x in range(70,168): for y in range(56,190): px=getPixel(picture,x,y) color = getColor(px) if distance(color,brown)<50.0: redness=getRed(px)*1.5 setRed(px,redness) show(picture) return(picture)

Get the range using MediaTools

Mirroring

Imagine a mirror horizontally across the picture,or vertically

What would we see? How do generate that digitally?

We simply copy the colors of pixels from one place to another

Mirroring a picture

Slicing a picture down the middle and sticking a mirror on the slice Do it by using a loop to measure a difference

The index variable is actually measuring distance from the mirrorpoint

Then reference to either side of the mirror point using the difference

Recipe for mirroring

def mirrorVertical(source): mirrorpoint = int(getWidth(source)/2) for y in range(1,getHeight(source)): for x in range(1,mirrorpoint): p = getPixel(source, x+mirrorpoint,y) p2 = getPixel(source, mirrorpoint-x,y) c = getColor(p2) setColor(p,c)

Can we do it with a horizontal mirror?

def mirrorHorizontal(source): mirrorpoint = int(getHeight(source)/2) for y in range(1,mirrorpoint): for x in range(1,getWidth(source)): p = getPixel(source,x,y+mirrorpoint) p2 = getPixel(source,x,mirrorpoint-y) setColor(p,getColor(p2))

Of course!

What if we wanted to copy bottom to top? Very simple: Swap the p and p2 in the bottom line

Copy from p to p2, instead of from p2 to p

def mirrorHorizontal(source): mirrorpoint = int(getHeight(source)/2) for y in range(1,mirrorpoint): for x in range(1,getWidth(source)): p = getPixel(source,x,y+mirrorpoint) p2 = getPixel(source,x,mirrorpoint-y) setColor(p2,getColor(p))

Messing with Santa some more

Copying pixels

In general, what we want to do is to keep track of a sourceX and sourceY, and a targetX and targetY. We increment (add to them) in pairs

sourceX and targetX get incremented together sourceY and targetY get incremented together

The tricky parts are: Setting values inside the body of loops Incrementing at the bottom of loops

Copying Barb to a canvasdef copyBarb(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 1 for sourceX in range(1,getWidth(barb)): targetY = 1 for sourceY in range(1,getHeight(barb)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

Transformation = Small changes in copying Making relatively small changes in this basic

copying program can make a variety of transformations. Change the targetX and targetY, and you copy wherever

you want Cropping: Change the sourceX and sourceY range, and

you copy only part of the program. Rotating: Swap targetX and targetY, and you end up

copying sideways Scaling: Change the increment on sourceX and sourceY,

and you either grow or shrink the image.

Copying into the middle of the canvas

def copyBarbMidway(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 100 for sourceX in range(1,getWidth(barb)): targetY = 100 for sourceY in range(1,getHeight(barb)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

Rotating the copydef copyBarbSideways(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 1 for sourceX in range(1,getWidth(barb)): targetY = 1 for sourceY in range(1,getHeight(barb)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetY,targetX), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

Cropping: Just the facedef copyBarbsFace(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 100 for sourceX in range(45,200): targetY = 100 for sourceY in range(25,200): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

Scaling the picture downdef copyBarbsFaceSmaller(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)/2)): sourceY = 25 for targetY in range(100,100+((200-25)/2)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 show(barb) show(canvas) return canvas

Scaling the picture updef copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas

Background subtraction

Let’s say that you have a picture of someone, and a picture of the same place (same background) without the someone there, could you subtract out the background and leave the picture of the person?

Maybe even change the background? Let’s take that as our problem!

Person (Katie) and Background

Background Subtraction Codedef swapbg(person, bg, newbg): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < 10: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)

Putting Katie in a Jungle

But why isn’t it alot better?

We’ve got places where we got pixels swapped that we didn’t want to swap See Katie’s shirt stripes

We’ve got places where we want pixels swapped, but didn’t get them swapped See where Katie made a

shadow

Another way: Chromakey

Have a background of a known color Some color that won’t be on

the person you want to mask out

Pure green or pure blue is most often used

I used my son’s blue bedsheet

This is how the weather people seem to be in front of a map—they’re actually in front of a blue sheet.

Chromakey recipedef chromakey(source,bg): # source should have something in front of blue, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

Can also do this with getPixels()def chromakey2(source,bg): # source should have something in front of blue, bg is the new background for p in getPixels(source): # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))

Example results

Just trying the obvious thing for Red

def chromakey2(source,bg): # source should have something in front of red, bg is the new background for p in getPixels(source): if getRed(p) > (getGreen(p) + getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))

Doesn’t always work as you expect

Let’s try that with green

def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness + blueness if getGreen(p) > getBlue(p) + getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

The same definition of green doesn’t work

Tweaking Chromakey

def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness and blueness if getGreen(p) > getBlue(p) and getGreen(p) > getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

That looks better

New functions for drawing on pictures

addText(pict,x,y,string) puts the string starting at position (x,y) in the picture

addLine(picture,x1,y1,x2,y2) draws a line from position (x1,y1) to (x2,y2)

addRect(pict,x1,y1,w,h) draws a black rectangle (unfilled) with the upper left hand corner of (x1,y1) and a width of w and height of h

addRectFilled(pict,x1,y1,w,h,color) draws a rectangle filled with the color of your choice with the upper left hand corner of (x1,y1) and a width of w and height of h

Making it easier to deal with the media folder

>>> setMediaFolder()New media folder: /Users/guzdial/mediasources/>>> print getMediaPath("barbara.jpg")/Users/guzdial/mediasources/barbara.jpg>>> print getMediaPath("sec1silence.wav")/Users/guzdial/mediasources/sec1silence.wav

Example picturedef littlepicture(): canvas=makePicture(getMediaPath("640x480.jpg")) addText(canvas,10,50,"This is not a picture") addLine(canvas,10,20,300,50) addRectFilled(canvas,0,200,300,500,yellow) addRect(canvas,10,210,290,490) return canvas

CS1315:Introduction to Media Computation

Sound Encoding and Manipulation

How sound works:Acoustics, the physics of sound

Sounds are waves of air pressure Sound comes in cycles The frequency of a wave is

the number of cycles per second (cps), or Hertz

(Complex sounds have more than one frequency in them.)

The amplitude is the maximum height of the wave

Volume and pitch:Psychoacoustics, the psychology of sound

Our perception of volume is related (logarithmically) to changes in amplitude If the amplitude doubles, it’s about a 3 decibel (dB) change

Our perception of pitch is related (logarithmically) to changes in frequency Higher frequencies are perceived as higher pitches We can hear between 5 Hz and 20,000 Hz (20 kHz) A above middle C is 440 Hz

“Logarithmically?”

It’s strange, but our hearing works on ratios not differences, e.g., for pitch. We hear the difference between 200 Hz and 400 Hz, as

the same as 500 Hz and 1000 Hz Similarly, 200 Hz to 600 Hz, and 1000 Hz to 3000 Hz

Intensity (volume) is measured as watts per meter squared A change from 0.1W/m2 to 0.01 W/m2, sounds the same

to us as 0.001W/m2 to 0.0001W/m2

Decibel is a logarithmic measure

A decibel is a ratio between two intensities: 10 * log10(I1/I2) As an absolute measure, it’s in comparison to threshold

of audibility 0 dB can’t be heard. Normal speech is 60 dB. A shout is about 80 dB

Digitizing Sound: How do we get that into numbers? Remember in calculus,

estimating the curve by creating rectangles?

We can do the same to estimate the sound curve Analog-to-digital conversion

(ADC) will give us the amplitude at an instant as a number: a sample

How many samples do we need?

Nyquist Theorem

We need twice as many samples as the maximum frequency in order to represent (and recreate, later) the original sound.

The number of samples recorded per second is the sampling rate If we capture 8000 samples per second, the highest frequency

we can capture is 4000 Hz That’s how phones work

If we capture more than 44,000 samples per second, we capture everything that we can hear (max 22,000 Hz)

CD quality is 44,100 samples per second

Digitizing sound in the computer

Each sample is stored as a number (two bytes) What’s the range of available combinations?

16 bits, 216 = 65,536 But we want both positive and negative values

To indicate compressions and rarefactions. What if we use one bit to indicate positive (0) or negative (1)? That leaves us with 15 bits 15 bits, 215 = 32,768 One of those combinations will stand for zero

We’ll use a “positive” one, so that’s one less pattern for positives

Each sample can be between -32,768 and 32,767

Working with sounds

We’ll use pickAFile and makeSound as we have before. But now we want .wav files

We’ll use getSamples to get all the sample objects out of a sound

We can also get the value at any index with getSampleValueAt

Sounds also know their length (getLength) and their sampling rate (getSamplingRate)

Can save sounds with writeSoundTo(sound,”file.wav”)

Demonstrating Working with Sound in JES

>>> filename=pickAFile()>>> print filename/Users/guzdial/mediasources/preamble.wav>>> sound=makeSound(filename)>>> print soundSound of length 421109>>> samples=getSamples(sound)>>> print samplesSamples, length 421109>>> print getSampleValueAt(sound,1)36>>> print getSampleValueAt(sound,2)29

Demonstrating working with samples>>> print getLength(sound)220568>>> print getSamplingRate(sound)22050.0>>> print getSampleValueAt(sound,220568)68>>> print getSampleValueAt(sound,220570)I wasn't able to do what you wanted.The error java.lang.ArrayIndexOutOfBoundsException has occuredPlease check line 0 of >>> print getSampleValueAt(sound,1)36>>> setSampleValueAt(sound,1,12)>>> print getSampleValueAt(sound,1)12

Working with Samples

We can get sample objects out of a sound with getSamples(sound) or getSampleObjectAt(sound,index)

A sample object remembers its sound, so if you change the sample object, the sound gets changed.

Sample objects understand getSample(sample) and setSample(sample,value)

Example: Manipulating Samples>>> soundfile=pickAFile()>>> sound=makeSound(soundfile)>>> sample=getSampleObjectAt(sound,1)>>> print sampleSample at 1 value at 59>>> print soundSound of length 387573>>> print getSound(sample)Sound of length 387573>>> print getSample(sample)59>>> setSample(sample,29)>>> print getSample(sample)29

Recipe to Increase the Volume

def increaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 2)

Using it:>>> f="/Users/guzdial/mediasources/gettysburg10.wav">>> s=makeSound(f)>>> increaseVolume(s)>>> play(s)>>> writeSoundTo(s,"/Users/guzdial/mediasources/louder-g10.wav")

Decreasing the volume

def decreaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 0.5)

This works just like increaseVolume, but we’re lowering each sample by 50% instead of doubling it.

Maximizing volume

How do we get maximal volume? It’s a three-step process:

First, figure out the loudest sound (largest sample). Next, figure out a multiplier needed to make that sound

fill the available space. We want to solve for x where x * loudest = 32767 So, x = 32767/loudest

Finally, multiply the multiplier times every sample

Maxing (normalizing) the sound

def normalize(sound): largest = 0 for s in getSamples(sound): largest = max(largest,getSample(s) ) multiplier = 32767.0 / largest

print "Largest sample value in original sound was", largest print "Multiplier is", multiplier

for s in getSamples(sound): louder = multiplier * getSample(s) setSample(s,louder)

Could also do this with IF

def normalize(sound): largest = 0 for s in getSamples(sound): if getSample(s) > largest: largest = getSample(s) multiplier = 32767.0 / largest print "Largest sample value in original sound was", largest print "Multiplier is", multiplier for s in getSamples(sound): louder = multiplier * getSample(s) setSample(s,louder)

Increasing volume by sample indexdef increaseVolumeByRange(sound): for sampleIndex in range(1,getLength(sound)+1): value = getSampleValueAt(sound,sampleIndex) setSampleValueAt(sound,sampleIndex,value * 2)

This really is the same as:def increaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 2)

Recipe to play a sound backwards

def backwards(filename): source = makeSound(filename) target = makeSound(filename)

sourceIndex = getLength(source) for targetIndex in range(1,getLength(target)+1): sourceValue = getSampleValueAt(source,sourceIndex) setSampleValueAt(target,targetIndex,sourceValue) sourceIndex = sourceIndex - 1

return target Note use of return for returning the processed sound

Recipe for halving the frequency of a sound

def half(filename): source = makeSound(filename) target = makeSound(filename)

sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex,

getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + 0.5

play(target) return target

This is how a sampling synthesizer works!

Here are the pieces that do it

Compare these twodef half(filename): source = makeSound(filename) target = makeSound(filename)


getSampleValueAt( source, int(sourceIndex)))

sourceIndex = sourceIndex + 0.5


def copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor(

getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas

Both of them are sampling

Both of them have three parts: A start where objects are set up A loop where samples or pixels are copied from one

place to another To decrease the frequency or the size, we take each

sample/pixel twice In both cases, we do that by incrementing the index by

0.5 and taking the integer of the index Finishing up and returning the result

Recipe to double the frequency of a sounddef double(filename): source = makeSound(filename) target = makeSound(filename) targetIndex = 1 for sourceIndex in range(1, getLength(source)+1, 2): setSampleValueAt( target, targetIndex,

getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 #Clear out the rest of the target sound -- it's only half full! for secondHalf in range( getLength( target)/2, getLength( target)): setSampleValueAt(target,targetIndex,0) targetIndex = targetIndex + 1 play(target) return target

Here’s the critical piece: We skip every other sample in the source!

What happens if we don’t “clear out” the end? Try it!

def double(filename): source = makeSound(filename) target = makeSound(filename) targetIndex = 1 for sourceIndex in range(1, getLength(source)+1, 2): setSampleValueAt( target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 play(target) return target

Splicing Sounds (page 81-85)

Splicing gets its name from literally cutting and pasting pieces of magnetic tape together

Doing it digitally is easy, but not short We find where the end points of words are We copy the samples into the right places to make the

words come out as we want them (We can also change the volume of the words as we

move them, to increase or decrease emphasis and make it sound more natural.)

Finding the word end-points

Using MediaTools and play before/after cursor, can figure out the index numbers where each word ends

Now, it’s all about copying

We have to keep track of the source and target indices.

targetIndex = Where-the-incoming-sound-should-startfor sourceIndex in range(startingPoint,endingPoint) setSampleValueAt( target, targetIndex,

getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1

def splicePreamble(): file = "/Users/guzdial/mediasources/preamble10.wav" source = makeSound(file) target = makeSound(file) # This will be the newly spliced sound targetIndex=17408 # targetIndex starts at just after "We the" in the new sound for sourceIndex in range( 33414, 40052): # Where the word "United" is in the sound setSampleValueAt(target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 for sourceIndex in range(17408, 26726): # Where the word "People" is in the sound setSampleValueAt(target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 for index in range(1,1000): #Stick some quiet space after that setSampleValueAt(target, targetIndex,0) targetIndex = targetIndex + 1 play(target) #Let's hear and return the result return target

The Whole Splice

What’s going on here?

First, set up a source and target. Next, we copy “United” (samples 33414 to

40052) after “We the” (sample 17408) That means that we end up at 17408+(40052-

33414) = 17408+6638=24046

Where does “People” start?

Next, we copy “People” (17408 to 26726) immediately afterward. Do we have to copy “of” to? Or is there a pause in there that we can make

use of?

Finally, we insert a little (1/441-th of a second) of space – 0’s

What if we didn’t do that second copy? Or the pause?

def spliceSimpler(): file = "C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\preamble10.wav" source = makeSound(file) target = makeSound(file) # This will be the newly spliced sound targetIndex=17408 # targetIndex starts at just after "We the" in the new sound for sourceIndex in range( 33414, 40052): # Where the word "United" is in the sound setSampleValueAt( target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 play(target) #Let's hear and return the result return target

Can we generalize shifting a sound into other frequencies?This way does NOT work:def shift(filename,factor): source = makeSound(filename) target = makeSound(filename)

sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + factor


Watching it not work

It’ll work for shifting down, but not shifting up. Why?>>> hello=pickAFile()>>> print hello/Users/guzdial/mediasources/hello.wav>>> lowerhello=shift(hello,0.75)>>> higherhello=shift(hello,1.5)I wasn't able to do what you wanted.The error java.lang.ArrayIndexOutOfBoundsException has occuredPlease check line 7 of /Users/guzdial/shift-broken.py

We need to prevent going past the end of the sound

def shift(filename,factor): source = makeSound(filename) target = makeSound(filename)


getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + factor if sourceIndex > getLength(source): sourceIndex = 1


Now we have the basics of a sampling synthesizer

For a desired frequency f we want a sampling interval like this:

Useful exercise: Build a shift function that takes a frequency as input.

Modules: Moving towards movies and animations The OS module offers a number of powerful

capabilities for dealing with files, e.g., renaming files, finding out when a file was last modified, and so on.

We start accessing the OS module by typing: import os

The function that knows about directories is listdir(), used as os.listdir() listdir takes a path to a directory as input.

Using os.listdir

>>> import os>>> print getMediaPath("barbara.jpg")C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\

barbara.jpg>>> print getMediaPath("pics")Note: There is no file at C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\picsC:\Documents and Settings\Mark Guzdial\My Documents\mediasources\

pics>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My

Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg',

'students7.jpg', 'students8.jpg']

Writing a program to title pictures

We’ll input a directory We’ll use os.listdir() to get each filename in the

directory We’ll open the file as a picture. We’ll title it. We’ll save it out as “titled-” and the filename.

A Working Titling Program

import os

def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)

Showing it work>>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students1.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students2.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students5.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students6.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students7.jpgProcessing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students8.jpg>>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics")['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg', 'titled-

students1.jpg', 'titled-students2.jpg', 'titled-students5.jpg', 'titled-students6.jpg', 'titled-students7.jpg', 'titled-students8.jpg']

Comparing before and after

What if you want to make sure you’ve got JPEG files?import os

def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file if file.endswith(".jpg"): picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)

Video Processing

Burst a movie into a series of JPEG frames MediaTools or QuickTime Pro will do this for you.

Process each frame using os.listdir

Lightening every frame of a movie

def lightenMovie(folder): import os for file in os.listdir(folder): picture=makePicture(folder+file) for px in getPixels(picture): color=getColor(px) makeLighter(color) setColor(px,color) writePictureTo(picture,folder+“L"+file)

Creating animations

Simply draw on JPEG frames And save each out as a consecutively numbered

picture

Simplest Animationdef AnimationSimple():

frames = 50rx = 0ry = 0rw = 50rh = 50for f in range(1,frames+1):

pic=makePicture(getMediaPath("640x480.jpg")) addRectFilled(pic,rx,ry,rw,rh,red)if(f < 10):

writePictureTo(pic,'test0%d.jpg'%(f))else:

writePictureTo(pic,'test0%d.jpg'%(f))rx = 5 + rxry = 5 + ry