5
Candy Carousel: Hear Your Food Sandra Y. Helsley, Derek Kan, Jenton Lee School of Information, University of California, Berkeley {syh, derek, jenton}@ischool.berkeley.edu ABSTRACT In this paper, we explore a tangible user interface to combine physical interaction and music generation, integrating com- monly found objects and audible pattern sequences. Using software written in Max/MSP/Jitter, MIDI sounds are gener- ated as a circular disc is rotated underneath a camera. This novel collaborative tabletop music sequencer allows for syn- chronous and asynchronous co-located and distributed inter- action. Author Keywords interaction, music, tangible interfaces ACM Classification Keywords H.5.m. Information Interfaces and Presentation (e.g. HCI): Miscellaneous General Terms Human Factors; Design; Measurement. INTRODUCTION Composing electronic music often involves heavy utilization of a computer, keyboard, and mouse. But when writing mu- sic with an instrument in hand, the musician has a very differ- ent physical experience. While electronic music composition affords a variety of advanced features, there is currently no adequate replacement of this tangible interaction. We sought to develop an interface which brings together the physical components of instrument manipulation and the flex- ibility of computerized music generation. Physical compo- nents allow hands-on interaction, while also providing the user with immediate visual feedback. One benefit of a physical system is that every day objects can be used as input. We were especially interested in incorporat- ing food items such as candy, giving a literal meaning to the phrase, “play with your food.” Having simple input artifacts might also encourage novice musicians to interact with the system. (not) Submitted to CHI 2013. Do not cite, do not circulate. To accomplish this, we designed the Candy Carousel, a phys- ical music sequencer system. We also developed a proto- type which we demonstrated during a project showcase for our class. This allowed us to informally test the system with users, and obtain feedback on interaction. THE CANDY CAROUSEL The Candy Carousel takes colourful candies as input, and uses software to estimate their positions on a flat surface. MIDI sounds are played according to these positions. Mul- tiple participants can generate music collaboratively, whether the composers are co-located or distributed. The system contains three main parts: a physical disc, a note reader, and a track projector. The disc sits on a small lazy susan, can be manually rotated, and has a surface upon which note markers can be placed. Concentric rings on the disc as- sist with precise sound creation, since the note played is de- pendent on the position of the candy relative to the centre of the circle. The reader uses Max/MSP/Jitter [6] to take input from a cam- era, and outputs sound when note markers pass through a pre- determined line. Because the system is manually controlled, the speed at which the disc is rotated determines the tempo; this manual control also allows for some record-scratching- like behaviour. If desired, it is possible to add a motor system for automatic rotation. Using a projector, users can display previously saved tracks onto the surface. This allows for multiple layers of input, while preserving specific note and timing details of a user’s composition. By saving created tracks and sharing them with others, collaboration can be asynchronous, as well as dis- tributed. Usage of the Candy Carousel can vary, though we expect that ideal use is in semi-structured jam sessions, to accompany other musical instruments. Because of its ease of use, it is also possible to incorporate the Candy Carousel in elementary schools and music classes. Input artifacts can be any discrete item or shape, so long as some white space is left around each object, to assist with item detection and size estimation. Currently the Candy Carousel system is set up to determine the positions of small, circular gumdrops, but is capable of reading colour from other sources, such as felt marker drawings or images. With some scaling, it is even possible to imagine interpreting and “hear- ing” larger circular shapes, such as dim sum dishes as they rotate around a table. 1

Candy Carousel: Hear Your Foodpeople.ischool.berkeley.edu/~derek/works/candyCarousel/candy-carousel.pdf · Candy Carousel: Hear Your Food Sandra Y. Helsley, Derek Kan, Jenton Lee

  • Upload
    others

  • View
    10

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Candy Carousel: Hear Your Foodpeople.ischool.berkeley.edu/~derek/works/candyCarousel/candy-carousel.pdf · Candy Carousel: Hear Your Food Sandra Y. Helsley, Derek Kan, Jenton Lee

Candy Carousel: Hear Your Food

Sandra Y. Helsley, Derek Kan, Jenton LeeSchool of Information, University of California, Berkeley

{syh, derek, jenton}@ischool.berkeley.edu

ABSTRACTIn this paper, we explore a tangible user interface to combinephysical interaction and music generation, integrating com-monly found objects and audible pattern sequences. Usingsoftware written in Max/MSP/Jitter, MIDI sounds are gener-ated as a circular disc is rotated underneath a camera. Thisnovel collaborative tabletop music sequencer allows for syn-chronous and asynchronous co-located and distributed inter-action.

Author Keywordsinteraction, music, tangible interfaces

ACM Classification KeywordsH.5.m. Information Interfaces and Presentation (e.g. HCI):Miscellaneous

General TermsHuman Factors; Design; Measurement.

INTRODUCTIONComposing electronic music often involves heavy utilizationof a computer, keyboard, and mouse. But when writing mu-sic with an instrument in hand, the musician has a very differ-ent physical experience. While electronic music compositionaffords a variety of advanced features, there is currently noadequate replacement of this tangible interaction.

We sought to develop an interface which brings together thephysical components of instrument manipulation and the flex-ibility of computerized music generation. Physical compo-nents allow hands-on interaction, while also providing theuser with immediate visual feedback.

One benefit of a physical system is that every day objects canbe used as input. We were especially interested in incorporat-ing food items such as candy, giving a literal meaning to thephrase, “play with your food.” Having simple input artifactsmight also encourage novice musicians to interact with thesystem.

(not) Submitted to CHI 2013.Do not cite, do not circulate.

To accomplish this, we designed the Candy Carousel, a phys-ical music sequencer system. We also developed a proto-type which we demonstrated during a project showcase forour class. This allowed us to informally test the system withusers, and obtain feedback on interaction.

THE CANDY CAROUSELThe Candy Carousel takes colourful candies as input, anduses software to estimate their positions on a flat surface.MIDI sounds are played according to these positions. Mul-tiple participants can generate music collaboratively, whetherthe composers are co-located or distributed.

The system contains three main parts: a physical disc, a notereader, and a track projector. The disc sits on a small lazysusan, can be manually rotated, and has a surface upon whichnote markers can be placed. Concentric rings on the disc as-sist with precise sound creation, since the note played is de-pendent on the position of the candy relative to the centre ofthe circle.

The reader uses Max/MSP/Jitter [6] to take input from a cam-era, and outputs sound when note markers pass through a pre-determined line. Because the system is manually controlled,the speed at which the disc is rotated determines the tempo;this manual control also allows for some record-scratching-like behaviour. If desired, it is possible to add a motor systemfor automatic rotation.

Using a projector, users can display previously saved tracksonto the surface. This allows for multiple layers of input,while preserving specific note and timing details of a user’scomposition. By saving created tracks and sharing them withothers, collaboration can be asynchronous, as well as dis-tributed.

Usage of the Candy Carousel can vary, though we expect thatideal use is in semi-structured jam sessions, to accompanyother musical instruments. Because of its ease of use, it isalso possible to incorporate the Candy Carousel in elementaryschools and music classes.

Input artifacts can be any discrete item or shape, so long assome white space is left around each object, to assist withitem detection and size estimation. Currently the CandyCarousel system is set up to determine the positions of small,circular gumdrops, but is capable of reading colour from othersources, such as felt marker drawings or images. With somescaling, it is even possible to imagine interpreting and “hear-ing” larger circular shapes, such as dim sum dishes as theyrotate around a table.

1

Page 2: Candy Carousel: Hear Your Foodpeople.ischool.berkeley.edu/~derek/works/candyCarousel/candy-carousel.pdf · Candy Carousel: Hear Your Food Sandra Y. Helsley, Derek Kan, Jenton Lee

RELATED WORKMany existing physical systems already allow users to takeevery day objects to create sound. In this section, we dis-cuss the projects that inspired the Candy Carousel. We wereprimarily interested in physical tabletop interfaces and musicsequencers.

Tabletop InterfacesThe reacTable [4], [5], by Kaltenbrunner, Jorda, et al., is anelectronic music generator, with a tangible tabletop surface.Users can move physical artifacts placed on the surface to cre-ate different sounds, while visual feedback indicates audiblerelationships between individual pieces.

Although the interactive pieces were specifically built for thereacTable, and are not ordinary objects, the open tabletop al-lows for multi-user play. Throughout the development of theCandy Carousel, we felt that integration of group interactionwas an important concept to preserve.

Andersen’s Mixxx [1], a software disc jockey (DJ) system,uses a turntable metaphor to allow for augmented contem-porary music mixing. Its design was intended to help betterunderstand the working practices of professional DJs, but itsphysical interface is nonetheless relevant to our project.

Many professional DJs continue to use physical turntables intheir work, despite the many digital options available. Thereare a number of possible reasons to explain this preference,but most importantly is the interaction options that the physi-cal interface affords.

Music SequencersThe Bubblegum sequencer [3] by Hesse and McDiarmid wasa music sequencer that used coloured gum balls placed on afour by sixteen grid create an audible pattern. Using a cameraplaced below the grid, position and colour are mapped andrecorded for audible play. Each gum ball could be mapped toa different set of sound samples.

Providing this regular visual gives the user a well defined, or-ganized space, allowing the user to ”see” the music; however,the user is limited to four tracks, and 16 notes per track.

Software based sequencers, such as the OscilloScoop [9], canprovide more feature flexibility. OscilloScoop is a music se-quencer application which allows users to interact with threecylinders on a tablet display. Each cylinder controls a differ-ent aspect of the sound, and changes are visible in the patternsof the cylinders’ walls.

Although the cylindrical implementation is an excellent indi-cator of the repetitions in a sequencer, the OscilloScoop is atablet-only application, which lacks any sort of physical af-fordance.

The Spinner Synth Prototype [7] was a vertical circular se-quencer which estimates the position of blue tape on a motor-ized wheel and plays notes once the tape passes a software-defined line. Like the Bubblegum sequencer, a camera is usedto detect colour on the surface and estimate its position.

Concentric circles on the surface show the boundaries be-tween playable notes. A motor rotates the Spinner Synth au-tomatically, providing a stable rhythm; however, the wheelcannot be manipulated manually, as a DJ’s turntable can.

DEMO PROTOTYPEWe created a demonstration prototype of the Candy Carouselfor a course project showcase. This version implements thephysical rotating disc with candy inputs, and the interpreta-tive software to generate sound. The setup of our prototype isvisible in Fig. 1, with the webcam hanging from a tripod overthe disc, and the software setup.

While some monitoring of the software was required, in orderto compensate for changes in lighting conditions, the systemwas mostly autonomous. For most of the demonstration, wemade the screen visible to participants, so they could, as oneperson said, ”see the magic.”

A video of our demo prototype is available at this location:http://youtu.be/ERk1rPoykqk.

The poster which accompanied our prototype is available atthis location: http://bit.ly/10Hb7jQ.

Figure 1. Setup of the Candy Carousel. A webcam hung over a rotatingplastic disc and was connected to a computer running Max/MSP/Jittersoftware.

Physical PrototypeThe Candy Carousel incorporates many of the features of itspredecessors. For the physical disc (Fig. 2), space around thedisc allows multiple hands to interact with the system. Byplacing gumdrops or other candies on the surface, users canenjoy the physical interaction that the system affords, withimmediate visual feedback.

The scalloped edges of the disc itself make it easy to grab andmove the surface both backwards and forwards, allowing freerotation and “scratching” behaviours.

Like the Spinner Synth, concentric circles on the surface areuseful for showing the repetitive nature of the sequencer. Thecircular lines also help with alignment of the candies, so thatprecise notes are possible, if desired.

2

Page 3: Candy Carousel: Hear Your Foodpeople.ischool.berkeley.edu/~derek/works/candyCarousel/candy-carousel.pdf · Candy Carousel: Hear Your Food Sandra Y. Helsley, Derek Kan, Jenton Lee

In addition to the circles, however, the surface is also dividedinto quadrants, separating the sequencer into four measures.However, all lines on the sequencer are not restrictive, and theonly limitation on placement and number of candy inputs is inthe processing power of the computer running the software.

A circular ”dead zone” in the centre does not emit sound, al-lowing some candies to be placed there for easy retrieval. Bycancelling out the centre, we also sought to make the outsidecircles as equal as possible in length. This also lent to themusical staff metaphor.

Max/MSP/JitterUsing a webcam hung from a tripod and connected to a com-puter, we were able to take input and translate it to MIDIsounds. Our software program was based off of two primarysources: code written in Max/MSP/Jitter and a third-partycomputer vision library called cv.jit [8]. Combining these al-lowed us to create visual software to estimate positions ofcandies as coloured centroids on a simulated circular surface(Fig. 3).

The circular rings were simulated in the software, incorpo-rating the physical dead zone. Careful placement of the discunder the camera was required to match up the circular shapewith the rotating surface. In our demonstration prototype, sizeof centroids is estimated, but is not used as an input, thoughthat is a potential consideration for future versions.

As the gum drops moved in a circle on the disc, our soft-ware allowed us to simultaneously change their positionson the circle. The appropriate MIDI note would be playedthrough speakers when an estimated centroid passed overa pre-determined rectangular area of pixels in the software,which we physically marked with a piece of tape. The tapedline was not read by the software, but instead was intended tobe a reference guide for users.

We adapted colour tracking code [2] to follow three selectedcolours, giving us three separate sampling tracks. Colour se-lection had to be specific for the lighting conditions at thetime. For our demos, we set one colour to a percussion track(cymbals and drums), and the other two to play piano sounds.

We originally set the two piano tracks to a C-major scale andG-major chord (G, B, and D notes), basing our setup in clas-sical Western music theory. However, the C-major scale cre-ated discordant sounds due to the sequential notes, and waseventually changed to a D-major (D, F#, and A notes) chord.

The selection of these chords was not random; historicallyin music, G-major and D-major keys are considered to lendpositive and joyous moods [10]. The distances between theresulting notes were mostly 3rds, perfect 4ths, and perfect5ths, giving us two harmonious sets of notes.

By selecting sets of MIDI note values, we were able to changethe keys if necessary, allowing us to change the atmosphere ofthe music generated. Additionally, a small sustain was addedto the piano tracks, to improve the simulated musical quality.

Figure 2. Surface of the Candy Carousel. Gum drops have been ar-ranged to create a pattern of colours.

Figure 3. A screenshot of part of the Max/MSP/Jitter program. In thisselection, the software is set to detect purple dots, and estimates theirpositions (below). MIDI notes are determined by the numbers on theright.

ProjectorFor our demo, we were unable to set up the projector forremote collaboration. In theory, a projector would be posi-tioned above or beside the Candy Carousel system, to projectlayers of tracks either saved from previous interactions withthe Candy Carousel, or from files sent by other collaborators(Fig. 4).

Users would be able to save notes placed on the circle, andmake recordings with the desired speed and direction of rota-tions. This data could be shared, or projected locally to createmultiple layers, thus avoiding issues with gum drop collision.

FEEDBACKOverall, initial feedback from two demo sessions were ex-tremely promising. After only a few words of explanation,users did not have any difficulty using the sequencer, and

3

Page 4: Candy Carousel: Hear Your Foodpeople.ischool.berkeley.edu/~derek/works/candyCarousel/candy-carousel.pdf · Candy Carousel: Hear Your Food Sandra Y. Helsley, Derek Kan, Jenton Lee

many participants were able to make harmonious soundingmusic within seconds, especially once the pre-set chords wereintroduced.

One musician liked the sounds he made using the CandyCarousel, recording his creation for future sampling. Anotheruser, upon learning about the G-major and D-major chordsettings, attempted to figure out which lines corresponded tospecific notes. Although we opted for the flexibility of havingno clef markings on the lines, he was able to discern octavesand individual notes based on knowledge of the chords pro-vided.

We expected that the system could be set up for use by mu-sicians with some knowledge of music theory, but the demosessions proved the interface to be simple enough for novices.The use of candy as an input was appealing to users of allages.

Participants were especially drawn to using the candies tomake coloured patterns and shapes on the surface, insteadof sequential lines as we expected. Regular usage includedgrouping of candies by colour or spacing them out in evenpatterns.

The sensitivity of the software encouraged users to creategroupings of candies, testing out the system’s ability to han-dle a jumble of colours. One user set a group of candies at theplayer line and proceeded to thump the table, creating minorvibrations that changed the sound. Another user was also ableto use the scallops on the disc to get very fine movements, re-sulting in a similar sound effect.

We had no expectations for participants based on age, but thethree children participating in the demo session were the mostenthusiastic users of the system. They were all capable ofmoving gum drops around simultaneously, turning the disc tosee how their input affected the sound.

One girl used the candies to “write” out letters (Fig. 5), andexclaimed as she rotated the disc, “This is what my namesounds like!”

FUTURE WORKThe demo sessions helped us to see what worked with ourprototype, but also brought to light some changes that wewould like to make in future revisions.

One of the most obvious problems was that of speed: spin-ning the disc too fast caused the candies to fly off of thesurface. One user commented that it would be interesting toplace all the candies in the centre and see how it sounded asthey spread out, but the weight of the gumdrops did not en-courage this behaviour.

Previous versions of the Candy Carousel used icing to anchorthe candies; however this made changing notes difficult. Thedemos showed the appeal of the ease with which one couldliterally sweep away candies and create a new composition.A concave surface or a slightly stickier medium could poten-tially prevent fly-away candies, while preserving the flexibil-ity of the current system.

Figure 4. Projector layers allow for multiple tracks. Each track consistsof physical gum drops. The positions of these candies are rememberedand projected on the surface while physical gum drops are placed for thenext track.

Figure 5. Four participants at a demo session. One young user spelledout her name, Milo, so she could “hear” it.

Because our software depended on the camera to select andultimately track an nearly monochromatic objects, changes inthe light source and intensity greatly affected reading notes onthe surface. This was especially true when we did try usingthe projector for our demo: notes could be played from a pro-jected image, however the variances in the projector displayitself wreaked havoc with the detection software.

We have considered a number of potential user studies thatcould be based on the Candy Carousel. The simplest studywould be to have users of different age groups perform vari-ous tasks with the system, in order to judge its usability, andto find flaws with the interaction design.

4

Page 5: Candy Carousel: Hear Your Foodpeople.ischool.berkeley.edu/~derek/works/candyCarousel/candy-carousel.pdf · Candy Carousel: Hear Your Food Sandra Y. Helsley, Derek Kan, Jenton Lee

Alternatively, we are particularly interested in learning whatis considered to be “musical,” especially in the minds ofyounger audiences. With the Candy Carousel, its ease of use,and the ability to change keys or moods, we could ask themto compose a song and see if the sequences they make matchtheir expectations of “good music.”

CONCLUSIONThe Candy Carousel successfully combines physical interac-tion and computerized music generation. Using our circularsequencer, every day objects can be transformed into musicalchords. A camera sends the video feed to software, which es-timates positions of candies placed on the surface, and createssounds when the candies pass a pre-determined line.

Seeing users turn the table and place candies without need-ing to be prompted demonstrated how a tangible user inter-face can provide much richer interactivity than a flat screen.Children seemed especially able to become engaged, whichsuggests a promising future for supporting young musicians.

We were extremely pleased with the resulting demo project,and the positive feedback from our users. Many users lovedthe idea of being able to use candy in the sequencer, whichmay have lowered barriers to entry and encouraged interac-tion. Luckily, only a few candies were eaten.

ACKNOWLEDGMENTSWe would like to thank Kimiko Ryokai, Laura Devendorf,and Elliot Nahman, for the feedback we have received as weiterated over this project.

It is also worth noting that at one point in the semester, our se-quencer was affectionately named after one of our team mem-bers, from which we devised the acronym J.E.N.T.O.N.: JustEnough Noise To Obtain Novelty. We would like to apolo-gize to Elliot for not keeping this name despite his preferencefor it.

REFERENCES1. Andersen, T. H. Mixxx: Towards novel dj interfaces. In

Proceedings of the 2003 conference on New interfacesfor musical expression, National University of Singapore(2003), 30–35.

2. Color tracking by setting a threshold for each colorchannel (jit.op problem).http://cycling74.com/forums/topic.php?id=38188,2012.

3. Hesse, H., and McDiarmid, A. Bubblegum sequencer.http://backin.de/gumball/, 2008.

4. Jorda, S., Kaltenbrunner, M., Geiger, G., and Alonso, M.The reacTable: a tangible tabletop musical instrumentand collaborative workbench. In ACM SIGGRAPH 2006Sketches, SIGGRAPH ’06, ACM (New York, NY, USA,2006).

5. Kaltenbrunner, M., Jorda, S., Geiger, G., and Alonso, M.The reactable*: A collaborative musical instrument. InProceedings of the 15th IEEE International Workshopson Enabling Technologies: Infrastructure forCollaborative Enterprises, WETICE ’06, IEEEComputer Society (Washington, DC, USA, 2006),406–411.

6. Max/msp/jitter. http://store.cycling74.com/s.nl/sc.2/category.2/.f.

7. Mets, M. Spinner synth prototype. http://www.cibomahto.com/2009/07/spinner-synth-prototype/,2009.

8. Max/msp blob tracking patch.http://nuigroup.com/forums/viewthread/878/,2008.

9. Snibbe, S., and Girling, L. Oscilloscoop.http://www.snibbestudio.com/oscilloscoop/, 2011.

10. Steblin, R. Affective key characteristics. http://www.wmich.edu/mus-theo/courses/keys.html.

5