Embed Size (px)
Citation preview
1
Demonstrations of Expressive Softwear and Ambient Media
Topological Media LabSha Xin Wei1, Yoichiro Serita2, Jill Fantauzza1, Steven Dow2, Giovanni Iachello2, Vincent Fiano2, Joey Berzowska3,
Yvonne Caravia1, Wolfgang Reitberger1, Julien Fistre4
1School of Literature, Communica-tion, and Culture / GVU CenterGeorgia Institute of Technology
[email protected], {gtg760j,gtg937i, gtg711j}@mail.gatech.edu
2College of Computing/GVU CenterGeorgia Institute of Technology
2 {seri, steven, giac, ynniv}@cc.gatech.edu,
3 Faculty of Fine ArtsConcordia University
Montréal, [email protected]
AbstractWe set the context for three demonstrations by describingthe Topological Media Lab’s research agenda We nextdescribe three concrete applications that bundle togethersome of our responsive ambient media and augmentedclothing instruments in illustrative scenarios.
The first set of scenarios involves performers wearing ex-pressive clothing instruments walking through a confer-ence or exhibition hall. They act according to heuristicsdrawn from a phenomenological study of greeting dy-namics, the social dynamics of engagement and disen-gagement in public spaces. We use our study of these dy-namics to guide our design of expressive clothing usingwireless sensors, conductive fabrics and on-the-body cir-cuit logic.
By walking into different spaces prepared with ambientresponsive media, we see how some gestures and instru-ments take on new expressive and social value. Thesescenarios are studies toward next generation TGarden re-sponsive play spaces [Sponge] based on gesturally pa-rameterized media and body-based or fabric-based ex-pressive technologies.
Keywords: softwear, augmented clothing, media choreog-raphy, real-time media, responsive environments, TGar-den, phenomenology of performance.
1. Context
The Topological Media Lab is established to study ges-ture, agency and materiality from both phenomenologicaland computational perspectives. This motivates an inves-
tigation of human embodied experience in solo and socialsituations, and technologies that can be developed for en-livening or playful applications.
The TML is the laboratory arm of a phenomenological re-search project concerning the substrate to heightened andeveryday performative experience. The TML is also af-filiated with a series of experimental performance instal-lations including TGarden and txOom, which have beenmanifested in six generations of productions of responsiveenvironments in 10 cities over the past four years[Sponge, FoAM].
Figure 0. Dancing in TG2001, a prototype TGarden, Rot-terdam, The Netherlands.
Our approach is informed by the observation that con-tinuous physicalistic systems allow improvisatory gestureand deliver robust response to user input. Rather thandisallow or halt on unanticipated user input, our dynami-cal models always provide expressive control and leavethe construction of meaning and aesthetics in the human’shands.
The focus on clothing is part of a general approach towearable computing that pays attention to the naturalizedaffordances and the social conditioning that fabrics, fur-niture and physical architecture already provide to oureveryday interaction. We exploit the fusion of physicalmaterial and computational media and rely on expert craftfrom music, fashion, and industrial design in order to
2
make a new class of personal and collective expressivemedia.
2. TML’s Research Heuristics
Perhaps the most salient notion and leitmotiv for our re-search is continuity. Continuous physics in time and me-dia space provides natural affordances which sustain in-tuitive learning and development of virtuosity in the formof tacit “muscle memory.” Continuous models allow nu-ance which provides different expressive opportunitiesthan those selected from a relatively small, discrete set ofoptions. Continuous models also sustain improvisation.Rather than disallow or halt on unanticipated user input,our dynamical sound models will always work. How-ever, we leave the quality and the musical meaning of thesound to the user. We use semantically shallow machinemodels.
We do "materials science" as opposed to object-centeredindustrial design. Our work is oriented to the design andprototyping not of new devices but of new species ofaugmented physical media and gestural topologies. Wedistribute computational processes into the environmentas an augmented physics rather than information tasks lo-cated in files, applications and “personal devices.”
One way to structure and design augmented physics is toadapt continuous mathematical models. The simplest inthe set theoretic sense are topological structures. Topo-logical media can richly sustain shared experience. Mediadesigned using heuristics drawn from topological struc-tures (continuous or dense structures rather than the spe-cial case of graph structures) and dynamical systems pro-vide alternatives to the scaffolding of shared experience.Our hypothesis is that simulated physics and qualitativeprocesses rigorously designed by topological dynamicalnotions offer a strong alternative to communication theo-ries based on the conduit metaphor [Reddy].
3. Applications and Demonstrations
We are pursuing these ideas in several lines of work: (1)softwear: clothing augmented with conductive fabrics,wireless sensing and image-bearing materials or lights forexpressive purposes; (2) gesture-tracking and mathemati-cal mapping of gesture data to time-based media; (3)physics-based real-time synthesis of video; (4) analogoussound synthesis; (5) media choreography based on statis-tical physics.
We demonstrate new applications that showcase elementsof recent work. Although we describe them as separateelements, the point is that by walking from an unpreparedplace to a space prepared with our responsive media sys-
tems, the same performers in the same instrumentedclothing acquire new social valence. Their interactionswith co-located less-instrumented or non-instrumentedpeople also take on different effects as we vary the locus(the locus of…) of their interaction.
3.1. Softwear: Augmented Clothing
Most of the applications for embedding digital devices inclothing have utilitarian design goals such as managinginformation, or locating or orienting the wearer. Enter-tainment applications are often oriented around control-ling media devices or PDAs, and high level semanticssuch as user identity [Aoki, Eaves] or gesture recognition[Starner].
We study the expressive uses of augmented clothing butat a more basic level of non-verbal body language, as in-dicated in the provisional diagram (Figure 1). The keypoint is that we are not encoding classes of gesture intoour response logic but instead we are using such diagramsas necessarily incomplete heuristics to guide human per-formers.
Performers, i.e. experienced users of our ”softwear” in-strumented garments will walk through the floor of thepublic spaces performing in two modes: (1) as human so-cial probes into the social dynamics of greetings, and (2)as performers generating sound textures based on gesturalinteractions with their environment. We follow the per-formance research approach of Grotowski and Sponge[Grotowski, sponge] that identifies the actor with thespectator. Therefore we evaluate our technology from thefirst person point of view. To emphasize this perspec-tive, we call the users of our technologies ‘players’ or‘performers’ (However, our players do not play games,nor do they act in a theatrical manner.) We exhibit fabric-based controllers for expressive gestural control of lightand sound on the body. Our softwear instruments mustfirst and foremost be comfortable and aesthetically plau-sible as clothing or jewelry. Instead of starting with de-vices, we start with social practices of body ornamenta-tion and corporeal play: solo, parallel, or collective play.
Using switching logic from movements of the body itselfand integrating circuits of conductive fiber with lightemitting or image bearing material, we push toward thelimit of minimal on-the-body processing logic but maxi-mal expressivity and response. In our approach, everycontact closure can be thought of and exploited as a sen-sor. (Fig. 1)
3
Figure 1. Solo, group and environmental contact circuits.
Demonstration A: Greeting Dynamics (Fantauzza, Ber-zowska, Dow, Iachello, Sha)
Performers wearing expressive clothing instruments walkthrough a conference or exhibition hall. They act accord-ing to heuristics drawn from a provisional phenomenol-ogical schema of greeting dynamics, the social dynamicsof engagement and disengagement in public spaces builtfrom a glance, nod, handshake, embrace, parting wave,backward glance. Our demonstration explores how peo-ple express themselves to one another as they approachfriends, acquaintances and strangers via the medium oftheir modes of greeting. In particular, we are interested inhow people might use their augmented clothing as expres-sive, gestural instruments in such social dynamics. (Fig.2)
Figure 2. Instrumented, augmented greeting.
In addition to instrumented clothing, we are making ges-tural play objects as conversation totems that can beshared as people greet and interact. The shared objectshown in the accompanying video is a small pillow fittedwith a TinyOS mote transmitting a stream of acceler-ometer data. The small pillow is a placeholder for thereal-time sound synthesis instruments that we have builtin Max/MSP. It suggests how a physics-based synthesismodel allows the performer to intuitively develop and nu-ance her personal continuous sound signature without anybuttons, menus, commands or scripts. Our study of theseembedded dynamical physics systems guides our designof expressive clothing using wireless sensors, conductivefabrics and on-the-body circuit logic.
Whereas this first demonstration studies the uses of soft-wear as intersubjective technology, of course we can alsomake softwear more explicitly designed for solo expres-sive performance.
Demonstration B: Expressive Softwear InstrumentsUsing Gestural Sound: (Sha, Serita, Dow, Iachello, Fis-tre, Fantauzza)
Many of experimental gestural electronic instrumentscited directly or indirectly in the Introduction have beenbuilt for the unique habits and expertises of individualprofessional performers. A more theatrical example isDie Audio Gruppe [Maubrey]. Our approach is to makegestural instruments whose response characteristics sup-port the long-term evolution of everyday and accidentalgestures into progressively more virtuosic or symbolicallycharged gesture.
In the engineering domain, many well-known examplesare mimetic of conventional, classical music performance.[Machover]. Informed by work, for example, at IRCAMbut especially associated with STEIM, we are designingsound instruments as idiomatically matched sets of fabricsubstrates, sensors, statistics and synthesis methods thatlie in the intersection between everyday gestures inclothing and musical gesture.
We exhibit prototype instruments that mix composed andnatural sound based on ambient movement or ordinarygesture. As one moves, one is surrounded by a corona ofphysical sounds “generated” immediately at the speed ofmatter. We fuse such physical sounds with syntheticallygenerated sound parameterized by the swing and move-ment of the body so that ordinary movements are imbuedwith extraordinary effect. (Fig. 3)
The performative goal is to study how to bootstrap theperformer’s consciousness of the sounds by such es-tranging techniques (estranging is a surprising and unde-fined word here) to scaffold the improvisation of inten-tional, symbolic, even theatrical gesture from uninten-tional gesture. This is a performance research questionrather than an engineering question whose study yields in-sights for designing sound interaction.
Gesturally controlled electronic musical instruments dateback to the beginning of the electronics era (see extensivehistories such as [Kahn]) .
Our preliminary steps are informed by extensive and ex-pert experience with the community of electronic musicperformance [Sonami, Vasulka, Wanderley].
4
Figure 3. Gesture mapping to sound and video.
The motto for our approach is “gesture tracking, not ges-ture recognition.” In other words we do not attempt tobuild models based on a discrete, finite and parsimonioustaxonomy of gesture. Instead of deep analysis our goal isto perform real-time reduction of sensor data and map itwith lowest possible latency to media texture synthesis toprovide rich, tangible, and causal feedback to the human.
Other gesture research is mainly predicated on linguisticcategories, such as lexicon, syntax and grammar. [Mac-Neill] explicitly scopes gesture to those movements thatare correlated with speech utterances.
Given the increasing power of portable processors, so-phisticated sub-semantic, non-classifying analysis has be-gun to be exploited (e.g. [VanLaerhoven]). We take thisapproach systematically.
Interaction Scenario
In all cases, performers wearing softwear instruments willinteract with other humans in a public common space.But when they pass through a space that has been sensi-tized with tracking cameras or receivers for the sensorstracking their gesture, then we see that their actions madein response to their social context take on other qualitiesdue to the media that is generated in response to theirmovement. This prompts us to build responsive mediaspaces using our media choreography system.
3.2. Ambient Media
After Krueger’s pioneering work [Krueger] with video,classical VR systems glue inhabitants’ attention to ascreen, or a display device and leave the body behind.Augmented reality games like Blast Theory’s Can YouSee Me Now put some players into the physical city envi-ronment, but still pin players’ attention to (mobile)screens. [Blast]
Re-projection onto the surrounding walls and bodies ofthe inhabitants themselves marks an important return toembodied social, play, but mediated by distributed andtangible computation.
The Influencing Machine [Hook] is a useful contrastingexample of a responsive system. The Influencing Ma-chine sketches doodles apparently in loose reaction toslips of colored paper that participants feed it. Like ourwork, their installation is also not based on explicit lan-guage. In fact it is designed ostensibly along “affective”lines. It is interesting to note how published interviewswith the participants reveal that they objectify the Influ-encing Machine as an independent affective agency. Theyspend more effort puzzling out this machine’s behaviorthan in playing with one another.
In our design, we aim to sustain environments where theinhabitants attend to one another rather than a display.How can we build play environments that reward repeatedvisits and ad hoc social activity? How can we build envi-ronments whose appeal does not become exhausted assoon as the player figures out a set of tasks or facts? Weare building responsive media spaces that are not predi-cated on rule-based game logic, puzzle-solving or ex-change economies [Bjork], but rather on improvisatoryyet disciplined behavior. We are interested in buildingplay environments that offer the sort of embodied chal-lenge and pleasure afforded by swimming or by workingclay
This motivates a key technical goal: the construction ofresponsive systems based on gesture-tracking rather thangesture-recognition. This radically shortens the compu-tational path between human gesture and media response.But if we allow a continuous open set of possible gesturesas input, however reduced, the question remains how toprovide aesthetically interesting, experientially rich, yetlegible media responses.
The TGarden environment [Sponge] that inspired ourwork is designed with rich physicalistic response modelsthat sustain embodied, non-verbal intuition and progres-sively more virtuosic performance. The field-basedmodels sustain collective as well as solo input and re-sponse with equal ease.
By shifting the focus of our design from devices to proc-esses, we demonstrate how ambient responsive media canenhance both decoupled and coordinated forms of playfulsocial interaction in semi-public spaces.
Our design philosophy has two roots: experimental theatertransplanted to everyday social space, and theories ofpublic space ranging from urban planners [PPS, Whyte] toplayground designers [Hendricks]. R. Oldenburg calls fora class of so-called “third spaces,” occupying a social re-gion between the private, domestic spaces and the van-ished informal public spaces of classical socio-politicaltheory. These are spaces within which an easier versionof friendship and congeniality results from casual and in-formal affiliation in “temporary worlds dedicated to theperformance of an act apart.” [Oldenberg]
5
Demonstration C: Social Membrane (Serita, Fiano, Re-itberger, Varma, Smoak)
How can we induce a bit more of a socially playful ambi-ence in a dead space such as a conference hotel lobby?Although it is practically impossible in an exhibition set-ting to avoid spectacle with projected sound or light, wecan insert our responsive video into non-standard geome-try or materials.
We suspend ( pace T. Erickson [Erickson]) a translucentribbon onto which we project processed live video thattransforms the fabric into a magic membrane. The mem-brane is suspended in the middle of public space wherepeople will naturally walk on either side of it. Peoplewill see a smoothly varying in time and space transforma-tions of people on the other side of the membrane. (Fig. 4)The effects will depend on movement, but will react ad-ditionally to passersby who happen to be wearing oursoftwear augmented clothing.
The challenge will be to tune the dynamic effects so thatthey remain legible and interesting over the characteristictime that a passerby is likely to be near the membrane, theaffect induces play but not puzzle-solving. Sculpturally,the membrane should appear to have a continuous gradi-ent across its width between zero effect (transparency)and full effect. Also it should take about 3-4 seconds fora person walking at normal speed in that public setting toclear the width of the inserted membrane.
Figure 4. Two players tracked in video, tugging at springprojected onto common fabric.
Above all the membrane should have a social Bernoullieffect that will tend to draw people on the opposite sidesto one another. The same effects that transform the otherperson’s image should also make people feel some of thesafety of a playful mask. The goal is to allow people togently and playfully transform their view of the other in acommon space with partially re-synthesized graphics.
Artistic Interest and Craft
We do not try to project the spectator’s attention into anavatar as in most virtual or some augmented reality sys-tems. Instead, we focus performer-spectator’s attention inthe same space as all the co-located inhabitants. Moreo-ver, rather than mesmerizing the user with media “ob-jects” projected onto billboards, we try to sustain human-human play, using responsive media such as calligraphic,gesture/location-driven video as the medium of shared
expression. In this way, we keep the attention of the hu-man inhabitants on one another rather than having themforget each other distracted by a “spectacular” object[Debord].
By calligraphic video we mean video synthesized byphysicalistic models that can be continuously transformedby continuous gesture much as a calligrapher brushes inkonto silk. Calligraphic video as a particular species oftime-based media is part of our research into the precon-ditions for sense-making in live performance. [Grotowski,Brooks].
4. Architecture
For high quality real-time media synthesis we need totrack gesture with sufficiently high data resolution, highsample rate, low end-to-end latency between the gestureand the media effect. We summarize our architecture,which is partly based on TinyOS and Max / MacintoshOSX, and refer to [ISWC, TML] for details.
Our previous 6 generations of wireless sensor platformshave oscillated between small custom–programmed mi-croprocessors with radio, and consumer commodityhardware with LINUX and 802.11b wireless Ethernet([Sha Visell MacIntyre, Sponge]). Prior work focused onAnalog devices ADXL202 accelerometers and videocamera-based body location tracking. The best samplerates of 1000Hz / accelerometer channel were obtainedusing custom drivers with LINUX on an Intrinsyc Cerfprocessor and wireless Ethernet. However, this solutionrequired a video camera battery that was too heavy forcasual movement.
Our current strategy is to do the minimum on-the-bodyprocessing needed to beam sensor data out to fixed com-puters on which aesthetically and socially plausible andrich effects can be synthesized. We have modified theTinyOS environment on CrossBow Technologies Micaand Rene boards to provide time series data of sufficientresolution and sample frequency to measure continuousgesture using a wide variety of sensory modalities. Thisplatform allows us to piggy-back on the miniaturizationcurve of the Smart Dust initiative [Kahn], and preservesthe possibility of relatively easily migrating some lowlevel statistical filtering and processing to the body.Practically this frees us to design augmented clothingwhere the form factors compare favorably with jewelryand body ornaments, while at the same time retaining thepower of the TGarden media choreography and synthesisapparatus. (Some details of our custom work are reportedin [ISWC].)
Now we have built a wireless sensor platform based onCrossbow’s TinyOS boards. This allows us to exploreshifting the locus of computation in a graded and princi-
6
pled way between the body, multiple bodies, and theroom.
Currently, our TinyOS platform is smaller but more gen-eral than our LINUX platform since it can read andtransmit data from photocell, accelerometer, magnetome-ter and custom sensors such as, in our case, customizedbend and pressure sensors. However, its sample fre-quency is limited to about 30 Hz / channel.
Our customized TinyOS platform gives us an interestingdomain of intermediate data rate time series to analyze.We cannot directly apply many of the DSP techniques forspeech and audio feature extraction because to accumulateenough sensor samples the time window becomes toolong, yielding sluggish response. But we can rely onsome basic principles to do interesting analysis. For ex-ample we can usefully track steps and beats for onsets andenergy. (This contrasts with musical input analysis meth-ods that require much more data at higher, audio rates.[Puckette])
The rest of the system is based on the Max real-time me-dia control system with instruments written in MSP soundsynthesis, and Jitter video graphics synthesis. (Fig. 5)
Figure 5. Architecture comprises clothing; sensing:TinyOS, IR camera; logic and physical synthesis: Max,
MSP, Jitter; projectors; sound reinforcement.
Technical Comment on Lattice Computation
Our research aims to achieve a much greater degree ofexpressivity and tangibility in time-based visual, audio,and now fabric media. In the video domain, we use lat-tice methods as a powerful way to harness models that al-ready simulate tangible natural phenomena. Such modelspossess the shallow semantics we desire based on ourheuristics for technologies of performance. A significanttechnical consequence is that such methods allow us toscale efficiently (nearly constant time-space) to accom-modate multiple players.
We briefly describe these lattice methods since they offera fair amount of play in our current ambient media instal-lations. Our lattice operators are modeled after the waveequation:
and the heat equation:
where F is a potential function on R2 x R:
We modified the Laplacian operator D by coefficients onthe second order and first order derivatives of F in orderto provide degrees of anisotropy and viscosity. Properlytuned, these parameters enable an even more physicalisticresponse to the people’s activity.
Another example is a class of operators that index time onstreaming video at the pixel level. Our time-shifting op-erator can be parameterized by functions of a secondstream of video, tracking data derived from live infra-redvideocamera, offering a physicalistic but entirely non-realistic response to people’s activity. People invariablyplay and move quite differently in concert with thesemodified dynamics when the media was projected backinto their surround physical space.
Since the movements and body-projections of the humanparticipants seed the dynamics of our media system, thepeople themselves are analog, distributed elements of thedistributed computation. Thus the people themselves as amatter of course inject liveness and playfulness into theirown space.
5. Acknowledgements
We thank members of the Topological Media Lab, and inparticular Junko Tsumuji, Shridhar Reddy, and KevinStamper for participating in the experimental constructionand its documentation. Tazama Julien helped adapt theTinyOS platform. Erik Conrad and Jehan Moghazyworked on the prior version of the TGarden.
We thank Intel Research Berkeley and the Graphics,Visualization and Usability Center for providing the ini-tial set of TinyOS wireless computers. And we thank theRockefeller Foundation and the Daniel Langlois Founda-tion for Art, Science and Technology for supporting partof this work.
This work is inspired by creative collaborations withSponge, FoAM, STEIM, and alumni of the Banff Centrefor the Arts.
7
6. References
Aoki, H., Matsushita ,S.. Balloon tag: (in)visible marker whichtells who's who. Fourth International Symposium on WearableComputers (ISCW’00), 77-86.
Berzowska, J. Electronic Fashion: the Future of Wearable Tech-nology.http://www.berzowska.com/lectures/e-fashion.html
Björk, S., Holopainen, J., Ljungstrand, P., Åkesson, K.P.. De-signing ubiquitous computing games – a report from a workshopexploring ubiquitous computing entertainment. Personal andUbiquitous Computing, 6, 5-6, (2002), 443-458, Springer-Verlag, 2002.
Blast Theory. Can You See Me Now?http://www.blasttheory.co.uk/v2/game.html
Brooks, P. The Empty Space. Touchstone Books, Reprint edi-tion. 1995.
Debord, G. Society of Spectacle. Zone Books. 1995.
Eaves, D. et al. NEW NOMADS, an exploration of WearableElectronics by Philips, 2000.
T. Erickson and W.A. Kellogg. Social Translucence: An Ap-proach to Designing Systems that Support Social Processes. ACM Transactions on Computer-Human Interaction , 7(1):59-83, March, 2000.
f0.am, txOom Responsive Space. 2002.
http://f0.am/txoom/.
Hendricks, B. Designing for Play, Aldershot, UK andBurlington, VT: Ashgate. 2001.
Höök, K., Sengers, P., Andersson, G. Sense and sensibility:evaluation and interactive art, Proceedings CHI'2003, ComputerHuman Interaction. 2003.
Grotoski, J., Towards a Poor Theater, Simon & Schuster, 1970.
Kahn, J. M., Katz, R. H., and Pister, K. S. J. . “Emerging Chal-lenges: Mobile Networking for 'Smart Dust',” Journal of Com-munications and Networks, Vol. 2, No. 3, September 2000.
Krueger, M. Artificial Reality 2 (2nd Edition), Addison-WesleyPub Co. 1991.
Maubrey, B. Die Audio Gruppe. http://home.snafu.de/maubrey/
Oldenburg, R. The Great Good Place. Marlowe & Company,1999.
Orth, M., Ph.D. Thesis, MIT Media Lab, 2001.
PPS, Project for Public Spaces, http://pps.org
Puckette, M.S., Apel, T., Zicarelli, D.D. Real-time audio analy-sis tools for Pd and MSP. ICMC 1998.
Reddy, M. J. The conduit metaphor. in Metaphor and Thought,ed. A. Ortony, Cambridge Univ Press; 2nd edition. 1993, pp.164-201.
Richards, T. At Work with Grotowski on Physical Actions.London: Routledge. 1995.
Sha, X.W., Iachello, G., Dow, S., Serita, Y., St. Julien, T., Fistre,J. Continuous sensing of gesture for control of audio-visual me-dia. ISWC 2003 Proceedings.
Sha, X.W., Visell, Y., MacIntyre, B., Media choreography usingdynamics on simplicial complexes. GVU Technical Report,Georgia Tech., 2003.
Sponge. TGarden, TG2001. Sponge. TGarden, TG2001.
http://sponge.org/projects/m3_tg_intro.html.
Starner, T., Weavr, J., Pentland, A. A wearable computer basedamerican sign language recognizer , ISWC 1997, pp. 130-137.
Topological Media Lab. Georgia Institute of Technology. Ubi-comp video. http://www.gvu.gatech.edu/people/sha.xinwei/ to-pologicalmedia/tgarden/video/gvu/TML_ubicomp.mov
Van Laerhoven, K. and Cakmakci, O., What shall we teach ourpants? Fourth International Symposium on Wearable Computers(ISCW’00), 77-86.
Whyte, W.H., The Social Life of Small Urban Spaces. Projectfor Public Spaces, Inc., 2001.
