7
Designing Continuous Multisensory Interaction Davide Rocchesso Dept. of Art and Industrial Design IUAV University of Venice Pietro Polotti Dept. of Computer Science University of Verona ABSTRACT We claim that continuous interaction and multisensory feed- back are key ingredients for successful interactive artefacts of the future. However, the complex web of sensors, ac- tuators, and control logic that is necessary for exploiting such ingredients opens tremendous challenges to designers, who are used to visual thinking and discrete interactions. A method of research through pedagogical examples, called basic design and developed in some post-Bauhaus design schools, has been proposed as an effective mean to tackle the complexity of contemporary interaction design. Three such exercises, each prototypical for a class of interactions, are proposed here. The sonic feedback is realized through parametric control of sound synthesis algorithms. Author Keywords Basic Interaction Design; Sound; Continuous Interaction. ACM Classification Keywords H.5.2 User Interfaces: Interaction Styles; H.5.2 User In- terfaces: Auditory (nonspeech) feedback; H.5.5 Sound and Music Computing. EMBODIED INTERFACES, CONTINUOUS INTERACTION In Human-Computer Interaction, the virtues of direct manip- ulation [29] have been widely appreciated and exploited. Its kernel principles can be summarized as: (i) Persistent rep- resentation of the object of interest; (ii) Physical actions or labelled buttons instead of complex syntax; (iii) Rapid incre- mental reversible operations whose impact on the object of interest is immediately visible. In traditional WIMP/GUI interfaces, however, the principles of direct manipulation are applied to a world where the three key components of interaction (model, control, and view) are largely all in the digital domain, as there is physical sep- aration between actions (mouse movements) and feedback (displayed output). An attempt to move control and view mostly into the physical realm led to Tangible User Inter- faces [16], which rely on really-direct manipulation of to- kens having some representational capabilities. The fact that users are manipulating tokens eliminates a level of indirec- tion, since part of the feedback is indeed where the action is. However, a hermeneutic level is often introduced, due to the nature of tokens as representations. This is what embod- ied interfaces [12] tend to avoid by reducing mediations at a minimum, in the spirit of phenomenological thinking. A disembodied interface, as most of existing machine inter- faces are, gives a schizoid perception and action in the world. For the auditory world, this was well understood and exten- sively described by Murray Schafer [27], who also coined the term schizophonia to indicate the separation from sound sources induced by recording and broadcasting means. The danger of schizophonia was felt much earlier by the com- poser Bel´ a Bart´ ok, who wrote in 1937 [6] that “[...] the less foreign bodies are interposing themselves between the hu- man body and the vibrating body or, the longest the time during which the human body controls the vibration is, the more the created musical sound will be immediate and, so to speak, human.” It is easy to generalize such observation to non-musical, everyday situations of interaction with arte- facts. The tightness of the control–display loop give a stronger sense of power which, albeit being subjectively desirable, may be abused. Schafer has been reported to say that it should not be allowed to have sounds without knowing where they come from, so that you can destroy the source if you don’t like it. Indeed, distruction seems to be a compelling outcome of large-scale marketing of partially-embodied in- terfaces. For example, as soon as the Nintendo Wii con- sole entered the market, a web site 1 was launched to collect the experiences of people damaging the remote controller or hurting themselves because of excessive engagement in games. Embodied interfaces tend to exploit continuous manipula- tions in a large extent. Before the industrial revolution, most of human actions in the world were essentially continuous. Opening a door meant grabbing the handle, turning it, and pushing it so that the door swings on its hinges. These con- tinuous actions are far more prone to expressive manipula- tion than simple button triggering, as found for example in modern elevator doors. Indeed, as studies in musical acous- tics show [17], expressiveness can be induced through on- off switches only by temporal fluctuations of repeated trig- gering patterns. Being inherently expressive, continuous ac- tions and gestures are supposed to be more “natural” than triggers. Naturalness here means that control is left to the human manipulator rather than transferred to some machin- ery. In Bart´ ok’s words “less foreign bodies are interposing”. According to the tightness of sensory feedback to the han- dle, control can be more or less direct/physical. For exam- ple, sailing using the tiller is in a sense more engaging than 1 http://wiihaveaproblem.com/ 3

Designing Continuous Multisensory Interaction - rice.iuav.itrice.iuav.it/123/1/Designing_Continuous_Multisensory_Interaction.pdf · In post-Bauhaus design schools, ... Moholy-Nagy,

Embed Size (px)

Citation preview

Designing Continuous Multisensory Interaction

Davide RocchessoDept. of Art and Industrial Design

IUAV University of Venice

Pietro PolottiDept. of Computer Science

University of Verona

ABSTRACTWe claim that continuous interaction and multisensory feed-back are key ingredients for successful interactive artefactsof the future. However, the complex web of sensors, ac-tuators, and control logic that is necessary for exploitingsuch ingredients opens tremendous challenges to designers,who are used to visual thinking and discrete interactions. Amethod of research through pedagogical examples, calledbasic design and developed in some post-Bauhaus designschools, has been proposed as an effective mean to tacklethe complexity of contemporary interaction design. Threesuch exercises, each prototypical for a class of interactions,are proposed here. The sonic feedback is realized throughparametric control of sound synthesis algorithms.

Author KeywordsBasic Interaction Design; Sound; Continuous Interaction.

ACM Classification KeywordsH.5.2 User Interfaces: Interaction Styles; H.5.2 User In-terfaces: Auditory (nonspeech) feedback; H.5.5 Sound andMusic Computing.

EMBODIED INTERFACES, CONTINUOUS INTERACTIONIn Human-Computer Interaction, the virtues of direct manip-ulation [29] have been widely appreciated and exploited. Itskernel principles can be summarized as: (i) Persistent rep-resentation of the object of interest; (ii) Physical actions orlabelled buttons instead of complex syntax; (iii) Rapid incre-mental reversible operations whose impact on the object ofinterest is immediately visible.

In traditional WIMP/GUI interfaces, however, the principlesof direct manipulation are applied to a world where the threekey components of interaction (model, control, and view)are largely all in the digital domain, as there is physical sep-aration between actions (mouse movements) and feedback(displayed output). An attempt to move control and viewmostly into the physical realm led to Tangible User Inter-faces [16], which rely on really-direct manipulation of to-kens having some representational capabilities. The fact thatusers are manipulating tokens eliminates a level of indirec-tion, since part of the feedback is indeed where the actionis. However, a hermeneutic level is often introduced, due tothe nature of tokens as representations. This is what embod-ied interfaces [12] tend to avoid by reducing mediations at aminimum, in the spirit of phenomenological thinking.

A disembodied interface, as most of existing machine inter-faces are, gives a schizoid perception and action in the world.For the auditory world, this was well understood and exten-sively described by Murray Schafer [27], who also coinedthe term schizophonia to indicate the separation from soundsources induced by recording and broadcasting means. Thedanger of schizophonia was felt much earlier by the com-poser Bela Bartok, who wrote in 1937 [6] that “[...] the lessforeign bodies are interposing themselves between the hu-man body and the vibrating body or, the longest the timeduring which the human body controls the vibration is, themore the created musical sound will be immediate and, soto speak, human.” It is easy to generalize such observationto non-musical, everyday situations of interaction with arte-facts.

The tightness of the control–display loop give a strongersense of power which, albeit being subjectively desirable,may be abused. Schafer has been reported to say that itshould not be allowed to have sounds without knowing wherethey come from, so that you can destroy the source if youdon’t like it. Indeed, distruction seems to be a compellingoutcome of large-scale marketing of partially-embodied in-terfaces. For example, as soon as the Nintendo Wii con-sole entered the market, a web site1 was launched to collectthe experiences of people damaging the remote controlleror hurting themselves because of excessive engagement ingames.

Embodied interfaces tend to exploit continuous manipula-tions in a large extent. Before the industrial revolution, mostof human actions in the world were essentially continuous.Opening a door meant grabbing the handle, turning it, andpushing it so that the door swings on its hinges. These con-tinuous actions are far more prone to expressive manipula-tion than simple button triggering, as found for example inmodern elevator doors. Indeed, as studies in musical acous-tics show [17], expressiveness can be induced through on-off switches only by temporal fluctuations of repeated trig-gering patterns. Being inherently expressive, continuous ac-tions and gestures are supposed to be more “natural” thantriggers. Naturalness here means that control is left to thehuman manipulator rather than transferred to some machin-ery. In Bartok’s words “less foreign bodies are interposing”.According to the tightness of sensory feedback to the han-dle, control can be more or less direct/physical. For exam-ple, sailing using the tiller is in a sense more engaging than

1http://wiihaveaproblem.com/

3

using the wheel to control the rudder. In the latter there is adecoupling that allows application of smaller forces, but lessfeeling about the changes in direction of the boat.

When sustained feedback is elicited by triggers, the personacting on the trigger has the impression of autonomous life,and experiences a sense of causality. For example, in com-puter jargon, exceptions are thrown, programs are launched,etc. . These are expressions similar to those used by the sub-jects of famous Michotte’s experiments to describe causalityin motion patterns [20]. In interfaces, causality induced byautomatization and triggers can be easily fooled by simulta-neous extraneous feedback. In other words, one may easilyget a fictitious sense of causality. Conversely, enactive andembodied interfaces are based on a closed loop based on mo-tor skills, where control is exerted via continuous and simul-taneous perception and action. Therefore, in such interfaces,experienced causality tends to correspond more closely tophysical causality.

APPROACHES IN DESIGNWhen designing objects for multisensory continuous inter-action, the controllable dimensions are many. There are ma-ny possibilities for augmenting objects, but sensors and ac-tuators should be considered together as they will certainlyinterfere in the action–perception loop. How can we tacklethis complexity? One possibility is to think in terms of basicphenomena, constructively. So, we should look for funda-mental interaction gestalts [32] that we exploit in “natural”interactions. Interaction gestalts naturally resonate with theconcept of movement primitives, as they are considered inthe literature of motor sciences as building blocks for com-plex motor skills [26]. Interaction gestalts may result fromabstraction of actual interactions.

In post-Bauhaus design schools, prominent designers andeducators such as Tomas Maldonado, Josef Albers, LaszloMoholy-Nagy, and Bruno Munari, organized their classesaround themes, by proposing exercises with well-defined ob-jectives and constraints [3]. Students acquired their basiccompositional skills by searching for solutions to the prob-lems, and by sharing and discussing the results. The ele-ments to work with could be raw materials, ready-made arte-facts, scientific facts, or algorithmic procedures. With thispractice of basic design, research on the design fundamen-tals was advanced while teaching, in a very effective way ifwe consider the complexity of the themes. For example, inAlbers’ color exercises [2], one may find a synthesis of cen-turies of scientific research on color perception. More re-cently, this practice has been extended to multisensory com-munication [25].

In interactive contexts, managing the complexity in designis even harder. Therefore, several contemporary theoristsand educators [3, 13] claim that basic design, although ina renovated form that we may call basic interaction design,is still a valuable method. In particular, as a method of in-quiry we proceed by analyzing actions, extracting interac-tion gestalts, designing exercises around a specific interac-tion gestalt. To exemplify this method, we start from an anal-

Figure 1. Temporal dynamics of connections. Representation by Kris-

tian Kloeckl (http://www.kristian-kloeckl.com/)

ysis of actions in the kitchen [33] and consider the actions in-volved in preparing a coffee. We notice the interaction prim-itive of screwing the parts of the moka together, we abstractit as a case of dynamic connection, and develop an exercisearound it. Then, we focus on cyclic gestures typically per-formed in the kitchen, for example when cutting vegetables.We reflect on the nature of cyclic gestures, as different fromdiscrete gestures, and on the continuous feedback that mayaccompany them. Finally, we propose an exploratory exer-cise aimed at developing an augmented sense of resistancewhen performing certain continuos actions, such as pouring.So far, the three proposed exercises have not been all fullydeveloped. However, they are indicative of a methodologythat we are trying to follow in our work on continuous mul-tisensory interaction.

THEME I: MULTI-STAGE YET CONTINUOUS COUPLINGConnections in DesignAn important topic in product design is that of “connec-tions”, i.e., how to create complex objects by joining sim-pler elements together. Different connections are character-ized by different temporal dynamics. The act of joining andsplitting parts unfolds in time, and the temporal explorationof the continuum between “attached” and “detached” can berepresented pictorially via traces, as in figure 1. The ver-tical axis could be interpreted as the degree of tightness inthe connection. Although being a continuum, this tightnessdimension can be categorically perceived in a small numberof discrete states. For example, torque measurement sys-tems2 discretize the state of a screw connection into threestages: (i) torqued too low, (ii) OK, (iii) torque too high. Forsuch systems, audio-visual signals have been designed to in-form symbolically about the state, using a color visual code(yellow, green, red) and an auditory code based on counting(one, two, or three beeps).

The moka and the three stages of couplingThe moka coffee maker is a piece of Italian design of thenineteen-thirties, now in widespread use worldwide. In or-der to prepare a coffee one has to fill the boiling chamber2For example, see Ingersoll-Rand products

4

with water, fill the filter with ground coffee, and connectthe three parts (boiling chamber, filter, and top container) bymeans of a screw connection. Tip sheets for moka pot brew-ing3 spend some words to describe how to screw the top andbottom together (“to get a tight fit is to hug the pot to yourchest while you grab the two parts and twist them together.This does require a fair amount of arm and hand strength.”)and how to detect deviations from proper tightness (“if youcan see steam escaping from the area where the top and bot-tom parts screw together”). The reason for such expressivedescriptions is that the gesture of closing the moka is con-tinuous, although it can be discretized into three stages: (i)loose, (ii) tight, (iii) too tight.

Evidently, it would be possible to apply some pressure ortorque sensor to a moka and provide the same kind of dis-crete audio-visual feedback as it is given by torque measure-ment tools. However, this is likely to reduce much of the fas-cination and emotional engagement that people experiencewhen exerting continuous manipulations of well-balancedmechanisms [9]. So, the basic design exercise that we arefacing here is

PROBLEM 1.

Theme Multisensory feedback for mechanical connections.

Objective Design the feedback for a screw connection, suchas found in the moka, in such a way that the right degreeof tightness in coupling can be easily reached.

Constraints The feedback should be continuous, non-sym-bolic, immediate to catch (or pre-attentional), and yet di-visible into three clear stages.

Audio-visual symbolic signals, as the ones found in torquemeasurement systems, would result in a disembodied feed-back, for two reasons: they enforce the discretization of ges-tures that are naturally continuous, and they rely on symboliccoding rather than on enactive experience. Notice that, in ba-sic design exercises, constraints are usually supported by arationale, although a different set of constraints may makeperfect sense as a different exercise.

If choosing between visual and auditory feedback for themoka problem, audition is certainly to be privileged because,in temporal patterns and tasks, it affords higher resolutionand it tends to prevail over possible non-coherent cues com-ing from other senses [15]. After this observation, the designspace is further constrained, and it reduces to sonic inter-active feedbacks satisfying the requirements of problem 1.Still, the possibilities are countless, and one may exploitmetaphors, analogies, or other design tricks to narrow thedesign space further.

The violin and the three qualities of playingBowed-string musical instruments, such as the violin, offeran interesting beacon to orient the approach to our sound de-sign problem. Starting from the requirement of three discrete3See, e.g., http://www.sweetmarias.com/

Minimum bow force

Maximum bow force

Sul ponticello

Brilliant

Sul tasto

0.01 0.02 0.04 0.06 0.1

0.001

0.1

1

Relative position of bow, ß

Rel

ativ

e fo

rce

RAUCOUS

NORMAL

0.2

0.01

HIGHER MODES

Figure 2. The Schelleng diagram, in the force–position plane, for con-

stant bow velocity

stages, violins indeed exhibit three main kinds of sonic qual-ity, which can be explored by varying the three principal con-trol variables: bow force onto the string, bow velocity, andbow position along the string. The oscillatory regime cor-responding to “good tone” production is called “Helmholtzmotion”. A graphical tool used to display at a glance the dif-ferent oscillatory behaviors of a bowed string is the so-calledSchelleng diagram [28], schematically drawn in figure 2 forconstant bow velocity. Two other regions are displayed onthe sides of the region of good-tone (normal) production.Given a bow position and velocity, lower forces produce a“surface sound”, eliciting higher modes. Higher forces, onthe other side, produce aperiodic “raucousness”. Similarly,the region of “playability” can be highlighted on the force–velocity plane, for constant bow position. Different modelsof bow–string interaction can be analyzed in terms of theoscillatory regimes they produce, and Schelleng diagramscan be produced experimentally [28]. The wider the Hel-moltz region results, the more easily playable the frictionmodel is. A model that displayed a wide playability region,while at the same time being so versatile to reproduce a largevariety of oscillatory regimes, is the elasto-plastic frictionmodel [5].

By exploring the space of control parameters we can gradu-ally step into different oscillatory regimes, and listeners areimmediately capable to catch the quality of the physical ac-tions associated with such sound manifestations. Namely,three distinct areas of effort (too low, ok, too high) can bereadily associated with the three main tone-quality areas de-scribed by the Schelleng diagram. So, the requirements ofour sound-design problem can be met.

Sonifying the mokaDuring a doctoral workshop at IUAV, we tackled problem 1by using preparatory materials (a disassembled moka, sen-sors and actuators, sound synthesis algorithms) to put to-gether convincing demonstrations that were experienced in-dividually and discussed in group. This is the kind of re-search through pedagogy, or pedagogy through research, that

5

Figure 3. Hacked moka with Max/MSP sonification patch

basic design practices recommend. We applied a force sen-sor between filter and gasket of a moka, and used the forcesignal to control the elasto-plastic friction model, implementedas a Max/MSP4 external within a supporting patch (see fig-ure 3). Although one may map the sensed force directlyto the force control parameter of the elasto-plastic model,clearer and more compelling transitions are obtained by co-varying other parameters of friction. Moreover, as a gen-eral criterion in the sonification of everyday objects, it ispreferable to make a sparse use of short sound pulses ratherthan modulate a sustained tone. Therefore, short frictionbursts were generated by a pseudo-random process whilethe moka components were being coupled. The resultingsound dynamically changes its timbral quality as the cou-pling becomes progressively tighter. More precisely, thetimbre moves gradually from a sound of glass harmonicafor loose coupling, to assuming a rubber quality for the tightstage. When the coupling enters a “too tight” state, the soundresembles that of a squeaking hinge. Since in this work wefocus on basic design methods rather than reproducible sci-entific experiments, we skip detailed report of the sound syn-thesis algorithm and parameters.

The sonified moka was experienced directly by ten peoplewho participiated to the workshop. The unanimous impres-sion was that sound enhances the screw connection and makesthe task more engaging. Feedback about the degree of clo-sure was reported to be very natural. The experience of act-ing directly on the moka was judged to be very different fromsimply listening to the produced sound. The auditory feed-back blends with the kinesthetic perception coming from ex-erting manual force on the moka components. Indeed, somepeople started to play with the moka and tried to push theboundaries of meaningful interaction, up to the point that, atthe end of session, the force sensor was broken due to exces-sive pressure. This highlights a problem that is common toany object that affords expressive interaction, at the point toelicit unexpected or excessive manipulation.

When considering sound augmentation of everyday artefacts,the problem of localization of the loudspeaker is quite se-rious. The question is whether to attach a loudspeaker to

4http://www.cycling74.com/products/maxmsp

the object or not. Physical constraints may impose a choicehere. Otherwise, using an embedded loudspeaker usuallyenhances the unity of experience, even though the ventrilo-quist effect [15] guarantees a certain degree of coherence inperception even for loudspeakers displaced from the artefact.We tried both options during the workshop. When an actua-tor was directly attached to the moka everybody agreed thatthe sound became really part of the object, not only becauseof its apparent point of emission, but also for the acousticproperties of the aluminum chambers of the moka, whichwere affecting the sound production.

THEME II: CYCLIC INTERACTIONSince the nineties some people started to look at continu-ous gestures in GUIs, like crossing targets or traversing hi-erarchical cascading menus, and tried to model them as lim-its of sequences of discrete gestures. The steering law wasderived and applied [1]. However, when the task requirescyclic movements, experiments show that these movementscan not be simply treated as a concatenation of discrete ges-tures, as their rhythmic nature allows better exploitation ofthe physical properties of the neuromotor system. Actually,Fitts’ law has been reported to break down in rhythmic tasks,where performance turns out to be supported by much higherindexes of performance [30].

Some other people looked at what is in between starting po-sition and final target in goal-directed movements, and dis-covered that there are kinematic patterns [8]. Specifically,when difficulty is raised in a reciprocal aiming task, velo-city profiles take an asymmetric bell shape, with an attackshorter than the release. So, there is much more behindpoint-like acts. The continuous support of discrete eventscontains much information that we are likely to exploit ineveryday activities. In particular, the expressive content ofour actions is largely mediated by these continuous gestures,even when they are aimed at discrete events, like cutting acarrot evenly into pieces.

In music performance, preparatory and ancillary continuousmovements are fundamental to convey the intended expres-sive character to a sequence of notes or drum strokes [10].When attending the performance of a drummer we integratevisual and auditory information. While the former is sup-ported by continuous profiles of an array of kinematic vari-ables, the latter comes in discrete bursts of impact noise.The visual continuous stream and the auditory burst-basedstream get integrated and synchronized according to differ-ent conditions. For example, the tempo of a rhythm andthe naturalness of visually-perceived movements are knownto influence the perceived synchrony [4]. Also, the weightgiven to different sensorial channels may depend on the de-gree of expertise. Expert drummers tend to trust auditoryfeedback more than visual feedback when evaluating audio-visual synchrony of drum gestures [22]. These observationsstrongly support the use of auditory display for providingfeedback in repetitive tasks. Indeed, as early as in 1991,Gaver, Smith, and O’Shea [14] simulated a bottling plantand demonstrated that auditory display can support rhyth-mic activities even in collaboratory environments.

6

A basic design exercise can be conceived to exploit the ef-fectiveness of a well-defined feedback in cyclic activities.In several traditions of food preparation, cutting carrots inpieces of various shapes is considered a sort of performa-tive activity, and sometimes even a key to meditation5. Sen-sorized kitchen tools may amplify such experience, so thatcutting precise rondelles becomes an act more easily achiev-able and enjoyable. To do that, we can play with the syn-chronization and balance between senses. The exercise mayread as

PROBLEM 2.

Theme Supportive feedback for cyclic continuous actions.

Objective Support coordination between the two hands in acyclic task such as cutting carrots in rondelles. The feed-back should give information about both the longitudinaltranslation of the carrot and the rhythmic action of cut-ting.

Constraints The feedback should be continuous, non-sym-bolic, pre-attentional, giving a sense of progress, and em-phasizing jitter in the coordinated movements.

We intend to run a sound design workshop around prob-lem 2. As far as technology is concerned, we already imple-mented a sonic knife and experimented with it as part of theGamelunch [11]. A similar tool was proposed for context-aware kitchen environments [18]. Detection of movementof the carrot can be done both by applying sensors on thecutting board or by mounting a camera on the knife.

THEME III: COUNTER-INTER-ACTIONSWhen continuous feedback is coupled to continuous move-ments, the coordination of different sensory channels is knownto affect the action [19]. For example, audition may con-vey a better sense of velocity or acceleration as comparedto vision or haptics, thus eliciting smoother trajectories ina target-reaching task [24]. Little research has been doneto investigate the emotional impact of continuous sounds oncontinuous gestures, even though it is recognized, in the areaof product sound quality, that evaluating the feel of an ob-ject in use is different from rating different sounds presentedover the headphones [31]. In the area of emotional design, ithas been observed that users interpret displays as emotion-ally expressive even if there is no explicit representation ofemotion [7]. So, it becomes possible to “distort” a display tomanipulate its emotional impact.

Playing with the relationship between different sensory chan-nels gives the opportunity to exploit the power of contradic-tion in multisensory interaction. A possible strategy is infact to distort one (or some aspects of one) of the displays,while preserving the natural characteristics of the other ones.Empirically, we observed that by properly designing a sonicfeedback, one can strengthen or resist an action. Produc-ing a sonic feedback, that is unexpected with respect to oneor more aspects of a specific interactive context, has inter-esting effects on the performed actions. By means of what5http://www.snowlight.com/keys.html

we could consider a manipulation of both cognitive expec-tations and emotional impact of the feedback, one is ableto stress the relevance of a specific portion of the informa-tion conveyed by the distorted channel. Experimental real-izations of counter-actions have been tested as part of theGamelunch installation [11]. In the Gamelunch, the manip-ulation of the sonic feedback does not affect the coherenceof the gesture, in the sense that when a more intense stateof the action takes place, the sound feedback reacts con-sistently in terms of loudness and/or increasing number ofevents. In other words, the feeling of effectiveness of the ac-tion is maintained. This has been experienced as fundamen-tal in order to preserve the veridicality of the sonic feedbackand avoiding the risk of arbitrariness that otherwise wouldoccur. Some other dimensions were considered, achievingan effect of distortion in two directions. The first one wasmore tightly related to the action. As an example, the ac-tion of pouring a liquid from a decanter was considered. Asolid friction sound was used to sonify the inclination of thedecanter, giving a feeling of a resisting force that contradictsthe (opposite) effect of the liquid flowing and of the decantergetting lighter. The second consisted in contradicting some-thing not directly related to the action as, for example, thematerial of the objects involved in the interaction. One casestudy was that of the action of stirring a soup. The sonicfeedback provided a sound similar to that of a tool mixinga sandy material, oppositely to the fact that one saw (andsmelled) a soup in the dish. Such experiences shed light onthe importance of the sonic feedback and of its coordinationwith the other sensory channels. Even if rigorous experi-mentation has not been conducted yet, it was encouraging tosee how most of the people who experienced the Gamelunchagreed on the effectiveness of the distortion in terms of ac-quiring consciousness of the role of the sonic feedback.

In terms of basic design, the problem of the resisting de-canter could be formulated as follows.

PROBLEM 3.

Theme Contradictory feedback for continuous actions.

Objective Design the feedback for a pouring action, so thata perceived resisting force contradicts the easiness of liq-uid streaming.

Constraints The feedback should be continuous, non-sym-bolic, immediate to catch (or pre-attentional), and consis-tent with the underlying physical process (e.g., more flowmore loudness)

Contradictory feedback may be exploited in contexts wheresome actions have potentially serious consequences. It iswell known that warning messages tend to be ignored inroutine situations [23]. It may well be the case that someadditional resistance introduced via multisensory emotionalstimulation in presence of critical gestures induces higher at-tention. This is particularly important in future objects, suchas cars, with a high degree of autonomy [21]. For example,the sustained experience of effort proved to be an effectivedesign ingredient to prevent modal errors in interfaces [23].

7

CONCLUSIONThe examples here proposed can be considered as exercisesin basic interaction design. Clearly, the kitchen scenariosare only instrumental to experimenting with continuous mul-tisensory interaction under well-defined constraints. Thereare many other environments and tools that may benefit fromcarefully-designed multisensory feedback, in terms of accu-racy, safety, affection and engagement.

AcknowledgmentsThis research was supported by the project CLOSED6 and by the ActionCOST-IC0601 on Sonic Interaction Design7. We wish to thank StefanoDelle Monache for sound design and Stefano Papetti for software develop-ment. Thanks to Carlo Bassetti for describing torque measurement to us.

REFERENCES1. J. Accot and S. Zhai. Beyond Fitts’ law: Models for

trajectory-based HCI tasks. In Proc. of CHI’97,Atlanta, Georgia, USA, 1997.

2. J. Albers. Interaction of Color. Yale University Press,New Haven, CT, 2006 (1st. ed. 1963).

3. G. Anceschi. Basic Design, fondamenta del design. InG. Anceschi, M. Botta, and M. A. Garito, editors,L’ambiente dell’apprendimento – Web design eprocessi cognitivi, pages 57–67. McGraw Hill, Milano,Italia, 2006.

4. R. Arrighi, D. Alais, and D. Burr. Perceptual synchronyof audio-visual streams for natural and artificial motionsequences. Journal of Vision, 6(3):260–268, 2006.

5. F. Avanzini, S. Serafin, and D. Rocchesso. Interactivesimulation of rigid body interaction withfriction-induced sound generation. IEEE Transactionson Speech and Audio Processing, 13(5):1073–1081,2005.

6. B. Bartok. Mechanical music. In B. Suchoff, editor,Bela Bartok – Essays, pages 289–300. University ofNebraska Press, Lincoln, NE, 1992. orig. 1937.

7. K. Boehner, R. DePaula, P. Dourish, and P. Sengers.Affect: from information to interaction. In CC ’05:Proceedings of the 4th decennial conference on Criticalcomputing, pages 59–68, New York, NY, USA, 2005.ACM Press.

8. R. Bootsma, L. Fernandez, and D. Mottet. Behind Fitts’law: kinematic patterns in goal-directed movements.International Journal of Human-Computer InteractionStudies, 61:811–821, 2004.

9. B. Buxton. Experience design vs. interface design.Rotman Magazine, pages 47–49, Winter 2005.

10. S. Dahl. On the beat: Human movement and timing inthe production and perception of music. Doctoralthesis, KTH School of Computer Science andCommunication, Stockholm, Sweden, 2005.

6http://closed.ircam.fr/7http://www.cost-sid.org/

11. S. Delle Monache, P. Polotti, S. Papetti, andD. Rocchesso. Gamelunch: A physics-based sonicdining table. In Proc. International Computer MusicConference, Copenhagen, Denmark, 2007.

12. P. Dourish. Where the Action Is: The Foundations ofEmbodied Interaction. MIT Press, Cambridge, Mass.,2001.

13. A. Findeli. Rethinking design education for the 21stcentury: Theoretical, methodological, and ethicaldiscussion. Design Issues, 17(1):5–17, 2001.

14. W. W. Gaver, R. B. Smith, and T. O’Shea. Effectivesounds in complex systems: the arkola simulation. InCHI ’91: Proceedings of the SIGCHI conference onHuman factors in computing systems, pages 85–90,New York, NY, USA, 1991. ACM Press.

15. S. Handel. Perceptual Coherence – Hearing andSeeing. Oxford University Press, New York, 2006.

16. H. Ishii and B. Ullmer. Tangible bits: towards seamlessinterfaces between people, bits and atoms. In CHI ’97:Proceedings of the SIGCHI conference on Humanfactors in computing systems, pages 234–241, NewYork, NY, USA, 1997. ACM Press.

17. P. Juslin, A. Friberg, and R. Bresin. Toward acomputational model of expression in performance:The germ model. Musicae Scientiae, Special Issue2001–2002: Current Trends in the Study of Music andEmotion:63–122, 2002.

18. M. Kranz, A. Schmidt, A. Maldonado, R. B. Rusu,M. Beetz, B. Hornler, and G. Rigoll. Context-awarekitchen utilities. In TEI ’07: Proceedings of the 1stinternational conference on Tangible and embeddedinteraction, pages 213–214, New York, NY, USA,2007. ACM Press.

19. J. Lagarde and J. Kelso. Binding of movement, soundand touch: multimodal coordination dynamics. ExpBrain Res, 173(4):673–688, 2006.

20. A. Michotte. The Perception of Causality. Basic, NewYork, 1963.

21. D. A. Norman. The Design of Future Things. BasicBooks, New York, 2007.

22. F. Pollick, K. Petrini, M. Russel, D. Rocchesso, C. H.Waadeland, S. Dahl, and F. Avanzini. Effects ofmusical expertise on perceived audiovisual synchrony.In Proc. European Conference on Visual Perception,Arezzo, Italy, 2007.

23. J. Raskin. The Humane Interface. ACM Press, 2000.

24. M. Rath and D. Rocchesso. Continuous sonic feedbackfrom a rolling ball. IEEE MultiMedia, 12(2):60–69,2005.

8

25. D. Ricco. Basic design sinestesico. fondamenti teorici.In G. Anceschi, M. Botta, and M. A. Garito, editors,L’ambiente dell’apprendimento – Web design eprocessi cognitivi, pages 97–110. McGraw Hill,Milano, Italia, 2006.

26. S. Schaal, A. Ijspeert, and A. Billard. Computationalapproaches to motor learning by imitation.Philosophical Transaction of the Royal Society ofLondon: Series B, Biological Sciences, 358:537–547,2003.

27. M. Schafer. Soundscape - Our Sonic Environment andthe Tuning of the World. Destiny Books, Rochester,Vermont, 1994.

28. S. Serafin. The sound of friction: real-time models,playability and musical applications. Doctoral thesis,CCRMA, Stanford University, 2004.

29. B. Shneiderman. Direct manipulation: a step beyondprogramming languages. IEEE Computer, 16(8):57–69,1983.

30. B. Smits-Engelsman, G. Van Galen, and J. Duysens.The breakdown of Fitts’ law in rapid, reciprocal aimingmovements. Experimental Brain Research,145(2):222–230, 2002.

31. C. Spence and M. Zampini. Auditory contributions tomultisensory product perception. Acta Acustica unitedwith Acustica, 92(17):1009–1025,November/December 2006.

32. D. Svanaes. Understanding Interactivity: Steps to aPhenomenology of Human-Computer Interaction. PhDthesis, NTNU, Computer Science Department,Trondheim, Norway, 2000. available at http://www.idi.ntnu.no/˜dags/interactivity.pdf.

33. Y. Visell, K. Franinovic, and D. Hug. Sound productdesign research: Case studies, participatory design,scenarios, and product concepts. Deliverable of ProjectCLOSED, Hochschule fur Gestaltung und KunstZurich, 2007. Available athttp://closed.ircam.fr/uploads/media/closed_

deliverable_3.1_public.pdf.

9