Perspectives on aesthetic computing

University of Westminster Eprints http://eprints.wmin.ac.uk Perspectives on aesthetic computing. Paul Fishwick1 Stephan Diehl2 Jane Prophet3 Jonas Lowgren4 1Department of Computer & Information Science & Engineering, University of Florida, U.S.A 2Computer Science, Catholic University Eichstätt, Germany 3Centre for Arts Research, Technology and Education, University of Westminster, 4School of Arts and Communication, Malmö, Sweden Copyright © [2005] MIT Press. This is an electronic version of an article published in Leonardo: Art Science and Technology, 38 (2). pp. 133-141, April 2005. Leonardo: Art Science and Technology is available online at: http://www.mitpressjournals.org/doi/abs/10.1162/0024094053722372 The Eprints service at the University of Westminster aims to make the research output of the University available to a wider audience. Copyright and Moral Rights remain with the authors and/or copyright owners. Users are permitted to download and/or print one copy for non-commercial private study or research. Further distribution and any use of material from within this archive for profit-making enterprises or for commercial gain is strictly forbidden. Whilst further distribution of specific materials from within this archive is forbidden, you may freely distribute the URL of the University of Westminster Eprints (http://eprints.wmin.ac.uk). In case of abuse or copyright appearing without permission e-mail [email protected].

wattsn

top stamp

wattsn

Middle

wattsn

Bottom

In collaboration with Leonardo, we participatedin the Aesthetic Computing workshop in the hills of south-western Germany in July 2002. The workshop [1], organizedby Paul Fishwick, Roger Malina and Christa Sommerer, wasone week in duration and was attended by over 30 represen-tatives of the following disciplines: art, design, computer sci-ence and mathematics. The purpose of the workshop was todefine this area and to try to bring to light key aspects of thefield from a variety of perspectives. An aesthetic computingmanifesto was recently published in Leonardo [2]. Aestheticcomputing is the application of art practice and theory to com-puting. Given this brief definition, several questions come tomind; we shall address these questions and then proceed withthe purpose of this paper, which is to cover sample projects inthe area and the different ways in which the authors proceedto accomplish their singular crafts.

One question regarding the name “aesthetic computing”may well be whether we can justify its existence. After all, isnot all computing within the realm of aesthetics, and is therenot already a significant number of projects that capture somecombination of computing and art?

Let us begin with the first question. We will define aesthet-ics broadly as a combination of cognitive and sensory modesof experience, according to its scope in philosophy [3]. Evi-dence of the employment of the purely cognitive aspect of aes-thetics can in fact be found within mathematics [4] andcomputing: One speaks of an elegant proof of a theorem, ora beautiful representation. With such qualifiers, the mathe-matician is usually referring to cognitively grounded aesthet-ics. Our intent is to explore aesthetics by means of the bridgethat separates the cognitive and sensory perspectives—it is bycrossing this bridge that we may enrich both design and com-puting. As Knuth points out in his discussion of Metafont,which underlies his TeX typesetting system, “Type design canbe hazardous to your other interests. Once you get hooked,you will develop intense feelings about letterforms” [5]. In-

terpreted more generally, Knuth isexpressing the idea of aesthetics asbeing more than the purely cogni-tive, even within computing, whichgrew out of mathematics. A textualsection of a computer program willhave both denotative as well as con-notative signifiers, and it is easy toimagine that the program mightalign itself with the goals of art byusing the types and variety of rep-resentations employed in the art world, thereby stretching thetraditional boundaries of what may be considered a usablecomputer program representation.

The second question, regarding the combination of sciencewith art, is addressed by first noting that, unlike computer ordigital art, the idea of aesthetic computing implies that art isaffecting—and reflecting—some aspect of computing. Thisresults in an agenda quite different from that found in manyother fields of digital art.

To acknowledge our interest in aesthetics in computing isto expand human-computer interaction and representationto reach into numerous areas of computer science, from theprimary operating systems interface to the interface used bycomputer scientists to create programs and by scientists to cre-ate models of geometry and dynamics. This step leads to twoobservations that partially justify the move toward aesthetic

©2005 ISAST LEONARDO, Vol. 38, No. 2, pp. 133–141, 2005 133

G E N E R A L A R T I C L E

Perspectives on Aesthetic Computing

Paul Fishwick, Stephan Diehl, Jane Prophetand Jonas Löwgren

Paul Fishwick (educator), Department of Computer & Information Science & Engineer-ing, University of Florida, Gainesville, FL 32611, U.S.A. E-mail: <[email protected]>.

Stephan Diehl (educator), Computer Science, Catholic University Eichstätt, Ostenstr. 14,85072 Eichstätt, Germany. E-mail: <[email protected]>.

Jane Prophet (artist, researcher), Centre for Arts Research, Technology and Education,University of Westminster, 70 Great Portland Street, London, U.K. E-mail:<[email protected]>.

Jonas Löwgren (designer, researcher), School of Arts and Communication, Malmö University, SE-205 06 Malmö, Sweden. E-mail: <[email protected]>.

A B S T R A C T

The authors present an intro-duction to the new inter-disciplinary area of aestheticcomputing and proceed todefine this area with examplesfrom each of their own disci-plines, practices and research.While several decades ofpublication and work haveresulted in significant advance-ments in art as implementedthrough technology, less empha-sis has been placed on studyingthe converse issue of art’seffect on computing, or “aes-thetic computing.” The authorspresent their individual work inthis area and then follow withbrief criticism of one another’swork to elucidate differentperspectives on the idea. Byapproaching the topic of aes-thetic computing in this manner,the paper serves as an introduc-tion to and survey and analysisof the field.

Fig. 1. Paul Fishwick, a2D static snapshot of aninteractive diagrammaticFSM interface. (© PaulFishwick)

computing: (1) aesthetics in computingare broader than the purely cognitive di-mension; and (2) the art-science conflu-ence embedded within the discipline ofinteraction design is broader than theprimary “desktop” interface.

The first observation is one of both on-tology and epistemology—we mightleverage existing aesthetic principles to-ward the sensory. Consider the case of“software patterns” [6] as one example.One can surely conclude that these pat-terns reflect abstractions of softwarestructures and that they are found lurk-ing in numerous software applications.There are two ways of looking at patterns,reflecting two ways of defining aesthetics:cognitive and material. What if the pat-terns revealed themselves in a way thatwas more attuned to material embodi-ment? This would allow us to build uponthe existing pattern literature and extendit, as well as to extend the pattern repre-sentations. The “factory” method es-poused by Gamma et al. could be thebasis for a two-dimensional or three-di-mensional scene that looks and operateslike a factory. The look and feel of the factory would improve with artistic in-fluence and guidance and would providestrong metaphorical cues. This sensorydimension seems lacking in the patternliterature; one generally finds that rep-resentation is limited to rectangles andarcs. And yet it is not only in the patternliterature that it is lacking, given that the visual minimalism of program struc-tures and the mathematical structuresunderlying them is fairly common incomputing.

The second observation implies thatinteraction design needs to concern itselfwith all questions of interaction and pre-sentation found in computing, includingthat of how to represent data and pro-gram structures, for instance. The emerg-

tion. In XML parlance, content refers toan abstract specification defined as a doc-ument tree, and presentation refers tothe way the tree is presented to the user—the way it looks and sounds. Thus, usingRUBE and guided by the XML philoso-phy, one may specify an equation andchoose to present the equation as eitherlinear text, a network or a 3D structure.The choice of which presentation to em-ploy can be determined by XML stylesheets and their associated transforma-tions.

RUBE’s architecture is based on open-source software and begins with author-ing toolkits: SodiPodi for 2D vectordrawing and Blender for 3D modeling.Let us consider the 3D pipeline begin-ning with Blender. The user creates a 3Dmodel in Blender and then uses a Pythonscripting interface, which allows attri-butions to be made regarding seman-tics. For example, one might point to an object and designate it as a state or a function. After the semantic assign-ment, an X3D (eXtensible 3D) file is cre-ated for the presentation and a specialXML file is created for specifying the for-mal model. After some XML transfor-mations, this XML file is translated intoJavascript or Java, whereby it can be rein-corporated into the VRML (Virtual Re-ality Modeling Language) file, resultingin an interactive VRML world. The 2Dtransformations are similar, except thatScalable Vector Graphics (SVG) is usedfor presentation.

Let us begin with a formal definitionof a Finite State Machine (FSM) M [14].These machines have states that are in-terconnected through transitions that gointo effect when an input to the machineis of a particular value. Here is a formaldefinition for M:

M = <Q, I, O, δ, λ>Q = {S1, S2, S3}, δ: Q × I→ Qδ (S1,0)=S1; δ (S2,0)=S2; δ (S3,0)=S3;δ (S1,1)=S2; δ (S2,1)=S3; δ (S3,1)=S2I = {0, 1}, λ: Q → OEven though this text might seem to be

the formal specification for M, it is actu-ally one of many types of presentation ofthe underlying formalism that is encodedin XML. In general, all presentations re-quire additional natural-language se-mantics if we are to make sense of them.Q is the state set for M, I the input set, Othe output set, δ the transition functionfrom one state to another and λ the out-put function.

Figure 1 illustrates our second presen-tation of the FSM. It has iconic proper-ties such that when the machine is in stateS2, the presentation of a circle for S2encodes the concept of a boundary and

ing areas of information visualization [7]and software visualization [8,9] representapproaches toward this goal, and yet theirrepresentations of information and soft-ware could benefit from greater empha-sis on a wider range of artistic expressionwithout sacrificing utility. So it is not thatthe area of design does not presently con-cern itself with incorporating aestheticsbut that the current level and degree ofthis incorporation need to be expandedbeyond those of the typical user interfaceat the operating systems level.

We now have a working idea of aes-thetic computing, and we need to ex-plore different approaches. There followsa statement on aesthetic computing byeach author, followed by a discussion ofone another’s views and statements.

PAUL FISHWICK: FORMALREPRESENTATIONSScopeIn applying aesthetics to computing, weneed to confine ourselves to some aspectof computing, or one of its subfields suchas human-computer interaction, visuali-zation or discrete structures, to name afew. At the University of Florida, we haveconstructed a software system calledRUBE [10–13], in which we have fo-cused primarily on representations inmathematics and computing notation,from the notation of algebraic and dif-ferential equations to that of programand data structures, informed, however,through an artistic sensibility. Our basicidea has been to build a system that al-lows a multiplicity of different notationsto be constructed so that one may see and hear the same underlying formalismin numerous ways. Not only do differ-ent people and cultural entities enjoyworking with a formalism using differentmetaphors, but a single person or groupcan also benefit from exposure to diversepresentations.

ImplementationRUBE therefore allows for different rep-resentations to be applied to a selectnumber of formal dynamic modelspecifications. Using RUBE, it is possibleto change the way formal models lookand sound. By formal models, I am re-ferring to a large class of models used tospecify systems that incorporate time foranalysis and simulation, such as finitestate machines, Petri networks, Markovmodels, queuing models and System Dy-namics graphs, as well as ordinary andpartial differential equations. RUBE usesXML (eXtensible Markup Language),which separates content from presenta-

134 Fishwick et al., Perspectives on Aesthetic Computing

Fig. 2. Paul Fishwick, a 3D agent-basedpresentation of the FSM using the VirtualReality Modeling Language (VRML). (© Paul Fishwick)

that which is inside it. That is, the graph-ical depiction of S2 is consistent with theunderlying metaphors of set theory,whereas the purely textual presentationdoes not capture these metaphors. More-over, Fig. 1 is incomplete on a non-inter-active medium, such as paper, since theadditional information encoded but notvisible in the text representation is ob-tained through minimal human-com-puter interaction (e.g. pressing a buttonto yield the text layer). Similarly, the ar-rows capture the notion of transitionfrom one state to another, since anyonewho has seen an arrow fly knows that it isaimed toward a target. The metaphors ofthe figure dramatically improve visuali-zation of the semantics of the machine,and so one is led to wonder whether em-ploying presentations with alternativeaesthetics might further improve the im-pact of the metaphor. The underlying as-sumption is that material aspects of levelsof representation are based largely onwhat is available for a society, as well aswhat is affordable and materially effi-cient. Consider Fig. 2 as a representationthat has only recently become possiblethrough computer graphics and the abil-ity to employ 3D components. The meta-phor of the circle as a boundary has beenreplaced by a small gazebo-like structure.The arrow in Fig. 1 is now shown as awoman walking from one state to anotheralong a lamp-lit walkway. While Fig. 2could be seen as introducing gratuitousimagery, it can alternatively be seen as a structure that reinforces the spatio-temporal metaphors that Fig. 1 only be-gins to reveal. The idea here is that anFSM, by its formal definition, is rich withmetaphor and that Fig. 2 exploits themetaphor by surfacing it through visual-ization, interaction and familiar icons.Movement from one state to anotherlooks like movement, and the thing mov-ing is more familiar to us than a mentalconstruct would be.

A host of philosophical issues comesinto play here. Is there not a need to en-force visual minimalism within this sortof structure? What are the cultural bar-riers that might prevent the adoption ofmodels like Fig. 2 in science and engi-neering?

With respect to the first issue, weshould note that it is quite possible tomaintain abstraction without requiringvisual minimalism. Within an artistic con-text, this can be seen when comparingand contrasting the genres of AbstractExpressionism and Surrealism. Both ofthese genres contain a wide variety ofworks that employ symbolism, iconog-raphy and rich semiotics even though

program text is written in an artificial lan-guage with a strict syntax and more orless well-defined semantics. Trying to un-derstand a real software system by read-ing its millions of lines of program text is a vain task. As a consequence, manytools have been developed to supportsoftware understanding. These tools relyon analysis and visualization techniques.In a famous paper, Turing Award recip-ient Fredrick Brooks even stated that“software is invisible and unvisualizable”because each kind of visualization onlyaddresses “one dimension of the intri-cately interlocked software elephant”[15]. These dimensions include the staticstructure of the software, its dynamicsand its evolution. To put it in otherwords, different kinds of visualizationsshow how software is encoded, how it behaves and how it is developed. In thefollowing I present and discuss exam-ples of visualizations for each of thesedimensions.

In Fig. 3 a graphical representation ofa program, its control-flow graph, isshown. In addition the graph containssome information computed by a pro-gram analysis. With the help of this visu-alization, developers can detect certainkinds of errors, so-called stack overflows,in their programs [16].

In Fig. 4, a snapshot of the animatedexecution of a sorting algorithm isshown. The window contains several rep-resentations or views of the data sorted.Algorithm animations are typically usedin education. Most of the animationtechniques do not scale for real softwaresystems.

Finally, Fig. 5 is a pixel map. In this ex-ample the color of the pixel at position(x,y) represents the number of times filesfx and fy have been changed together rel-ative to the total number of times file fxhas been changed. From this figure thedeveloper can see how strongly differentfiles are coupled. We call this kind of cou-

the visual presentations are strikingly dif-ferent.

The expression of an abstract state inan FSM, for example, need not requirethat the state be presented visually in aminimalist fashion. The key task is tostrengthen the metaphor of what itmeans to be a state and the correspon-ding elements of boundary that go alongwith it.

The second question, about culturalbarriers, may lie at the heart of the aes-thetic computing challenge. Computerscientists have been educated with mini-malist figures and text, and so it maycome as a shock to realize that our rep-resentations of formal objects are not asconstrained as we may have thought. Un-til the era of computer graphics and fastcomputers, we had little need to inquireabout what initially appeared to be exoticways to encode formal knowledge. How-ever, this is a challenge not only for com-puter scientists but also for artists, sinceartists should be encouraged to considerthe computer and computing practicesas subject material in addition to raw ma-terial. This suggestion of formal struc-tures as raw or subject material may strikesome artists as a modernist-era agenda;however, the computer and its mathe-matical foundations enable the creationof significantly higher complexity as atool, or as a subject, than paint, paletteknife or chisel ever could.

STEPHAN DIEHL: SOFTWARE VISUALIZATIONSoftware is neither matter nor energy. Itis just a kind of information. Matter andenergy are media that carry informationand thus software. For new software to bedeveloped or existing software to be un-derstood, it has to be projected into a hu-manly readable form—the program text.Note that the program text is not the soft-ware, just a representation thereof. The

Fishwick et al., Perspectives on Aesthetic Computing 135

Fig. 3. Stephan Diehl, programstructure with analysis results. (© Stephan Diehl)

pling evolutionary [17], because it isbased on the change histories of files, todistinguish it from the logical couplingusually used in software engineering. Asthe files are sorted by their containing di-rectories, the pixels form blocks. Theseblocks indicate that files within a direc-tory are coupled, that is, often changedtogether. Software developers are mainlyinterested in the outliers: those pixelsrepresenting couplings between files indifferent directories, such as those la-beled “Patches” in Fig. 5. Outliers can bea sign of a bad system architecture.

We have now seen three very differentkinds of visualizations as they are used forunderstanding software and its develop-ment process.

Now what about the aesthetics of thesevisualizations? It may come as a surpriseto the artistic reader, but there actuallywere some aesthetic criteria involved inthe computing of these visualizations:

In Fig. 3 the number of edge crossingsand bends has been reduced, and di-rected edges are mostly drawn down-wards. In Fig. 4 color is used consistentlyin the different views. While the algo-rithm actually performs discrete changes,the animation of these changes is exe-cuted smoothly. In Fig. 5, color codingbased on the heat metaphor was origi-nally used: The color red showed thehighest coupling, blue low coupling andwhite no coupling at all. In other words,hot files are those that are often changedtogether.

The sole purpose of the above-men-tioned aesthetic criteria is to produce vi-sualizations that convey the informationas clearly and effectively as possible. As a consequence of concentration on theautomatic generation and usability of vi-

pothesis-driven ethos of the medical re-search lab to reflexive practice in the artstudio to the empirically driven environ-ment of mathematics. The aesthetic con-text of each discipline is equally diverse.Visualization in the laboratory differsfrom visualization in the art studio. In themedical laboratory, representation is usu-ally taken literally, as scientific illustra-tion. In the art studio, “representation”is a term and process framed by culturaland art historical theories of represen-tation (for example, an image, sound orobject can signify something without ac-tually sounding or looking anything likeit). In Cell, we document, develop andevaluate the interdisciplinary collabora-tive process itself, including discussion ofour different aesthetic values.

Through studio and laboratory visits,Theise and I identified significant aes-thetic differences. In cell biology the“photographs” of tissue slides have atruth-status and are accepted as “proof”of experiments within papers. Thebeauty of these representations is also val-ued. Figure 6 shows one of Theise’s im-ages, representing skin tissue from afemale mouse who received a bone mar-row transplant from a male mouse. Thetissue has been dyed, as is typical in suchexperiments, and blue nuclei of hair fol-licle lining cells surround the orange,autofluorescent hair shaft (large arrow).Two of these nuclei contain fluorescentlylabeled Y-chromosomes (small arrows)indicating that they derive from the do-nated male bone marrow, not from thefemale’s own original cells. Thus, bonemarrow stem cells have given rise to skin-type lining cells.

As is typical of contemporary artists, Iwas educated to resist beautifying arti-facts and taught that photographic rep-resentations have no truth-status and thattheir meanings are subjective rather thanobjective. Alife systems that have a graph-ical output pose further challenges to no-tions of representation. Such outputs areat least semi-autonomous; images canendlessly change (some might say evolve)as they are produced in real time to rep-resent the software running beneaththem, which is constantly changing as acomplex system of interactions betweenagents takes place.

Converging: Focusing on Behavior across TimeTheise’s research examines the plasticityof adult stem cells and their function, us-ing processes including the analysis ofspecimens of cell tissue. Because the tis-sue is dead at the time it is analyzed, itrepresents a frozen moment in time,

sualizations, the current research on soft-ware visualization is rich in the differentproperties of software that have been vi-sualized but poor in the spectrum of vi-sual metaphors used: boxes, circles, linesand color.

JANE PROPHET: INTERDISCIPLINARITY ANDAESTHETIC DIFFERENCEI consider aesthetic computing througha discussion of the project Cell. Cell is aninterdisciplinary collaboration betweenan artist (myself), a liver pathologist(Neil Theise), a mathematician (Markd’Inverno), a computer scientist (RobSaunders) and a curator (Peter Ride).Cell explores new approaches to the representation of cell behavior, usingmathematics to bridge the gap betweenscientific theory and computer visualiza-tion. Related literature is found in Good-sell’s research [18]. Results include solelyauthored and co-authored papers inpeer-reviewed medical, mathematicalmodeling and simulation journals; math-ematical models of a new paradigm ofstem cell behavior; dynamic simulationsof the mathematical model; art installa-tions; and illustrations of cells and theirbehavior generated using artificial life(Alife) techniques. Here my focus is onthe graphic outputs from Cell’s agent-based Alife simulations.

ContextCell’s practical research context rangesfrom Theise’s medical laboratory to d’In-verno’s and Saunders’s mathematical andcomputer science labs and to my art studio. Each setting has particular em-bedded methodologies, from the hy-


Fig. 4. Stephan Diehl,animation of theHeapSort algorithm.(© Stephan Diehl)

from which researchers hope to under-stand another aspect of stem cell behav-ior and extrapolate further hypothesesto test. My experience as an artist work-ing with time-based media and Alife sug-gested a different approach to assessingstem cell behavior. We therefore de-veloped an Alife engine to enable thescientist to study simulated stem cell be-havior as it happens within the complexsystem of a wider community of cell typesand enzymes. Stills from the real-time 3Dgraphics driven by our simulation (Fig.7) show images that draw on the aes-thetics of medical illustration. The userclicks on a moving cell to see a data read-out. Figure 7a shows photorealistic semi-transparent cells moving, dividing anddying in 3D space. Figure 7b illustratesinformation available when a user clickson a cell or activates a global commandthat makes the data for all cells visible.

The initial Cell collaboration expandedto include d’Inverno, who determinedthe mathematical rules that describestem cell behavior as proposed by Theiseand modified by me (that stem cells canevolve into mature cells of other or-gans such as skeletal muscles, bones andbrains, with unexpected plasticity). Saun-ders interpreted the mathematical modelproducing real-time graphical displayswith me. The aesthetic (designed bySaunders and Prophet) is influenced bymedical and scientific illustration.

Aesthetics and Conceptual ArtThe notion of “an” aesthetic is chal-lenged in Cell. As in much conceptual art,the idea behind Cell (namely, modelingthe behavior of stem cells) and the meansof producing it (via interdisciplinary col-laboration) are more important than the

screen as a framing device; stem cells are“born” at the center of the screen, mov-ing to the edges as they age, divide anddie. The line and single tone for each cellis deliberately abstracted and looks inor-ganic. This is more aesthetically pleasingto me—it makes no attempt to suspendthe viewer’s disbelief. It is obvious thatthis work is computer generated.

By contrast, the 3D version has beeninfluenced by the aesthetics of medicalillustration and its goal of explaining viaprecise observation of the appearance ofthings; this is at odds with my emphasison the behavior of things (in this case,stem cells). This version (using the sameunderlying Alife simulator but with a dif-ferent graphic output) is characterizedby its photorealism: detailed surface ren-dering, depth and transparency (Fig. 7).This was pleasing to Theise, as it was fa-miliar to peers used to seeing video andstills of actual cells recorded via micro-scopes. However, Theise might deter-mine system-level changes in behaviorthat could influence wet laboratory workmore easily via the abstract 2D version.Anecdotal evidence suggests that ab-straction prompts the viewer to look forpatterns of behavior rather than focusingon the activity of individual cells.

JONAS LÖWGREN: INTERACTION DESIGNAND AESTHETICSInteraction design is concerned withshaping the use qualities of digital arti-facts. Another way of putting it would beto say that interaction design is design ofthe digital materials, where the word de-sign is used in the strong sense of ex-ploring all aspects of a possible future:aesthetic and ethical aspects as well asstructural and functional.

It may seem odd, and in fact it is odd,to talk about aesthetic computing as if itwere something new and hitherto unex-plored. All computing is aesthetic in thesense that all use of digital artifacts en-tails aesthetic reactions. To be sure, manycontemporary digital artifacts tend toelicit aesthetic reactions along the linesof frustration, indifference or boredom,but these are aesthetic reactions never-theless, and as designers, we are free toaim for other kinds of reactions if we like.

Of course, there are historical reasonsfor the existence of such blind spots. Theacademic roots of interaction design stemfrom the field of human-computer in-teraction, where the main focus has al-ways been on the task-oriented use ofdigital artifacts and in particular on itsefficiency and absence of errors. It is not

finished work or its (fixed) appearance:“In conceptual art the idea or concept isthe most important aspect of the work . . .all planning and decisions are made be-forehand and the execution is a per-functory affair. The idea becomes themachine that makes the art” [19].

Conceptual art can be defined as the“appreciation for a work of art becauseof its meaning, in which the presentationof shape, colour and materials have novalue without the intentions of the work”[20]. If conceptual art has an aesthetic,it is the dematerialization of the art-object; the object only has value as a ma-terialization of the idea, not in and ofitself. Mathematics and computing sci-ence can both operate without materi-ality and can describe the immaterial,which is one reason why there may be amutual attraction between computer sci-entists, mathematicians and artists work-ing conceptually using digital media.

Aesthetics and VisualizationTo date, the real-time graphic represen-tation arising from Cell’s complex adap-tive system (Fig. 7 and Color Plate A No.2) is not what I would consider a piece offine art in its own right. This is scien-tific visualization, or graphics, informedby my aesthetic framework. I want thegraphic look and feel to reflect the un-derlying software, to draw attention tothe essence of the idea or concept. Fromthis standpoint the 2D Java version(Color Plate A No. 2) is a more satisfyingoutcome than the 3D version. Here, mycolor palette has white stem cells. Cells in each lineage tree are a particular hue,becoming darker as they differentiatethrough division. Lines signify geneswithin circles defining cells. I use the


Fig. 5. Stephan Diehl,number of simultaneouslychanged files. (© StephanDiehl)

surprising that the image of computeruse is sometimes simplified to the extentthat useful, efficient and error-free use is seen as the whole picture. The disre-garded parts of the picture, including aes-thetic qualities of the use, will thenappear as new when they are broughtinto consideration.

How, then, should aesthetic use quali-ties be dealt with in interaction design?A common fallacy is to equate aestheticswith pleasing visual design. To be sure,there are design situations in which theimmediate visual impression is an im-portant factor in determining the out-come of the interaction. A good exampleis a web shop with one-time customerswho ideally should spend money on theirfirst and only visit. The visual design ofthe web shop’s front page is crucial in es-tablishing the right combination of de-sire and credibility to actually makecustomers enter, shop and pay.

In nearly every other case, however, weneed to realize that a digital artifact isconstituted primarily not by its static vi-sual design but by its dynamic gestalt—the character of the interaction it al-lows over time. Digital design materialsare temporal in this respect, at least as much as they are spatial. Below are threeexamples of aesthetic qualities in the useof digital artifacts. They serve as illustra-tions of how the aesthetics of interactiondesign can be approached and articu-lated for further debate as well as forpractical application in concrete designsituations.

Pliability is the quality of plasticity, ofmalleability of the digital artifact underthe hands of the user. A set of informa-

ical “window” fills again with metaphori-cal condensation. If Granny happens tosee the signs before they are gone, shecan reply in the same way. A sign on thedisplay disappears completely in a fewhours.

The strength of the Hazed Windowsconcept is not in its focus on lightweight,emotional communication, which hasbeen realized thousands of times before,but rather in the elegant questioning ofthe hidden core assumptions of com-puter-mediated communication. Whenwe deal with e-mail, we devote our full at-tention to it. We expect messages to per-sist until we choose to file or delete them.In short, we approach the communica-tion situation as a binary task. The HazedWindow hints at the possibility of a newmiddle ground, a new approach to com-puter-mediated communication in textand pictures that exhibits a much greaterdegree of fluency.

Seductivity refers to the captivatingqualities of a digital artifact. Followingthe seminal analysis by Khaslavsky andShedroff [26], seduction is described an-alytically as a process of enticement, re-lationship and fulfillment. Enticementconcerns grabbing attention and makingan emotional promise. The subsequentrelationship is based on making progresswith small fulfillments and more prom-ises, possibly lasting for a long time. Thefulfillment, or ending, involves makinggood on the final promises and endingthe experience in a positive and memo-rable way.

It should be clear from the descriptionabove that seductivity, in the sense usedhere, is a quality that crucially dependson the dynamic gestalt, the temporalqualities of the interaction. It has noth-ing to do with sexually explicit pictureson the front page of a web site. The ex-ample offered by Khaslavsky and Shedroffis the Visual Thesaurus [27] by PlumbDesign. It is a web application that du-plicates the contents of a traditional the-saurus but takes on entirely new qualitiesby virtue of its interactive properties. Theuser explores words, their synonyms andeventually the transient nature of lan-guage itself by navigating a beautifully an-imated network of words and theirinterrelations.

Khaslavsky and Shedroff argue that theVisual Thesaurus is seductive in the sensethat it offers surprising novelty, goes be-yond obvious needs and expectationsand creates an emotional response dueto its visual and interactional beauty (en-ticement). It connects to personal goalsthrough the fascination of words andconcepts and promises to fulfill those

tion is pliable to the user if it feels like aresponsive material, a matter for inquirythat can be manipulated and experi-enced in a tactile sense. Pliability con-tributes to a highly involved process ofexploration where the loop betweensenses, thought and action is rapid andphysical rather than elaborate and ab-stract. I make a small, quick move—thematerial shapes and responds—I noticesomething new—I make another move—and so on. Ahlberg and Shneidermanwere the first to articulate this quality,under the label of “tight coupling” [21].

In the interpretation above, pliabilityconcerns the micro-qualities of interac-tion, the qualities of the surface. Exam-ples of interaction design aiming atpliability include the influential conceptof dynamic queries [22] as well as themore recent interaction technique Sens-A-Patch [23]. Another aspect of pliabil-ity has to do with the user’s capacity toact freely and shape the material accord-ing to the larger situation at hand, suchas annotating the margins of a paperform to communicate something outsidethe rigid boundaries of the form itself.As Henderson and Harris [24] point out,this kind of deep pliability is often un-necessarily lost in the transition from pa-per to computer systems in, for example,administrative work. It can be straight-forwardly observed that many existingadministrative systems could be extendedto accommodate free-form annotationand the equivalent of sticky notes.

Fluency as an aesthetic quality of digi-tal artifacts is brought to the fore in re-lation to the increasingly pervasive digitalinfrastructure. Use is not necessarily a pri-mary activity at the focus of attention; itis not a binary variable in which a digitalartifact is either used or not. With ubiq-uitous and mobile computing, it becomesmore of a dance among multiple repre-sentations and mediations. Streams ofinformation flow between center and pe-riphery as we move through the shiftingenvironments of everyday life and work.Transitions need to be graceful andnondisruptive.

As a simple but conceptually powerfulexample of fluency, consider the HazedWindows concept by interaction de-sign students Trine Freiesleben, MiskaKnapek and Henrik Moberg at MalmöUniversity [25]. The idea is simply to of-fer a more lightweight and transientcommunication channel, for instance be-tween a little girl and her granny who livein different cities. The girl draws or writeswith her finger on a display in her home.Her signs are redrawn at Granny’s displaybut gradually fade away, as the metaphor-


Fig. 6. Hair follicle with lining cells derivedfrom bone marrow. (© Neil Theise)

goals (relationship). The casual viewermay discover deeper meanings in look-ing up a word by apprehending the mul-tidimensional and dynamic relationshipsbetween concepts (fulfillment).

An Analysis of PerspectivesAs suggested in the title of this paper, thereader is meant to come away with thediffering views of four authors. What arethe connections, and is there a synthesisamong the varying views? What waslearned during the workshop? Startingwith the second question, we learned thatthe terms aesthetics and computing arecomplex and that aesthetic computing isnot merely the straightforward applica-tion of one field to another but also a self-reflective activity. By juxtaposing twowords in a new way, we suggest that theart, design and computing communitiesmust reevaluate the very meaning of thewords aesthetics and computing and notonly what it means for them to be com-bined as a phrase.

To summarize the key perspectives al-ready presented:

• Fishwick stresses the use of aestheticstyle and genre within traditionalnotations for computing artifacts.Aesthetics in both computing andmathematics can be pushed in the di-rection of artistic styles to augmentcognitive aesthetics.

• Diehl discusses the use of visualiza-tion and a specific type of aestheticfor software design and execution di-agrams. There is already a great dealof research on applying visualization,and one particular sort of aesthetic(optimality), to software design.

• Prophet defines a new type of teamcollaboration, a kind of “aestheticcomputing in action,” in which trans-

representations without sacrificing ab-straction. Mathematics and software areconcerned not with visual abstraction but with conceptual abstraction. For ex-ample, a stack is a common abstract data structure used in computing. Thata stack can be represented in 3D using an aesthetic form that is meaningful to a group of users (that is, a culture) doesnot defeat the conceptual abstraction.Rather, the 3D concretization of the stack brings to the surface the inherentmetaphor that led to 2D diagrammaticstack representations in the first place;the concept of a stack is rife with meta-phor—using artistic styles and famil-iar objects brings these metaphors to life within the same abstract conceptualsphere of knowledge.

The key conflict is cultural—scientistshave been trained with visually abstractrepresentations for the sake of materialand labor economy. With the advent ofnew display technologies, it is no longerinconvenient to form novel 3D repre-sentations while simultaneously preserv-ing abstraction. There is a need forcomputer scientists to embrace a widerdefinition of aesthetics [28], one influ-enced substantially by the arts. This cul-tural transition may be difficult but canbe gradually facilitated by the kind of evolutionary collaboration described byProphet, in which artists and scientistswork closely together. In such a fashion,all parties become accustomed to abroader definition of aesthetics. Theneed for this broader definition is re-inforced by Löwgren’s discussion of in-teraction qualities, as well as recentmovements in the design community[29,30], where emotion, style and visualappeal are found to be equally as impor-tant as function. There are additional re-

disciplinarity leads to new collabora-tions, influencing the computingprocess. Applying aesthetics to com-puting is a matter of process and hu-man integration, not just a staticresult from the application.

• Löwgren broadens the traditionalview of usability to include the aes-thetic qualities of the digital designmaterial. As a community, computingprofessionals must balance cognitiveand sensory qualities in design.

For the most part, there is generalagreement among the authors that eachsection corresponds to a separate axisupon which aesthetics can be seen to ap-ply to computing; however, there is onesignificant issue raised by Fishwick andDiehl that deserves explanation. Diehldiscusses the state of the art in softwarevisualization and emphasizes the currentway in which software designers generallyview aesthetics—as optimality considera-tions inherent within the visual design.For example, an aesthetic can be definedto minimize the number of crossings ina 2D graph, or to ensure that all linesleading into a node are equidistant. Diehlalso quotes Brooks’s idea that “softwareis invisible.” Diehl’s views on characteris-tics of design accurately reflect how thevast majority of computing professionalsview aesthetics and computing artifacts—as primarily mental constructs and, ifnot mental, focused on formal qualities(“formal” is used here as in the artist’slexicon).

Fishwick takes issue with confining therole of aesthetics to such a narrow inter-pretation and suggests that future repre-sentations for software models shouldcapture more personalized and stylis-tic forms, given that new technologieshave increased our ability to afford 3D


Fig. 7. Jane Prophet, Cell 3D Alife representations (a, left; b, right). (© Jane Prophet. Rendering design by Rob Saunders.)

sults in the foundational cognitive roleof metaphor [31]. If richer designs fostera metaphorical understanding, then suchdesigns ought to be encouraged and in-vestigated. We further suggest that whatis relevant for product design is also rel-evant for the design of more abstractcomputer programs, data structures andmodels. Both types of designs reflect anecessary mixture and balance of formand function.

Certainly, optimal designs for achiev-ing a goal or satisfying a particular pur-pose are still useful when considering theproblem of design from the perspectiveof performance. However, this functionalbias of design must be carefully balancedwith what one learns by exploring aes-thetics in the arts: Design is also aboutuser choice, culture and the freedom toincorporate arbitrary interactive 2D or3D icons to represent formal structure.Asking, for example, whether it is moreappropriate to use a square versus a cir-cle in a design presupposes that there ex-ists an optimal design condition, when inmany instances, the design condition isstrongly oriented toward user preferenceand not performance. Designs must beuseful, yet this does not imply universalusability for an objective purpose, but in-stead utility for a segment of the popu-lation. This type of free-form design iscommon in automobiles, architectureand kitchen products; they all must bothbe usable and contain aesthetic qualitiesfor whoever purchases them. We contendthat software and formal structures usedin computing are no different in this re-gard. Representation, in general, hasalways been driven by the state of tech-nology and the economy. When all onehas is clay, then clay marks must be usedfor representation. As our display and in-teraction technologies evolve, our rep-resentations can also advance, not toreplace older representations, but to aug-ment them in the same way that new me-dia have augmented, or remediated [32],old media.

CONCLUSIONSOne of the key tensions in our discussionis the interplay between that which artistsproduce versus that which computer sci-entists produce. Artists have an agendabased on a wide variety of styles and aes-thetics, from formalism and cultural ex-ploration to capturing social, political oreconomic aspects of phenomena. Com-puter scientists primarily are after high-utility artifacts; if something is useless,most mathematicians and computing

less our current state of progress. An il-lustrative example is our introduction ofthe dimension of utility as one of the waysin which the fundamental values of thedisciplines of art and computing differ.

The most advanced integrative stage,according to Jantsch, is interdisciplinar-ity. It involves direct cooperation in bothdirections where the outcomes typicallycould not be achieved entirely withoutany of the disciplines involved. It also en-tails the formation of new concepts, prac-tices and values that transgress thetraditional boundaries of these disci-plines. It is our firm conviction that theencounter between art and computingholds the potential for interdisciplinarityin this strong sense, and our ongoing di-alogue might represent a step in that di-rection.

Acknowledgments

The authors are indebted to Dagstuhl, Germany,where many of the ideas in this paper flourished ina synergistic weeklong session. We also would like toacknowledge the Leonardo organization for their co-sponsorship of the event. Paul Fishwick would like tothank his graduate students in the RUBE Project(Minho Park, Jinho Lee and Hyunju Shim), the Na-tional Science Foundation, working via grant EIA-0119532, and the Air Force Research Laboratory,working via grant F30602-01-1-05920119532. JaneProphet’s contribution is part of the Cell collabora-tion and has been written following discussion withMark d’Inverno, Peter Ride, Rob Saunders, NeilTheise and Katrina Jungnickel. Cell research has beenconducted with awards from the Wellcome Trustsciart, Shinkansen Future Physical “BioTech” and theQuintin Hogg Trust.

References

1. Aesthetic Computing Dagstuhl Seminar 02291,15–19 July 2002, <http://www.dagstuhl.de/02291>.

2. The Dagstuhl Aesthetic Computing Workshop,“Aesthetic Computing Manifesto,” Leonardo 36, No.4 (2003) p. 255.

3. M. Kelley, ed., Encyclopedia of Aesthetics, 4 vols. (Ox-ford, U.K.: Oxford Univ. Press, 1998).

4. J. Hadamard, The Psychology of Invention in the Math-ematical Field (Princeton, NJ: Princeton Univ. Press,1945).

5. D. Knuth, The METAFont Book (Reading, MA: Ad-dison-Wesley, 1986).

6. E. Gamma et al., Design Patterns: Elements of ReusableObject-Oriented Software (Reading, MA: Addison-Wesley, 1995).

7. S. Card, J. Mackinlay and B. Shneiderman, eds.,Readings in Information Visualization: Using Vision toThink (San Francisco, CA: Morgan Kaufmann, 1999).

8. S. Diehl, ed., Software Visualization (New York:Springer Verlag, 2001).

9. J. Stasko et al., eds., Software Visualization: Pro-gramming as a Multimedia Experience (Cambridge, MA:MIT Press, 1998).

10. T. Kim, J. Lee and P. Fishwick, “A Two-Stage Mod-eling and Simulation Process for Web-Based Model-ing and Simulation,” ACM Transactions on Modelingand Simulation 12, No. 3, 230–248 ( July 2002).

11. P. Fishwick, “Aesthetic Programming: Crafting

professionals will tend to shy away fromit. However, there is a line that stretchesfrom “no use” to “full use,” if there canbe such unambiguously defined things.To see this, we need to step back to thedefinition of aesthetic computing: the ap-plication of art practice and theory tocomputing.

There is no reason why this applicationmust be targeted on artifacts of high gen-eral utility. By exploring the boundariesand interstices of the “use range,” wethink that we can enhance both art andcomputing. Artists will become more fa-miliar with elements of computing, suchas data structures, programs and archi-tectures, and even the core mathemati-cal structures upon which computing isfounded. For example, what artists pro-duce [33,34] reflects a computing es-sence, whether or not the result is ofimmediately obvious utility. To the extentthat a piece makes one reflect, it is usefulin exploration, creativity and education,and so even the term “use” or “usability”becomes suspect. Thus, aesthetic com-puting for the artist can range from soft-ware art (a fairly new movement definedby artists producing their own programsor languages) to representational art inwhich the computing element becomesthe subject of the art, rather than the ma-terial for the art as in software art. It isthis ability to weave through the webs ofutility and computing that makes aes-thetic computing a unique enterprise.

For the computer scientist, what theartists produce will be fertile ground forrepresentation, interaction design andhuman-computer interfaces in general.While through software art some artistsare becoming computer scientists of asort, we can also observe the converse sit-uation as some computer scientists be-come artists.

On a more general level, the en-counter between art and computing maybe studied in terms of disciplinary rela-tions. Jantsch [35], the originator of suchconcepts as multi- and interdisciplinarity,views disciplinary integration as an evo-lutionary hierarchy. If a traditional disci-plinary approach is specialization inisolation, then multidisciplinarity sim-ply refers to adding different disciplineswithout any direct cooperation betweenthem.

The next step up the evolutionary lad-der is crossdisciplinarity, where one dis-cipline supports the other within theother’s own discipline. Our discussionabove, and the identification of fieldssuch as software art and aesthetic com-puting, might suggest that this is more or


Personalized Software,” Leonardo 35, No. 4, 383–390(2002).

12. J. Hopkins and P. Fishwick, “Synthetic HumanAgents for Modeling and Simulation,” Proceedings ofthe IEEE 89, No. 2, 131–147 (February 2001).

13. J. Hopkins and P. Fishwick, “Exploiting the Agent-Based Metaphor in Software Visualization using theRUBE System,” Journal of Visual Languages and Com-puting 14, No. 1, 97–117 (February 2003).

14. P. Fishwick, Simulation Model Design and Execution:Building Digital Worlds (Englewood Cliffs, NJ: Pren-tice-Hall, 1995).

15. F.P. Brooks, “No Silver Bullet,” IEEE Computer 20,No. 4, 10–19 (April 1987).

16. aiCall StackAnalyzer, AbsInt GmbH <http://www.aicall.de>.

17. T. Zimmermann, S. Diehl and A. Zeller, “How His-tory Justifies System Architecture,” Proceedings of In-ternational Workshop on Principles of Software EvolutionIWPSE 2003, Helsinki, Finland, 1–2 September 2003(IEEE Computer Society Press, 2003).

18. D. Goodsell, <http://www.scripps.edu/pub/goodsell/>.

19. Sol LeWitt, “Paragraphs on Conceptual Art,” Art-forum 5, No. 10, 79–83 ( June 1967).

20. J. Kosuth, in G. Guercio, ed., Art after Philosophyand After: Collected Writing 1966–1990 (Cambridge,MA: MIT Press, 1991).

21. C. Ahlberg and B. Shneiderman, “Visual Infor-mation Seeking: Tight Coupling of Dynamic QueryFilters with Starfield Displays,” Human Factors in Com-

31. G. Lakoff and M. Johnson, Philosophy in the Flesh:The Embodied Mind and Its Challenge to Western Thought(New York: Basic Books, 1999).

32. J. Bolter and R. Grusin, Remediation: Understand-ing New Media (Cambridge, MA: MIT Press, 1999).

33. J. Maeda, Design by Numbers (Cambridge, MA:MIT Press, 1999).

34. Processing Language: <http://www.processing.org>.

35. E. Jantsch, Technological Planning and Social Fu-tures (London: Associated Business Programmes Ltd,1972).

Manuscript received 15 October 2003.

Corresponding author Paul Fishwick is withthe Computer and Information Science De-partment at the University of Florida.

Stephan Diehl is with the Computer ScienceDepartment at the Catholic University Eich-statt, Germany.

Jane Prophet is Director of the Centre for ArtsResearch Technology and Education(CARTE), University of Westminster, London.

Jonas Löwgren is with the School of Arts andCommunication, Malmö University, Malmö,Sweden.

puting Systems (CHI ’94 Proceedings) (New York:ACM Press, 1994) pp. 313–317.

22. C. Ahlberg, C. Williamson and B. Shneiderman,Dynamic Queries for Information Exploration: An Imple-mentation and Evaluation. Human Factors in Comput-ing Systems (CHI ’92 Proceedings) (New York: ACMPress, 1992) pp. 619–626.

23. J. Löwgren, “Sens-A-Patch: Interactive Visualiza-tion of Label Spaces,” in E. Banissi et al., eds., Pro-ceedings of the Fifth International Conference onInformation Visualization (IV2001) (Los Alamitos, CA:IEEE Computer Society, 2001) pp. 7–12. Web:<http://webzone.k3.mah.se/k3jolo/Sens-A-Patch>.

24. A. Henderson and J. Harris, “Beyond Formalism:The Art and Science of Designing Pliant Systems,” inK. Kaasgard, ed., Software Design and Usability (Copen-hagen, Denmark: CBS Press, 2000).

25. T. Freiesleben, M. Knapek and H. Moberg, Hazed Windows, 2001 <http://webzone.k3.mah.se/kid01016/projekt_forskningstema.htm>.

26. J. Khaslavsky and N. Shedroff, “Understandingthe Seductive Experience,” Communications of theACM 42, No. 5, 45–49 (1999).

27. Visual Thesaurus: <http://www.plumbdesign.com/thesaurus>.

28. D. Norman, Emotional Design: Why We Love (orHate) Everyday Things (New York: Basic Books, 2003).

29. Norman [28].

30. P. Jordan, Designing Pleasurable Products: An Intro-duction to the New Human Factors (Boca Raton, FL:CRC Press, 2000).


CALL FOR PAPERS

Live Art and Science on the Internet

The Internet has become a venue and medium for art as a means to broadcast ideas to a worldwide audi-ence. Leonardo and Guest Editor Martha Wilson seek texts on the subject of “Live Art and Science on theInternet” for a series of special sections in the international journal Leonardo, both in print and online.

As artists and others produce live art on the Internet, liveness, presence, mediatization, online activism,surveillance and identity/gender, among other issues, are being explored. We seek texts documenting suchwork, as well as texts on the history of this field of practice and on the vocabulary being used to describe it.We also seek texts from scientists who have used the Internet to conduct science investigations live on-line.

Guest Editor Martha Wilson and her peer review committee seek Statements (500 words plus one image describing one work), Notes (2,500 words plus 6 images describing a body of work), Galleries (750-word curator’s introduction plus up to 10 images by individual artists, each with a 200-word caption) and Articles (5,000 words plus 12 images). Texts describing the work of a living artist or scientist must be written by the artist or scientist him/herself, with a co-author if necessary.

This call for papers is open for 2004, 2005 and 2006.

Please send an initial statement of interest with a brief explanation of your project to Martha Wilson:<[email protected]>. For author guidelines, follow the link “Info for Authors” on LeonardoOn-Line <www.leonardo.info>.

Documents

Perspectives on aesthetic computing