Embed Size (px)
Citation preview
Digital Creativity2002, Vol. 13, No. 2, pp. 65–70
1462-6268/02/1302-0065$16.00© Swets & Zeitlinger
Visual communication andinteraction
Janni Nielsen
Copenhagen Business School, Denmark
Visual communication is the essential founda-tion for the design of IT interfaces but there canbe very different understandings of this topic.Different academic traditions have theirperspectives and their vocabularies throughwhich the topic is conceptualised and under-stood. Practitioners have other concerns andhave specific tools and techniques that enablethem to communicate visually.
Though technological developmentshave led to new interdisciplinary approaches,these developments have yet to produce a trulyinterdisciplinary foundation. Because the field isstill new we each draw, sensibly enough, onwhat we already know and have mastered whenapproaching this new area.
The following introduction to theinterdisciplinarity of visual communication andinteraction presents perspectives from film andmedia analysis, from communication studies,architecture, the development of digital tech-nologies and informatics. It should be read as anattempt to contribute to the interdisciplinarydevelopment of a theoretical framework forunderstanding visual interaction. However,embedded in the framework is a need forfurther qualification — by integrating a psycho-logical understanding of the human being as anemotive, sensuous, bodily, intellectual andexistential being. A possible framework for theinitial steps in this work is suggested.
1. Fragmentary evidence: affordance forconstruction of assumptions and interpretations
In his article, Interacting with pictures: film,narrative and interaction, Steven Boyd Davisdiscusses the knowledge gained in film makingand how the design of interactive media games
may benefit. His perspective is that of the film-maker, and the manufacturer of an imaginaryvision. He reminds us that film is essentiallypictorial, and it does not show, but ratherconstructs an aspect of vision which the filmviewer needs or wants in order to make sense ofthe film. The film maker does this by offering,“fragmentary evidence, organised with a view toaffording certain assumptions and interpreta-tions, and the film viewer (partly on the basis ofshared conventions) duly makes those interpre-tations”.
The tools in the creation of the fragmen-tary evidence are the shot, the camera view (e.g.close up, the camera as the eye of the spectator)the information and the affective expressivity aswell as the contextualisation of the shot. Theseare brought together in the editing, and thecontextual information of a shot, the authorstates, provides what is psychologically neces-sary. From here, Boyd Davis turns to interactivemedia games and suggests that it may be fruitfulto evaluate them within the same frameworkthat is as essentially pictorial. He argues that thedesign ideally must make the users “believe thatthey are interactive observers of a world”. BoydDavis points out that one of the problems hereis the constraining spatiality in interactivemedia. The author suggests that a way ofexploring this is the ‘make believe’ which is to befound in the ‘spatial maturity’ of film making.The essence of this is ‘psychological immersion’.
The paper is written from the perspec-tive of the film maker, but the film viewer isever present. But the film viewer is not anobserver, she is a participant and the meaningconstruction she engages in is intimately tied tothe visual viewing. However, in relation to
ed
ito
ria
l
Dig
ital C
reat
ivity
, Vol
. 13,
No.
2Nielsen
6 6
interactive media games it must also be tied tointeraction, the user’s visual interaction with themedia. Behind Boyd Davis’ theory lays theassumption that the human being is capable ofmaking sense out of the film maker’s fragmentedevidence. This is true, but not specific to filmviewing. It is a general human competence ofmeaning construction. Human beings live a lifewhich — seen from outside — is fragmented:fragmented information, fragmented interac-tion, fragmented communication, whichfragment our actions, etc. Seen from inside thehuman mind, however, it is coherent because ofthe human ability to make sense. We areengaged in a constant process of sense making,of seeing coherence where coherence was absentand constructing life as meaningful. This lifebecomes meaningful because we are not observ-ers; we live life and understand it from anembodied point of view. We are immersed inlife, whatever actions we engage in — we are notoutside it (Winograd and Flores 1987).
Boyd Davis brings into the concept ofvisual communication the role of the tools andtechniques of film making. He also points toessential dimensions in the viewing of the filmmaker’s imaginary vision. However we needtools in order to understand how the psycho-logically necessary is presented in fragmentaryevidence, and how this affords the necessaryconstruction of assumptions and interpretationsin the film viewer?
2. The inexpressible aesthetic functionThe paper by Lisbeth Thorlacius, A model ofvisual, aesthetic communication focusing on websites, may bring us a possible step closer to a toolwhich will enhance our understanding of howthe psychologically necessary is presented infragmentary evidence. Thorlacius’ paper istheoretical and focuses on the visual andaesthetic aspects of communication. The aim isto develop a foundation for the analysis (andconstruction) of web sites. Concurrent tointroducing a visual, aesthetic communicationmodel the author also identifies a second aim: a
theoretical discussion of the functions oflanguage in Roman Jacobson’s communicationmodel and their relation to visual communica-tion.
Thorlacius introduces and distinguishesbetween the expressive function, which relatesexclusively to the addresser, and the emotivefunction, which is ascribed to both the addresserand addressee (the film maker and the filmviewer, the designer and the user, etc). Theemotive function is further analysed andspecified as: the addresser may posses/may notposses expression of emotions which he evokesin the addressee; or expressions of emotions mayunintentionally be evoked in the addressee bythe addresser. This differentiation is interestingbecause it opens the possibility of an in-depthanalysis of the emotive and aesthetic expressionsand perceptions in both the film maker and thefilm viewer. Hence the ‘psychologically neces-sary’ which Boyd Davis speaks of, and which ispresented in fragmentary evidence, may alsocome within our grasp.
Thorlacius also develops the concept ofthe formal aesthetic function, which is under-stood as the ability of visual language to com-municate aesthetic expression — the expressible.This concept conditions its opposite, namely theinexpressible aesthetic function, the origin ofwhich is the senses and feelings. It is defined asthe ability of visual language to communicatethat which cannot be classified. The latter isimportant because the author’s interest is themutual (not the individual) experience ofinexpressibility. It is the addresser and theaddressee’s (or the designer’s and the user’s)experience of not being able to express thatwhich they perceive. Returning to Boyd Davis’analysis, these concepts may give us a handle foranalysing the emotive expressions of the filmmaker that afford a process of necessary con-struction of ‘assumptions and interpretations’ inthe film viewer.
Thorlacius’ paper is written from theperspective of communication theory. However,embedded in the text is a perception of the
6 7
Digital C
reativity, Vol. 13, No. 2
Editorial
human being on the other side of the screen.Thus she talks of the inexpressible which is notjust the designer’s experience —but also theuser’s experience of not being able to expressthat which they are perceiving. Yet Thorlacius,when turning to the concept of interaction,defines it from the point of view of the user, butas physical — not as interpersonal or as mental— interaction. It may be very operational toidentify interaction as physical. However, thecommunication of a web site cannot be sepa-rated from the mental interaction. It seems tome that a web site has as a prerequisite, that ahuman being starts interacting with it. Some-body has to key in the site address, somebodyhas to click and navigate inside the site, but aprerequisite to this is that that somebodyinteracts psychologically with the site. It is notsufficient for a hand to click on the mousebuttons in order for the communications towork. Here the architectural concept of visualcommunication as embodied spaces, the essenceof which is immersion, may help us.
3. Embodied perceptual experience technologiesAndreas Lueschers’ perspective is architectureand the educational models (design representa-tions) as opposed to those of the computer. Thephysical trace is the title of his paper and hefocuses on image representations (in architec-ture, theatre, painting, sculpture, etc). It isespecially “the representation of three or moredimension of information on two-dimensionaldisplay surfaces and the simulation of a fullspatial experience with horizontal, vertical andtemporal extensions” which the discussionmoves around. Luescher introduces us tohistorical evidence of visualisation techniques ofenvironmental representation, which incorpo-rate possibilities for “sensual engagement,material exploration, and a connection with thereal”. This involves synthesis of touch, soundand movement, and experience, because theserepresentations require the “participation of thespectator; for dynamic, physical involvement inthe process of creation”. Experiences which
would result in bodily, sensual, emotional andintellectual understandings.
One such representation is the pano-rama, for example in paintings, the experienceof which is “qualitatively different from lookingat a picture isolated within a frame… Thepanorama made it necessary to move not onlyone’s eyes and head, but also one’s body in orderto assimilate the vast continuous picture”.Hence the history of representations alsobecomes the history of immersion in the image,and it is not just the history of visuals but also ofembodied perceptual experience technologies.This is also the case in architectural designwhere experiments with representations turnedexperiences into bodily, sensuous, time andspatial constructions. This was and is done byletting students construct in 1:1 scale in solidmaterials, instead of just drawing 1:10 on (flat)paper. From this basis Luescher stages thediscussion and he questions the claimed revolu-tionary potential of the computer of imagingand the promise of immersion. He points outthat computer representations remain screen-bound, small-scale and impenetrable and treatsarchitectural constructions as drawings. Not thatLuescher rejects the computer in education —but he argues that when and how to use com-puters must be seriously contemplated.
With Luescher the representationsbecome not just visual but material and theinteraction is both physical and psychological.The architectural concept of visual communica-tion as embodied spaces, which requires immer-sion, takes the visual communication out of thespectator or observer perspective and into theperspective of the actor — interacting withphysical material. With this the embeddedassumption of a psychological being inLueschers’ description becomes more visible.Knowledge of the psychological faculties at workhere and the subjective experience would be yetanother, but essential contribution to ourunderstanding of the interaction. However, weneed to be able to understand not just theembodied experience in the construction of the
Dig
ital C
reat
ivity
, Vol
. 13,
No.
2Nielsen
6 8
physical spatial representations, and the embod-ied movement in physical spatial representa-tions. Exploring when and how to use comput-ers may be done by studying the representationsof visual communication on computer and theexperience of persons interacting with thesevirtual representations. Keyboard and screenbased computers can never be embodied spaces.However, I will take the liberty of rewordingLueschers’ statement into an entirely differentclaim; computers offer representations, whichembed possibilities for sensual engagement,virtual exploration of material and a connectionwith the electronic images and animations of thedesigner’s imaginary vision. These representa-tions require the participation of the user andthe experiences may enhance intellectual,sensual, emotional and even bodilyunderstandings. However, exploration of andexperiments with the medium are necessary ifwe are to develop a better understanding.
4. Movements in virtual spaces and that whichcannot be represented
The paper, The eye of the user: the influence ofmovement on users’ visual attention, by HellePetersen and Janni Nielsen reports on one suchexperiment. They explore the communicativepotential of movement in interface design.Concurrently they discuss their exploration oftwo specific user test techniques: eye trackingand mind taping.
The representations they work with arevisualisations of movements, which weredesigned to:1.catch the eye of the user;2.direct the user’s attention and provide an
aesthetic experience; and3.enable the user to decode relevant information
without distraction, irritation or prevention.In experimental terms, Petersen and Nielsendefine movements as graphical objects, whichshift location, transform shape or changetexture. They draw on a classification of move-ment in film production, and distinguish
between primary movements (moving object),secondary movements (screen motion) andtertiary movements (movement between pages).These movements may be initiated automati-cally, in which case the user has no control, orthe movements may be initiated by the user.The authors develop a set of six guidelines fordesign and report on the design and testing oftwo kinds of movement: the automatic move-ment of objects (primary movement), and userinitiated movement between pages (tertiarymovement) followed by an automatic movingobject (primary movement).
They ask the essential question: whatgoes on in the user’s mind, and they describe thetwo techniques they use in the study: eyetracking, using a headset with video recordingsof respondent’s eye movement, and mind tape.The latter may be described as a conversationalinterview, where recordings of the respondent’seye movements are the object and act as thetrigger for a conversational interview betweenrespondent and researcher.
The analysis — though the data is rathercursory — seems promising: movement holdsmuch promising potential as a communicativetool in interface design. The techniques fortesting not only give access to cognitive proc-esses that run associatively while the userinteracts with the system, but equally interest-ing, the techniques seem to give us some insightinto tacit processes in human cognition — or tothat which cannot be expressed.
5. Using visualisation of performer’s movementin physical space to control light and sound
Before we get to the possible contribution froma psychological theory to the concepts of visualinteraction, I want to introduce Jeff Burke’swork at UCLA’s HyperMedia Studio. Hisapproach is also experimental, and he exploresnew digital technologies for traditional produc-tion of theatre, film and television. In his paper,Interactive performance environments and thevisualisation of actor movement, he describes thedevelopment of performance spaces with
6 9
Digital C
reativity, Vol. 13, No. 2
Editorial
wireless sensing systems that adapt to the actionsof performers. In essence it is a visualisation, anautomatic diagramming of human motion as itdevelops over time. Burke wanted to use theperformer’s position and movement to controltheatrical lighting and sound. For a productionof Ionesco’s Macbett he developed a system ofnetworked software modules where the perform-ance tracking was done with a small wirelessmicrophone worn by each performer. Hewanted to understand, not just how the per-former moved but also how long the performerwould stay at different locations on the scene.Experiments with the two-dimensional visualisa-tion system allowed him to see how the stagewas used over time. With the aid of thevisualisations and the possibility of determiningthe position of the performer he developed somevery special magic staffs for the witches inMacbett. He built the position and motiontracker into the foam head of the staff, and tothe audience the power of the two witches musthave been a delightful fright. The movement ofthe staff controlled the sound and lightning.One witch would conjure powerful thunder andlighting by raising her staff quickly in the air —the speed and strength of her thrust controllingthe effect. The other witch could create ripplesof darkness, colour shifts and the sound ofwhirling wind by swirling her staff. Again theintensity of the effect was controlled by thepower of her movements.
To understand the relationship betweenthe physical space and the performer’s move-ment is important to theatre scholars andpractitioners, but is not limited to this field. Forthe creators of the buildings we work and livein, understanding the relationship between thephysical spaces that the architect designs and theway people move in them may improve thefunctionality — seen from a user’s point of view.For interface design, understanding the relationbetween physical space and the movement ofobjects becomes important for the design ofdynamic representations, which may embedpossibilities for sensual and emotional engage-
ment. Maybe even the possibility of virtualexploration of the electronic images and theanimation of the designer’s imaginary vision.The fundamental requirement is humanparticipation and studying the interaction withsymbolic representations may enhance ourunderstanding of how the computer-mediatedrepresentations in the interface interact withintellectual, sensual, emotional and bodilycognitive processes.
6. Visual interactionVisual communication is mainly understood bythe authors as visual, though it may be calledimage representation, visual language, pictures,or pictorial depending on the academic perspec-tive. In their discussion of visual communicationthe common theme which emerges is thequestion of representation — and I would sayespecially, that which cannot be represented.
That which can be represented seems tobe that which we can see, feel and/or touch andmove into. However, it is the other theme, thatwhich cannot be represented, which emerges,unsurprisingly, as the concept which is onlylightly discussed. It is labelled the psychologi-cally necessary, the psychological immersion, theemotive function, the inexpressible aestheticfunction, or described as psychological interac-tion, and as tacit processes in human cognition.But the concepts are only cursorily addressed. Isthis because being that which cannot be repre-sented, it also becomes that which cannot bediscussed? The theory of visual aestheticcommunication with its explanation offers us away out. Thorlacius argues that inexpressibleaesthetic function, the origin of which is sensesand feelings, is the ability of visual language tocommunicate that which cannot be classified.And this is a mutual experience of inexpressibilitywhich both the designers and the users experi-ence, despite the fact that they are perceiving theinexpressible.
But how does this joint experience comeabout? The understanding of film making as theconstruction of an imaginary vision, and the
Dig
ital C
reat
ivity
, Vol
. 13,
No.
2Nielsen
7 0
discussion of the role and the affordance ofrepresentation in architectural design, mayenhance our understanding of immersion.However, whether talking about representationsin film, in computers, yes, even in the case ofthe physical spaces talked of in architecturalrepresentations, the concept of immersion is apsychological category. Though architecturalrepresentations do embed body, senses andemotions (as such the process of immersion isenhanced) they are architectural mock-up —not the real thing. If we turn to the interactiveperformance environment — the actor and theeffect of her movements on stage — herimmersion is obviously physical but alsopsychological. However, Burkes’ focus is thedeveloper and the user of the digital technolo-gies; the visualisation — which is the object ofdiscussion here — is an abstract visual represen-tation of body, movement and time. Thedeveloper of digital technologies may sit downin the theatre and observe how the technologyfunctions and thus experience the actor movingon stage, the lightning, the thunder — but it isnot the developer who is moving. To experiencethe move he has to project his senses and hisbody onto the stage. The representation he dealswith is a visualisation, an automatic diagram-ming of motion over time in a given space.
Space and movement, which is anothertheme emerging in the papers, becomes the issuehere. In the papers space may be perceived asphysical and virtual space. Movement may takeplace in any of the spaces, and is understood asphysical bodies moving in space, the projectionof body and movement into a space by person,the movement of physical objects or movingimagined physical objects in space. But move-ment of bodies in space becomes irrelevantwhen talking about visual communication —because visual communication and interactionhas as a prerequisite immersion and this immer-sion is psychological — or as one of the authorswrites — mental.
7. Concluding reflectionsThere are two actor perspectives in the papers.One is the creator, the designer, the architectand what they create, or design. It is how theydo it to obtain what they intend which is thekey issue in visual communication and interac-tion. But it remains uninteresting without theother actor, the user, the film viewer, thestudent. What they experience and how theyexperience is the other side of the coin. Thetheoretical framework of visual communicationand interaction seems to need a theory of thehuman being. Integrating a psychologicalunderstanding of the human being as anemotive, sensuous, bodily and intellectualexistential being could maybe help us under-stand better how the designer, the developer, thearchitect creates that which cannot be repre-sented but also help us understand how theinexpressible — that which cannot be repre-sented but is the psychologically necessary — isexperienced and how that experience unfolds.
References
Boyd Davis, S. (2002) Interacting with pictures:film, narrative and interaction. Digital Creativity13(2) 71–84.
Burke, J. (2002) Interactive performance environ-ments and the visualisation of actor movement.Digital Creativity 13(2) 122–128.
Keller, E. F. (1983) A feeling for the organism – the lifeand work of Barbara McClintock. Freeman, SanFrancisco.
Luescher, A. (2002) The physical trace. DigitalCreativity 13(2) 99–108.
Petersen, H. and Nielsen, J. (2002) The eye of theuser: the influence of movement on users’ visualattention. Digital Creativity 13(2) 109–121.
Thorlacius, L. (2002) A model of visual, aestheticcommunication: focusing on the web. DigitalCreativity 13(2) 83–98.
Winograd, T. and Flores, F. (1987) Computers andcognition. Addison Wesley, Reading, Mass.
