MIKOLAJ SZATKO & ARTUR JANAS

CONSUMED

CONTENTS

WHAT’S BEHIND....................................9

SOUNDS & VIDEO & INTERACTIONS...................21

A TOUCH OF TECHNOLOGY...........................35Conductive ink...............................................38Live video colour tracking...................................41Capacitive Sensor & Arduino..................................41

Max/MSP/Jitter and Interaction models........................43

OUTRO..........................................48

6

PREFACE This small book was created to document our work for Creative Computing module (level 10) taught by Mr Tom Flint as a part of BSc Interactive Media Design and BSc Digital Media courses at Edinburgh Napier University.

Our aim was to combine our skills to construct an engaging and original installation piece that would expand on our experience and knowledge of digital and interactive media.We developed a well structured concept providing a theoretical background for the piece. This enabled us to shed light on the dynamics of today’s urbanism and consumption in a new way.It was the consumption of technology and everyday objects that prompted us to simply title the piece Consumed.

9

WHAT’S BEHIND?

Consumed is an interactive audio-visual installation, a responsive sound scultpure. It features objects embodying no longer used technology and everyday waste, which people as urban consumers exploit, digest, and dispose of in a thoughtless fashion. All items are suspended in the air by cables - another prevalent element of the urban realm.

It is primarily this process of constant consumption that drove us to create the piece, being a commentary on the urban way of living as an ever-changing one. There is no stopping of today’s widespread need to improve on the technology we use, exchange trends and fashions so that all markets can work and produce newer and newer artifacts, gadgets etc etc.Because of this insatiable need for the-new-and-cool, older technology becomes obsolete and is consigned to oblivion.There will always be enthusiasts of such products, but the facts are that a vast majority of world’s population in 50 years will probably not remember about the Walkman’s existence.

It is not our objective to criticise or praise this process. We are just pointing the participant’s attention towards the objects that contribute to the shape of this installation, in the hope of reestablishing any long-gone connections or evoking memories. All objects ask to be remembered; they have been designed by somebody and should not become empty meaningless pieces of plastic.

8

The objects used in Consumed are: A DETERGENT BOTTLE, A COKE CAN, A STILLETO HEEL, A CASSETTE TAPE, A VHS CASSETTE, A LIGHT BULB, A REMOTE CONTROL, A PLAYSTATION CONTROLLER, A WALKMAN, and A CARDBOARD SCULTPURE (discussed later).They all are urban icons, which have been a part of, or have shaped, the urban lifestyle and entertainment over the last few decades. They are symbols of enjoyment, technological advancements, personalisation, power, and wealth. But also: waste, degradation, mindless consumerism. They embody nostalgia. Pairing carefully designed objects with rubbish is our way of raising awareness that they all become unified in the ruthless process of digestion and disintegration of items no longer needed in our world.

The articles are hung up in a more or less circular fashion.The cardboard scultpure, mentioned before, is the centre of this constellation. The abstract, futuristic shape of the sculpture is our way of conveying our fascination with urban forms, but also with materials - they do not always are iron and concrete. Cardboard is ubiquitous.Provisional solutions are commonplace. Hence the makeshift character of this figurative-non-figurative object.

The next pages show the items in their full beauty.They are wrapped in various colours to express their character, to ephasize their anonymity and individuality at the same time.The colours also help at abstracting the objects from their usual form, changing the way they can be perceived and endowing them with an after-life.

10

21

SOUNDS & VIDEO & INTER-ACTIONS

Upon being touched, the Consumed objects trigger off sound.The sonic layer consists of representations of the urban environment. We did not recreate real, concrete, physical sounds though, but decided to develop an array of audio metaphors - glitches, abstract noises and various musical notes.You will not hear cars or pneumatic drills.

It was our plan to produce sounds that would not be strictly connected with the objects prescribed to them. However, we did not want to create a audio-visual disconnection which could put off a potential participant. So, there is a link, however abstract or metaphorical it may seem. For example, the Walkman will produce a musical sound, so will the cassette tape. The can will produce a more clunky, rubbishy sound, etc. Working with these abstract analogies, we created sounds that correspond to the objects.

The central object (visible opposite) generates a continuous, low-frequency buzz. Even when no one participates in the installation and touches the articles, there will be a sound - similarly to the urban environment, which is never silent. There are fans, heaters, transistors etc.

An urban life demands from us to accept and get used to an unyielding attack of sonic events, whose artificial nature

make the urban space so special and unique. This specific form of urban music, happening on its own, was one of major points of reference and inspiration for Consumed.

There is scope for the audience to participate in creation of music through interaction with the installation. Thanks to the possibility of multiple simultaneous engagements, several persons can interact with the piece at the same time, and work towards a more articulate, deliberate structure. The outcome relies solely on the cooperation between participants. Of course, a more chaotic, unstructured soundscape can be achieved with ease. A rhythmic, structured, but improvised composition can be reached, challenging the users to communicate with each other, either through gestures or verbally. The sounds attached to the objects will allow for countless rhythmic patterns as well as elementary melodies. This interaction between individuals and objects, and among participants themselves create the basis for the entertainment quality of the piece.

The notion of play in Consumed links the piece to the notion of escapism. Through emotional engagement, the participants will experience a sense of active immersion. The audience is hoped to become absorbed in the activity of interacting with physical objects, whether on an individual or group level. This act makes room for a paradoxical occurrence: by being immersed in the piece relying on triggering noises and urbanism, one can strive to escape from it, or at least gain a new perspective on the concrete, harsh and noisy surroundings.

In order to give the piece a more abstract character, and add another dimension to its visual layer, as well as create

22 23

interest in the visitors, a responsive projection is to be displayed onto the objects and adjacent wall. There are several levels of the projection. The basic one is a short video loop of a busy freeway, manipulated so that two images from two different times of the day are superimposed on top of each other. The video reinforces the idea that the urban environment never stops. Regardless of time and city, the urban currents ruthlessly push everything forward, without a clear goal in mind.

This sequence can be manipulated by the participant’s interaction. The audience will be challenged to find linkages between objects they touch. Upon finding such a linkage, i.e. simultaneously touching two connected items, a special sound will be triggered and more importantly, the colour scheme of the video will change from black & white to one of the colours of RGB colour model.This works in accord with the idea of colouring of the items. The seemingly realistic video sequence suddenly undergoes a transition, which alters its character - not only the content’s character, but the projection’s one, too. The black & white sequence rapidly becomes coloured - but only within one of the RGB channels. The space is altered, is extended beyond usual perception.

There are three pairs of items responsible for this interaction: the Walkman and the cassette tape; the remote control and the VHS cassette; the Coke can and the detergent bottle.

There is one more, final mode of interaction in Consumed. The remaining objects - the light bulb, the stilleto heel and the Playstation controller touched together can elevate the experience to a whole new realm - an escape from the urban space, into the abstraction of the natural world. Simultaneous touch of these items triggers not only a completely new

set of sounds assigned to the objects (for a limited time only), preceded by a characteristic sound. It also changes the projection to a colourful display of leaves displayed under a layer of digital noise and deconstructed layer of digital TV broadcast.It is our encouragement to stop for a short moment at least, and ponder not only over human’s relationship with the steel, glass, concrete, noise and buzz, the gadgets, trends and entertainment, but also with nature, which in fact, is also a part of the urban environment. It may become forgotten like the discarded objects one day though. Consumed asks the viewer to notice it.

The choice of the objects is not accidental. To us, the shoe represents the act of passage, movement. The light bulb is the symbol of innovation - eureka! The controller equals control and power. The viewer acquires CONTROL to MOVE to a NEW IDEA. Consumed is not only an interactive playful installation. It is a riddle (a difficult one) to find linkages between the objects.

Next pages show a basic prototype of the installation as well as stills from the video sequences. The prototype includes only a small number of un-coloured objects in a makeshift setting.

24

A TOUCH OF TECHNOLOGY

The installation is designed so that the participants can trigger various sounds by tapping on the objects hung in a space. This model brings the question of how the responsiveness of the objects to touch may affect the reception of the artwork on a visceral level, and how the users may engage in order to create collaborative interactions.

An overview of the installation

35

In accordance to our concept, a slightest tap on, for instance, a hanging Walkman or an empty Coke, can would produce an abstract sound. This form of interaction needsto be very subtle, intuitive and enjoyable. The technology has to make the connection between the viewer and the object with regard to these requirements. Otherwise the interaction may become unpleasant and clunky.

There are different technologies that have been investigated in the context of these requirements. There are ten objects that trigger sounds independently. One of the methods was to track colors of multiple objects with a camera. However, this method is vulnerable to errors and misinterpretation of the inflowing data, and it is rather ‘hungry’ for processing power. The method could not be used for tracking multiple objects, but it is still viable to control only one object that produces synthesized continuous buzz.

Scan to view multiple objects color tracking test video

36 37

Amongst other methods investigated in the context of their ability to trigger interactions with the objects was a Piezo transducer. Piezo needs much less processing power in comparison to video tracking of multiple objects. It can detect knocks and other sound waves that are created, for instance, when an object is tapped. The use of Piezo, however, caused issues in relation to accuracy and mounting the transducers onto the objects. A person interacting with an object would have to make an effort to trigger a sound. A one-point interaction would lack on continuity, as the object reacts only on sound waves created at the moment of a tap or a knock, and the rather unattractive form of interaction would jeopardize the chances of multiple objects being activated at the same time by the viewers.

A similar verdict has been decided for the potential use of pressure sensor in this project: again, this type of interaction would lack the lightness of the interaction required by the model. Another quite comprehensible approach involved the use of accelerometers to measure the movement of the objects. As opposed to Piezo and pressure sensors, accelerometers could trigger an action with even slightest movement. This poses a problem that a rush of air could make the sounds play. The technology needed to be more directly connected to the use by human and ideally not based on movement – for instance, as if the objects were capacitive touch buttons that respond only to human touch.

Conductive ink

Conductive ink provides exactly the sort of connection that is needed to create a direct relationship between the viewer and the objects. Based on conductive ink, a one-point interaction is activated only when somebody passes an electric charge, that is, if you read this you have statistically quite a good chance of activating a sound in the installation. Objects may be touched continuously and send a trigger for as long as they are being touched, which increases the likelihood of a viewer or group of viewers to discover hidden interactions. The viewers may be encouraged by this to try how combinations of objects create different output. An object covered with the ink is connected to a microcontroller that sends the value of electric charge passed to a computer when an object is touched. The assumptions about how the interaction with the artwork can be realised with the use of conductive ink. It conforms to our requirements for easy and enjoyable interactions that are responsive and effortless. It also

38

can be painted over with paint, so it does not affect the aesthetics of the piece, and does not need much processing power to work.

if this was conductive ink

and this was a real Arduino

This setup could send information to a computer that would react to the input by, for instance, playing various sounds.

39

Live video color tracking

Live camera input and color tracking utilize jit.findbounds object in Max/MSP/Jitter programming environment. The method can be used to find the bounds of a specified color on incoming image matrix. An object in the installation (the tenth object), painted in distinctive color, can be tracked this way. Its coordinates may control a grain synthesizer that is used to make the buzzing sound. Although there may be some interference of the tracking caused by the viewers (they may wear the same color blouse as the tracked object, for instance) or by the video projection, which will overlay part of the tracked object), any disruption will not profoundly change the meaning and the reception of the project.

Capacitive Sensor & Arduino

Technically, the interaction with the use of conductive ink is possible by utilizing a microcontroller, namely Arduino, which deals with the incoming data. The code in Arduino uses external library called CapacitiveSensor by Paul Badger. It is available at Arduino Playground (http://playground.arduino.cc//Main/CapacitiveSensor?from=Main.CapSense).An Arduino board holds this code in its memory together with some extra code needed for serial communication, which allows for sending the received input to another program on the computer, namely Max/MSP/Jitter. The latter deals with playing of the sounds and video.

Capacitive sensor needs some wiring and resistors to process the electric impulses on the input. The CapacitiveSensor class uses one digital pin from Arduino as receive pin,

41

and another as send pin. A receive pin is the sensor pin, so an object would hang down from the other end of the cable connected to this pin. Arduino has 14 digital pins, but the first two cannot be used when serial communication is utilized. This leaves 12 pins for CapacitiveSensor’s disposal. Figure below shows complete set up:

An overview of complete circuit built using Arduino, a breadboard,

jumper cables, and 10k ohm resistors

In practice, one send pin can be used with 11 different objects and all of them would work fine except for one important detail: only three of the objects touched at once would trigger sounds. Touching the fourth object at the

42

same time stops the other objects’ triggers or does not trigger its own interaction. This situation has been resolved by building three separate circuits where 3 receive pins share one send pin. It sums up to 12 digital input pins available in Arduino. As you noticed, there are 9 objects accommodated in this way. The tenth object used in the installation is the one that makes buzzing sound and is controlled by a camera tracker. This way, all of the objects can be triggered at exactly the same time - although this is quite unlikely to happen, the model covers for all eventualities.

Arduino’s digital pins diagram showing the distribution of send and receive pins that create three separate circuits. The white boxes on Figure 5 are send pins - each has three receive pins /

objects connected to it.

43

Max/MSP/Jitter and Interaction models

Now that the circuits have been organized and the send pins send the triggering data, serial communication is used to pass on this data from Arduino to another program, Max/MSP/Jitter. This Integrated Development Environment (IDE) has been chosen because it offers established methods of communication using serial port, namely a function called serial. Max includes methods that decode the incoming bytes from ASCII format that is used by serial communication

protocol back to default line format. Furthermore, Max/MSP/Jitter can manipulate the sounds used with the objects, the video used for the projection, and the live video from camera tracking of the tenth object. Although the latter can be used completely independent, Arduino, the objects, and the video projection are interdependent, so to use one common programming environment is pretty convenient.

There are four types of interaction in the project: one point interaction with one object at a time, two point interaction with two objects at a exactly the same time, three point interaction with three objects at exactly the same time, and a final interaction based on object’s movement (color tracking).

The point interactions are grouped in relation to how many objects are touched at a time. The nine objects may be touched separately, one by one. They may all be touched at the same time, which would trigger their secondary functions. There are three couples among these objects that if touched simultaneously, produce a new distinctive sound and change the video projection slightly for a period of time. The three objects that are left will change the video and audio more radically if touched at the same time. This type of interaction will produce a new theme evident in the change of the video projection to completely different video, and all default sounds that are triggered when one object is touched at a time will be substituted for a given period of time. This will let the viewer experience the installation in a very different way than it was experienced before. The figure opposite depicts a possible arrangement of the objects in the space:

44

one-point interaction triggers one sound, default projection is

displayed

two-point interaction triggers one sound on top of default sounds, modifies the current projection for a given period of time

three-point interaction triggers new theme for a period of time: one point interactions trigger different set of soundsthe projection video is changed to another video

the object controlled with video colour tracker

In the Max/MSP/Jitter the audio is played with sfplay object and video projection uses jit.qt.movie to play video and jit.charmap to over-impose color overlays when two-point interaction is triggered. A system of buttons and objects called gswitch are used to route different behaviors within the program and initiate actions.

45

OUTRO

The tangibility of the interaction with the objects, the videos and sounds altogether compose a unique experience and create a entrancing aura. The connection with urban entertainment via the use of various objects ‘consumed’ by our culture and utilizing them in the installation as abstract sound-triggering objects make up for oneiric reality that engages the viewer experientially. The abstract soundscape and projections are accessed by the tactile interaction with the objects by one person or at least two people, which makes for a social experience that can be shared and experienced together with others. It is the ultimate aim of the implemented technology to make these interactions as easy, natural, and enjoyable as possible.

All of the files used in the making of this project, together with comprehensive comments are available in the folders: resources/Arduino and resources/Max. High-resolution screenshots of the code used are available in the folder resources/screenshots on a DVD-ROM attached.

48

MIKOLAJ SZATKO

I divide my time between film-making, music production, photography, new-media art and education. While all that is digital has never been my biggest interest, I have developed a more profound understanding of what’s going on, and I must admit, I’m liking it more and more. Some things should remain analogue, physical and traditional, and I will stick to that, but I now appreciate that digital art can do a lot of amazing stuff. And these worlds can merge - wow. A very-soon-to-be graduate of Digital Media at Edinburgh Napier University. Keep on moving.

http://www.swallowsflylow.tumblr.com

50

ARTUR JANAS

I am on a journey bursting through the world of technology that stimulates my endorphin levels with a blend of coding and design skills. I am interested in many aspects of user experience design, from application design to designing experiences with physical computing. I was always intrigued by thought of connecting physical world with computing in more direct and unparalleled ways. I believe in minimalism in many aspects of life, including technology. At the moment of writing this I am (almost!) fresh out of the Edinburgh Napier University where I graduated from Interactive Media Design. Oh, and I can’t play on a trumpet. Yet.

http://www.arturjanas.co.uk/

52

2013 Mikolaj Szatko & Artur Janas