View
0
Download
0
Category
Preview:
Citation preview
plastic and figurative signs and in any semiotic analysis of information visualization resides in the fact that
we do not always deal with figurative images. Indeed, even though we might be working on iconic
productions such as photographs or film sequences, their analysis through image processing methods
requires to interpret resulting images that could be disrupting, abstract or non-sense for a fresh eye.
Previous investigations on information visualizations that depict relationships in form of a network
diagram, flow/pie/bar chart, treemap, stack graph, or any other visual method, prove their validity because
they tackle symbolic problems (Tufte, 1997). They also recall that images are not universal but
combination of analog and conventional modes of reading (Eco, 1997). Regarding indexicality, while
diagrams keep an indexical relationship by means of quantification and data, in visual media the indexical
signs are related to the materiality of visual images.
Cases of study In this section we present some techniques that have been employed for media visualization in order to
detect some hints and first discussions toward reading strategies of visual productions. We employ the
term ‘reading’ applied to images as an umbrella term to refer to seeing in a comprehensive way and to
understanding a code of meaning.
Pixelation and color scheme
In 1982, Adele Goldberg and Robert Flegal introduced the term ‘pixel art’ to refer to a new kind of images
produced by using Toolbox, a Smalltalk-80 drawing system designed for interactive image creation and
editing (Goldberg & Flegal, 1982). The technique consisted on applying a mask and a combination rule.
The mask could be a square (pixel) or an 8-bit dot and the combination rule would replace the original
image with the mask.
Nowadays, we can simply pixelate an image by selecting Pixelate under the Filter menu of most image
editors, from Adobe Photoshop to the online Pixlr1. The resulting image is a colored grid where squares
take its sampled color from the original image. Users are able to decide on the size of pixels. As we can
see, iconic properties of the image start to disappear and plastic signs become more evident. Of course,
this is not to say that plastic signs are not evident at first glance, we do see them but we often focus on
the identification of figures before colors, shapes and spatial composition. Figure 1 shows an example of
a pixelated image.
1 http://pixlr.com/
Figure 1. Pixelation of an image
Cultural uses of pixelation can be found in artistic styles such as pointillism. Other uses have been seen
in broadcast TV, often to hide some parts of the image. For us, this technique helps to get rid of figurative
properties and assists on developing a visual literacy grounded on plastic signs. In late 2012, we taught a
course on Digital Culture to undergraduate students of Information and Communication at University of
Paris 13. Among other activities, students were asked to pixelate images in order to make evident plastic
signs such as colors and their properties. Later we employed free and easy to use software such as
Rainbow 1.52, an add-on for Firefox Web browser. Our aim was to facilitate the understanding of different
arrangements of color. The color scheme of the image above can be seen as Figure 2.
Figure 2. Color palette of an image
A color scheme represents a color synthesis of an image. It shows the main colors, often arranged by
frequency, i.e. the most predominant will appear at the farther left. But the scheme can be observed in
more detail. Figure 3 shows a fragment of the complete color scheme, now called color palette. It is
important to note the table can be arranged according to visual properties on each column.
2 https://addons.mozilla.org/en-us/firefox/addon/rainbow-color-tools/
Figure 3. Complete color palette (fragment) of an image. Arranged by the visual feature white contrast.
From this image we also note some visual features extracted by Rainbow 1.5 that pertain to the chromatic
category: red, green, blue, saturation, lightness, white contrast, black contrast, luminosity. Recent
projects evidence the existence of 399 visual features that can be extracted and quantified using software
like MatLab and FeatureExtractor3.
Image slicing
One of the first uses of image slicing is slit-scan photography. Among other pioneers, William Larson
produced, from 1967 to 1970, a series of experiments on photography called ‘figures in motion’. The trick
was to mount a thin slit in front of the camera lens to avoid the pass of light into the film. Thus the image
is only a part of an ordinary 35mm photograph. Moreover, if the photographer sets the camera on a
movable artifact and makes a long exposure of the picture, the result could be seen indeed as a figure in
motion. Golan Levin has been compiling a list on projects and precursors on slit-scanning (Levin, 2005).
Recently, it is possible to take photos directly as slit-scan images. Consider iPhone apps such as Slit-
Scan Camera by funnerLabs4. But we can also generate a slit-scan from a live or recorded video
sequence by using Processing. Figure 4 shows a fragment of the film Inception (Nolan, 2010).
3 http://code.google.com/p/softwarestudies/wiki/FeatureExtractor 4 http://funnerlabs.com/apps/slitscan
Figure 4. Slit-scan generated with Processing from a sequence from the film Inception
It might be clear to perceive the slit-scan effect. Yet for this particular case it could not be appealing
because it was done for a short duration of time. Perhaps interesting patterns may be discovered for
entire productions. Figure 5 illustrates a slit-scan, also called orthogonal cut in the software ImageJ. The
input was the video “Come as you are” by the band Nirvana (Kerslake, 1992) but converted into an image
sequence.
Figure 5. Orthogonal perspective of the music video “Come as you are”
But how would it look the visualization of an entire film according to this technique? The Tumblr site called
moviebarcode offers an index of a bunch of movies transformed to slit-scans. Among those we can see
the result for Inception.
Figure 6. Inception as slit-scan from the Tumblr site moviebarcode
Slit-scans are arranged progressively according to the time of an animated visual production. In the case
of the video-clip we note variations of colors and forms, but in the case of a film slits have been reduced
drastically and we can only see colors. An analytic reading would focus on contrasts of colors. A synthetic
reading would be related to textures created by colors. For a viewer familiar to the original production, she
could relate colors at any single position of the slit-scan to passages of the film, and that is because she
has previous experience with the object. From this image, contrasts of colors of an entire production
could then be schematized, pixelated, or we can apply other technique for visual feature extraction in
order to make comparisons on styles and rhythm.
Image montage
In 2004, Brendan Dawes presented ‘Cinema Redux’, a project aimed at showing what he calls a visual
fingerprint of an entire movie. The main idea was to decompose an entire film into frames and then to
arrange them as rows and columns, like a mosaic. In some sense, this technique is related to slit-
scanning in that is a zoom out of an entire production. This means we can observe 2 hours of images in
one single image. We replicated the same technique in Figure 7. We chose the video-clip ‘Come as you
are’ by Nirvana (Kerslake, 1992). First, we generated the image sequence at 10 frames per second
resulting in 2,257 image frames. Then, with ImageJ, we rendered an image montage.
Figure 7. Image montage of the music video “Come as you are”
While the source image is the same as in Figure 5, the result is very different. The chromatic categories
are the same but their spatial composition changes. We are indeed in front of a fingerprint, in terms of
Dawes: an indexical sign in terms of Pierce. To read this kind of image it is useful to keep in mind the
action or event or sound that relates to each position (a pragmatic perspective). Then a new relationship
to the object would rise, in other words, a new semantic of the indexical sign. A simple strategy consists
on looking for patterns in time by means of an analytic reading, which implies to visually group sequences
by colors and shapes. Do they repeat in another part? If yes, then we are in front of a pattern. In our case,
we chose deliberately the number of columns and rows to depict 6 seconds per row. Direction deploys
according to timing and should be read from left to right and top to bottom. We may observe that
saturated colors appear in the beginning, then low brightness, then again bright colors and at last a fusion
of both. Does the same pattern exist for other video clips? Is there a relationship between visual pattern
and music tempo? More investigations and comparisons are required to answer this question.
To discuss another example, Figure 8 compares two sequences that employ CGI in the film Inception
(Paris slow-motion explosions vs. Paris fold-over sequences). In this particular case, we can identify a
pattern of time: the amount of frames by shot is more or less the same. We can also appreciate that shots
include less quantity of shapes in the beginning and become more complex in the end. We may also
advance a partial conclusion. Nolan, together with visual effects supervisors Franklin, Corbuld, Bebb, and
Lockley, insert ‘rest zones’, i.e. shots without CGI effects in between shots with CGI effects.
Figure 8. Comparison of two image montages from the film Inception
Image averaging
Image averaging is related to the work of Sirovich and Kirby on ‘Eigenfaces’ in 1987 (Sirovich & Kirby,
1987). More recently, Jason Salavon has also produced series of images by averaging 100 photos of
special moments (Salavon, 2004). This method implies to flatten multiple images into one according to
the average of some chromatic value. Figure 9 shows the average of a shot in Inception that includes CGI
by taking into account the maximum brightness of 30 frames.
Figure 9. Average of max brightness of a 1-second CGI shot in Inception
With ImageJ it is easy to average other values by applying simple statistical analysis: minimum, median,
medium, or standard deviation. As we can observe, this technique is about extracting and quantifying
visual properties and then to perform statistical analysis.
It might be supposed this technique is more adequate when applied to short sequences or on those with
small changes on time. Averaging an entire contemporary Hollywood film would produce an image closer
to pure white or gray because of the quantity of frames and the movement and changes inside a frame.
However, it is possible to average only a small sample of the whole collection of frames. In Figure 11 we
show the result of averaging maximum brightness of the entire video clip ‘Smells like teen spirit’ by
Nirvana (Bayer, 1991). The image sequence was created by exporting the video at a rate of 10 frames
per second. We then imported it by incrementing by 10.
Figure 10. Average of sampled images of the music video “Smells like teen spirit”
In this case, larger forms appear in the middle of the frame. They also constitute the brightest region. A
more detailed examination, taking into account slit-scan and image montage of the same video-clip, leads
to conclude for this video an ideographic content. The figure of Kurt Cobain appears in the center, shiny
bright and high contrast inside an obscure atmosphere. However, these appearances are not numerous.
They are mainly three for the whole duration: the first around the start, the second around the middle and
the third at the end.
We present one last example. We have averaged the complete catalog of Thrasher covers, a popular
magazine among skateboarding culture. There are more than 350 images, ranging from 1981 to 2012. As
it can be appreciated the position of logo has not experienced major changes; the same can be said for
bar codes on the bottom part.
Figure 11. Average brightness of all covers of Thrasher Magazine
2D plotting
Plotting data values on two axes is a common type of representing information visually. Generally
speaking, the technique consists on locating on a Cartesian plane the crossing points according to data
values. Values can be positive or negative and they often allow seeing different positions at the same
time, so the decision maker can see variations and symbolic behavior.
Manovich introduced 2D media plotting. It consists on extracting values of visual features and storing
them in a tabulated data file. This file can also be amplified with metadata such as author, year, place and
other geographical, social or historical information. To give an example, we can extract all hue values for
a bunch of images and then mapping each of them on the X axis. The position on Y would depend on
other visual feature or on visual metadata, for instance a year. In that manner we can observe the
evolution of hue during a certain period of time.
The software ImageJ includes basic capabilities for extracting visual features. Together with
ImageMeasure and ImageShape, both scripts by the Software Studies Initiative, we can assemble an
important corpus of eidetic and chromatic visual features, about 112. As an example, consider Figure 12.
We have used ImageJ to plot 503 skateboards by Powell Peralta, produced between 1978 and 2012.
Figure 12. X axis represent hue values. Y axis stands for median brightness
What happens when you have more than 100 visual features measured? Is it possible to make an
average and to plot them? Manovich has done so by using R language. But in any case, the resulting
image depicts images themselves instead of dots and circles. We can of course zoom in and zoom out to
see details (the resolution of a media visualization is often larger than 8,000 pixels width). It is also
possible to distinguish curves and density zones. Consider figure 12 to approach a semantic reading.
Density is related to clustering and perceived through the dimension of formemes. Direction can be seen
through curves and lines traced along the graph: images go up and down. Repetition patterns can be
seen through repetitions of coloremes and texturemes.
3D structures
One of the common forms of 3D structures are 3D plot diagrams. Plotting on 3 axes has been useful for
multivariate analysis. Regarding visual information, colorimetric analysis was a pioneer in developing 3D-
based scales to visualize different color spaces. For instance, using ImageJ we can interactively visualize
an image in different color spaces. Figure 13 shows the color space RGB (red, green, blue).
Figure 13. 3D color analysis of an image in histogram mode
How to read such an image? Dimension of bubbles represent the frequency of that color, i.e. its
recurrences in the image. The position in space of the bubbles is determined by its value on red, blue,
and green channels (each of them ranging from 0.0 to 2.0). The departing point in the bottom angle is, by
default, brightness, going from 0, which is black, to 255, which is white.
Of course 3D plot is not limited to an organization inside a cube volume. Figure 14 shows two more
spaces: HSB and HDD.
Figure 14. HSB and HDD color spaces
Besides 3D plotting, we can also take advantage of 3D spaces to stack images. In this case, we simply
overlap images from a sequence. Figure 15 shows a stalk of the all shots including CGI visual effects in
the fortress explosion sequence of the film Inception.
Figure 15. Stalked sequence of images.
Now, imagine we synthetize colors for each image in a sequence and then we stack them and animate
them. Figure 16 shows two frames from a process named 3D surface plot in ImageJ.
Figure 16. 3D surface plot of a stalked image sequence
In this case, we plot the whole surface of the frame and we only see plastic signs. Remember you are
viewing the frame as if it were lying on the ground and not in front of you. It may be observed that the
highest points correspond to brighter tones, while the darker are located in the bottom. The color of the
surface is mapped from the original image. That means yellow and red stand for the explosion effect.
Gray and white are mostly snow, buildings, and the sky.
Finally, we are currently working on what we call ‘motion structures’. For us, motion structures are 3D
digital models created out of video sequences. Our workflow consists on transforming images to 8-bit
black and white, then to stalk them and to remove dark background. Figure 17 shows three views of the
same the object. It is the Paris-fold over sequence seen as motion structure.
Figure 17. Three views of a motion structure
Basically, the resulting visual production puts in evidence the shape of movements. Instead of analyzing
camera movements or character performances, we trace the transformations within the image itself. This
allows us to explore the form of indexical signs of moving pictures, to construct shapes, and to explore
them in 3D fashion. We rely on the software ImageJ to manipulate an image sequence and convert it into
a 3D model but it can be easily exported in DAE format and hence imported in other 3D navigating and
editing environment such as MeshLab, Maya, Blender, Sculptris or personal-developed applications with
Processing, OpenFrameworks, Quartz Composer and alike.
Figure 18. Motion structure imported into Sculptris
Conclusion
In this article we have proposed first steps toward a visual semiotic foundation to approach media
visualizations. The basic units for us are visual features. They are material properties of the visuality of
images. Indeed, whether contrast, mean values, standard deviation, energy, entropy, among others, they
can all be considered ‘plastic formants’. The usefulness of visual features is, essentially, to elaborate a
visual vocabulary of digital images and to foster, consequently, a more rigorous visual literacy in readers.
Moreover, we can also tackle visual features from the point of view of objects and super-objects: they can
be implied, associated, inherited between them. This facilitates a more flexible analytic/synthetic
perspective when dealing with external metadata that integrates as visual feature. As we saw, metadata
might be useful for 2D media plotting; they can be parameters to be chosen when producing an image
montage or an image average (number of frames, number of rows, visual feature to highlight).
In any case, the visual codes available are still weak. We need textual hints besides images. It is still
early to talk about visual methods besides traditional diagrams (pie/bar charts). But once we have defined
our main building blocks and general codes, we could agree on assembling larger projects that take into
account social, cultural, and political context; we could mix and remix techniques; we could create new
software and interfaces. The importance of conducting media visualizations is to navigate in the unknown;
to delegate some pre-conceptions about an object to an external agent (a software or programming
code). Perhaps this is what new images such as media visualizations are about. Perhaps, like abstract
art, they come with an ideological and connotative side: by conferring importance to materiality they are in
the same arena of the new realism (Galloway, 2012) or as alien phenomenology (Bogost, 2012).
Currently we are dedicating our efforts to construct a more comprehensible perspective to couple
semiotics, software, and media visualization. First, we believe it is fundamental to take into account
recent advances on what has been called ‘pictorial semiotics’ (Sonesson, 1989). Second, the
developments on cognitive semiotics must also be regarded, along with its methods and theories. Both
pictorial and cognitive semiotics are important because they take into account perceptual processes.
Finally, some investigations that already exist on scientific imagery (Fontanille, 2010) and on semiotics of
software (Andersen, 1997) provide valuable insights to be studied more deeply.
References
Andersen, P. (1997). A Theory of Computer Semiotics. Cambridge, MA: Cambridge University Press.
Bayer, S. (1991). “Smells like teen spirit”. DGC Records.
Bogost, I. (2012). Alien Phenomenology: or What It’s Like to Be a Thing. Minneapolis, MN: University of
Minnesota Press.
Dawes, B. “Cinema Redux”. Available online: http://brendandawes.com/projects/cinemaredux (retrieved
January 5, 2013).
Eco, U. (1997). Kant e l’ornitorinco. Milan: Bompiani.
Fontanille, J. (2010). “Réglages iconiques & esthétisation: notes sur l’interprétation de l’imagerie
scientifique“. Online:
http://www.unilim.fr/pages_perso/jacques.fontanille/articles_pdf/visuel/reglagesiconiquesimageriescientifi
que.pdf
Galloway, A. (2012). Les Nouveaux Réalistes. Paris: Léo Scheer.
Goldberg, A. & R. Flegal (1982). “Pixel Art” in Communications of the ACM. Vol. 25. No. 12. December.
New York: ACM Press.
Greimas, A.J. (1984) "Sémiotique figurative et sémiotique plastique" in Actes Sémiotiques. VI, 60. Paris:
Groupe de recherches sémio-linguistiques.
ImageJ. Available online: http://imagej.nih.gov/ij/ (retrieved January 5, 2013).
Kerslake, K. (1992). “Come as you are”. DGC Records.
Levin, G. (2005). “An Informal Catalogue of Slit-Scan Video Artworks and Research”. Available online:
http://www.flong.com/texts/lists/slit_scan/ (retrieved January 5, 2013).
Manovich, L. (2011). “Media Visualization: Visual Techniques for Exploring Large Collections of Images
and Video”. Available online: http://manovich.net/DOCS/media_visualization.2011.pdf (retrieved January
5, 2013).
Mitchell, W.J.T. (2005). What Do Pictures Want? Chicago: The University of Chicago Press.
Mondrian. Available online: http://stats.math.uni-augsburg.de/Mondrian/ (retrieved January 5, 2013).
moviebarcode. Available online: http://moviebarcode.tumblr.com/post/3508193540/inception-2010-prints
(retrieved January 5, 2013).
Nolan, Ch. (2010). Inception. Warner Bros Pictures.
O’Rourke, M. (2012). “Visualizations of Impressionist artists”. Available online:
http://lab.softwarestudies.com/2012/04/visualizations-of-impressionist-artists.html (retrieved January 5,
2013).
Rainbow 1.5. Available online: http://harthur.github.com/rainbow/ (retrieved January 5, 2013).
Salavon, J. (2004). “100 Special Moments”. Available online: http://salavon.com/work/SpecialMoments/
(retrieved January 5, 2013).
Sirovich, L. & M. Kirby (1987). "Low-dimensional procedure for the characterization of human faces" in
Journal of the Optical Society of America. Vol. 4. No. 3, pp. 519–524.
Slit-Scan Camera. Available online: http://funnerlabs.com/apps/slitscan (retrieved January 5, 2013).
Software Studies Initiative.
Sonesson, G. (1989). Pictorial concepts. Inquiries into the semiotic heritage and its relevance for the
analysis of the visual world. Lund: ARIS/Lund University Press.
Tufte, E. (1997). Visual Explanations: Images and Quantities, Evidence and Narrative. Chesire,
Connecticut: Graphics Press.
Recommended