Embed Size (px)
Citation preview
Grimace project documentation
March 17, 2009
Oliver Spindler Thomas Fadrus
0100611 / 066 935 0075129 / 033 532
Vienna University of Technology
Institute for Design and Assessment of Technology
Supervisor: Ao.Univ.-Prof. Dr. Peter Purgathofer
1
Contents
1 Introduction 3
1.1 Emotion model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Related work 4
3 Design 6
3.1 Graphical approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 The Uncanny Valley . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3 Our approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4 Face model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4 Development 11
4.1 Selected technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2 Iterative development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5 Technical details 13
5.1 Muscles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.3 Emotions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.4 Splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.5 Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.6 Stroke styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.7 Facedata file format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.8 Deployment and use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.9 Class diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
6 Results 24
6.1 Conclusion and future directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
References 29
2
1 Introduction
Emotions are important guides in our lives. Neglected since Plato as something to be overcome by
rational thought, newer research shows emotions play a key role in problem-solving and sense-
making (e.g. Ekman 2003, Norman 2004). Emotions are an aspect of affect, which is an umbrella
term for cognitive phenomena apart from rational thought. Other kinds of affect include moods
or personality traits.
Information entities, be it words, objects, ideas, music pieces or photographs, carry meaning.
An entity’s meaning can be seen as the sole reason for us to get interested in it; “meaningless”
and “pointless” are synonyms. Meaning is frequently seen to be comprised of two complemen-
tary aspects, denotation and connotation (e.g. Ogden et al. 1969; Garza-Cuarón 1991; Barthes
1996). Denotation is the actual, or, intended meaning of something; the meaning of words you
find in dictionaries, the plot of a film, the score of a music piece. Affective connotation is the
emotion or feeling which we believe to be communicated by an entity, or which is aroused in us
when exposed to the entity. With the rise of experimental psychology, this aspect has received
more attention. An important step was the work of Osgood (1957), who quantitatively measured
the affective connotation of words and from their results inferred a semantic space or affective
space. Nowadays, connotation usually refers to this kind of meaning.
Seeing the importance emotional information plays in our understanding of the world, it
should be natural to use this kind of information in the description of content on the web. How-
ever, content description is currently focused on denotation, while there are only few attempts
to target affective connotation.
A notable exception is the common usage of emoticons in textual communication. It is no
coincidence that we use emoticons, which are symbolic abstractions of facial expressions, to
express our emotional state and the affective connotation of our textual messages. Animals and
humans across all cultures express emotions through facial expressions. This hardwired relation
between emotions and facial expressions makes information about emotions an ideal candidate
for describing affect on the web.
Emoticons show that detailed reproduction of facial expression is not necessary to convey
emotions unambiguously. Reduced detail and complexity allows to focus on those features
which are necessary to effectively display a specific facial expression. Our hypothesis is that
concentrating on relevant facial features shows emotions more clearly.
1.1 Emotion model
The nature of emotions is an ongoing scholarly debate. Over the years, many different expla-
nations have been put forward, resulting in various emotion models. Most models are based
on one of two major approaches. In the dimensional approach, emotions are being described by a
small number of independent dimensions – usually two or three. For instance, the circumplex
model (Russell, 1980) describes emotion via the dimensions “valence” and “arousal”.
The categorical approach, on the other hand, assumes a finite number of basic emotions, which
describe innate emotional reactions. These basic emotions are believed to have developed evo-
lutionary, guiding human behaviour in a world of unforeseeable events (Ekman, 1999, Sloboda
and Juslin, 2001).
3
Grimace was based on the book Making Comics by Scott McCloud (2006), a manual for artists
on how to draw comics. Chapter 2 of the book deals with how to convicingly draw facial
expressions. In this chapter, McCloud develops an emotion model which follows the categorical
approach, postulating 6 basic emotions:
Anger Joy Surprise Disgust Sadness Fear
Table 1: The basic emotions defined by McCloud (2006)
This list is, in turn, based on the research of Paul Ekman. In a series of cross-cultural exper-
iments, Ekman et al. (1972) showed photographs of facial expressions to members of different
cultures. Since the posed expressions could be judged accurately, he inferred the universality of
several specific emotions. While the existence and the number of basic emotions is still debated,
his results show that these 6 emotions can be judged correctly.
Artists who want to draw convincing portraits of humans need to be expert observers of facial
expression. The Artist’s Complete Guide to Facial Expression by Gary Faigin (1990), an excellent
guide to drawing detailed facial expressions, is based on a categorical model. McCloud (2006)
takes visual cues for his model from Faigin’s work and shows how to depict emotion through
facial expressions in the world of comics, offering the ideal framework for our work.
In McCloud’s model, these basic emotions can be blended to achieve complex emotions. He
compares this process to the way arbitrary colours can be mixed from three primary colours.
Accordingly, he calls the 6 basic emotions primaries, and blendings of two basic emotions secon-
daries. He gives example depictions and names for all primary and secondary emotions, which
we used as the basis for our work. He asserts that mixtures can occur in arbitrary intensity and
might even include three emotions.
1.2 Goals
The goal of the project Grimace was to build a facial expression display using web technology
which effectively conveys emotions as depicted by McCloud (2006, p.83-85). This includes any
primary (or basic) emotion and any secondary emotion (blendings of two basic emotions) in
arbitrary intensity. The result should be a free component which can be easily integrated into
other projects to enable the addition of emotional expressiveness to interactive systems.
2 Related work
Realistic approaches
The development of dynamic face models is an important research area. Most of this work is
being undertaken in the field of affective computing, which aims to enhance interactive systems
with affective awareness and expressiveness. In a principal work for this research area, the
4
goal was defined as “making machines less frustrating to interact with.” (Picard, 1997, p. 214)
Interactive systems should recognise the affective state of the user and adapt their output and
behaviour accordingly.
One commonly proposed solution is the use of embodied conversational agents (ECA). In such
an environment, the system interacts with the user through an avatar with human-like appear-
ance and expressiveness. ECA strive to be believable conversation partners. Therefore, they need
to be able to express their emotions through facial expressions. Since an ECA also speaks, an-
other requirement is appropriate facial animation which supports the impression that the ECA
is actually speaking. Usually, an ECA is modeled as a 3D head.
Wang (1993) undertook an early yet impressive attempt at building a three-dimensional face
model, which predates affective computing. The face consists of a set of hierarchical b-spline
patches, which can be manipulated by simulated muscles.
Pan et al. (2007) focus on the notion that the basic emotions as postulated by Ekman (see
section 1.1) are not the ones which are actually needed for conversations with believable agents.
They developed a 3D character which expresses affective states like agreement, interest, thinking
and uncertainty through facial expressions and animated gestures. Ochs et al. (2005), on the
other hand, base their three-dimensional ECA on basic emotions, but allow them to be blended
to achieve more subtle affective states.
Albrecht et al. (2005) depart from the concept of basic emotions and base their agent on a
dimensional approach. Their agent supports a text-to-speech system. The system analyses a
text for certain terms whose coordinates for three affective dimensions are stored in a table.
With these values, the system augments the spoken text with appropriate facial expressions.
Zhang et al. (2007) follow a similar approach. In their system, high-level affective dimensions
are translated into MPEG 4 Facial Animation instructions. This standard is described by Pandzic
and Forchheimer (2002) in detail.
There are publicly available ECA frameworks. The CSLU Toolkit1 is one example. It is a
comprehensive framework which facilitates application development with the included ECA.
Another example is Xface2, which achieves an impressive level of realism. It makes use of the
MPEG 4 Facial Animation standard.
Comic-like approaches
The approaches described above usually employ characters which are designed for a high level
of realism. Such level of realism, however, is not necessary for this project. We aim to unam-
biguously express emotions through facial expressions. McCloud (2006) shows that a certain
level of abstraction is possible without any loss of expressiveness (see section 3). There are a few
attempts to develop face models which aim to achieve such abstracted comic-like appearance.
Bartneck (2001) conducted a study on the well-suitedness of different representations of facial
expressions. Three representations at different abstraction levels were compared; photographs
of a real face, an embodied conversational agent from the CSLU Toolkit, and highly abstracted
emoticons (black & white, 10× 9 pixels in size). Subjects rated facial expressions of these repre-
sentations for convincingness and distinctness. Results showed that the very abstract emoticon
1http://cslu.cse.ogi.edu/toolkit/2http://xface.itc.it/
5
was perceived to be as convincing as the photograph, while distinctness was rated as decreasing
with increasing abstraction levels.
Tanahashi and Kim (1999) developed a comic-like face model. Their model is designed to
express four out of six basic emotions as defined by Ekman. They also experiment with exagger-
ation of the features and with the addition of symbols to achieve a higher level of expressiveness.
The symbols employed are highly culturally dependent. Iwashita et al. (1999) also follow a comic
approach in their system for caricature development. An interesting point is the use of a survey
to improve the validity of the system, in which they asked subjects to pick the most expressive
and convincing representation out of a few alternatives.
Schubert (2004) describes a very simple comic-like face model, which only consists of eyes
and mouth. These features are represented by parabolic functions. His model is based on a
dimensional approach. The shape of the mouth indicates valence, the degree to which eyes
are opened represents arousal. The model is used to visualise emotions expressed by music.
Emotions are very short experiences which are not constant over the duration of a song. The
model expresses these changes in affect.
Discussion
Our goal is not to build a conversational agent; we do not include animation or a speech com-
ponent. The realistic three-dimensional approaches are designed for a purpose quite different
from our goals. Although embodied conversational agents can express emotions through facial
expressions, we believe that expressiveness can be increased further. For instance, the ECA cited
above do not include wrinkles in their design. However, wrinkles are an essential aspect of some
facial expressions (disgust, for example).
The comic-like models we have found, on the other hand, reduce the facial complexity to
an extent which results in a loss of expressiveness. Tanahashi and Kim try to counter this with
the addition of symbols. This is something we wish to avoid, because symbols are culturally
dependent, while cross-cultural research has shown that facial expressions on their own are
universally understood.
This brief survey leads to the principles on which we built our system.
3 Design
3.1 Graphical approach
Our main premise was to keep everything simple and minimalist. We also applied this principle
to the visual aspects of Grimace. Inspired by the approach applied in figure 1 by McCloud, we
decided that we need to make our facial representation somewhat comic-like but also natural
enough to be able to stand on its own, because we did not want to use symbols to augment it.
As seen in figure 1, it is not necessary to use an entirely realistic representation to achieve
emotional readability. The first three images all show the same emotion, which can be reliably
identified as an anxious state of mind. In fact, it even gets easier to identify this state when the
head is simplified. However, this cannot be done ad infinitum, because there is a point when
too much of the facial features are omitted and the intended emotion may not be recognized
6
Figure 1: McCloud (2006, p.96)
any more. In figure1, this point is reached a little bit left of the dotted line. The face has been
stripped of all the wrinkles and even the primary features like the eyes mouth and nose are mere
dots and lines. It is difficult to read the emotion on the face. Symbolic augmentation, in this case
drops of anxious sweat emenating from the face, is needed to make it work again.
We did not want to use symbols in our implementation, because this would make blending
different emotions together far more difficult. Furthermore, symbols aren’t as easily recognised
across different cultures (see section 1.1).
Omitting all the unnecessary wrinkles and features makes the face easier to recognize and it
would also ease the implementation and aid the performance of the system. Furthermore, if the
face was too humanlike the viewer wouldn’t tolerate if something wasn’t completely right. The
comic-like face is better suited for our approach because it would allow us to work more freely.
3.2 The Uncanny Valley
Another reason why we opted for a comic-like approach is the notion of The Uncanny Valley.
The concept was introduced in a short essay by robot-researcher Masahiro Mori (1970). In this
essay, he describes the phenomenon that when something (e.g. a robot) looks almost human-
like, it causes aversion rather than sympathy. As it can be seen on figure 2, this only happens at
the very end of the human likeness graph. Mori also states that movement amplifies the aversive
effect. So that something that looks almost like a human and moves in an almost human fashion,
causes a repulsive reaction in an observer.
A highly realistic 3D representation of a human would be very near to the right wall of
the valley, but obviously deep in the eerie area of the uncanny. In contrast, a comical face
would be located somewhere on the left slope of the ascent to humanoid robot, which puts it
comfortably out of the uncanny area. That’s exactly why we chose to use a comic-like appearence
for Grimace, because it is far more difficult to get something to look right if it’s supposed to
be very humanlike. Small errors in the appearance or movement can have disastrous effects,
because we are so used to seeing human beings around us all day, that slight deviations can
cause a repulsive reaction.
Bartneck et al. (2007) conducted a study in which they tried to find out if Mori’s predictions
were accurate. They used pictures of robots at different levels of human-likeness, as well as
pictures of real humans. It has to be noted that Bartneck et al. measured the level of likeability in
their study, not the level of familiarity which Mori described. Their findings did not confirm
the predicted raise in likeability. More abstract depictions were perceived as more likeable
than pictures of real humans. They note that knowledge about whether the photo showed a
7
still
100%50%
zombie
prosthetic handcorpsehuman likeness
industrial robot
stuffed animal
healthyperson
uncanny valley
bunraku puppet
humanoid robot
moving
fam
iliar
ity
Figure 2: Graphical representation of the Uncanny Valley. Taken from Geller (2008), based onMori (1970)
real human or a robot did not have an influence on likeability. Instead, results were highly
dependent on the perceived beauty of the stimulus. From their results, they infer that it might be
more accurate to speak of an Uncanny Cliff.
Geller (2008) gives an up-to-date examination of the concept. He notes that there are a
number of examples which contradict Mori’s predictions. One important factor which influences
familiarity is the depiction of the eyes. There might be an uncanny threshold; the eyes need to
be very well done to be acceptable. He closes with a recent quote of Mori, in which he, too,
questions the predicted raise in familiarity. Mori now believes that the human ideal is best
expressed by the calm facial expressions of Buddha statues.
3.3 Our approach
So the first step was to find a simplified version for each of the basic emotions laid out in table 1.
To do this, we used the representations of McCloud (2006), omitted some wrinkles and removed
the plasticity (see figure 3). As it was difficult to judge which ones could be left out, it would
have been helpful to conduct an early experiment with these reduced representations, to verify
that all the minimized faces still conveyed the intended emotion.
8
Figure 3: Simplified faces
Wrinkles that form when muscles in the face are contorted are as important for emotion-
recognition as the features themselves. It was also necessary to take some time and find out
which wrinkles comprise the minimal set to accurately represent all the emotional states and
still be manageable in terms of implementation complexity. As it can be seen in figure4 we
first identified the wrinkles that occur in most of the basic emotions, like the wrinkles in purple
around the mouth or the ones around the eyes in red, blue and green. Than we incrementally
added the ones that help to recognise a certain emotion and added them to the set.
Figure 4: Essential wrinkles
Basically wrinkles are crevices that form when skin is compressed through muscle tension.
To simulate this behaviour of appearing and disappearing wrinkles we used opacity. When a
wrinkle would form on a human face we would slowly raise the opacity of the correspoding
spline.
We realised that most of the facial expressions are fairly symmetrical and decided to model
only half of the face and then mirror all of it over to the other side. This would ease development
and also reduce the redundancy in the system.
3.4 Face model
We believe that closely following the biological process of how emotions result in facial expres-
sions increases the credibility and clarity of the displayed emotion. In humans and animals
alike, facial expressions result from the contraction of facial muscles. Therefore, the face model
was implemented following a muscle-based approach. The muscles are basically a system of
influences. When tension is applied to them they deform the facial features they are attached to.
These muscles are themselves influenced by emotions. These are defined as groups of muscles
which are tensioned if an emotion is applied to the face. The tension is calculated with a mathe-
matical function which was specified to match the non-linear behaviour of human muscles. This
approach is explained in detail in section 5.5.
Facial features had to be translated into graphical elements that could be transformed algo-
rithmically. We found the necessary combination of accuracy and flexibility in Bézier splines. All
9
facial features and muscles are represented by one or more splines. Figure 5 shows the first at-
tempt to represent the facial features with a minimal number of Bézier splines. This early model
proved to be not capable of expressing all necessary facial expressions and was augmented in
later iterations. The shape of Bézier splines is determined by a very small number of control
points. The idea was to connect the virtual muscles to these control points in such a manner that
contraction of the muscles would result in natural looking transformations of the splines.
Figure 5: First attempt to represent facial features via Bézier splines.
If a specific emotion is dialed in, the system checks which muscles are involved and calculates
the corresponding tension through the function specified for this particular set of emotion and
muscle. If necessary, it blends multiple muscle-tensions together if they are attached to the
same feature point. Furthermore, if more than one emotion is applied, Grimace also handles the
interpolation based on a priority assigned to each muscle. After the calculation phase the result
is used to deform the splines which make up the features.
emotions
featuresstatic dynamic
muscles
Figure 6: Face model
10
4 Development
4.1 Selected technology
Grimace has been developed in Actionscript 3 and is being deployed as a Flash / SWF file.
Actionscript 3 is the scripting language used by Adobe Flash and Adobe Flex. Though not
advertised, the language can be used on its own. The technology was selected for several reasons:
• Free: The Flex 3 SDK is available open source under the Mozilla Public License. It contains
MXMLC, an Actionscript 3 compiler, which allows generation of SWF files entirely through
Actionscript 3 code, without the need for an IDE like Adobe Flash IDE or Adobe Flex
Builder.
• Optimised for dynamic vector graphics: Flash originated as a vector-based animation tool and
offers comprehensive vector-drawing functions.
• Optimised for the web: Flash is a web-centric technology which delivers small file sizes and
can be conveniently integrated in web projects.
• Ubiquity and consistency: The Flash player is available for all major operating systems and
has an extremely high install base. SWF files are displayed in exactly the same way across
platforms and browsers.
4.2 Iterative development
After the visual style had been laid out we started with the technical implementation. As already
stated we tried to keep the face model as simple as possible to keep it managable and also
performant. This approach should be applied throughout the whole system, so we started again
with a minimum set of splines to define the features. First we modeled the eyes, as they are one
of the simpler forms in the human face. They are fairly static and basically just need to open
and close. This is an oversimplified assumption, but it sufficed to take the complexity of the
modelling process away.
In the course of developing the face model, these simple splines where reworked and ex-
tended to cover the whole gamut of facial expressions. This gamut represents the entirety of
facial expressions that can be covered with a certain set of emotions through blending. We had
to model our system so that it would cover as much of this gamut as possible. To achieve this,
we iteratively added splines and tried to model every possible facial expression with them. If the
simple spline turned out to be insufficient it was extended and reevaluated until it was sufficient
to represent all of the basic emotions.
The next step was to define the muscles that would influence the features to form the respec-
tive facial expressions. Again it was our premise to keep the muscular system as uncomplex
as possible without sacrificing expressiveness. Muscles were defined and linked to the spline
points of their corresponding feature. Then we had to try and match all the basic emotions with
this setup and when it wasn’t possible to mimic all the emotional states it had to be adjusted
and the trial phase had to be repeated until every necessary facial expression could be formed
with the defined muscular system.
11
0 0.25 0.5 0.75 1
0.25
0.5
0.75
1
Anger
Tension
Polynomial mapping
Sine mapping
Figure 7: Muscle tensions were plotted and interpolated for each emotion.
After that a similar procedure had to be executed to model the wrinkles and tie them to the
muscle system. Unfortunately it wasn’t possible to simply connect the wrinkles to the same
muscles that moved the feature splines, because the underlying physics are far more complex
than our minimalist system could reproduce, so we had to define a seperate muscle system
for the wrinkles because we needed very fine-grained control over how they moved. This also
proved to be very helpful at a later stage when it was necessary to blend different emotions
together, because we had separate control over the muscle systems.
When we had verified that the system was able to represent the basic emotions in this static
setup we had to put it all into motion. McCloud (2006) again proved to be a valuable resource
at this stage. He already defined how a face changes with different emotional intensities very
precisely. We manually modelled four gradual steps for every emotion, visually aligning them
with McCloud’s illustrations of the according state. Then we printed out the corresponding
muscle tensions and put them into a table. After this step we had a numerical representation of
the motion flow of a certain emotion from neutral to a fully expressed state.
For every muscle we dialed these values into Grapher.app and either manually matched a
mathematical function to the point set or used so called “curve fitting” the get an interpolated
polynomial function. For example, figure 7 shows the relationship of two muscles with anger.
The indicated forms are approximated by two mathematical functions. These functions were
then implemented via mappings (see section 5.5). Now we could define an emotion as a set
of muscle influences with a mapping, which specifies how the tension changes with different
intensities of this emotion.
After this step had been repeated for every muscle, every emotion and all the wrinkles we
had a pretty good system to dial in a basic emotion in a continous way and get a meaningful
representation of the defined state. The only thing left to do was to make the combination of
emotions work as well. In parts this was automatically achieved by averaging the mathemat-
12
ical functions that formed the basic emotions, but human emotions aren’t just mathematical
functions so a mixture of different emotions is hardly ever just an average state between the
corresponding basic emotions. To cope with this a priority was assigned to every muscle. The
ones with higher priority where favored in the blending process. A lot of manual work was
required to determine the priorities so that every mixture resulted in a meaningful facial expres-
sion. This was quite tedious, but our system proved to be very manageable and our commitment
to simplicity helped a lot in this phase.
When the face was capable of expressing any primary or secondary emotion, the component
was adapted for deployment. This included the addition of a JavaScript API, which allows full
control over the face’s capabilities, and the construction of a project website.
5 Technical details
Grimace follows a muscle-based approach and thus mimics the way biological faces operate.
In human and animal faces, facial expressions result from contraction of facial muscles. Facial
muscles are, unlike muscles in any other part of the body, only fixed to bones at one end, while
the other one is attached directly into the facial skin. This unique property allows the wide range
facial expressions humans are capable of displaying.
Our face model consists of three major components: emotions, muscles and features.
• Features, which are the visible elements of a face. Features can be transformed by muscles.
Typically, this includes dynamic features like eyes, eyebrows, mouth and wrinkles, as well
as static features like nose and head outline.
• Muscles, which are the link between emotions and features. The shape of a muscle is
defined by a spline and when contracted can move an arbitrary number of control points
along its path.
• Emotions, which are the high-level concept that influences a number of muscles in an arbi-
trary fashion. Each emotion affects specific regions of the face and results in familiar facial
expressions.
Figure 8 illustrates how these three components work together to achieve a facial expression. It
shows the mouth and its surrounding muscles for a neutral expression and four states of anger.
The shape of the mouth is represented by two features, upper lip and lower lip (shown
as ). A feature consists of several control points. The mouth is surrounded by several muscles.
A muscle has a defined path (shown as ) and a current tension (the dot along its path).
Each control point can be influenced by multiple muscles (influences are ). When a muscle is
contracted, it moves its tension dot along its path. Any control point which is influenced by the
muscle is then moved, which results in a change of the feature’s shape. Now, when an emotion
is present – the example shows the influence of anger – several muscles contract simultaneously.
In the following, each of these components and their underlying technologies are being de-
scribed. This is followed by a brief description of how Grimace can be put to use in other
projects. A complete overview of all the classes is given in a UML-style class diagram.
13
0.0 0.25 0.5 0.75 1.0
Figure 8: Influence of anger on muscles surrounding the mouth.
5.1 Muscles
As explained before, facial muscles are fixed to a bone at one end, and attached to skin at the
other end. When muscles contract, they shorten and thus pull the skin towards the point where
they are attached to the bone.
We simulate this behaviour. The shape of a muscle is defined by a spline (see section 5.4).
However, unlike real muscles, muscles in Grimace have no width. The tension parameter of a
muscle corresponds to the position t ∈ [0, 1] along the spline. Thus, t = 0 is a completely relaxed
muscle, while t = 1 represents maximum tension. A muscle can be defined with the parameter
initTension, which defines the neutral state for this muscle. This defaults to 0, but in some
cases, a neutral face – i.e. no emotion is active – results in contracted muscles. An example is
Levator palpebrae, which controls the upper eye lid. Since the eyes are halfway open in neutral
state, this muscle is defined with initTension.
The tension of a muscle, or rather, the distance between the points Q(t = initTension) and
Q(t = tension) influences the position of feature nodes (see section 5.2: Node influences). In
turn, the tension of a muscle is calculated from the emotions which exhibit an influence on the
muscle (see section 5.3).
Finally, Muscles are grouped into instances of MuscleGroup. This grouping is optional, but
currently muscles are divided into feature muscles and wrinkle muscles, defining additional
muscles which simulate the wrinkles resulting from when feature muscles contract. This is a
point where we had to leave an accurate biological representation to achieve the desired facial
expression gamut.
5.2 Features
Features, the visible parts of a face, can be transformed by muscles. The Feature class encapsu-
lates distinct facial features, e.g. the upper lip, an eyebrow or a wrinkle.
A feature is comprised of one or more segments, which are instances of the FeatureSegment
class. The shape of a segment is defined by a spline. Thus, a feature can take an arbitrary shape
by connecting several segments.
14
Node influences
A spline has two endpoints and 0 or more control points, referred to as nodes. Every point is
represented by the FeatureNode class and can be influenced by an arbitrary number of muscles.
For every node-muscle influence, a weight parameter is stored.
For n registered muscles, the position of node N is evaluated in the following way: For each
registered muscle M we calculate the distance between the muscle’s position resulting from its
current tension v, and the position resulting from its initial tension t. The distance is scaled by
the respective weight factor w. The node’s initial position N0 is then translated by the resulting
vector.
N = N0 +n
∑i=1
(wi · (Mi(v)−Mi(t)))
Fills
Features can also be filled arbitrarily, represented by the FeatureFill class. Fills can also be
influenced by muscles, thus adding a way to add animation. For every fill, a FeatureNode
represents a pivot point, which can then be influenced by muscles like any other node and
moves the whole fill when translated.
Alpha mapping
Not every feature is constantly visible. Wrinkles result from tightening of facial skin and thus
only become visible when certain muscles are contracted. To simulate this behaviour, the visibil-
ity of features can be mapped to the tension of a muscle. The relation is not direct but mediated
through mappings (see section 5.5). In this way, the feature opacity can be controlled flexibly.
5.3 Emotions
Emotions are the high-level concept which we aim to display via facial expressions. In real faces
as well as in our implementation, the 6 basic emotions we have implemented result in distinct
facial expressions, which have been said to be recognisable cross-culturally (see section 1.1).
The presence of an emotion is represented by a parameter value ∈ [0, 1], where value = 0
means the emotion is not present, and value = 1 represents maximum influence of an emotion.
When an emotional state is present, it results in simultaneous contraction of a set of muscles.
This contraction does not follow value linearly. For instance, some features only start to be
influenced when an emotion is strongly present, while others are continously influenced, but
more strongly in early than in later stages. Therefore, for every emotion-muscle influence, we
have defined a mapping which allows flexible control over how a muscle is contracted for an
emotion state (see section 5.5).
Our emotion model subscribes to the idea that complex emotions are in fact mixtures of
basic emotions. If two or more emotions are present simultaneously, more than one set of
muscles is affected. However, since different emotions sometimes influence the same muscles,
influences have a priority parameter. When a muscle is influenced by more than one emotion
simultaneously, priority defines the influence of each emotion on the final tension of a muscle.
15
For instance, a genuine smile not only influences the shape of the mouth, but also results in
squinting of the eye. However, a result of feeling surprised are widely-opened eyes. If joy and
surprise are experienced together, the eyes remain open, because surprise has a stronger influence
on the eyes than joy. This is represented by different priorities.
Given an influence of n emotions, with emotion values vi, influence priorities pi and raw
emotion tensions ti, the final tension of a muscle is calculated as:
t =n
∑i=1
vi · pi · ti ·1
n∑
i=1(vi · pi)
5.4 Splines
Spline is the common term for the use of piecewise parametric curves in computer graphics.
A spline has two endpoints and may have control points in between. With splines, complex
shapes can easily described or approximated by very few points. Bézier curves are a form of
parametric curves which are commonly used in vector-drawing and animation software. It is a
notable property of Bézier curves that the curve does not run through the control points but is
merely pulled towards them.
All shapes in Grimace – facial features and muscles – are based on straight lines and Bézier
curves. They offer an easily understandable way to model the face, and the selected technology
offers native support for these types of splines. Facial features are all visible components of the
face, e.g. the eyes, the mouth or wrinkles. Each feature consists of one or more segments, and
the shape of each segment is defined by exactly one spline. In addition, muscles are also based
on Bézier curves; the shape of each muscle is defined by exactly one spline.
Splines implement the ISpline interface. The interface defines the getPoint(t) method,
which calculates the location of a point along the spline given the position t ∈ [0, 1], where t = 0
is the starting point of the spline, and t = 1 is the endpoint.
The following splines are available for muscles and facial features:
12
12
18
78
12
12
12
12
383
8
C0=Q0
Q1
R1S1
T1
Q2=R0
R2=S0
S2=T0
C3=T2
C1 C2
Q0
Q1 Q2=RA
Q3=R0
C1 continuity
S0=R3
R1R2
S1=RB
S2
S3
Figure 9: Line
12
12
18
78
12
12
12
12
383
8
C0=Q0
Q1
R1S1
T1
Q2=R0
R2=S0
S2=T0
C3=T2
C1 C2
Q0
Q1 Q2=RA
Q3=R0
C1 continuity
S0=R3
R1R2
S1=RB
S2
S3
Figure 10: Quadratic Bézier
12
12
18
78
12
12
12
12
383
8
C0=Q0
Q1
R1S1
T1
Q2=R0
R2=S0
S2=T0
C3=T2
C1 C2
Q0
Q1 Q2=RA
Q3=R0
C1 continuity
S0=R3
R1R2
S1=RB
S2
S3
Figure 11: Cubic Bézier
Line
A spline which connects two endpoints with a straight line. Flash offers the native drawing
method lineTo for this spline type.
16
Quadratic Bézier
A Quadratic Bézier curve has one control point. Flash offers the native drawing method curveTo
for this spline type.
The parametric form of a Quadratic Bézier curve is:
Q(t) = P0(1− t)2 + P1 · 2t(1− t) + P2 · t2 , t ∈ [0, 1]
Cubic Bézier
A Cubic Bézier spline has two control points and offers great control over the curve form. If two
or more Cubic Bézier splines are concatenated, they offer enough flexibility to draw all necessary
facial features, including the mouth, which demands the greatest flexibility.
The parametric form of a Cubic Bézier curve is:
Q(t) = P0(1− t)3 + P1 · 3(1− t)2t + P2 · 3(1− t)t2 + P3 · t3 , t ∈ [0, 1]
Flash does not offer a native drawing method for Cubic Béziers. However, the form can
be approximated by lower-complexity curves like Quadratic Bézier splines. The more lower-
complexity curves are used, the more accurate the form of the approximated curve becomes.
12
12
18
78
12
12
12
12
383
8
C0=Q0
Q1
R1S1
T1
Q2=R0
R2=S0
S2=T0
C3=T2
C1 C2
Q0
Q1 Q2=RA
Q3=R0
C1 continuity
S0=R3
R1R2
S1=RB
S2
S3
Figure 12: Fixed Midpoint approximation
We have selected the Fixed Midpoint approximation method described by Groleau (2002). It
approximates a Cubic Bézier with four Quadratic Béziers and offers a good trade-off between
accuracy and calculation complexity. The approach is illustrated in figure 12. Given the four
points of a Cubic Bézier C to be approximated, endpoints and control points for Quadratic
Béziers Q, R, S and T are calculated in the following way:
17
H0 =C0+C1
2 + C1+C22
2=
C0 + C2
4+
C1
2
H1 =C1+C2
2 + C2+C32
2=
C1 + C3
4+
C2
2
Q0 = C0 ; Q1 =5C0 + 3C1
8
R1 =7H0 + H1
8; R2 =
H0 + H1
2
Q2 = R0 =Q1 + R1
2
S0 = R2 ; S1 =H0 + 7H1
8
T1 =3C2 + 5C3
8; T2 = C3
S2 = T0 =S1 + T1
2
In our implementation, the spline can be used like a regular Cubic Bézier with two endpoints
and two controlpoints, while the approximation is handled internally by the class.
Joiner
For some facial features, i.e. the shape of the mouth or several wrinkles, a single Cubic Bézier
curve does not suffice. In these cases, two or more curves may be joined together to form a curve
with additional flexibility. In these cases, one feature consists of more than one segment.
Parametric continuity Cn is a description of the smoothness of concatenated parametric curves:
• C0: curves are joined.
• C1: first derivatives are equal.
• C2: first and second derivatives are equal.
Without additional measures, connected Bézier curves only offer C0 continuity. If two connected
splines are to appear as a single and coherent curve, however, at least C1 continuity is necessary.
12
12
18
78
12
12
12
12
383
8
C0=Q0
Q1
R1S1
T1
Q2=R0
R2=S0
S2=T0
C3=T2
C1 C2
Q0
Q1 Q2=RA
Q3=R0
C1 continuity
S0=R3
R1R2
S1=RB
S2
S3
Figure 13: Joiner
18
The Joiner spline is a Cubic Bézier spline whose control points are calculated from the control
points of adjacent splines to achieve C1 continuity. The concept is illustrated in figure 13.
A Joiner spline R is constructed from two endpoints R0, R3 and two additional points RA,
RB. These additional points are used to calculate the necessary control points R1, R2 to achieve
C1 continuity in both endpoints. R1 and R2 lie on the lines formed by−−−→R0RA and
−−−→R3RB. The
distance of the control points from the respective endpoints on their respective axis is derived
from the distance between the endpoints.
Typically, RA and RB are set to the nearest control points of adjacent splines. For instance, if
Cubic Bézier Q ends in R0, then RA would be set to Q2. Likewise, if Cubic Bézier S starts in R3,
then RB would be set to S1.
The Joiner class is also used for mirroring. Assume a mirror through the vertical axis at
position x = 0, which results in horizontal mirroring. To ensure a smooth curve, R′(x = 0) must
be 0. If R0 = (x = 0, y = y0), this can be achieved by setting RA = (x < 0, y = y0). Then, R0
and RA form a horizontal line, which places R1 at (x > 0, y = y0) and results in zero slope for
x = 0. When the curve is now horizontally mirrored at this point, C1 continuity is achieved.
5.5 Mappings
Each emotion influences a different set of muscles. McCloud offers drawings for each basic emo-
tion in 4 intensity levels. These drawings were used as references, which we wanted to match.
For each emotion and each intensity level, muscles were adjusted to match the reference draw-
ing. The values of the muscle tensions were saved for each intensity level. Plots of the muscle
tensions showed that the relationship is a different one for each combination of muscle and emo-
tion. In some cases, the relation is a linear one – heightening the level of an emotion increases a
muscle’s tension. More often than not, however, the relation is much more complicated. In order
to achieve credible muscle tensions, this relationship, only indicated by 5 points (neutral and 4
intensity levels for each emotion), needs to be interpolated.
We represent the relationships by a number of mathematical functions, which we call Map-
pings. A Mapping takes a few parameters which influence the resulting function in a flexible
way to approximate the form of the underlying relationship. The IMapping interface is merely a
wrapper for a low-level mathematical function with one parameter and only has one method:
function y(x:Number):Number;
Every registered emotion-muscle influence is represented by a Mapping. The y-method takes
the current value of an emotion as parameter x and returns the current tension for the muscle.
Another relation represented by Mappings is the visibility of Features. Some features –
wrinkles – only become visible when a muscle is contracted. Representing this relationship
through Mappings allows fine-grained control over the opacity.
Three mapping types are currently available:
SineMapping
This form of Mapping is defined by four parameters. The function returns y0 for x < x0, and y1
for x ≥ x1. For x0 ≤ x < x1, the curve interpolates between y0 and y1, following the form of a
sine function. This results in a smooth transition between the two states.
19
0 x0 x1
y1
y0
Figure 14: SineMapping
y(x) =
y0 x < x0(
0.5 · sin(
π ·(
x−x0x1−x0
+ 1.5))
+ 0.5)· (y1 − y0) + y0 x0 ≤ x < x1
y1 x ≥ x1
GaussMapping
This mapping represents the Gaussian function and is used in cases where a muscle is only
contracted for intermediate values of an emotion, but not for low or high values.
0 µ
a0
a1
a2
(a) Influence of scale factor a
0 µ
σ2=0.005
σ2=0.01
σ2=0.02
(b) Influence of variance σ2
Figure 15: GaussMapping
y(x) = a · 1σ√
2πe−
(x−µ)2
2σ2
The mapping takes three parameters: value = a; mean = µ; variance = σ2
PolynomialMapping
This is a direct representation of a polynomial function. It can approximate any necessary form
by increasing the order of the polynomial. However, the function is hard to configure manually.
20
In practice, we used the curve-fitting methods of Grapher.app, which calculates a polynomial
interpolation of desired order for a given point set.
y(x) = anxn + an−1xn−1 + · · ·+ a2x2 + a1x + a0
5.6 Stroke styles
The shape of features is represented by splines. Stroke styles determine how the splines are
visually represented.
If no stroke style is set, the spline is simply stroked by a constant width brush. However, in
many cases, this does not deliver favourable results. Stroke styles implement the IStrokeStyle
interface. The interface’s draw method supplies the style with the spline to be drawn.
BrushStyle
Currently, BrushStyle is the only stroke style available. It simulates the characteristic form of a
brush; thin lines at the start, getting thicker towards the center, and again thinner towards the
end. This corresponds to the parameters startWidth, maxWidth and endWidth.
From the spline to be stroked, two splines are derived which define the shape of the stroke.
One spline defines the upper edge, the other one defines the lower edge. In every point of the
base spline, a normal is drawn. On each normal, the positions of the points of the upper and
lower splines are shifted; points of upper spline to the left, points of lower spline to the right.
Thus, maxWidth does not directly represent the actual thickness of the resulting stroke, but the
distance of the control points. The concept is illustrated in Figure 16.
13t=
23t=
startWidth
endWidth
max
Width
maxW
idth
Figure 16: BrushStyle applied to a Cubic Bézier spline
21
5.7 Facedata file format
Faces are entirely defined through external files which are loaded at runtime. This allows the
development of faces which look entirely different to the standard face we developed. Additional
emotions can also be implemented.
A complete set of Facedata defines the following:
• Features, which are the visible elements of a face. Features can be transformed by muscles.
Typically, this includes dynamic features like eyes, eyebrows, mouth and wrinkles, as well
as static features like nose and head outline.
• Muscles, which are the link between emotions and features. The shape of a muscle is
defined by a spline and when contracted can move an arbitrary number of control points
along its path.
• Emotions, which are the high-level concept which influence a number of muscles in an
arbitrary fashion. Each emotion affects specific regions of the face and results in familiar
facial expressions.
• Overlays, which are optional graphical elements added on top of the face to add additional
personality to the face. In the standard model, the hairdo is an overlayed vector graphic.
Pixel-based graphics can also be included.
Facedata is an XML-based file format. Currently, no graphical editor is available; Facedata has
to be edited manually. A corresponding DTD is kept up-to-date3 with the current capabilities of
Grimace and allows face developers to validate their files through an XML validation service4.
Since the definitions can become quite large and data have to be edited manually, Facedata
definitions can be spread across files. The loadFacedata API method takes an array of URLs as
parameter, loading the files in the supplied order.
5.8 Deployment and use
Grimace is a self-contained component which enables the addition of facial expressions to soft-
ware projects. Being written in Actionscript 3, the component is deployed as SWF file and can
be opened by Adobe Flash Player 9 and upwards. The component can be downloaded from the
project website and includes detailed instructions and demo files.
Control of the component is offered by an API, which is compatible with JavaScript and
Actionscript 3. The recommended method is to embed Grimace into web pages and control it
through JavaScript via the API. Through embedding, Grimace can also be controlled via Action-
script 3. Apart from pure AS3, this includes Flex and Flash (from version CS4 upwards). The
AS3 API is basically identical to the JavaScript API but less tested.
Customisation
The download package includes a complete face in the form of a set of Facedata XML files. We
encourage the development of new faces based on these definitions. Currently, no graphical
3The latest version of the DTD can always be found at http://grimace-project.net/dtd/latest.dtd4e.g. http://www.validome.org/xml/
22
editor is available, values need to be edited manually. However, the package also includes
Facemap.swf, the tool we used to develop the face definition. The tool allows to show muscles
and their current tension, include underlayed pictures which offer reference, and allows the
output of the current state of all components.
5.9 Class diagram
Data input
External API
AbstractSpline
ISpline<<interface>>
QuadraticBezierLine CubicBezier
Joiner
IMapping<<interface>>
PolynomialMapping
SineMapping
GaussMapping
Feature
1..*
1
FeatureSegment
Grimace
FeatureController
0..*
1
1..*
1
FeatureNode
IStrokeStyle<<interface>>
0..1
11
1
MuscleGroup
MuscleController
0..*
1
Muscle
0..*
1
1
1
Emotion
EmotionCore
0..*
1
0..*
1
1
1
0..*
0..*
<<moves>>
<<visibility>>
<<contracts>>
<<shape>> <<shape>>
XMLFactory
FacedataLoader
XMLDraw
ExternalCommands
JSHandler
ASHandler
Geom
BrushStyle
FeatureFill
0..*
1
<<pivot>>
Figure 17: Class diagram
23
6 Results
We have developed a software component which can display all primary and secondary emo-
tions as depicted by McCloud. Furthermore, primaries can be blended in arbitrary intensities,
thus covering states not covered before.
The component has been released to the public under a Creative Commons licence. A project
website5 has been implemented. The website features a demo application that allows visitors
to express arbitrary blendings of any two emotions. A download package is available, which
includes the component, demo applications for all supported programming environments and
comprehensive documentation on how to use the component.
Public reactions to the project were notedly positive, shown in a large number of approv-
ing comments. Scott McCloud kindly featured our project on his blog on February 25, 2009,
emphasising that facial expressions should be taught in school, for which our project might be
very useful. We are also very thankful to Mr. McCloud for his encouraging words and useful
comments about our work at an intermediate stage of the project.
The resulting face is shown in figure 18 with a neutral expression. Figure 19 shows the 6
emotional primaries at four intensity levels. In figure 20, any combination of two primaries
(both at 75% intensity level) is shown.
Figure 18: Neutral expression
5http://grimace-project.net
24
Joy
Sadness
Disgust
Anger
Fear
Surprise
Figure 19: Primary emotions in 4 intensity levels
25
Joy + Surprise
Joy + Fear
Joy + Disgust
Joy + Anger
Joy + Sadness
Surprise + Disgust
Surprise + Sadness
Surprise + Fear
Surprise + Anger
Fear + Sadness
Fear + Anger
Fear + Disgust
Sadness + Anger
Sadness + Disgust
Disgust + Anger
Figure 20: Secondary emotion blendings of intensity level 3
26
6.1 Conclusion and future directions
The described software component Grimace displays emotions through a comic-like face. We
believe that the display of emotional information is a valuable addition to information resources,
and facial expressions are a natural way of expressing this kind of information. The work of
McCloud (2006) was used as guide and visual reference throughout the design and development
process. We believe to have found a useful compromise between simplicity and necessary detail.
We include all facial features which are necessary to convey an emotion while omitting the rest.
The component was developed using web technology, which allows easy deployment. We
defined an API which allows convenient integration into other projects without the need for
knowledge about technical details. All configuration data is loaded from external files, which
use an XML-based file format. The file format is fully documented and allows full customisation
of all aspects – features, muscles and emotions. The component is stable and ready for use
for the intended purpose. A project website was implemented, from which the component and
documentation can be downloaded.
While we believe that the goal of the project has been achieved in a satisfactory manner, there
are many areas which remain to be addressed, a few of which will be outlined in the following.
First of all, the current face model can be further optimised. We had to add additional
muscles to the principal facial muscles in a few cases to achieve the desired expressiveness.
However, it might be possible to reduce the number of necessary muscles by optimising the
definition of the actual muscles.
Furthermore it would be interesting to see, if the intended emotions are actually recognized
in the facial expressions Grimace produces. In order to verify this an online experiment was
conducted. A website was designed and built to make the survey easily available to a large au-
dience. The setup consisted of a number of facial expressions rendered by Grimace, covering the
whole emotional gamut. Next to the face the 6 basic emotions (see figure 1) where listed and the
participants had to specify which emotions they associated with the shown expression. About
200 people from around the world participated in the experiment. The analysis of the collected
data is in progress and the findings may be incorporated in the next iteration of Grimace to
improve the readability of the emotions.
Calder et al. (2000) show that comprehensibility of facial expressions can be increased further
if the characteristic features of an expression are exaggerated. Our model has comic-like appear-
ance, and it might be possible to make our model even more expressive if we allow a certain
level of unrealistic, cartoon-like exaggerated expressions.
Customisation and extension of the current face model would become much easier if a graph-
ical editor was available. First and foremost, such an editor should facilitate customisation of
visible features of a face. Currently, the control points for features need to be entered manually
in XML files. These are the parts which can be exchanged easily. The relationships between
muscles and emotions, however, need considerable attention and are quite tedious to change.
So far, the system can only display facial expressions which represent emotional states. Of
course, humans can communicate much more through their faces, which can be easily observed
by studying the wide range of facial expressions which actors can display. Facial expressions
which cannot be expressed currently include doubt or agreement. The Facial Action Coding
System, or FACS, (Ekman et al., 1978) describes a comprehensive framework of all possible
27
facial movements. If the range of possible facial expressions was to be extended, this framework
would offer a good basis.
This would also mean a departure from the mirroring of facial features. Right now, facial
features are completely symmetrical. In FACS, asymmetrical movement of features is possible.
Ideally, the system would still mirror those parts that are symmetrical and only consider the
differences to the symmetrical state when necessary.
28
References
I. Albrecht, M. Schröder, J. Haber, and H. Seidel. Mixed feelings: expression of non-basic
emotions in a muscle-based talking head. Virtual Reality, 8(4):201–212, 2005. URL http:
//www.springerlink.com/index/G407T21751T81161.pdf.
K. Balci. Xface: Open Source Toolkit for Creating 3D Faces of an Embodied Conversational
Agent. In Smart Graphics: 5th International Symposium, SG 2005, Frauenwörth Cloister, Germany,
August 22-24, 2005: Proceedings. Springer, 2005.
R. Barthes. Denotation and connotation. The Communication Theory Reader, pages 129–133, 1996.
C. Bartneck. How Convincing is Mr. Data’s Smile: Affective Expressions of Machines. User
Modeling and User-Adapted Interaction, 11(4):279–295, 2001.
C. Bartneck, T. Kanda, H. Ishiguro, and N. Hagita. Is The Uncanny Valley An Uncanny Cliff? In
Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communica-
tion, RO-MAN, pages 368–373, 2007.
S. Brennan. Caricature generator: The dynamic exaggeration of faces by computer. Leonardo, 18
(3):170–178, 1985.
A. Calder, D. Rowland, A. Young, I. Nimmo-Smith, J. Keane, and D. Perrett. Caricaturing
facial expressions. Cognition, 76(2):105–146, 2000. URL http://linkinghub.elsevier.com/
retrieve/pii/S0010027700000743.
P. Ekman. Basic emotions. Handbook of cognition and emotion, pages 45–60, 1999. URL http:
//www.vhml.org/theses/wijayat/sources/writings/papers/basic_emotions.pdf.
P. Ekman. Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emo-
tional Life. Times Books, 2003.
P. Ekman, W. Friesen, and P. Ellsworth. Emotion in the Human Face: Guidelines for Research and an
Integration of Findings. Pergamon, 1972.
P. Ekman, W. Friesen, J. Hager, and A. Face. Facial action coding system. 1978.
N. Etcoff and J. Magee. Categorical perception of facial expressions. Cognition, 44(3):227–40,
1992. URL http://www.ncbi.nlm.nih.gov/pubmed/1424493.
G. Faigin. The Artist’s Complete Guide to Facial Expression. Watson-Guptill, 1990.
B. Garza-Cuarón. Connotation and Meaning. Mouton De Gruyter, 1991.
T. Geller. Overcoming the uncanny valley. IEEE Computer Graphics and Applications, 28(4):11–17,
2008.
T. Groleau. Approximating cubic bezier curves in flash mx, 2002. URL http://
timotheegroleau.com/Flash/articles/cubic_bezier_in_flash.htm.
29
S. Iwashita, Y. Takeda, and T. Onisawa. Expressive facial caricature drawing. In Fuzzy Systems
Conference Proceedings, 1999. FUZZ-IEEE’99. 1999 IEEE International, volume 3, 1999. URL
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=790143.
S. McCloud. Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels. Harper-
Perennial, 2006.
M. Mori. The uncanny valley. Energy, 7(4):33–35, 1970. URL http://graphics.cs.ucdavis.
edu/~staadt/ECS280/Mori1970OTU.pdf.
D. Norman. Emotional Design: Why We Love (or Hate) Everyday Things. Basic Books, 2004.
M. Ochs, R. Niewiadomski, C. Pelachaud, and D. Sadek. Intelligent expressions of emotions. In
1st International Conference on Affective Computing and Intelligent Interaction ACII. Springer, 2005.
URL http://www.springerlink.com/index/7007280gtq412j0h.pdf.
C. Ogden, I. Richards, and U. Eco. The Meaning of Meaning: A Study of the Influence of Language
Upon Thought and of the Science of Symbolism. Routledge & Kegan Paul, Ltd., 1969.
A. Ortony and T. Turner. What’s basic about basic emotions. Psychological Review, 97(3):315–331,
1990. URL http://www.cs.northwestern.edu/~ortony/papers/basic%20emotions.pdf.
C. Osgood. The Measurement of Meaning. University of Illinois Press, 1957. URL
http://books.google.com/books?hl=en&lr=&id=Qj8GeUrKZdAC&oi=fnd&pg=PA1&dq=
osgood+measurement&ots=RFI2_XNI8d&sig=hv5zzkO69BJWzCIK-37hS8QoecU.
X. Pan, M. Gillies, T. Sezgin, and C. Loscos. Expressing complex mental states through facial
expressions. LECTURE NOTES IN COMPUTER SCIENCE, 4738:745, 2007. URL http://www.
springerlink.com/index/l8607854jt5q23l9.pdf.
I. Pandzic and R. Forchheimer. MPEG-4 Facial Animation: The Standard, Implementation and Appli-
cations. Wiley, 2002.
R. Picard. Affective Computing. MIT Press, 1997. URL http://books.google.at/books?
hl=en&lr=&id=GaVncRTcb1gC&oi=fnd&pg=PP11&dq=picard+affective+computing&ots=
F1k6rlAaab&sig=qxVU7LSWnrL3XWmOthw7YX3cC-U.
J. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161–
1178, 1980. URL http://content.apa.org/journals/psp/39/6/1161.
E. Schubert. Emotionface: Prototype facial expression display of emotion in music. In Proc. Int.
Conf. On Auditory Displays (ICAD), 2004. URL http://music.arts.unsw.edu.au/aboutus/
research/Schubert/ICAD04SchubertEmotionFace.pdf.
J. Sloboda and P. Juslin. Psychological perspectives on music and emotion. Music and emotion:
Theory and research, pages 71–104, 2001.
S. Tanahashi and Y. Kim. A comic emotional expression method and its applications. In TENCON
99. Proceedings of the IEEE Region 10 Conference, volume 1, 1999. URL http://ieeexplore.
ieee.org/xpls/abs_all.jsp?arnumber=818417.
30
C. Wang. Langwidere: A Hierarchical Spline Based Facial Animation System with Simulated Muscles.
PhD thesis, University of Calgary, 1993.
S. Zhang, Z. Wu, H. Meng, and L. Cai. Facial expression synthesis using pad emotional param-
eters for a chinese expressive avatar. LECTURE NOTES IN COMPUTER SCIENCE, 4738:24,
2007. URL http://www.springerlink.com/index/118766717256766j.pdf.
31
