Embed Size (px)
Citation preview
Phil. Trans. R. Soc. B (2009) 364, 3497–3504
doi:10.1098/rstb.2009.0165
Review
* Autho
One conof emot
The face is not an empty canvas: how facialexpressions interact with facial appearance
Ursula Hess1,*, Reginald B. Adams Jr2 and Robert E. Kleck3
1Department of Psychology, University of Quebec at Montreal, CP 8888, Station A, Montreal,Quebec, Canada H3C 3P8
2The Pennsylvania State University, University Park, PA, USA3Dartmouth College, Hanover, NH, USA
Faces are not simply blank canvases upon which facial expressions write their emotional messages.In fact, facial appearance and facial movement are both important social signalling systems in theirown right. We here provide multiple lines of evidence for the notion that the social signals derivedfrom facial appearance on the one hand and facial movement on the other interact in a complexmanner, sometimes reinforcing and sometimes contradicting one another. Faces provide infor-mation on who a person is. Sex, age, ethnicity, personality and other characteristics that candefine a person and the social group the person belongs to can all be derived from the facealone. The present article argues that faces interact with the perception of emotion expressionsbecause this information informs a decoder’s expectations regarding an expresser’s probableemotional reactions. Facial appearance also interacts more directly with the interpretation offacial movement because some of the features that are used to derive personality or sex informationare also features that closely resemble certain emotional expressions, thereby enhancing or dilutingthe perceived strength of particular expressions.
Keywords: emotion; facial expression; facial appearance
Facial expressions of emotion have long been of inter-est to philosophers and psychologists. Darwin’s (1872/1965) seminal work On the Expression of the Emotions inMan and Animals was a first attempt to systematicallyunderstand emotion expression and its meaning. Inthis book, he proposed a number of explanations asto why certain facial and bodily behaviours communi-cate certain emotions. In doing so, Darwin had nodoubt that the expressive behaviour that he describedwas part of an underlying emotional state, that is,that emotion expressions have communicative valuespecifically because they are outward manifestations ofan inner state. Darwin assumed clear parallels andantecedents to human emotions in the emotions ofanimals and our humanoid ancestors, and conse-quently considered emotion expressions to beuniversal across human cultures.
1. WHAT DO FACIAL EXPRESSIONS EXPRESS?Yet, right from the beginning, Darwin’s view ofemotion expressions as the visible part of an under-lying emotional state was disputed and rejected bythose who considered facial expressions as learnedsocial signals that varied across cultures. Indeed,research in the first-half of the twentieth century
r for correspondence ([email protected]).
tribution of 17 to a Discussion Meeting Issue ‘Computationions in man and machines’.
349
produced inconsistent results regarding whether evenmembers of the same culture could accurately inter-pret the facial expressions of their social group. Thisdisparity in findings led Bruner and Tagiuri in their1954 Handbook of social psychology review article tostate that ‘ . . . the evidence for the recognizability ofemotional expressions is unclear’ (p. 634). Researchby Ekman and colleagues (e.g. Ekman et al. 1969,1987; Ekman 1972) as well as Izard (e.g. Izard1971) caused the pendulum to swing back towardsDarwin’s view in establishing that certain ‘basic’emotion expressions are indeed universally recognized.
Yet, even if certain expressions are recognized assignalling certain emotions, this does not necessarilymean that the expressions are in fact the output of anunderlying affective state. In this vein, Fridlund (1994)emphasized that for emotion expressions to be trulyuseful as a communicative signal they should be linkedto the organism’s social motives rather than to quasi-reflexive emotions, and concluded that emotionexpressions should be considered as unrelated to anunderlying emotional state. A series of studies publishedsince Fridlund made this argument have shown thatemotional facial expressions function both as symptomsof an underlying state and as communicative signalsrelevant to social motives (e.g. Hess et al. 1995; Jakobset al. 1999a,b, 2001; Manstead et al. 1999).
The observation of—not always insubstantial—differences in emotion recognition rates betweencultures led more recently to the notion of cultural
7 This journal is # 2009 The Royal Society
3498 U. Hess et al. Review. The face is not a canvas
dialects (Elfenbein & Ambady 2003; Elfenbein et al.2007). Specifically, it has been argued that there is auniversal language of emotion but with local dialectsthat differ subtly from each other. These dialect dif-ferences are small enough to allow high levels ofrecognizability across cultures, yet large enough sothat judgements are faster and more accurate forperceivers who are familiar with these subtle variations.
2. FACIAL EXPRESSIONS IN RESEARCHAND IN EVERYDAY LIFEIn summary, to date there is little doubt regarding thefact that emotional facial expressions can be recog-nized across cultures. Also, observers consider themto be an honest signal of an underlying emotion(Fiske 2004). Yet, it is important to note that thestimuli used in research on the recognition of facialexpressions of emotion generally have two featuresthat set them apart from the expressions we see inthe faces of the people we interact with on a dailybasis. First, participants see the same expression onseveral faces and their success in recognition is aver-aged across these exemplars and second, theexpressions that are used are intense to a degreerarely encountered in everyday life.
As regards the first issue, Wiggers (1982) noted thatthe same combination of facial actions on two differentmodels yields different recognition rates. These obser-vations led to the methodological advice to use morethan one model in emotion expression research butno effort was made to understand the source of theseacross-model differences.
As regards the second issue, full-blown facialexpressions are by far the exception in everyday inter-actions and seldom take the canonic form employed asstimuli by emotion researchers (e.g. Ekman et al.1969). Indeed, facial expressions can be expressedpartially or they can be the result of blends thatconvey different emotions at the same time (Ekman &O’Sullivan 1991). Hence, the facial expressions wecommonly encounter are weak, elusive or blended,resulting in a signal that often is ambiguous andrequires substantial interpretative work. Thus, themajority of the research referred to above neglectstwo pertinent elements of actual emotion recognitionin interactions. First, the impact of the morphologyof the face on how expressions appearing on any oneface will be interpreted has not been explored.Second, the challenge posed by the more ambiguousexpressions that most people show in most situationshas not been adequately addressed. These will bediscussed in more detail below.
3. TWO STRATEGIES FOR THE RECOGNITIONOF EMOTIONThere are two principal strategies for the decoding ofemotion displays (Kirouac & Hess 1999). First, pat-tern matching can be used to draw inferencesregarding an expresser’s presumed emotional stateusing a strategy where specific features of theexpression are associated with specific emotions(Buck 1984). Thus, upturned corners of the mouth
Phil. Trans. R. Soc. B (2009)
or lowered brows are recognized as smiles or frownsand a perceiver can thus conclude that the individualis happy or angry, respectively. This approach breaksdown when the features are either too weak to beclassified or lead to contradictory conclusions—suchas would be the case when a person both smiles andfrowns at the same time. Pattern matching is the prin-cipal strategy tested in the studies referred to above,where participants typically rate the emotions fromintense expressions shown by several unknown others.
The second decoding strategy depends upon theknowledge that the perceiver possesses regardingthe sender and/or the social situation in which theinteraction is taking place. This information permitsthe perceiver to take the perspective of the encoderand helps him or her to correctly infer the emotionalstate that the sender is most likely experiencing.Thus, learning that someone’s car was vandalizedwould lead to the expectation that the person is prob-ably angry. This in turn would influence expectationsregarding the likely expression shown—dependingalso on knowledge about the temperament of theperson. If the sender and the receiver know eachother well, the receiver usually is aware of the sender’spersonality, beliefs, preferences and emotional style.This knowledge then permits the receiver to take theperspective of the sender and to deduce how thesender would most likely react in the given situation.Thus, we may expect more intense anger from acholeric person than from an easy-going person. Butwhat happens when we do not know the otherperson well or at all?
Studies in which people are asked to judge the likelypersonality of complete strangers show that people canand do draw conclusions about a person’s personalityfrom no more information than what is provided bythe face (e.g. Ambady et al. 1995). Yet more impor-tantly, the face tells us a great deal about the socialcategories into which our interaction partners fit.That is, faces tell us the sex, age and race of theother person and this knowledge can be used by obser-vers to predict the likely emotional reactions of thesender. Thus, for example, when imagining that anindividual’s car had been vandalized, participants pre-dicted that a man would show anger but a womansadness (Hess et al. 2005).
A further example of the impact on the decoding ofsuch generalized beliefs based on social group mem-bership inferred from facial appearance is providedby Matsumoto et al. (1999), who studied judgementsof emotion expressions by the Americans andJapanese. The Americans are usually encouraged toshow emotions, especially positive emotions and tendto show an emotion more intensely than is warrantedby the underlying feeling state. This is not the casein Japan. Consequently, the Americans attribute lessintense underlying emotional arousal to expressionsof the same objective intensity than do the Japanese,that is, they ‘correct’ their estimate of a person’s feelingstate based on the belief that people are likely toexaggerate their expressions.
The social group context can impact not only on theknowledge used to interpret an expression but also onthe specific nature of the face on which they are
Review. The face is not a canvas U. Hess et al. 3499
displayed. Thus, facial morphological differencesbetween men and women or members of differentracial groups may enhance or obscure some expressiveelements and hence bias pattern matching. The facialmorphology of women and younger individuals, forexample, appears to enhance the cues associated withhappiness, whereas those of men and older individualsenhance the cues associated with anger (Becker et al.2007; Hess et al. 2009b). Further, certain facial con-figurations make neutral faces appear to showemotions. Thus, a shorter distance between the eyesand mouth, more typical for male faces, leads to theperception of an angrier face (Neth & Martinez 2009).
4. THE FUNCTIONAL EQUIVALENCEHYPOTHESISIn summary, there are at least two important reasonsas to why the same facial expressions shown by twoindividuals may not be interpreted in the same way.First, the beliefs that we have about the individualsmay lead us to different conclusions regarding theirlikely underlying emotional state and second, facialfeatures and facial expressions may interact such thatpattern-matching errors are made.
As regards the latter, Darwin (1872/1965) firstsuggested that some emotion expressions may actuallyimitate morphological features. For example, he notedthat piloerection and the utterance of harsh sounds by‘angry’ animals are ‘voluntarily’ enacted to make theanimal appear larger and hence a more threateningadversary (see pp. 95 and 104). This notion of aperceptual overlap between emotion expressions andcertain trait markers, which then influences emotioncommunication, has been more recently taken up byZebrowitz (Zebrowitz & Montepare 2006) as well asby Hess et al. (2007a,b, 2008, 2009a). Specifically,Hess et al. (2007a,b) proposed the notion that someaspects of facial expressive behaviour and morphologi-cal cues to dominance and affiliation are equivalent intheir effects on emotional attributions, the functionalequivalence hypothesis. In what follows, we will explorethe impact of beliefs on the one hand and the impactof facial morphology on the other on perceptions ofemotions in men and women.
5. BELIEFS ABOUT THE EMOTIONALITYOF OTHERSPeople hold stereotypical beliefs about the emotions ofothers based on a number of characteristics, which arereadily—albeit not always accurately (Ambady et al.1995)—discernable from a person’s face. In additionto beliefs about ethnicity, as in the Matsumoto et al.(1999) study described above, they also hold beliefsabout the emotions of others based on their sex, ageand personality.
Thus, for example, women are expected to smilemore and in fact also do smile more. By contrast,men are expected to show more anger but do notseem to in fact do so (Fischer 1993; Brody & Hall2000). These expectations are socialized early andcan have dramatic consequences on the perception ofemotion in others. Even children as young as 5 years
Phil. Trans. R. Soc. B (2009)
tend to consider a crying baby as ‘mad’ when thebaby is purported to be a boy but not when it is pur-ported to be a girl (Haugh et al. 1980). Thus, the‘knowledge’ that the baby was a boy or a girl, biasedthe perception of the otherwise ambiguous emotiondisplay. In line with these expectations, expressionsfrom a standardized set of expressions—assuringequivalence across genders—were rated as moreintense when anger was shown by a male actor andwhen happiness was shown by a female actor thanvice versa (Hess et al. 1997). That is, the stereotypicalview that associates anger more strongly with menand happiness more strongly with women biases thedecoding of emotional behaviour.
People also have beliefs about age and emotionality.In a recent study, we showed participants photos ofindividuals from four different age groups (18–29,30–49, 50–69 and 70þ) and asked them to indicatehow likely they thought that the person shown in thephoto would express each of four emotions (happiness,sadness, anger and fear) in everyday life. Theresponses differed with regard to both sex and age.Thus, as they get older, men were perceived to beless likely to show anger, whereas the reverse was thecase for women. Men were also perceived as morelikely to show sadness as they got older.
An individual’s perceived social dispositions areanother source of strong beliefs that can impact thedecoding of emotional expressions. Certain relativelystatic facial features are strongly associated with domi-nance and affiliation. Specifically, a high forehead, asquare jaw and thicker eyebrows have been linked toperceptions of dominance (e.g. Keating et al. 1981a,b;Zebrowitz 1997) whereas a rounded baby-face is bothfeminine and perceived as more approachable (Berry &Brownlow 1989) and warm (Berry & McArthur1986), central aspects of an affiliative or nurturingorientation. These behavioural tendencies are also per-ceived as predictive of an individual’s emotionality.Thus, dominant individuals are believed to be morelikely to show anger than are submissive ones, whereasaffiliative individuals are believed to be more likely toshow happiness (Hess et al. 2005).
6. FACIAL APPEARANCE AND BELIEFS ABOUTTHE EMOTIONALITY OF MEN AND WOMENThe beliefs about men’s and women’s emotionalityand beliefs about the emotionality of dominant andaffiliative individuals are not independent. Specifically,Hess et al. (2005) could show that some of the stereo-typical beliefs about men’s and women’s emotions canin fact be traced to beliefs about dominant and affilia-tive individuals. They asked separate groups ofparticipants to rate men’s and women’s neutral faces,either with regard to how dominant or affiliative theyappeared, or with regard to the likelihood that theperson in the photo would show various emotions ineveryday life. Men’s faces were perceived as moredominant in appearance and men were rated as morelikely to show anger, disgust and contempt. By con-trast, women’s faces were rated as more affiliative inappearance and women were expected to be morelikely to show happiness, surprise, sadness and fear.
3500 U. Hess et al. Review. The face is not a canvas
Mediational analyses showed that the tendency toperceive women as more likely to show happiness, sur-prise, sadness and fear was in fact partially mediatedby their higher level of perceived affiliation and lowerlevel of perceived dominance. The tendency to per-ceive men as more prone to show anger, disgust andcontempt was partially mediated by both their higherlevel of perceived dominance and their lower level ofperceived affiliation.
These beliefs are also normative. For example,when presented with a vignette that describes aperson who just learned that their car was vandalized,participants rated the person as very likely to beangry—regardless of whether the person was describedas a man or a woman (Hess et al. 2005). However,even though a man was then expected to show hisanger, a woman was expected to show sadnessinstead—but not if she was described as highly domi-nant. In this latter case, showing anger was expectedfor men and women equally. In a similar vein, menwere expected to show less happiness unless theywere described as high in affiliation, in which casethey were expected to smile even more than women.Thus, the judgement of the appropriateness ofshowing anger or happiness was heavily dependenton the perceived dominance and affiliation of the pro-tagonist, and not just the product of gender categorymembership per se.
In summary, some of the beliefs that people holdabout men and women—and which influence thedecoding of facial expressions shown by men andwomen—can in fact be traced to differences in facialappearance, specifically, to differences in perceivedfacial dominance and affiliation. These differences inturn are owing to the morphological variations thatcharacterize the faces of men and women. Moreover,as we will show below, these differences between thefacial structures of men and women have an evenmore direct impact on facial expression perceptionwhen it comes to pattern matching.
7. PATTERN MATCHINGSpecifically, the facial morphology that makes a faceappear male or female and in turn dominant andaffiliative interacts directly with the movement patternsthat characterize specific emotional expressions.Importantly, some of the perceptual cues that markanger expressions, such as lowered eyebrows andtight lips, mimic features also associated with domi-nance. On the other hand, high eyebrows andsmiling in happiness expressions reinforce affiliativefeatures. The perceptual overlap between facialexpressions of anger and dominance on the one handand facial expressions of happiness and affiliation onthe other was recently demonstrated by Hess et al.(2009b). They used a double oddball task, where par-ticipants had to identify neutral expressions of highlydominant- and highly affiliative-appearing individualsembedded in either a series of angry, or happy, or fear-ful faces. In this type of task, participants are slower toidentify faces as neutral if they share perceptual space(face space; Valentine 1991, 2001) with the emotionalfaces into which they are embedded. That is, the more
Phil. Trans. R. Soc. B (2009)
anger and dominance look alike, the harder it is toidentify a dominant neutral face within a series ofangry faces. By contrast, the identification of theaffiliative neutral faces is comparatively easy. The con-verse is the case for affiliative faces embedded in aseries of happy expressions.
As predicted, participants were slower to identifydominant faces embedded in angry expressions andaffiliative faces embedded in happy expressions thanvice versa. This finding supports the notion that theperceptual markers for anger and dominance, as wellas happiness and affiliation, have some appearancecharacteristics in common. This implies that for allintents and purposes, a highly dominant face looksangry even when no actual facial movement is present.By contrast, highly affiliative neutral faces look happy.Put another way, the facial configurations that createimpressions of dominance and affiliation are thesame that make a face appear to show anger and hap-piness in line with the findings of Neth & Martinez(2009). These perceptual similarities between domi-nance/anger and affiliation/happiness then can beexpected to bias the perception of these emotions,especially when facial expressions are weak andambiguous.
The perceptual overlap between dominant facialmarkers and expressive markers of anger on the onehand, and affiliative facial markers and expressive mar-kers of happiness on the other, also implies that maleand female faces will be reacted to differently. As men-tioned above, men’s faces are perceived as moredominant and women’s as more affiliative. In fact,the high forehead, square jaw and thicker eyebrowsthat have been linked to perceptions of dominance(e.g. Keating et al. 1981a,b; Zebrowitz 1997) are alsomore typical for men’s faces (Brown & Perrett 1993;Burton et al. 1993). On the other hand, a roundedbaby-face with large eyes is more feminine (Brown &Perrett 1993; Burton et al. 1993), perceived as moreapproachable and warm (Berry & Brownlow 1989),and is more typical for women’s faces.
Further, anger expressions signal dominance on thepart of the expresser, whereas happiness expressionssignal affiliation (Knutson 1996; Hess et al. 2000).In turn, perceptions of the dominance and affiliationtendencies of others are relevant to the approach/avoidance dimension. Specifically, in hierarchical pri-mate societies such as ours, highly dominant alphaindividuals pose a certain threat insofar as they canclaim territory or possessions (e.g. food) from lower-status group members (Menzel 1973, 1974). Hencethe presence of a perceived dominant other shouldlead to increased vigilance and preparedness for with-drawal (Coussi-Korbel 1994). By contrast, affiliation isrelated to nurturing behaviours and should lead toapproach when the other is perceived to be high onthis behavioural disposition.
Because anger, dominance and male sex markers onthe one hand and happiness, affiliation and female sexmarkers on the other overlap perceptually in the facespace and are functionally equivalent, anger shownby women and happiness shown by men can beexpected to elicit different reactions from observers.Specifically, when anger is shown on a highly dominant
Review. The face is not a canvas U. Hess et al. 3501
face, the threat signal of the expression and the threatsignal derived from facial morphology are congruentand reinforce each other. By contrast, when anger isexpressed on a highly affiliative face, the two signalscontradict each other and hence weaken the overallthreat message. The converse is true for happinessexpressions (Hess et al. 2007a,b). Following this lineof argument, the female anger expression can beviewed as a combination of an appetitive face with athreatening expression. Male anger, on the otherhand, represents a less ambiguous example of a threatstimulus. Conversely, female happiness is a clearerappetitive stimulus than male happiness.
Hess et al. (2007a,b) tested this hypothesis using astartle-reflex methodology. They assessed both theeye-blink reflex in response to an acoustic startleprobe and the postauricular reflex in response to thesame sound. The eye-blink reflex to a sudden acousticprobe is modulated by emotional state (e.g. Vrana et al.1988; Lang et al. 1990; Lang 1995), such that when anindividual is exposed to a threatening or withdrawalinducing stimulus, the reflex is potentiated. Conver-sely, a pattern opposite to that of the eye-blink reflexis found for the postauricular reflex, the muscleresponse that serves to pull the ears back and up(Berzin & Fortinguerra 1993). Specifically, individualsshow an augmented postauricular reaction to anacoustic startle probe when exposed to appetitivestimuli (Benning et al. 2004).
Congruent with the notion that dominance andaffiliation signals from the face and facial expressionsof anger and happiness interact perceptually, eye-blink startle was potentiated for male anger facescompared with neutral and happy faces, as well ascompared to female anger faces. By contrast, the post-auricular reflex was potentiated for female happinessfaces and attenuated during male anger faces, com-pared to neutral faces as well as to male happinessfaces. Thus, anger, even though this expression signalsthreat (e.g. Aronoff et al. 1988), potentiated eye-blinkstartle only when shown by a man, that is, shown on aface suggesting social dominance. Conversely, thepostauricular reflex was potentiated preferentially forfemale happiness expressions.
In brief, facial features and facial expressions inter-act when it comes to the perception of emotionexpressions. The studies reported above focused onmale and female faces because these represent a natu-ral category differing in facial dominance andaffiliation. But obviously individuals within each sexdiffer on these dimensions and hence we wouldexpect, for example, anger to be more threateningwhen shown on a highly dominant female face andconversely male anger to be less so when shown on ahighly affiliative face. It is important to note that notonly do men and women differ with regard to thesedimensions, but other groups do so as well. Thus,for example, age changes faces such that men’s facesare perceived as increasingly dominant until a veryold age, when they appear as more affiliative.Women’s faces also increase in apparent dominanceas they age. The impact of these age-related changesin facial appearance on emotional attributions iscurrently being investigated in our laboratories.
Phil. Trans. R. Soc. B (2009)
In summary, both beliefs and facial morphologyhave an impact on the perception of the facial move-ment involved in emotional expressions. In everydaylife, these two sources of influence will often be con-founded. Thus, as we have seen, male faces appearas generally more dominant, masculine and maturethan female faces and hence perceptually overlapwith anger expressions. Conversely, social roles aresuch that women are expected to feel less anger andmore fear and happiness. Yet, whereas beliefs basedon social roles are based on such relatively more malle-able factors as the distribution of power and nurturingversus agentic roles between the genders, facialappearance-based effects are owing to the relative dis-tribution of facial appearance cues associated withperceived dominance and affiliation across genders.That is, these two factors, albeit confounded inour reality, actually represent conceptually differentexplanatory factors.
8. THE ROLE OF BELIEFS AND MORPHOLOGYFOR PERCEPTIONS OF EMOTIONALITYFor all intents and purposes, it is impossible to disentan-gle the unique contribution of gender differences inpower, status, social roles and facial appearancewith regard to perceived emotionality in our society.In Western countries, men tend to occupy powerfulsocial positions in politics and business, and in manycountries of this world, they exclusively occupythese positions. Women not only exclusively bearchildren but also overwhelmingly are responsiblefor their upbringing, thereby assigning themselves anurturing role.
However, it is not uncommon in science fiction toquestion gender roles and to imagine worlds wheresuch roles are different from ours. This may includethe addition of genders other than male and femaleor the redistribution of child rearing tasks (e.g.Cogenitors, in Star Trek Enterprise episode no. 48).Hess et al. (submitted) therefore created a science-fiction scenario in which a planet (Deluvia) is inhab-ited by members of a race that has three genders:male, female and carer. They manipulated socialroles by describing the male and female as exactlyequal in social dominance, whereas a submissive andnurturing role was assigned to the carer who wasdescribed as being entirely responsible for the bearingand upbringing of the young. Facial appearances of themembers of each gender were manipulated to be high,medium or low in facial cues to dominance. Partici-pants read a description of Deluvia and itsinhabitants and then rated the likelihood that aDeluvian would experience various emotions. Theresults showed that social roles and facial appearanceshad varying but comparable impact on these percep-tions. Sex per se, however, did not influence theratings significantly. These findings suggest thatthe beliefs we have about men’s and women’s emotion-ality are indeed a composite of the emotions that areassociated with nurturing versus agentic social roleson the one hand, and the conclusions we draw abouta person’s emotional behaviour based on the socialsignals that facial morphology transmits on the other.
3502 U. Hess et al. Review. The face is not a canvas
9. THE FACE AS A SOCIAL SIGNALLINGSYSTEMThe line of research presented here shows that boththe face and facial expressions of emotion have socialsignal value and that these signals interact in complexways. Importantly, this means that when we perceiveand react to the emotional facial expressions ofothers, it really matters who shows what and inwhich context. The appearance of the sender, whatwe know—or think we know—about them and thesituation, and the expressive movements themselves,all contribute to this process. In the present context,we have focused on expressive movements capturedin stills, that is, the departure from the neutralexpression, however, the dynamic aspects of facialmovement, such as the speed of onset and offset oraccompanying head movements also add to the picture(Boker et al. in press).
The present article has focused on the behaviouraltendencies of dominance and affiliation and theirrelation to sex on the one hand and the facialexpressions of anger and happiness on the other. How-ever, dominance and affiliation are not the onlypersonality characteristics that can be deduced fromthe face and that could interact with our interpret-ations of emotion expressions. Thus, for example,Becker et al. (2007) found that perceived masculinityinfluences the decoding of anger expressions; also,facial maturity interacts with perceptions of fear(Marsh et al. 2005; Sacco & Hugenberg 2009).Specifically, Becker et al. (2007) proposed thatanger has evolved to mimic masculinity, whereas hap-piness has evolved to mimic femininity. They couldshow that faces which vary in masculinity cues (asmanipulated by the eyebrow ridge) are rated as moreangry to the degree to which they are perceived asmore masculine. That is, they found some physicaloverlap between masculine morphology and an angryappearance. Further, Marsh et al. (2005) found thatfear shares perceptual overlap with baby-facedness,whereas anger shares overlap with a mature appear-ance, a finding more recently replicated by Sacco &Hugenberg (2009).
In a related vein, anger and happiness have alsobeen associated with another evolutionarily importantbehavioural intention, trustworthiness (Todorov2008). Thus, trustworthy faces that expressed happi-ness were perceived as happier than untrustworthyfaces, and untrustworthy faces that expressed angerwere perceived as angrier than trustworthy faces(Oosterhof & Todorov 2009).
This slowly increasing list of evolutionarilyimportant behavioural dispositions that people inferfrom facial traits and that interact directly with theperceptions of facial emotion expressions underlinesthe importance of emotions for social signalling.Specifically, the interpretation of facial features assignalling behavioural intentions is an extension ofthe signal value of facial expressions (Zebrowitz et al.2003; Hess et al. 2007a,b, 2008; Todorov 2008).What this means in the context of decoding facialexpressions is that the face is not a blank canvas butis like a musical instrument that imbues the expressionwith its own timbre.
Phil. Trans. R. Soc. B (2009)
10. IMPLICATIONS FOR COMPUTATIONALMODELLINGThe importance of facial morphology for the interpret-ation of and reactions to emotional facial expressionshas relevance for the computational modelling ofhuman faces. Specifically, humans tend to treat com-puters in these interactions largely as they wouldtreat humans (Reeves & Nass 1996). In this frame-work, computer agents and robots have beendesigned that not only can interpret human emotionsbut also can signal emotions via facial expressions(e.g. Koda & Maes 1996; Breazeal 2003; Pelachaud &Bilvi 2003). However, the research presented abovesuggests that human beings do not restrict themselvesto facial expressive information when judging theemotions of others. Rather they use all the availableinformation provided by the face, and these additionalsources of information can reinforce or obscure theemotional message transmitted by the facialexpressions. It can be expected that this applies alsoto the avatars and agents used in human computerinterfaces. Thus, an agent with a very square mascu-line face may not be a good choice if warmth andcare are to be transmitted.
Preparation of this manuscript was supported by grantLX0990031 from the Australian Research Council to U.H.,as well as a grant from the Social Science and HumanitiesResearch Council to U.H. and a National ScienceFoundation grant no. 0544533 to R.E.K., U.H. and R.B.A.
REFERENCESAmbady, N., Hallahan, M. & Rosenthal, R. 1995 On judging
and being judged accurately in zero-acquaintance situ-ations. J. Pers. Soc. Psychol. 69, 518–529. (doi:10.1037/0022-3514.69.3.518)
Aronoff, J., Barclay, A. M. & Stevenson, L. A. 1988 The rec-
ognition of threatening facial stimuli. J. Pers. Soc. Psychol.54, 647–665. (doi:10.1037/0022-3514.54.4.647)
Becker, D. V., Kenrick, D. T., Neuberg, S. L., Blackwell,K. C. & Smith, D. M. 2007 The confounded nature ofangry men and happy women. J. Pers. Soc. Psychol. 92,
179–190. (doi:10.1037/0022-3514.92.2.179)Benning, S. D., Patrick, C. J. & Lang, A. R. 2004 Emotional
modulation of the post-auricular reflex. Psychophysiology41, 426–432. (doi:10.1111/j.1469-8986.00160.x)
Berry, D. S. & Brownlow, S. 1989 Were the physiognomistsright? Personality correlates of facial babyishness.Pers. Soc. Psychol. Bull. 15, 266–279. (doi:10.1177/0146167289152013)
Berry, D. S. & McArthur, L. Z. 1986 Perceiving character in
faces: the impact of age-related craniofacial changeson social perception. Psychol. Bull. 100, 3–10. (doi:10.1037/0033-2909.100.1.3)
Berzin, F. & Fortinguerra, C. R. 1993 EMG study of theanterior, superior, and posterior auricular muscles in
man. Anatomischer Anzeiger 175, 195–197.Boker, S. M., Cohn, J. F., Theobald, B. J., Matthews, I.,
Mangini, M., Spies, J. R., Ambadar, Z. & Brick, T. R.In press. Something in the way we move: motion, notperceived sex, influences nods in conversation. J. Exp.Psychol.
Breazeal, C. 2003 Emotion and sociable humanoid robots.Int. J. Hum.—Comput. Stud. 59, 119–155. (doi:10.1016/S1071-5819(03)00018-1)
Brody, L. R. & Hall, J. A. 2000 Gender, emotion,and expression. In Handbook of emotions, 2nd edn
Review. The face is not a canvas U. Hess et al. 3503
(eds M. Lewis & J. M. Haviland), pp. 447–460.New York, NY: Guilford Press.
Brown, E. & Perrett, D. I. 1993 What gives a face its gender?
Perception 22, 829–840. (doi:10.1068/p220829)Bruner, J. S. & Tagiuri, R. 1954 The perception of people.
In Handbook of social psychology, 2 (ed. G. Lindzey),pp. 634–655. Cambridge, MA: Addison-WesleyPublishing.
Buck, R. 1984 Nonverbal receiving ability. In The communi-cation of emotion (ed. R. Buck), pp. 209–242. New York,NY: Guilford Press.
Burton, A. M., Bruce, V. & Dench, N. 1993 What’s the
difference between men and women? Evidence fromfacial measurement. Perception 22, 153–176. (doi:10.1068/p220153)
Coussi-Korbel, S. 1994 Learning to outwit a competitor inmangabeys (Cercocebus torquatus torquatus). J. Comp.Psychol. 108, 164–171. (doi:10.1037/0735-7036.108.2.164)
Darwin, C. 1872/1965 The expression of the emotions in manand animals. Chicago, IL: The University of ChicagoPress. (Originally published, 1872).
Ekman, P. 1972 Universals and cultural differences infacial expressions of emotion. In Nebraska Symposium onMotivation, 1971 (ed. J. Cole), pp. 207–283. Lincoln,NE: University of Nebraska Press.
Ekman, P. & O’Sullivan, M. 1991 Who can catch a liar? Am.Psychol. 46, 913–920. (doi:10.1037/0003-066X.46.9.913)
Ekman, P., Sorenson, E. R. & Friesen, W. V. 1969 Pan-cultural elements in facial displays of emotion. Science164, 86–88. (doi:10.1126/science.164.3875.86)
Ekman, P. et al. 1987 Universals and cultural differences in thejudgements of facial expressions of emotion. J. Pers. Soc.Psychol. 53, 712–717. (doi:10.1037/0022-3514.53.4.712)
Elfenbein, H. A. & Ambady, N. 2003 Universals and culturaldifferences in recognizing emotions. Curr. Direct. Psychol.Sci. 12, 159–164. (doi:10.1111/1467-8721.01252)
Elfenbein, H. A., Beaupre, M. G., Levesque, M. & Hess, U.2007 Toward a dialect theory: cultural differences in theexpression and recognition of posed facial expressions.Emotion 7, 131–146. (doi:10.1037/1528-3542.7.1.131)
Fischer, A. H. 1993 Sex differences in emotionality: fact orstereotype? Feminism Psychol. 3, 303–318. (doi:10.1177/0959353593033002)
Fiske, S. T. 2004 Social beings: a core motives approach to socialpsychology. New York, NY: Wiley.
Fridlund, A. J. 1994 Human facial expression: an evolutionaryview. New York, NY: Academic Press.
Haugh, S. S., Hoffman, C. D. & Cowan, G. 1980 The eye ofthe very young beholder: sex typing of infants by young
children. Child Dev. 51, 598–600. (doi:10.2307/1129302)Hess, U., Kappas, A. & Banse, R. 1995 The intensity of
facial expressions is determined by underlying affectivestate and social situation. J. Pers. Soc. Psychol. 69,280–288. (doi:10.1037/0022-3514.69.2.280)
Hess, U., Blairy, S. & Kleck, R. E. 1997 The intensity ofemotional facial expressions and decoding accuracy.J. Nonverbal Behav. 21, 241–257. (doi:10.1023/A:1024952730333)
Hess, U., Blairy, S. & Kleck, R. E. 2000 The influence of
expression intensity, gender, and ethnicity on judgmentsof dominance and affiliation. J. Nonverbal Behav. 24,265–283. (doi:10.1023/A:1006623213355)
Hess, U., Adams Jr, R. B. & Kleck, R. E. 2005 Who mayfrown and who should smile? Dominance, affiliation,
and the display of happiness and anger. Cogn. Emotion19, 515–536. (doi:10.1080/02699930441000364)
Hess, U., Adams Jr, R. B. & Kleck, R. E. 2007a When twodo the same it might not mean the same: the perceptionof emotional expressions shown by men and women.
Phil. Trans. R. Soc. B (2009)
In Group dynamics and emotional expression (eds U. Hess &P. Philippot), pp. 33–52. New York, NY: CambridgeUniversity Press.
Hess, U., Sabourin, G. & Kleck, R. E. 2007b Postauricularand eye-blink startle responses to facial expressions.Psychophysiology 44, 431–435. (doi:10.1111/j.1469-8986.2007.00516.x)
Hess, U., Adams, R. B. J. & Kleck, R. E. 2008 The role of
perceived emotion in first impressions. In First impressions(eds N. Ambady & J. J. Skowronski), pp. 234–254.New York, NY: Guilford Press.
Hess, U., Adams Jr, R. B. & Kleck, R. E. 2009a Inter-group
misunderstandings in emotion communication. InIntergroup misunderstandings: impact of divergent socialrealities (eds S. Demoulin, J.-P. Leyens & J. F. Dovidio),pp. 85–100. New York, NY: Psychology Press.
Hess, U., Adams Jr, R. B. & Kleck, R. E. 2009b The categ-
orical perception of emotions and traits. Soc. Cogn. 27,319–325. (doi:10.1521/soco.2009.27.2.320)
Hess, U., Adams Jr, R. B. & Kleck, R. B. Submitted. Theinfluence of gender, social roles and facial appearanceon perceived emotionality.
Izard, C. E. 1971 The face of emotion. New York, NY:Appleton-Century-Crofts.
Jakobs, E., Manstead, A. S. R. & Fischer, A. H. 1999aSocial motives and emotional feelings as determinants offacial displays: the case of smiling. Pers. Soc. Psychol. Bull.25, 424–435. (doi:10.1177/0146167299025004003)
Jakobs, E., Manstead, A. S. R. & Fischer, A. H. 1999b Socialmotives, emotional feelings, and smiling. Cogn. Emotion13, 321–345. (doi:10.1080/026999399379212)
Jakobs, E., Manstead, A. S. R. & Fischer, A. H. 2001 Socialcontext effects on facial activity in a negativeemotional setting. Emotion 1, 51–69. (doi:10.1037/1528-3542.1.1.51)
Keating, C. F., Mazur, A. & Segall, M. H. 1981a A cross-
cultural exploration of physiognomic traits of dominanceand happiness. Ethol. Sociobiol. 2, 41–48. (doi:10.1016/0162-3095(81)90021-2)
Keating, C. F. et al. 1981b Culture and the perception ofsocial dominance from facial expression. J. Pers. Soc.Psychol. 40, 615–625. (doi:10.1037/0022-3514.40.4.615)
Kirouac, G. & Hess, U. 1999 Group membership and thedecoding of nonverbal behavior. In The social context ofnonverbal behavior (eds P. Philippot, R. Feldman &E. K. Coats), pp. 182–210. Cambridge, UK: Cambridge
University Press.Knutson, B. 1996 Facial expressions of emotion influence
interpersonal trait inferences. J. Nonverbal Behav. 20,165–182. (doi:10.1007/BF02281954)
Koda, T. & Maes, P. 1996 Agents with faces: the effects ofpersonification of agents. In Proc. Hum.–Comp.Interaction ‘96, London, UK, August 1996.
Lang, P. J. 1995 The emotion probe: studies of motivationand attention. Am. Psychol. 50, 372–385. (doi:10.1037/
0003-066X.50.5.372)Lang, P. J., Bradley, M. M. & Cuthbert, B. N. 1990
Emotion, attention, and the startle reflex. Psychol. Rev.97, 377–395. (doi:10.1037/0033-295X.97.3.377)
Manstead, A. S. R., Fischer, A. H. & Jakobs, E. 1999 The social
and emotional functions of facial displays. In The social con-text of nonverbal behavior. Studies in emotion and socialinteraction (eds P. Philippot, R. S. Feldman & E. J. Coats),pp. 287–313. New York, NY: Cambridge University Press.
Marsh, A. A., Adams Jr, R. B. & Kleck, R. E. 2005 Why do
fear and anger look the way they do? Form and socialfunction in facial expressions. Pers. Soc. Psychol. Bull.31, 73–86. (doi:10.1177/0146167204271306)
Matsumoto, D., Kasri, F. & Kooken, K. 1999 American–Japanese cultural differences in judgements of expression
3504 U. Hess et al. Review. The face is not a canvas
intensity and subjective experience. Cogn. Emotion 13,201–218. (doi:10.1080/026999399379339)
Menzel Jr, E. W. 1973 Leadership and communication in
young chimpanzees. In Precultural primate behavior (ed.J. E. W. Menzel), pp. 192–225. Basel, Switzerland:Karger.
Menzel Jr, E. W. 1974 A group of young chimpanzees in a one-acre field. In Behavior of nonhuman primates (eds A. M.
Schrier & F. Stollnitz), pp. 83–153. San Diego, CA:Academic Press.
Neth, D. & Martinez, A. M. 2009 Emotion perceptionin emotionless face images suggests a norm-based
representation. J. Vis. 9, 1–11. (doi:10.1167/9.1.5)Oosterhof, N. & Todorov, A. 2009 Shared perceptual basis
of emotional expressions and trustworthiness impressionsfrom faces. Emotion 9, 128–133. (doi:10.1037/a0014520)
Pelachaud, C. & Bilvi, M. 2003 Computational model of
believable conversational agents. In Communication inMultiagent systems: agent communication languages andconversation policies (ed. M.-P. Huget), pp. 300–317.Heidelberg, Germany: Springer.
Reeves, B. & Nass, C. 1996 The media equation. Stanford,
CA: CSLI Publications.Sacco, D. F. & Hugenberg, K. 2009 The look of fear and
anger: facial maturity modulates recognition of fearfuland angry expressions. Emotion 9, 39–49. (doi:10.1037/a0014081)
Todorov, A. 2008 Evaluating faces on trustworthiness: anextension of systems for recognition of emotions signaling
Phil. Trans. R. Soc. B (2009)
approach/avoidance behaviors. Ann. NY Acad. Sci. 1124,208–224. (doi:10.1196/annals.1440.012)
Valentine, T. 1991 A unified account of the effects of distinc-
tiveness, inversion and race on face recognition. Q. J. Exp.Psychol. Hum. Exp. 43A, 161–204.
Valentine, T. 2001 Face-space models of face recognition. InComputational, geometric, and process perspectives on facialcognition: contexts and challenges (eds M. J. Wenger &
J. T. Townsend), pp. 83–114. Hillsdale, NJ: Erlbaum.Vrana, S. R., Spence, E. L. & Lang, P. J. 1988 The startle
probe response: a new measure of emotion? J. Abnorm.Psychol. 97, 487–491. (doi:10.1037/0021-843X.97.
4.487)Wiggers, M. 1982 Judgements of facial expressions of
emotions predicted from facial behavior. J. NonverbalBehav. 7, 101–116. (doi:10.1007/BF00986872)
Zebrowitz, L. A. 1997 Reading faces: window to the soul?Boulder, CO: Westview Press.
Zebrowitz, L. A. & Montepare, J. M. 2006 Theecological approach to person perception: evolutionaryroots and contemporary offshoots. In Evolution andsocial psychology (eds M. Schaller, J. A. Simpson &
D. T. Kendrick), pp. 81–114. New York, NY:Psychology Press.
Zebrowitz, L. A., Fellous, J., Mignault, A. & Andreoletti, C.2003 Trait impressions as overgeneralized responses toadaptively significant facial qualities: evidence from con-
nectionist modeling. Pers. Soc. Psychol. Rev. 7, 194–215.(doi:10.1207/S15327957PSPR0703_01)
