1

Wearable Affective Life-Log System forUnderstanding Emotion Dynamics in Daily Life

Byung Hyung Kim and Sungho Jo, Member, IEEE

Abstract—Past research on recognizing human affect hasmade use of a variety of physiological sensors in many ways.Nonetheless, how affective dynamics are influenced in the contextof human daily life has not yet been explored. In this work, wepresent a wearable affective life-log system (ALIS), that is robustas well as easy to use in daily life to detect emotional changes anddetermine their cause-and-effect relationship on users’ lives. Theproposed system records how a user feels in certain situationsduring long-term activities with physiological sensors. Based onthe long-term monitoring, the system analyzes how the contextsof the user’s life affect his/her emotion changes. Furthermore,real-world experimental results demonstrate that the proposedwearable life-log system enables us to build causal structures tofind effective stress relievers suited to every stressful situation inschool life.

Index Terms—Causality, Deep learning, Daily life, EEG, Emo-tional Lateralization, Emotion Recognition, Physiological Signals,PPG, Lifelog, LSTM

I. INTRODUCTION

PEOPLE experience various emotions from a single eventin different situations. For instance, today’s coffee is not

always the same as yesterday’s coffee. The cup of coffee wedrank today may not be as enjoyable as the cup of coffeewe drank yesterday. While drinking coffee generally helps toreduce a person’s stress, the stress-relieving effects of coffeemay vary from day to day for many reasons. For a person wholikes calm and quiet surronding, a cup of coffee drunk todayin a crowded coffee shop with distracting background noise islikely to be less enjoyable than a cup of coffee drunk yesterdayin the quiet kitchen of one’s own home. This instance showsthat a person can have different emotional responses to thesame life events in different circumstances.

Why and how does a person experience various emotionsfrom a single event under different situations? Answering thisquestion could improve human life in a variety of ways, as byimproving physical health. People who suffer from depressionare more vulnerable to heart disease than people with nohistory of depression. Therefore, discovering life elementsrelated to depression and offering guidance to avoid suchelements can help sufferers to lessen their suffering and lead ameaningful life. In response to this question, recent researcheson recognizing human affect has made use of a varietyof physiological sensors in many ways. Such miniaturizedphysiological sensors and advanced mobile computing tech-nologies have enabled people to monitor their physiological

B. Kim and S. Jo are with the School of Computing, KAIST, Republic ofKorea. e-mail: {bhyung, shjo}@kaist.ac.kr.

S. Jo is the corresponding author.

signals continuously in daily life with so-called “everydaytechnology” [1], [2], [3], [4]. Using these sensors, featuressuch as heart rate variability, pulse oximetry, and galvanic skinresponse have been used to capture emotional changes so as tohelp us understand the etiology of mental health pathologiessuch as stress.

However, how affective dynamics are influenced in thecontext of human daily life has not yet been explored. Moststudies [5] have been limited to laboratory environmentsassessed by self-report tools [6] or specialized questionnaires.These methods, however, provide evidence only on immediateaffect from a single event and provide only a limited under-standing of affective dynamics. In our study, understandingaffective dynamics surrounding daily life mainly means dis-covering the causal relations between life contents and humanaffect in daily life. This understanding includes the correctaffect elicitation mechanism and its effect on the emotionalresponse. However, building reliable automated systems forunderstanding affective dynamics is a challenging probleminvolving various noises, a low signal-to-noise ratio (SNR) ofphysiological signals, inter- and intra-subject variability, andusability.

A. Contribution

Solving these challenges, we present a wearable affectivelife-log system, ALIS, which helps to bridge the gap betweenthe low-level physiological sensor representation and the high-level context sensitive interpretation of affect. The proposedsystem records how a user feels in certain situations duringlong-term activities using physiological sensors. Based on thelong-term monitoring, the system analyzes how the contextsof the user’s life affect his/her emotion changes. Notably,the contributions of the proposed system, as an alternative toexisting works, are:• A wearable life-log system for capturing human affect:

Easy to use in daily life. The proposed system is designedto provide comfort and performance to users during theirlong-term activities. Our device offers user advancedusability to users in their daily life.

• A physiological affect model for capturing and tracingaffective dynamics in real-world scenarios: By extendingour previous work on physiological affect-based emotionrecognition [7], we present a physiological affect networkto learn affective dynamics with minimal supervision.Unlike previous works [5], [8] that focused on a spe-cific personalized assessment for every event, our modelcaptures affective states and continuously traces their

arX

iv:1

911.

0107

2v2

[cs

.AI]

5 N

ov 2

019

2

changes, exploiting unlabeled and unbalanced real-worlddata through semi-supervised learning.

• An asymmetric causal influence model of the life content-affect relationship: We present an asymmetric measure toidentify the causal relationship between affective contentsand human emotion in daily life. The model for the firsttime allows users to understand when, what, and howtheir surroundings affect them unawares in their daily life.

• Extensive experiments to evaluate our proposed systemin a long-term series of life-logging over several daysin real-world scenarios, through the evaluation of whichwe demonstrate how emotion dynamics are influenced bythe affective contents of human daily life using variouscontextual information acquired over up to 60 consecutivedays.

B. Problem Statements and Challenge Issues

Our goal is to determine affective causality in daily lifeby analyzing emotional states and life contents encounteredin various situations. We present the problem formulationof modeling the causal relation between life contents andemotional changes in the proposed ALIS. Without loss ofgenerality, we treat the problem as a combination of two-stageproblems. The first stage is the emotion recognition problem.Given physiological signals, the problem is the identificationof the correct emotional state as

y = argmaxy∈YP (y|X1,X2, . . . ,Xt), (1)

where Xt is a segment of physiological signals at time t andY is the set of emotional states such as happiness, surprise,anger, fear, and sadness.

The second stage is the affective causality learning problem.The problem is formulated as a graph

G = (M, C, E), (2)

whereM∈ RNs×Ne×T is the emotion occurrence matrix, Nsis the number of situations and Ne is the number of emotionalstates. C ∈ RNs×Nc×T is the life contents occurrence matrix,where Nc is the number of contents and eij ∈ E indicates theaffective causal effect of sequence Ci on sequence Mj .

The proposed ALIS is built upon our previous work onrecognizing human emotion [7]. Although the system hasshown merits in discriminating physiological signals in differ-ent emotional states, non-trivial and challenging issues existwhen the model is applied to demonstrate its efficacy in dailylife. There are two main challenging problems in real-worldenvironments as follows.

1) Limited and unbalanced labeling problem in emotionrecognition (C.1): Despite the reliablility of the prior workin emotion recognition, obtaining accurate results of emotionrecognition becomes a severe challenge when physiologicalsignals accompany unreliable labels. In real-world scenarios,only a few labels are available where humans rate their feelingin indoor environments using self-reporting tools. The limitedand unbalanced class labels deteriorate the performance ofclassifiers in a broad range of emotion classification problemsin (1).

2) Sparsity problem in affective causality identification(C.2): Discovering affective causality is essential to the un-derstanding of users’ affective dynamics in their daily life.However, it is unclear whether emotional causality exists inreal-world situations where a user encounters life events inlong temporal sequences. For the most part the issue of howto identify the causality direction and discover the influence oflatent factors are challenging, as emotions are usually affectedby various complex and subtle factors in daily life.

ALIS is designed to solve the two problems (1) and (2),addressing the challenge issues (C.1) and (C.2) along withadvanced wearable sensor technologies for multi-modal ac-quisition toward a deeper understanding of how humans actand feel while interacting with their real-world environment.

The rest of this paper is organized as follows: We describeprior work in Section II with respect to emotion recognition,causal inference, and life-logging technologies for monitoringlong-term activities without loss of usability. Section III coversthe preliminaries of our previous work and causal inference.In Section IV, we present our system, describing the systemdesign and framework. Sections V and VI evaluate our systemon synthetic, public, and real data sets. Lastly, we concludesthis paper with a discussion of future work in Section VII.

II. RELATED WORKS

A. Physiology and Multi-Modality for Recognizing HumanAffect

Several physiology-based studies of human affect havebeen conducted that have advanced significantly in manyways over the past few decades [9], [10]. Along with thisresearch area, we focus on using machine learning to identifyphysiological patterns that corresponds to the expressions ofdifferent emotions. In particular, EEG measures the electricalactivity of the brain by recording its spontaneous electricalactivity with multiple electrodes placed on the scalp. Despiteits low spatial resolution on the scalp, its very high temporalresolution, non-invasiveness, and mobility are valuable in real-world environments such as clinical applications [11], [12].

Most EEG-based emotion recognition systems have ex-tracted and selected EEG-based features through electrodeselection based on neuro-scienctific assumptions. Zheng etal. have collected EEG signals above the ears and presenteda framework to classify four emotions [5]. They extractedEEG features from the six symmetrical temporal electrodesand showed that EEG has the advantage of classifying happyemotions. Similarly, Wang et al. investigated the characteris-tics of EEG features to assess the relationships with emotionalstates [13]. Their study showed that the right occipital and theparietal lobes are mainly associated with emotions in the alphaband, while the temporal lobes are associated with emotionsin the beta band. Furthermore, the left frontal and righttemporal lobes are associated with emotions in the gammaband. Petrantonakis and Leontios developed adaptive methodsto investigate physiological associations between EEG signalsegments and emotions in the time-frequency domain [14].They exploited the frontal EEG asymmetry and the multidi-mensional directed information approach to explain causality

3

between the right and left hemispheres. These results haveshown that emotional lateralization in the frontal and temporallobes can be a good differentiator of emotional states.

EEG-based emotion recognition systems have often shownimproved results when different modalities have beenused [15], [16], [17]. Among the many peripheral physiolog-ical signals, PPG, which measures blood volume, is widelyused to compute heart rate (HR). It uses optical-based tech-nology to detect volumetric changes in blood in peripheralcirculation. Although its accuracy is considered lower thanthat of electrocardiograms (ECGs), it has been used to developwearable biosensors in clinical applications such as detectingmental stress in daily life [18] due to its simplicity. HR, aswell as heart rate variability (HRV), has been shown to beuseful for emotion assessment [19], [20], [21]. Over the pasttwo decades, some reports have shown that HRV analysis canprovide a distinct assessment of autonomic function in boththe time and frequency domains. However, these assessmentsrequire high time and frequency resolutions. Due to theserequirements, HRV has only been suitable for analyzing long-term data. Several researchers have focused on overcomingthis limitation. Valenza et al. [8] have recently developeda personal probabilistic framework to characterize emotionalstates by analyzing heartbeat dynamics exclusively to assessreal-time emotional responses accurately.

In these studies, distinct or peaked changes of physiologicalsignals in the time or frequency domains at a single instan-taneous time have been considered candidates. However, thisapproach is limited and cannot be used to fully describe emo-tion elicitation mechanisms due to their complex nature andmultidimensional character. To overcome this problem, thiswork formulates emotion recognition as a spectral-temporalphysiological sequence learning problem.

B. CausalityUnderstanding causal direction is essential to predicting the

consequence of any intervention from a group of observationsamples and is critical to many applications, including biologyand social science [22]. We should note that causal learning isdifferent from mainstream statistical learning methods in thatit aims to discover the data-generation mechanism instead ofcharacterizing the joint distribution of the observed variables;that is the most significant difference between causality andcorrelation. Discovering emotional influence has been a focusfor testing whether the correlation exists in real-world applica-tions. Wu et al. [23] developed a model to learn the emotionaltags of social images and to verify the correlation betweenuser demographics and emotional tags of social images. Jia etal. [24] exploited the social correlation among images to inferemotions from images. Their results showed that the emotionsevoked by two consecutive images can be associated with eachother from the same user uploads. Yang et al. [25] reported thatthe ability to emotionally influence others is closely associatedwith a user’s social roles in image social networks.

C. Life-Log Technologies for Monitoring Long-Term ActivitiesSelf-tracking through logging various personal metrics is

itself is not new. For instance, people with chronic conditions

such as diabetes have long been monitoring their personalmetrics to figure out how daily habits influence their symp-toms. From a physiological point of view, EDA has beenused to measure emotional changes for over 125 years [2].However, these methods have all been limited to highly con-trolled environments because of the complex and cumbersomemachinery involved. In order to monitor long-term activitiesin any environments, wearable technologies are required tobe “as natural and unnoticeable as possible”. Over the pastdecade, there have been significant advances in miniaturizingEDA sensors. Pho et al. [26] developed a compact and low-costwearable EDA sensor to monitor long-term activities duringdaily activities outside of a laboratory. Garbarino et al. [1]presented a wearable wireless multi-sensor device, “EmpaticaE3,” which has four embedded sensors: PPG, EDA, 3-axisaccelerometer, and temperature. The device is a new wrist-wearable sensor that is suitable for real-life applications. Wear-able design techniques also have been developed satisfyingthe conditions for wearability. Williams et al. [3] presentedSWARM, a wearable affective technology which includes amodular soft circuit design technique. SWARM is simply ascarf, but its modular actuation components can accommodateusers’ sensory capabilities and preferences so that the userscan cope with their emotions.

III. PRELIMINARIES

A. Deep Physiological Affect Network (DPAN) for Recogniz-ing Human Emotions

DPAN describes affect elicitation mechanisms used to detectemotional changes reflected by physiological signals. It takestwo-channeled EEG signals underlying brain lateralization anda PPG signal as inputs and outputs a one-dimensional vectorrepresenting emotional states scaled from 1 to 9. Suppose thatDPAN obtains the physiological signals at time N over aspectral-temporal region represented by an M×N matrix withP different modalities. From the two modalities of EEG andPPG sensors, physiological features Bt and Ht are extractedfrom the respective sensors as follows.

Bt = ξrl ◦(ζl − ζr)

(ζl + ζr), (3)

where ‘◦’ denotes the Hadamard product. (ζl−ζr)(ζl+ζr)

representsthe spectral asymmetry and the matrix ξrl is the causalasymmetry between the r and l EEG bipolar channels. Thebrain asymmetry feature Bt describes the directionality andmagnitude of emotional lateralization between the two hemi-spheres.

DPAN extracts the heart rate features Ht over the M ×Nspectral-temporal domain, where frequencies with peaks inthe PSD of the PPG signal are regarded as candidates ofthe true heart rate from the PPG signal Pt at each timeframe t. These data form a candidate set over time. Theobservation at a given time can then be represented by a tensorX ∈ RM×N×P , where R denotes the domain of the observedphysiological features. Then the learning problem then is theidentification of the correct class based on the sequence oftensors X1,X2, . . . ,Xt.

y = arg maxy∈Y

P (y|X1,X2, . . . ,Xt), (4)

where Y is the set of valence-arousal classes. Fig. 1 shows

4

Fig. 1. (a) An overview of DPAN. After every time interval N , the proposedDPAN first extracts two physiological features (brain lateralized and heartbeatfeatures) and constructs a spectral-temporal tensor. These features are then fedinto ConvLSTM to compute affective scores of emotions via our proposedloss model, temporal margin-based loss (TM-loss). The output at the finalsequence is selected to represent an emotion over a 2-dimensional valence-arousal model for the entire sequence. (b) The discriminative margin mt(y)(red line) of an emotion y started at t0. The margin mt(y) is computed as thedifference between the ground truth affective score st(y) (blue line) and themaximum scores maxy′ 6=y st(y

′) (dashed blue line) of all incorrect emotionstates between t0 and t. The model becomes more and more confident inclassifying emotion states until the time tc. However, after the time tc, Lt

are non-zero due to the violation of the monotonicity of the margin.

the entire overview of DPAN for the recognition of emo-tions. To solve the learning problem, DPAN feeds spectral-temporal tensor-based physiological features into ConvLSTMsto compute affective scores of emotions via the proposed lossmodel, temporal margin-based loss(TM-loss). The proposedTM-loss aims to learn the progression patterns of the emotionsin training for developing reliable affect models. The TM-lossis a new formulation based on the temporal margin betweenthe correct and incorrect emotional states. The reasoning forusing the formulation is as follows:• When more of a particular emotion is observed, the model

should be more confident of the emotional elicitation asthe recognition process progresses.

The function constrains the affective score of the correctemotional state to discriminate its margin, which does notmonotonically decrease with all the others while the emotionprogresses.

Lt = − log st(y) + λmax(0, maxt′∈[t0,t−1]

mt′(y)−mt(y)), (5)

where − log st(y) is the conventional cross-entropy loss func-tion commonly to train deep-learning models, y is the groundtruth of emotion rating, st(y) is the classified affective scoreof the ground truth label y for the time t, and mt(y) is thediscriminative margin of the emotion label y at time t.

mt(y) = st(y)−max{st(y′)|∀y′ ∈ Y, y′ 6= y}. (6)

λ ∈ Z+ is a relative term to control the effects of the discrim-inative margin. As described in (5) and (6), a model becomes

Fig. 2. An Overview of ALIS. ALIS consists of an Affective Contents Col-lector (ACC), Affective Dynamics Network (ADNet), and Affective CausalityNetwork (ACNet). ACC collects an user’s contextual information in situationswith frontal images and emotion measured by EEG and PPG signals. Giventhis information, ADNet detects emotional changes, which are used as an inputwith frontal images for ACNet to discover the causal relationship betweenemotions and situations.

more confident in discriminating between the correct state andthe incorrect states over time. With this function DPAN isencouraged to maintain monotonicity in the affective score asthe emotion training progresses. As shown in Fig. 1(b), afterthe time tc, the loss becomes non-zero due to the violation ofthe monotonicity of the margin. Note that the margin mt(y)of the emotion y spanning [t0, t] is computed as the differencebetween the affective score st(y) for the ground truth y and themaximum classification scores maxy′ 6=y s(y

′) for all incorrectratings at each time point in [t0, t].

B. Conditional Independence Test

To discover the causality between affective dynamics andlife contents and answer (C.2), we test the conditional inde-pendence with three sequences Ai, Aj , and Ak. Suppose thethree sequences have the same length T , the test is to verifythe statistical significance of the statement Ai ⊥ Aj |Ak. Con-sidering each triplet (Ai(t), Aj(t), Ak(t)) for each 1 ≤ t ≤ Tas a sample over three variables, a 3-D contingency table Crecords the number of triplet samples on the three variablessuch that

Copq = |{1 ≤ t ≤ T |Ai(t) = o,Aj(t) = p,Ak(t) = q}|. (7)

Note that the expectation of Copq under the null hypothesiscan be estimated by:

E(Cops) = (C∗pqCo∗q)/(C∗∗q), (8)

where C∗pq , Co∗q , and Cop∗ are the marginals of the countswith Ai, Aj , Ak, respectively. For the test, we use the standardG2 conditional independence test, which returns the Kullback-Leibler divergence between the distributions of Copq andE(Copq) over all three variables:

G2 = 2∑o,p,q

CopqlnCopq

E(Copq), (9)

which follows a χ2 distribution with degree of freedom (|Ai|−1) ∗ (|Aj | − 1) ∗ |Ak|. For removing the sparsity in the tableC, the degree of freedom is penalized by the number of zerocells as in [27].

5

Fig. 3. The Affective Contents Collector and its components; IMU, EEG,PPG sensors, and a tiny frontal camera. The location of two EEG electrodes(F3, F4) on the 10-20 international system.

IV. WEARABLE AFFECTIVE LIFELOG SYSTEM

The Wearable Affective Lifelog System (ALIS) consists ofthree main parts: The Affective Contents Collector (ACC),Affective Dynamics Network (ADNet), and Affective Causal-ity Network (ACNet). Fig. 2 shows the entire framework ofALIS. First, ACC gathers contextual information continuouslysurrounding the wearer in daily life. The logged data are thentransferred into ADNet, which aims to find answers aboutunder which situations and to what extent a human feelsthe elicited affect. ACNet discovers the dynamic causal re-lationship between the situation faced and the human emotionexplored by ADNet. It provides an intuitive understanding ofaffect dynamics for users. The following sections describe thedetails of each component.

A. Affective Contents Collector for Logging Data

ACC is a simple device designed to be easily wearable forusers to act freely in everyday situations (See Fig. 3) as wellas to collect human affect correctly. Since human affect issophisticated and subtle, it is vulnerable to personal, social,and contextual attributes. The noticeability and visibility ofwearable devices could elicit unnecessary and irrelevant emo-tions. Therefore, recording human affect should be unobtrusivewhen measured in the natural environment. To design anunnoticeable device, we imitated the design of existing easy-to-use wireless headsets. We note that the term “unobtrusivedevice” in this paper means that it is not easily noticed ordoes not draw attention to itself. The term does not implythat our device aims to be small or concealable. This easy-to-use device provides comfort and performance to users duringlong-term activities.

Everyday technology requires wearable systems to have theunprecedented ability to perform the comfortable, long-term,and in situ assessment of physiological activities. However, thedevelopment of practical applications is challenging becauseof the cumbersomeness of the equipment that requires multiplechannels to obtain reliable signals and the complexity ofsetting up experiments. To use body sensors with a guaranteeof both reliability and simplicity, we designed the proposeddevice with a tiny PPG sensor and a minimum configuration ofa two-channel EEG, the lowest number of channels necessary

Fig. 4. An overview of ADNet. The model first trains physiological signalsassociated with their labels y on the dataset Ds by DPAN (red lines). Themodel then uses the learned parameters on the dataset D for predicting noisypseudo-labels y (blue lines), which are fed into a sub-network Label Cleanerconditioned on physiological features χ from ConvLSTMs. Within the LabelCleaner network, a residual architecture learns the difference between thenoisy y and clean labels y (green lines). Finally, the model predicts cleanedlabels y penalized by a joint loss function (LC-loss).

to learn patterns of lateralized frontal cortex activity involvedin human affect.

While satisfying the two criteria, our device consists ofmultimodal sensors to capture various emotions surroundingdaily life as follows:• Frontal Camera for Collecting Visual Contents: Visual

information has been widely used to detect situationsfaced by the user. The study of recognizing scenes andactivities by analyzing images from a camera has pro-vided an understanding of contextual information. Hence,in our system, a small frontal viewing camera with a 30fps sampling rate (Raspberry Pi Zero Camera) was usedto record images.

• Small Physiological Sensor to Capture Human Affect:The analysis of patterns of physiological changes hasbeen increasingly studied in the context of affect recog-nition. To capture this information, we used a two-channel EEG sensor on OpenBCI on the left and righthemispheres and a small ear-PPG sensor, with samplingrates of 250 Hz and 500 Hz, respectively.

B. Affective Dynamics Network for Recognizing Emotions

ADNet aims to solve the problem in (1), addressing thechallenging questions (C.1) on a real-world large dataset.Suppose the dataset D naturally comprises a large number ofunlabelled physiological signals P and a subset Ds where asmall number of ground-truth labels y by self-reporting areavailable. Our network first learns the representations of thephysiological signals P according to the affective labels on thesubset Ds by DPAN [7] and produces noisy pseudo-labels yon the dataset D. Given the dataset D = {(Pi, yi), ...} and itssubset Ds = {(Pi, yi, yi), ...}, the proposed framework jointlylearns to reduce the label noise and predict more accuratelabels y on the dataset D.

Fig. 4 shows the overview of ADNet. The network firstpredicts sets of affective labels y = (V, A) on the dataset Dupon physiological features learned by DPAN on the subsetDs. While DPAN has shown its superiority in classifying

6

emotions, its predicted labels has contained noise. To over-come this issue, ADNet comprises a sub-network, whichlearns to map noisy labels y to clean labels y, conditionedon physiological features from ConvLSTMs in DPAN. Thenetwork is supervised by the human reported labels y andfollows a residual architecture so that it only needs to learnthe difference between the noisy and clean labels. In particular,to handle the sparsity in noisy labels, ADNet encodes theemotional occurrence y of each of the d classes in valenceV and arousal A ratings into a pair of d-dimensional vector[0, 1]d. Similarly, the model projects the physiological featuresX in to a low dimensional embedding, then all embeddingvectors from the two modalilties are concatenated and trans-formed with a hidden linear layer followed by a projectionback into the high dimensional label space. At the same time,the network shares the physiological features extracted byConvLSTMs in DPAN and learns to directly predict cleanlabels y following a sigmoidal function. The predicted labelsy are supervised by either the output of DPAN or a human.

To train the ADNet we formulate a joint loss function asbelow:

Lc =∑i

|yi − yi| −∑j

[uj log(yj) + (1− uj) log(1− yj)], (10)

where uj is yj when the SAM-ratings are available, otherwiseyj . The LC-loss Lc is the combination of 1) the differencebetween the cleaned labels and the corresponding ground truthverified labels and 2) the cross-entropy to capture the differ-ence between the predicted labels and the target labels. Thecross-entropy term is only propagated to yj . The cleaned labelsy are treated as constants with respect to the classification andonly incur gradients from the LC-loss.

C. Affective Causality Network

ACNet aims to solve the problem in (2). The goal is to de-rive the affective causality by analyzing emotional changes andthe user’s relevant situations. Suppose the emotion sequenceM(1 : T ) is a binary stochastic process during a discrete timeinterval [1, T ], where T is the maximal length of the interval.Then, an element Mj in the sequence M(1 : T ) is a binaryindicator of the occurrence of a certain emotion j at time t.With this notation, Mη

j is generated with each concatenatedelementMη

j (t) = (Mj(t−η), . . . ,Mj(t)) at each timestampt ≤ T . In the same way, an element Ci in a situation sequenceC(1 : T ) is a binary indicator of the occurrence of certainsituation i at time t. Based on these notations, the time-varyingaffective network and the affective causality are defined withthe formulation of the related learning task.

Definition IV.1. The time-varying affective network is denotedas G = (M, C, E), where Ci and Mj are the situation i andthe emotional state j sequences, and eij ∈ E indicates theaffective causal effect of the sequence Ci on the sequenceMj .

Definition IV.2. Given n situation and emotion sequences{C1, . . . , Ci,M1, . . . ,Mj} from a user, the directed graph G isconstructed in the underlying affective causality model whileaddressing confounding factors.

By representing each of the emotion and situation variablesas a node, ACNet could be formulated as a directed graphG over the variable Ci ∪Mj such that each edge Ci → Mj

indicates the affective causal effect of sequence Ci on sequenceMj . In the network, there is a parameter associated with anode or an edge between Ci andMj . Based on the descriptionsabove, a network generally consists of two components: 1) adirected graph G used to model the causal structure among thesequences and 2) association parameters on the edges used tomodel the causal phenomena. These two components charac-terize the global and local properties of the interaction betweenemotions and life contents, respectively, the combination ofwhich provides a general guideline ruling the events happeningon the two sequences.

Affective causality is a computational asymmetric measureto determine the causal relationship between affective dynam-ics and situations. The computation is based on conditionalindependence testing to detect the relationship with latentconfounders underneath the observational two sequences. Thecausal structure learning method is inspired by Cai et al.’swork [22], but differs in that the ACNet has two different vari-ables M and C as input nodes, considering sparse relations.The proposed model analyzes this causal problem, considering1) with and 2) without confounding factors.

1) Learning without confounding factors: The interactionwithout any latent factors can observe that the state Ci attimestamp t− 1 affects the state of Mj at timestamp t, whilethe reverse does not hold. This observation can be formalizedin the language of statistical testing with Lemma 1.

Lemma 1. Given two dependent sequences Ci →Mj withouta connected latent variable, the following asymmetric depen-dence relations hold: 1) there exists a delay ηc satisfyingCi ⊥ M1

j |Cηci and there does not exist a delay ηm satisfying

Mj ⊥ C1i |Mηmj .

Proof. Suppose ηm and ηc are the influence lags of M andC, respectively. In the case of the 1), the state of Ci at t timeis determined only by its previous states Cηci . Hence, Ci ⊥M1

j |Cηci naturally holds. In the case of the 2), there is no

variable ηm to Mj ⊥ C1i |Mηmj because the state of Mj at t

time is directly influenced the previous state of Ci at t−1.

2) Learning with confounding factors: Sequences under theexistence of confounding factors 1) behave similarly becauseof common characteristics or 2) interact independently overthe timeline. Both the situation Ci and emotionMj sequencesare affected by a constant latent variable. In such a case,Ci(t) is thus dependent on the Mj(t− 1) given any previousstates of Ci. Similarly,Mj(t) also depends on Ci(t−1) givenany previous states of Mj . The confounding factor, whichis itself an independent variable over time, is underneath thetwo sequences under test. When this case occurs, a positivestatistical dependence between C and M is observed. But thestates of Ci(t) could be completely independent ofMj(t−1)given its previous states Ci(t − ηc − 1 : t − 1), and Mj(t)could also be independent of Ci(t − 1) given its previousstates Mj(t− ηm− 1 : t− 1). The following lemma could be

7

recognized using the same group of conditional independencetests.

Lemma 2. If there is a latent factor between Ci and Mj ,the following symmetric relations hold: 1) there does not existany delay η satisfying Ci ⊥ M1

j |Cηci nor Mj ⊥ C1i |M

ηmj ,

2) there exists delay ηm and ηc satisfying Ci ⊥ M1j |C

ηci and

Mj ⊥ C1i |Mηmj , respectively.

Proof. For 1) they are dependent on each other in the givencondition set without the latent factor because Ci and Mj aredependent on the latent factor. For 2), suppose ηc and ηmbe the self-influence lag of Ci andMj , respectively, the latentfactor at time t is independent of the latent factor at time t−1.Therefore, Ci ⊥M1

j |Cηci and Mj ⊥ C1i |M

ηmj hold.

Based on the above lemmas, the following theorem con-firms the correctness of asymmetric measure used for causaldirection detectioni with the following statements.S1: ∃ηc satisfying Ci ⊥M1

j |Cηci

S2: ∃ηm satisfying Mj ⊥ C1i |Mηmj

Theorem 1. Given two sequences Ci and Mj , the follow-ing propositions on the causal structure between the twosequences hold: 1) Ci →Mj in G, if S1∧¬S2, 2) 1) Ci ←Mj

in G, if ¬S1 ∧ S2, and 3) there is a latent factor between Ciand Mj , if S1 ∧ S2 or ¬S1 ∧ ¬S2.

Proof. Assuming there are only three directions between twosequences Ci and Mj ; 1) Ci → Mj , 2) Mj → Ci, and3) Ci ← H → Mj . H is a latent factor. For the firstcase, S1 ∧ ¬S2 ⇒ Ci → Mj , and the causal structure ofS1 ∧ ¬S2 based on Lemma 1 cannot be Ci ← Mj . There isno confounding factor between the two sequences. Therefore,the causal structure must be Ci →Mj .

Algorithm 1 Affective Causality Direction LearningInput: Ci : situation i sequenceMj : emotion state j sequenceα : confidence thresholdT : maximal timestampF : affective pair set {Ci,Mj}

Output: The directed graph GInitialization : set G as empty set;

1: for each i and j in a pair F in a situation do2: if @F ′ ∈ F − {Ci,Mj}, Ci ⊥Mj |F ′ then3: Test S1 on Ci and Mj ;4: Test S2 on Mj and Ci;5: if S1 ∧ ¬S2 then6: Add i→ j into G;7: else if ¬S1 ∧ S2 then8: Add j → i into G;9: else

10: Add i← H→ j into G;11: end if12: end if13: end for14: return G

3) Affective Causality Direction Learning Algorithm: Al-gorithm 1 describes the asymmetric relations on all dependentsequence pairs of situation C and emotionial state M anddetects the directions of causal edges on the underlying theaffective model G. In this algorithm, the affective sequenceset F = {C,M} along with the maximal timestamp T , thenumber of sequence N , and the confidence threshold α areused as inputs for the test. Given the inputs, each affectivepair is tested to see whether each is independent of the otherconditioned on other variables. The complexity of Algorithm1 is determined by the dependent pair detection and the testof S1 and S2.

V. EVALUATION ON SYNTHETIC DATASET

In this section, we examine the robustness of the proposedALIS on the two datasets: a public dataset and a syntheticdataset for quantitative evaluation of ADNet in emotion recog-nition and ACNet in causality identification.

A. Affective Dynamics Network Performance

For quantitative evaluation of ADNet in emotion classi-fication, we performed realistic experiments by deliberatelymanipulating labels on a public dataset called DEAP [15].

1) DEAP dataset: DEAP is a public dataset of physiolog-ical signals to analyze emotions quantitatively on a 2D planealong with valence and arousal as the horizontal and verticalaxes. Its physiological signals are recorded from 32-channelEEGs at a sampling rate of 512Hz using active AgCl electrodesplaced according to the international 10-20 system and 13other peripheral physiological signals from 32 participantswhile they watched 40 one-minute-long excerpts of musicvideos. The dataset rates emotions with respect to continuousvalence, arousal, liking, and dominance scaled from 1 to 9and discrete familiarity on scales from 1 to 5 using Self-Assessment Manikins [15].

2) Dataset Configuration: Since ACC gathers EEG signalsfrom a pair of electrodes and a PPG signal, we retrieved datafrom the eight selected pairs of electrodes and a plethys-mograph on the DEAP dataset. The 64 combinations ofphysiological signals per video generates 81,920 physiologicaldata points. We split the dataset into fifths: one-fifth for testingand the remaining for training and validation. The data werehigh-pass filtered with a 2 Hz cutoff frequency using EEGlaband the same blind source separation technique as in [7] forremoving eye artifacts in EEG signals. A constrained inde-pendent component analysis (cICA) algorithm was applied toremove motion artifacts in PPG signals. The datasets for train-ing, testing, and validation were subject-independent, whichmeans they were chosen entirely randomly while keeping thedistribution of ratings balanced. We used the balanced datasetsto evaluate the performance of our models and other methodson solving the limited amount of clean-label problems. Toevalutate the unbalanced labeling problem, we stochasticallychanged the amount of physiological signals associated witha label while varying the distribution of labels in the trainingdataset. The unbalanced dataset comprised p percentage ofphysiological signals according to a pair of random labels in

8

valence and arousal, with the others distributed equally. Thetest data remained unperturbed to allow us to validate andcompare our model to other methods. The highlighted one-minute EEG and plethysmography signals were split into 6frames of 10 seconds each. They were down-sampled to 256Hz and their power spectral features were extracted.

3) Evaluated Methods and Metrics: We evaluated the per-formance of our ADNet, comparing the results with state-of-art methods that have shown their performance on the DEAPdataset in terms of loss functions and the number of layers. Ourproposed system and the following comparative models consistof 256 hidden states and 5×5 kernel sizes for the input-to-stateand state-to-state transition. They were trained by learningbatches of 32 sequences and back-propagation through timefor ten timesteps. The momentum and weight decay were setto 0.7 and 0.0005, respectively. The learning rate starts at 0.01and is divided by 10 after every 20,000 iterations. We alsoperformed early-stopping on the validation set.• Model A: 1-layered ConvLSTMs with a softmax layer

as a baseline classifier. The results from the modelrepresents a typical performance of deep neural networks(DNNs).

• Model B: 1-layered ConvLSTMs with the temporalmargin-based loss function as in [7]. The model hasshown its superiority in recognizing human emotions,increasing the distinctiveness of physiological character-istics between correct and incorrect labels.

• Model C: The model is 4-layered ConvLSTMs with asoftmax layer. The model was implemented as in [28].

All models learn physiological signals to classify d2(d =2, 3, 4) affective states, which are subdivided into each ofd discrete ratings for valence and arousal on 2-D emotionalspace. We choose the mean average precision (MAP) asa metric to evaluate the performance of our system withadditional Precision, Sensitivity, and Specificity to report howcommonly occurring classes in a training set affect the modelperformance.

4) Evaluation Results: Fig. 5 shows that our proposedsystem performed better than the alternatives on the balanceddataset over all dimensions (d = 2, 3, 4) unless the amountof clean labels was very large (> 0.7). In general cases,the model C achieved the next best classification results.Its performance increased rapidly as the number of SAM-rated labels increased. On the other hand, model A con-sistently reported the lowest precision consistently over allconfigurations. This consistency could imply that model Awas overfit and overconfident to some labels. With respect tooverfitting, although having a specialized loss function (modelB) and increasing the number of layers in DNNs (model C)improved discriminative power to classify different emotions,the limited number of clean labels seems to lead to overfittingfor many classes. The three comparative methods showedworse precision when the number of labels d increased, alle-viating the problem of overfitting. Our approach, on the otherhand, does not face this problem. In particular, it performedsignificantly better than the other methods when limited labelsare available from 0.2 to 0.4. Fig. 6 provides a closer lookat how different label frequencies affect the performance in

emotion recognition when the label balances were collapsed byp in terms of precision, sensitivity, and specificity. The resultsshow that the unbalancing in labels increased by the numberof labels generally deteriorated the classification performance.In particular, the vertically stacked layers in model C led itto misunderstand the physiological characteristics associatedwith a major numbers of labels. While the other three modelsbecame overfit to common labels and reduced the overallaccuracies, our approach is most effective for very commonclasses and shows improvement for all but a small subset ofrare classes.

B. Affective Causality Network Performance

1) Experimental Setup: To evaluate the affective causalityidentification on the proposed ACNet, a synthetic data set wasgenerated. In the data set, pairs of affective sequences F aregenerated by simulating a Poisson point process with 10 minper timestamp and ε as occurrence frequency for situations Cand emotions M per day. Each sequence is influenced by itscausal nodes as the time-dependence influence function withexponential probability p(4t, η) = ηe(−4tη), where 4t is thetime interval between t and the causal sequence’s most recentstate, and η is the average influence lag.

2) Evaluated Methods and Metrics: We evaluated the per-formance of ACNet, comparing it against the transfer entropy(TR) method [29] and Granger’s causality (GC) method [30].Two groups of causal structures were designed for experi-ments. The first group has the causal structures of directedgraphs without latent variables, and the second group consistsof directed graphs with latent factors. Given n sequences andthe average in-degree dg of the graph, a pair of two emotionand situation nodes and a directed edge between the pairsinto the graph are selected until there are dg · n edges in thegraph. For the second group, confounding factors are selectedby nc independent pair of nodes and an additional latent factorH underlying two edges is added into the causal graph. Wechoose Precision, Recall, and F1-score as metrics to evaluateeach type of causality.

3) Results: Fig. 7 shows the performance of ACNet andthe two other methods on different causal structures and datageneration parameters with varying occurrence frequenciesε, and average influence lag η. Overall, ACNet consistentlyoutperformed the other methods. As shown in Fig. 7(a), theoccurrence frequency reflects the sparsity of the sequencessuch as the number of timestamps with recorded situations.The performance of TR and GC declines with increasingsparsity. On the other hand, ACNet maintained its performanceat around 0.7 even when there was only one occurrence inthree hours. This result implies that our model is effective insolving the causal discovery problems on the sparse affectivesituation sequences. Fig. 7(b) shows the sensitivity with differ-ent influence lags. The recall of our model decreased with theincreasing average influence lags because larger influence lagsmeans longer dependence on the previous states. The precisionof our model was not sensitive to the influence lag, keeping aperformance between 0.65 and 0.8, while the precision of othermethods decreased as the influence lag increased. These results

9

Fig. 5. Test classification performance on the DEAP dataset with varying the fractions of SAM rated labels over different number of classes (valence, arousal).

Fig. 6. Test classification performance on the DEAP dataset with varying label frequencies over different classes.

Fig. 7. Affective identification performance on the synthetic dataset varying occurrence frequency, average influence lag, and numbers of confounder parameters

imply our model is capable of catching long-term dependencefor learning causal direction.

VI. EVALUTATION ON REAL-WORLD DATASET

Underlying the quantitative analysis of our model on thepublic and synthetic datasets, we first built a real-world datasetcalled the Affective Lifelog Dataset, where participants usedour ACC in their daily life. We then evaluated the performanceof the proposed ALIS with respect to physiological discrim-ination in emotion recognition and user agreement in causalidentification.

A. Affective Lifelog Dataset

1) Data Acquisition: The dataset consists of two modalitiesof physiological signals, accelerometer signals, and frontalimages obtained by ACC from 16 male and 5 female universitystudents aged from 21 to 35 (26.4 ± 4.87) years. Our require-ment for participation was to perform at least one common taskof a university student, such as conducting research, takingclasses, or having a discussion with colleagues. We requiredthem to wear the device over six hours per day for up to 45days with $10 compensation per day. This experiment wasapproved by the Institutional Review Board (IRB) in HumanSubjects Research.

To identify individual specific affective contents and assign

10

Fig. 8. Affective Situation Dataset D and its subset Ds which contains SAM-rated situations. (b) Example images in the dataset D. (c) Proportion of thesubset Ds and the dataset D. Ns = 378 for the subject 1.(d) Distribution ofthe SAM-rated situations in valence and arousal labels on the subset Ds.

the SAM ratings to them as ground-truth labels, we askedthe participants if they had any affective contents that hadelicited a specific feeling, and how the contents changed theiremotion before and after the elicitation. The life contentswere perceived as breakthroughs to change their mentality.The changes were rated by the SAM scaled from 0 to 6 forarousal and valence ratings. Furthermore, we retrieved thefollowing affective contents manually which are consideredto potentially affect mental status stress: 1) watching movies,2) drinking coffee, 3) drinking green tea, 4) hanging out withfriends, 5) playing games, 6) drawing a picture, 7) taking awalk, 8) reading a book, 9) eating food, 10) studying at adesk, 11) reading research papers on a computer monitor, 12)conducting researches in a laboratory, and 13) playing withmedia devices. In such situations, the participants performedthe SAM ratings every five days.

2) Affective Lifelog Dataset: Fig. 8 summarizes the Af-fective Lifelog Dataset D. The situations rated by the SAMconsist of a subset Ds with pairs of emotion labeling y = (V ,A) scaled from 0 to 6 for valence V and arousal A ratings.The labeling y = (V , A) is used as ground-truth to evaluate theperformance of our proposed system. As shown in Fig. 8(c),the dataset has a limited amount of labeling data. The ratios ofthe dataset D and the subset Ds are 0.418 from all participants.Furthermore, the distribution of labels is unbalanced. Thesechallenge issues are consistent with our understanding describein (C.1) and (C.2). Fig. 8(b) shows some situations in thedataset D from our real-world experiments. Participants haveexperienced various situations in daily life such as driving acar, reading a research paper, playing games, and taking awalk.

B. Experimental Setup

1) Train/Test/Evaluation on the Dataset D: From the real-world dataset D, we grouped affective labels (V , A) invalence and arousal into seven discrete affective states: NVHA,NVMA, NVLA, UVLA, PVLA, PVMA, and PVHA, com-prising low (LA), mid (MA), and high (HA) arousal and

TABLE IPERFORMANCE OF ADNET FOR CLASSIFYING THE SEVEN EMOTIONAL

STATES AND ACNET FOR IDENTIFYING THE AFFECTIVE CAUSALITIES ONTHE DATASET D.

ACNet ADNetNVHA NVMA NVLA UVLA PVLA PVMA PVHA

0.74 0.61 0.59 0.55 0.64 0.52 0.56 0.55

negative (NV), neutral (UV), and positive (PV) valence ratings.We should note that two states (UVMA and UVHA) wereomitted since their occurrence was extremely low. 40% of thedataset Ds was used for training. We then retrieved 10,000pieces of physiological data per affective state, which is70,000 pieces of physiological data on total of seven statesfor every participant on the dataset D. ADNet first predicts yon the dataset Ds and produces pseudo-labels y with learningphysiological patterns X . Then, the model uses the patternswith the SAM-rated rating y to produce the clean labels y.The test data remained unperturbed to validate and compareour model to other methods. Since physiological signals, inparticular, EEG signals, are vulnerable to motion artifacts [31],we developed a strategy to improve the quality of EEG signalsby abandoning EEG signals highly correlated with motionartifacts rather than separating and removing motion artifactsin EEG signals occurring due to body movement [32], [33]. Topursue this strategy, we subdivided the EEG signals into twogroups separated by varying the accelerometer data. From eachof the two groups we extracted the following EEG features: 1)Mean power, 2) Maximum amplitude, 3) Standard deviation ofthe amplitude, 4) Kurtosis of the amplitude, and 5) Skewnessof the amplitude. The features are metrics to describe keycharacteristics of clean EEGs [34]. After representing thefeatures into two-dimensional space using principal componentanalysis (PCA), we computed the Bhattacharyya distancebetween the two groups over the two-dimensional space asa differentiator between the clean EEG and the contaminatedEEG based on the maximum distance between the two groups.

2) Network Settings: ADNet is composed of two-layerednetworks with 512 and 256 hidden states and has 5×5 size ofkernels for the input-to-state and state-to-state transitions. Totrain the network, we used learning batches of 32 sequences,set the learning rate as 0.01 initially, and divided the rateby ten after every 20,000 iterations. The weight decay andmomentum were set to 0.0005 and 0.6, respectively. Back-propagation through time was performed for ten timesteps.We also performed early-stopping on the validation set. ForACNet, we only selected pairs of the affective sequence Fwhich were associated with clean physiological signals fromall participants with every 10 min as a timestamp.

C. Evaluation Results

1) Performance of ALIS on Recognizing Emotions andIdentifying Affective Causalities: TABLE I reports the averageprecision in recognizing emotions and identifying affectivecausalities over all participants on the dataset D. The perfor-mance on the UVLA state showed the highest results over allparticipants.

11

Fig. 9. Performance in emotion recognition and affective causality identifi-cation on the real-world dataset D increasing the dataset size over differentemotional states and affective situations

This implies that when participants are in a UVLA state,such as calm and relaxed feelings, their physiological signalsfluctuate in common patterns, which helps ALIS to learn theircharacteristics. It can be also attributed that the percentagesof affective situations rated with the labels UV and LA werehigher than others on the dataset D. On the other hand, theperformance in classifying valence ratings associated with lowarousal ratings besides LAUV, such as NVLA and PVLAstates, were relatively lower than other arousal ratings. Inparticular, PVLA had the lowest accuracy. This observa-tion may imply that classifying emotions by valence canbe improved by considering their associations with arousal.Affective causalities in the 13 situations were identified witha precision of 0.74. Although ACNet identified affectivecausalities by regarding the prior results of ADNet recognizingaffective states, the performance was consistently higher thanon detecting emotional states. This can be attributed to ACNetworking well on eliminating false causal relationships built onwrong emotions.

Unlike the DEAP dataset, our real-world dataset D recordseveryday activities with physiological signals. Hence, the inter-and intra-day variability in physiological signals can determinethe performance of ALIS in understanding affective dynamics.Fig. 9 shows the performance in emotion recognition andcausality identification on different amounts of physiologicaldata in days. The accuracy generally improved when sufficientdata was available. ADNet classified HA states better whenmore of their associated signals were provided. On the otherhand, classifying negative states such as NVLA and PVLAshowed the smallest improvement with increased data sizes.These results could imply that the affective dynamics invalence requires the development of elaborate deep learning ar-chitectures more than the provision of sufficient physiologicaldata. Since causal identification depends on the prior emotionrecognition, the performance in emotion recognition is anotherkey in causality. However, this experiment shows that oursystem does not face this problem. It consistently achievedhigh scores with small increaments.

2) Analysis of Emotion Recognition: Fig. 10(a) shows themultimodal physiological features X in a two-dimensionalspace obtained using t-distributed stochastic neighbor em-bedding (t-SNE) [35], a popular technique for unsupervised

Fig. 10. (a) Clustering of the UVLA states from the participants usingt-SNE applied to physiological features X . (b) The shared physiologicalrepresentation visualized by the grand average of the feature X over frequencybands in the seven affective states.

dimensionality reduction and visualization. The features of dif-ferent participants in an affective state cluster in the projectedspace, revealing their high variety. Despite their heterogeneity,ADNet is capable of capturing some shared physiologicalcharacteristics. To investigate the physiological phenomenaobserved when emotions are classified using ADNet, wevisualize the grand average of brain lateralization featuresin (3) across participants in valence and arousal ratings. Weshould note that heart-related features have served as essentialelements reflecting the function of ANS. However, in thissection, we focus more on brain lateralization, as it has rela-tively large inter- and intra-subject variability. Fig. 10(b) showsthe commonly shared frequencies over all participants in theseven emotional states. We found that alpha and beta bandswere activated when most emotional states except UVLA andPVLA were elicited. Strong negative related feelings such asNVHA and NVMA led to an increase in physiological changesin alpha and beta. The other two states were characterizedby either the theta or gamma band. This indicates emotionalreactions in real-world situations lead to the activation of phys-iological signals in alpha and beta while stability in emotionmaintains physiological signals in the theta or gamma bands.Although several alternatives have been suggested in reportson the neuro-physiological correlates of affective states, thisresult is in line with our previous work [7]. Our findingsmay also be justified by the fact that when some negativebut approach-related emotions, such as “anger” that would belateralized to the left hemisphere, are induced, they lead toincreases in alpha band activity in the left anterior and theleft temporal regions in the beta bands. We also found thatemotional lateralization has also affects on reacting emotionsin cases of arousal correlated with valence. When increasingarousal from low to high within the same level of valencestates, there are slight increments in the theta band. Thisobservation may reflect the inter-correlations between valenceand arousal, as reported in [15].

3) Causality Discovery: To better understand the overallcausal pairs in daily life, we report three types of causalnetworks of the four frequent stress relievers on the datasetD. We choose the four most frequent situations: “Studyingat a desk,” “Playing games,” “Drinking coffee/green tea in acafeteria,” and “Watching movies.” The situations and their

12

Fig. 11. Causal Structures between the four situations and the seven emotionalstates on the real-world dataset D. Nodes from 1 to 7 represent affective statesNVHA, NVMA, NVLA, UVLA, PVLA, PVMA, and PVHA. Situation nodesare denoted as S, G, C, M and L for Studying on a desk, Playing Games,Drinking coffee/green tea in a cafeteria, Watching movies, and a latent factor.Red lines indicates the asymmetric causality from the situation to the emotionstates. Blue indicates the opposite causalities. Green lines show the latentfactors between the situation and emotional states.

causality were manually browsed in the results of our system.Fig. 11 shows the causality structure over all participants.Edges are asymmetric affective causalities between two nodes.Numbers on edges are percentages of causalities affected bythe relationship. Note that the graph does not include cascadeflows; namely, in case of 1 → 2 → 3, it has only a causalrelation between node 1 → 2 and 2 → 3, but the causalitydoes not propagate 1→ 3.

Participants changes their behaviors when they feel specificemotions. First of all, we found that while studying at adesk, most users felt the emotional states NVLA and UVLA,which are close to calm and stress, but any feeling did notaffect students’ desire to study. This phenomenon indicatesthat the activity can be a major emotional cause regulatingtheir feelings of being stressed in repeated school routines.In other words, the participants has studied either habituallyor for no particular emotional reason, but they were stressedby studying. Interestingly, these types of stressed feelingsled them to change their behaviors. Some people had coffeewhen their feelings rated as having negative valence and lowarousal. Furthermore, in line with affective causality, drinkingcoffee had a causal effect on overcoming emotional negativ-ity, increasing valence. While most activities causes a givenemotion to a particular feeling, “Watching a movie” affectedmultiple emotional states. This can be attributed to individualemotional acceptance of a movie or characteristics of differentmovie genres. Similarly, when users play games, they feeleither NVMA or PVMA emotional states. The polarity invalence from the two emotional states implies playing games isaccompanied by emotional elements of fatigue, while it helpsto lead positive feeling. From a few users, we found there existtwo hidden factors between emotions and situations. Whenusers had the latent factor L2, they played the game whilefeeling excited. Similarly, the latent factor L1 bridged users todrink coffee/green tea with a happy feeling. Neither connectionhad been established without the two factors.

VII. CONCLUSION

We presents a new wearable system to detects emotionalchanges and find causational relationships in daily life, based

on the new affective model of interaction behavior. By apply-ing our proposed model to real-world dataset, our approach isable to find meaningful causal connections between emotionsand behaviors, even when confounder variables potentiallyaffect human emotions and behaviors. In the future, we willexplore the possibilities of social interaction behavior causedby individual emotional changes. It is also interesting andeven more challenging to effectively implement causalityidentification in the complex human behaviors and situationalanalysis of daily life.

ACKNOWLEDGMENT

This work was supported by Institute for Information &communications Technology Promotion(IITP) grant funded bythe Korea government(MSIT) (No.2017-0-00432, No.2017-0-01778).

REFERENCES

[1] M. Garbarino, M. Lai, D. Bender, R. W. Picard, and S. Tognetti, “Empat-ica e3a wearable wireless multi-sensor device for real-time computerizedbiofeedback and data acquisition,” in Wireless Mobile Communicationand Healthcare (Mobihealth), 2014 EAI 4th International Conferenceon. IEEE, 2014, pp. 39–42.

[2] R. W. Picard, S. Fedor, and Y. Ayzenberg, “Multiple arousal theory anddaily-life electrodermal activity asymmetry,” Emotion Review, vol. 8,no. 1, pp. 62–75, 2016.

[3] M. A. Williams, A. Roseway, C. O’Dowd, M. Czerwinski, and M. R.Morris, “Swarm: An actuated wearable for mediating affect,” in Pro-ceedings of the Ninth International Conference on Tangible, Embedded,and Embodied Interaction. ACM, 2015, pp. 293–300.

[4] O. Yuruten, J. Zhang, and P. H. Pu, “Predictors of life satisfactionbased on daily activities from mobile sensor data,” in Proceedings of theSIGCHI Conference on Human Factors in Computing Systems. ACM,2014, pp. 497–500.

[5] W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki, “Emotionmeter:A multimodal framework for recognizing human emotions,” IEEETransactions on Cybernetics, no. 99, pp. 1–13, 2018.

[6] M. M. Bradley and P. J. Lang, “Measuring emotion: the self-assessmentmanikin and the semantic differential,” Journal of behavior therapy andexperimental psychiatry, vol. 25, no. 1, pp. 49–59, 1994.

[7] B. H. Kim and S. Jo, “Deep physiological affect network for the recog-nition of human emotions,” IEEE Transactions on Affective Computing,2018.

[8] G. Valenza, L. Citi, A. Lanata, E. P. Scilingo, and R. Barbieri, “Revealingreal-time emotional responses: a personalized assessment based onheartbeat dynamics,” Scientific reports, vol. 4, p. 4998, 2014.

[9] R. R. Cornelius, The science of emotion: Research and tradition in thepsychology of emotions. Prentice-Hall, Inc, 1996.

[10] D. Sander, D. Grandjean, and K. R. Scherer, “A systems approach toappraisal mechanisms in emotion,” Neural Networks, vol. 18, no. 4, pp.317–352, 2005.

[11] S. Smith, “Eeg in the diagnosis, classification, and management of pa-tients with epilepsy,” Journal of Neurology, Neurosurgery & Psychiatry,vol. 76, no. suppl 2, pp. ii2–ii7, 2005.

[12] N. Lovato and M. Gradisar, “A meta-analysis and model of the relation-ship between sleep and depression in adolescents: recommendations forfuture research and clinical practice,” Sleep Medicine Reviews, vol. 18,no. 6, pp. 521–529, 2014.

[13] X.-W. Wang, D. Nie, and B.-L. Lu, “Emotional state classification fromeeg data using machine learning approach,” Neurocomputing, vol. 129,pp. 94–106, 2014.

[14] P. C. Petrantonakis and L. J. Hadjileontiadis, “Adaptive emotionalinformation retrieval from eeg signals in the time-frequency domain,”IEEE Transactions on Signal Processing, vol. 60, no. 5, pp. 2604–2616,2012.

[15] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi,T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysisusing physiological signals,” IEEE Transactions on Affective Computing,vol. 3, no. 1, pp. 18–31, 2012.

13

[16] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis ofeeg signals and facial expressions for continuous emotion detection,”IEEE Transactions on Affective Computing, vol. 7, no. 1, pp. 17–28,2016.

[17] R. Subramanian, J. Wache, M. Abadi, R. Vieriu, S. Winkler, and N. Sebe,“Ascertain: Emotion and personality recognition using commercial sen-sors,” IEEE Transactions on Affective Computing, 2016.

[18] Z. Zhang, Z. Pi, and B. Liu, “Troika: A general framework forheart rate monitoring using wrist-type photoplethysmographic signalsduring intensive physical exercise,” IEEE Transactions on BiomedicalEngineering, vol. 62, no. 2, pp. 522–531, 2015.

[19] H. Chigira, A. Maeda, and M. Kobayashi, “Area-based photo-plethysmographic sensing method for the surfaces of handheld devices,”in Proceedings of the 24th Annual ACM Symposium on User InterfaceSoftware and Technology (UIST). ACM, 2011, pp. 499–508.

[20] Y. Lyu, X. Luo, J. Zhou, C. Yu, C. Miao, T. Wang, Y. Shi, and K.-i.Kameyama, “Measuring photoplethysmogram-based stress-induced vas-cular response index to assess cognitive load and stress,” in Proceedingsof the 33rd Annual ACM Conference on Human Factors in ComputingSystems. ACM, 2015, pp. 857–866.

[21] D. Sun, P. Paredes, and J. Canny, “Moustress: detecting stress frommouse motion,” in Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems. ACM, 2014, pp. 61–70.

[22] R. Cai, Z. Zhang, Z. Hao, and M. Winslett, “Understanding socialcausalities behind human action sequences,” IEEE transactions onneural networks and learning systems, vol. 28, no. 8, pp. 1801–1813,2017.

[23] B. Wu, J. Jia, Y. Yang, P. Zhao, J. Tang, and Q. Tian, “Inferringemotional tags from social images with user demographics,” IEEETransactions on Multimedia, vol. 19, no. 7, pp. 1670–1684, 2017.

[24] J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, and J. Tang, “Can we understandvan gogh’s mood?: learning to infer affects from images in socialnetworks,” in Proceedings of the 20th ACM international conferenceon Multimedia. ACM, 2012, pp. 857–860.

[25] Y. Yang, J. Jia, B. Wu, and J. Tang, “Social role-aware emotion contagionin image social networks.” in AAAI, 2016, pp. 65–71.

[26] M.-Z. Poh, N. C. Swenson, and R. W. Picard, “A wearable sensor forunobtrusive, long-term assessment of electrodermal activity,” BiomedicalEngineering, IEEE Transactions on, vol. 57, no. 5, pp. 1243–1252, 2010.

[27] G. W. Imbens and D. B. Rubin, Causal inference in statistics, social,and biomedical sciences. Cambridge University Press, 2015.

[28] S. Tripathi, S. Acharya, R. D. Sharma, S. Mittal, and S. Bhattacharya,“Using deep and convolutional neural networks for accurate emotionclassification on deap dataset.” in AAAI, 2017, pp. 4746–4752.

[29] G. Ver Steeg and A. Galstyan, “Information transfer in social media,”in Proceedings of the 21st international conference on World Wide Web.ACM, 2012, pp. 509–518.

[30] S. Seth and J. C. Principe, “Assessing granger non-causality usingnonparametric measure of conditional independence,” IEEE transactionson neural networks and learning systems, vol. 23, no. 1, pp. 47–59, 2012.

[31] I. Daly, M. Billinger, R. Scherer, and G. Muller-Putz, “On the automatedremoval of artifacts related to head movement from the eeg,” IEEETransactions on neural systems and rehabilitation engineering, vol. 21,no. 3, pp. 427–434, 2013.

[32] I. Daly, F. Pichiorri, J. Faller, V. Kaiser, A. Kreilinger, R. Scherer,and G. Muller-Putz, “What does clean eeg look like?” in Engineeringin Medicine and Biology Society (EMBC), 2012 Annual InternationalConference of the IEEE. IEEE, 2012, pp. 3963–3966.

[33] S. M. Alarcao and M. J. Fonseca, “Emotions recognition using eegsignals: a survey,” IEEE Transactions on Affective Computing, 2017.

[34] R. Jenke, A. Peer, and M. Buss, “Feature extraction and selectionfor emotion recognition from eeg,” IEEE Transactions on AffectiveComputing, vol. 5, no. 3, pp. 327–339, 2014.

[35] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journalof machine learning research, vol. 9, no. Nov, pp. 2579–2605, 2008.

Byung Hyung Kim received the B.S. degree in com-puter science from Inha University, Incheon, Korea,in 2008, and the M.S. degree in computer sciencefrom Boston University, Boston, MA, USA, in 2010.He is currently working toward the Ph.D. degreeat KAIST, Daejeon, Korea. His research interestsinclude affective computing, brain-computer inter-face, computer vision, assistive and rehabilitativetechnology, and cerebral asymmetry and the effectsof emotion on brain structure.

Sungho Jo (M’09) received the B.S. degree inschool of mechanical & aerospace engineering fromthe Seoul National University, Seoul, Korea, in 1999,the S.M. in mechanical engineering, and Ph.D. inelectrical engineering and computer science from theMassachusetts Institute of Technology (MIT), Cam-bridge, MA, USA, in 2001 and 2006 respectively.While pursuing the Ph.D., he was associated withthe Computer Science and Artificial IntelligenceLaboratory (CSAIL), Laboratory for InformationDecision and Systems (LIDS), and Harvard-MIT

HST NeuroEngineering Collaborative. Before joining the faculty at KAIST, heworked as a postdoctoral researcher at MIT media laboratory. Since Decemberin 2007, he has been with the department of computer science at KAIST,where he is currently Associate Professor. His research interests includeintelligent robots, neural interfacing computing, and wearable computing.