Dynamic Neuro-Symbolic Knowledge Graph Constructionfor Zero-shot Commonsense Question Answering

Antoine Bosselut ♦♥ Ronan Le Bras ♦ Yejin Choi♦♠

♦Allen Institute for Artificial Intelligence ♥Stanford University♠Paul G. Allen School of Computer Science & Engineering, University of Washington

{antoineb,ronanlb,yejinc}@allenai.org

Abstract

Understanding narratives requires reasoning about implicitworld knowledge related to the causes, effects, and states ofsituations described in text. At the core of this challenge ishow to access contextually relevant knowledge on demandand reason over it.In this paper, we present initial studies toward zero-shot com-monsense question answering by formulating the task as in-ference over dynamically generated commonsense knowledgegraphs. In contrast to previous studies for knowledge integra-tion that rely on retrieval of existing knowledge from staticknowledge graphs, our study requires commonsense knowl-edge integration where contextually relevant knowledge isoften not present in existing knowledge bases. Therefore, wepresent a novel approach that generates contextually-relevantsymbolic knowledge structures on demand using generativeneural commonsense knowledge models.Empirical results on two datasets demonstrate the efficacy ofour neuro-symbolic approach for dynamically constructingknowledge graphs for reasoning. Our approach achieves signif-icant performance boosts over pretrained language models andvanilla knowledge models, all while providing interpretablereasoning paths for its predictions.

1 IntroductionUnderstanding narratives requires reasoning about all the im-plicit, but trivially inferable, details of a situation based onlyon what is explicitly stated in text. A statement as simpleas “they went to the club” instantly invokes a bank of com-monsense expectations: they had to get dressed, they weregoing dancing, they likely had drinks, and so forth. Thesereasoning capabilities are missing in most existing neurallanguage understanding models that learn task-specific rep-resentations without acquiring rich background knowledgeabout the social and physical world.

In response, recent work has investigated augmenting deeplearning models with retrieval mechanisms over large-scalecommonsense knowledge graphs (Mihaylov and Frank 2018;Bauer, Wang, and Bansal 2018; Paul and Frank 2019). How-ever, these approaches assume an entity linking step betweenthe written text and knowledge graph. By canonicalizing en-tities, they discard key context surrounding the input, andoften retrieve semantically irrelevant knowledge (e.g., a “club”being a blunt weapon is irrelevant to the earlier situation).

Kai knew that things were getting out of control and managed to

keep his temper in check

X keeps X’s temper

X keeps ___ under control

X sweats

X avoids a fight

X wants to show strength

X keeps X's ___ in check

Link to static Knowledge Graph

Kai wants to avoid trouble

Generate dynamic graph with COMET

Kai intends to be calm

Kai is viewed as cautious

Kai stays calm

bad links

context-free knowledge

contextual knowledge

no linking

Figure 1: Previous approaches for accessing knowledge linksituational contexts to static knowledge graphs. Our workgenerates knowledge dynamically from neural knowledgemodels.

In this paper, we propose to generate new knowledge that iscontextually relevant instead of retrieving existing knowledgeas is. Bosselut et al. (2019) recently introduced CommonsenseTransformers (COMET), a new framework for training neu-ral representations of knowledge graphs. This new class ofneural knowledge model provides a powerful representationaltool for connecting commonsense knowledge to downstreamtask models. Because COMET represents knowledge graphsneurally, it can generate commonsense inferences for any en-tity that can be encoded by the neural model (i.e., describedwith language). With no need to canonicalize context entitiesto link to a static knowledge graph, the knowledge model canbe queried directly with complex compositional structures,and even full narrative contexts. This flexibility has led themto be used out-of-the-box in a variety of settings requiringcontextual knowledge, such as sarcastic comment genera-tion (Chakrabarty et al. 2020), therapy chatbots (Kearns et al.

arX

iv:1

911.

0387

6v2

[cs

.CL

] 3

0 O

ct 2

020

2020), and story plot generation (Ammanabrolu et al. 2020).In this work, we use COMET to construct context-relevant

knowledge graphs that can be reasoned over for common-sense question answering. Given a raw context, COMETgenerates commonsense inferences that provide world knowl-edge about the situation depicted in the context. These in-ferences can be used as additional context to score answercandidates or to generate additional inferences. By generat-ing new inferences and connecting them to the raw contextand answers, COMET dynamically constructs a symbolicgraph of commonsense knowledge. The raw context is theroot node, answer choices are leaf nodes and generated com-monsense inferences provide intermediate nodes betweenthem, instantiating different reasoning paths between the con-text and answers. Using COMET generated scores as factorsweighting these paths, we propose new inference algorithmsto reason over the generated graph and identify the mostlikely answers to questions about the situation.

We evaluate our approach in a zero-shot setting on the SO-CIALIQA (Sap et al. 2019b) benchmark, a question answeringdataset for evaluating social commonsense, and the STO-RYCS benchmark (Rashkin et al. 2018), a story understand-ing dataset. Empirical results show that our neuro-symbolicapproach, COMET - DynaGen, outperforms purely neurallarge-scale pretrained language models (Radford et al. 2018,2019) and knowledge models that evaluate QA examplesdirectly without dynamically generating an intermediate sym-bolic commonsense knowledge graph (i.e., reasoning withCOMET with no inference hops).

2 Dynamic Knowledge Graph Constructionfor Question Answering

Our approach uses a knowledge model, COMET (Bosse-lut et al. 2019), to dynamically construct a context-relevantcommonsense knowledge graph about a presented situation.COMET is trained using transfer learning from large-scalepretrained language models (Radford et al. 2018) to knowl-edge graphs. When trained on the ATOMIC knowledge graph(Sap et al. 2019a), it learns to generate social commonsenseinferences of situations depicted in text. Importantly, unlikestatic knowledge graphs (e.g., ConceptNet; Speer, Chin, andHavasi 2017), which require canonicalizing input entities tolink to the graph, COMET represents knowledge neurally, al-lowing it to generate commonsense for arbitrary input forms.

In Figure 1, for example, the context “Kai knew thingswere getting out of control and managed to keep his tem-per in check” is unlikely to be found in any existing knowl-edge graph. It describes a very specific situation. However,COMET can parse this full context and generate common-sense knowledge about Kai’s reactions and motivations, suchas “Kai stays calm” or “Kai wants to avoid trouble,” as down-stream inferences. We exploit this generalization propertyof knowledge models to dynamically construct knowledgegraphs for presented situations that can be reasoned over toanswer commonsense questions about them.

Notation. Formally, we assume a dataset of examples, eachwith an associated context c describing a situation, a ques-

tion q asked about that situation, and a set of n possibleanswersA = {a0, ..., an−1} to that question. Each answer iscomposed of multiple tokens Y a = {y1, ..., y|a|}.Generating Commonsense Inferences. We generate com-monsense inferences for a situational context c by concatenat-ing the context with relation types from the ATOMIC knowl-edge graph and using COMET to produce candidates G. Eachcandidate g ∈ G is associated with a score φg that approxi-mates the model’s confidence in the inference:

φg =1

|g|

|g|∑t=1

logP (xt|x<t, c, r) (1)

where xt are the tokens of g, |g| is the token length ofg, r is an arbitrary commonsense relation type for whichCOMET can generate inferences, and:

P (xt|x<t, c, r) = COMET(c, r, x<t) (2)where the tokens of c and r are concatenated with the tokensx<t to be input to COMET. Any generation g ∈ G condi-tioned on c can be seen as a 1-hop commonsense inferenceof c.

Using a Markov assumption, we can generalize this ap-proach by conditioning on generated commonsense infer-ences to generate G`, a set of `-hop inferences from c:

φ`g = φ`−1g +1

|g`|

|g`|∑t=1

logP (xt|x<t, g`−1, r) (3)

where φ`g is a generation score for any g` ∈ G`, g`−1 is anarbitrary inference from G`−1, the set of inferences of theprevious hop, and φ`−1g is the generation score of that seedinference. Using this approach, we can use COMET to con-struct a graph where commonsense inferences g are nodes.For an arbitrary node g`, its parent is the node from the pre-vious level G`−1 that COMET conditions on to generate g`.The children of g` are nodes generated when COMET condi-tions on g` to generate new commonsense inferences. We setg0 = c because the context is the root node of the graph, andφ0g = 0 because the original context c is deterministic.

Answers as Leaf Nodes. The final step in constructing theknowledge graph is to connect the answer choices a ∈ A tothe generated commonsense inferences. We initialize a nodein the graph for each answer choice a and connect it as a childnode to each commonsense inference in the graph: g ∈ G` for` ∈ [0, L) where L is the number of levels in the final graph.In Figure 2b, we see that the answer choices A = {relieved,scared, anxious} are connected to the root node and eachgenerated commonsense inference in the L = 2 level graph.

3 Knowledge Graph ReasoningBeing designed as a conditional language model,COMET can also be used to score candidate com-monsense inferences. We use this property to score answercandidates a ∈ A conditioned on the generated common-sense inferences g ∈ G that are connected to them. The

context node

generated node

answer node

Kai knew that things were getting out of

control and managed to keep his temper in check

Kai wants to avoid trouble

Kai is viewed as cautious

Kai intends to be calm

relieved

scared

anxious

(a) COMET receives the context c and generates new common-sense inferences to construct a local graph of knowledge aboutthe situation (Section 2).

ℓ = 0ℓ = 1

ϕ1g1

ϕ1g2

ϕ1g1a1

ϕ1g1a3

ϕ1g3a2

ϕ1g3a1

ϕ1g3a3

ϕ1g3

ϕ1g1a2

ϕ1g2a1

ϕ1g2a3

ϕ1a1max

ϕ0a1 ϕ0

a2 ϕ0a3

generation factor (Eq. 1, 3)

layer aggregate (Eq. 7, 8)

answer factor (Eq. 4, 6)

a1g1

g2c

g3

a2

a3a1 a2 a3

(b) Our inference algorithms reason over the graph by aggregat-ing commonsense paths to answer questions about the situation(Section 3).

Figure 2: Our approach consists of dynamically constructing a local commonsense knowledge graph about a presented situation.This graph can be used to reason about the different questions about the situation.

scores from COMET are used to initialize factor nodesbetween each generated commonsense inference (at all levelsof the graph) and each answer choice. Using these scores,and scores between commonsense inferences (Eqs. 1, 3), asa set of factors, our generated knowledge graph implicitlyencodes a factor graph that can be reasoned over to evaluateeach answer candidate.

Computing Answer ScoresCOMET is originally trained to maximize the conditional log-likelihood of the tokens of a target entity e2 from a knowledgegraph tuple (e1, r, e2). As a result, the knowledge model canmeasure the log-likelihood of a candidate entity e2 given asource entity e1 and relation r. For a given example, we treateach answer candidate a as an e2 candidate for COMET,map the parent nodes of a (e.g., g nodes) to be equivalent toe1, and set the question q as r, allowing COMET to evaluateeach answer candidate according to its implicit knowledgerepresentations. For each answer a ∈ A, we define a factorbased on each token’s conditional log-likelihood as computedby COMET:

φga =1

|a|

|a|∑s=1

logP (ys|y<s, g, q) (4)

where ys corresponds to the token in a at time step s, y<s isall the tokens preceding ys in a, and |a| is the total numberof tokens making up a. In this way, for any QA example,we define a set of factor nodes φga connecting the answer

candidates a ∈ A to the commonsense inferences g ∈ Ggenerated by COMET about the situational context c.

Overcoming Answer Priors. Because certain answer candi-dates have a high probability of occurring for certain ques-tions regardless of the context (e.g., happy is a commonanswer for questions about emotional reactions), we redefineφga (Eq. 4) in terms of the point-wise mutual informationbetween the commonsense path g and answer a:

φga ∝ PMI(a, g|q)

φga =1

|a|

|a|∑s=1

(logP (ys|y<s, g, q)

− logP (ys|y<s, q))

(5)

where logP (ys|y<s, q) is the log-likelihood of each tokenin the answer given only the question and previous answertokens. We describe our approximation of this distribution inAppendix B.

InferenceEach φ`g scores a unique reasoning path at a particular depth` in the graph. The composition γgφ`g + γgaφ

`ga can then be

seen as scoring a path to a particular answer. To find the mostlikely answer, we marginalize over all paths to the answers atlayer `:

φ`a = f({γgφ`g + γgaφ`ga : g ∈ G`}) (6)

where φ`g (Eq. 3) and φ`ga (Eq. 5) are the path and answerscore, respectively, for generation g ∈ G`. γg and γga arehyperparameters balancing the contribution of both scores.Because the path and answer scores are log-probabilities, weset f as the LogSumExp, yielding Eq. 6 as a variable elimi-nation over g ∈ G`. We also define an extremum estimatorover the distribution of generated inferences G`:

φ`amax= max

g∈G`γgφ

`g + γgaφ

`ga (7)

At a high level, φ`amaxcan be interpreted as approximating

the likelihood of answer a given a singular reasoning path:{c → g1 → · · · → g` → a}, rather than by computing anaggregation of all paths in the graph to the answer (Eq. 6).

Once the answer scores at different levels in the graphare computed, {φ`a}L0 , the final score for each answer can beevaluated by averaging over the graph levels ` ∈ [0, L):

logP (a|q, c) ∝ φa =

L∑`=0

β`φ`a (8)

a = argmaxa∈A

φa (9)

where a is the selected best answer by the approach, L isthe number of generation hops made by the COMET model(i.e., the number of levels in the graph), φ`a is the score thatis propagated from each hop of the constructed knowledgegraph, and β` is hyperparameter scaling the contribution ofeach hop score. We note that φ0a is the result from evaluatingthe answer candidates directly against the original context c,and that φ`a is replaced by φ`amax

if the extremum estimator(Eq. 7) is used instead of variable elimination (Eq. 6).

4 Experimental SetupWe evaluate our approach in a zero-shot experimental setting.It is a well-studied phenomenon that neural methods trainedon crowdsourced data often learn to shortcut reasoning toarrive at a correct answer (Gururangan et al. 2018; Li andGauthier 2017). We use a zero-shot setting to simulate themodel having to reason about situations it has never encoun-tered before, forcing it to construct reasoning graphs fromexplicit knowledge it can generate (e.g., knowledge learnedby COMET), and precluding it from learning dataset-specificartifacts. As such, we do not use training data to update modelparameters. Furthermore, any result presented on the test setdoes not have hyperparameters tuned on the development set.

Datasets and ProcessingWe evaluate our method on two datasets: SOCIALIQA (Sapet al. 2019b) and STORYCS (Rashkin et al. 2018).

SOCIALIQA. The SOCIALIQA dataset evaluates a model’sability to understand the social dynamics underlying situa-tions described in short text snippets. Each example in thedataset consists of a context, a question about that context,and three multiple choice answers. An example from the

dataset is shown in Figure 2. We outline pre-processing stepsfor the data in Appendix A.

STORYCS. The STORYCS dataset consists of short 5-sentence stories with annotated motivations and emotionalresponses whose labels are drawn from classical theories ofpsychology (e.g., Plutchik 1980). We map the emotion classi-fication task to a QA task by posing an individual questionfor each emotion label (disgust, surprise, fear, anger, trust,anticipation, sadness, joy) that must be predicted for eachexample. We outline this procedure in Appendix B.

Experimental SettingsHyperparameters. We use most of the same hyperparam-eters to train the COMET model on the ATOMIC knowledgegraph as in Bosselut et al. (2019). However, we use GPT2-345M (Radford et al. 2019) as the pretrained language modelthat seeds COMET and freeze the position embeddings sowe can generalize to longer contexts. We note that the SO-CIALIQA dataset is partially derived from ATOMIC knowl-edge base tuples. However, we do not allow ATOMIC tuplesused to seed SOCIALIQA evaluation examples to be usedas training examples for COMET. We provide more detailsof this splitting in Appendix A. The number of levels in thegraph L is set to 2. As we operate in the zero-shot setting, wedo not tune hyperparameters. For the SOCIALIQA dataset,we set γg = γga = 1.0 and β` = 1.0 ∀`. For STORYCS, wedo the same except that γg = 0. Unless stated otherwise, weuse argmax decoding to generate inferences from COMET,and use variable elimination over the graph to select answers.

Prediction. To predict an answer on the SOCIALIQA dataset,we use Equation 9. Prediction for STORYCS is less straight-forward, as the task is originally binary multi-label classifica-tion. To make a prediction, we treat φa (Eq. 8) for each labelj independently and select an answer based on whether φa,jis above a label-specific threshold, κj . To avoid violating thezero-shot setting (i.e., tuning thresholds on the developmentset), we select the threshold using the score at the percentileof the positive label distribution (e.g., if the joy emotion ispresent for 20% of examples, we set the threshold at the scoreof the 20th percentile of the CDF). Thresholds are reportedin Appendix Table 10 for each label.

5 SOCIALIQA StudyBaselines. As baselines in the SOCIALIQA study, we uselarge-scale pretrained language models: GPT (Radford et al.2018), GPT2-117M, GPT2-345M, and GPT2-762M (Rad-ford et al. 2019). To adapt these language models optimallyto the QA task, question-answer pairs are automatically con-verted to a templated form, a process we outline in Ap-pendix B. We also report the results of a model, COMET- Direct, that only uses φ0a to select answers (i.e., answersare evaluated with respect to the context with no dynamicgraph construction). Additionally, we compare against theSELF-TALK model of Shwartz et al. (2020), which queriespretrained language models to generate additional detailsabout a presented situation and appends these to the originalcontext. Finally, we report the result of supervised BERT

Situation Most Contributing Paths in Graph Answers

Jesse drove Ash to the airport and droppedthem off at the airport with ease.

Jesse wants to go homea) drained 3b) went to the ticket counterc) dropped me off at the airport

Jesse wanted to be helpfula) drained

How would Jesse feel afterwards? b) went to the ticket counter 7c) dropped me off at the airport

After jumping off the roof of his houseQuinn had trouble breathing.

Quinn gets hurta) foolish 3b) patientc) light-headed

Quinn wants to get medical helpa) foolish

How would you describe Quinn? b) patientc) light-headed 7

Alex took notice of the children who weresinging at the playground.

Alex is happya) hurt the childrenb) joy 3c) tell the children to stop

Alex wants to go homea) hurt the children

What will happen to Alex? b) joyc) tell the children to stop 7

Taylor was close to winning the game.Taylor ran straight for home plate.

Taylor wants to celebratea) try to get over that they did winb) celebrate the win 7c) wanted to score

Taylor wants to be homea) try to get over that they did win

What will Taylor want to do next? b) celebrate the winc) wanted to score 3

Table 1: Example contexts, paths, and answers for the COMET - DynaGen model on SOCIALIQA. We bold the predicted answerand its most contributing path. We italicize the most likely answer for each path. Incorrect high-scoring answers for a path arehighlighted in red 7 and correct answers are highlighted in blue 3. We only present a subset of the generated paths. On average,graphs generated using argmax decoding as the graph construction algorithm yield 10.6 nodes and 26.4 edges (Table 3).

Model Dev Acc. Test Acc.Random 33.3 33.3GPT 41.8 41.7GPT2 - 117M 40.7 41.5GPT2 - 345M 41.5 42.5GPT2 - 762M 42.5 42.4SELF-TALK 46.2 43.9COMET - Direct 48.7 49.0COMET - DynaGen 50.1 52.6BERT-large (sup.) 66.0 63.5RoBERTa-large (sup.) 78.1 77.0Human 86.9 84.4

Table 2: Accuracy on the development and test sets of SO-CIALIQA. COMET - DynaGen is our model.

(Devlin et al. 2018) and RoBERTa (Liu et al. 2019) models,and random and human baselines from Sap et al. (2019b).

Overall Performance. We report the main results of our SO-CIALIQA study in Table 2. First, our approach achieves anabsolute improvement of ∼10.2% over the top performinglanguage model baseline, GPT2-762M, showing the impor-tance of using knowledge models to represent commonsense.Additionally, our approach of dynamically constructing aknowledge graph on demand (COMET - DynaGen) performsbetter than using the knowledge model to directly evaluateanswers (COMET - Direct) by∼3.6%, highlighting the value

in representing more complex reasoning paths. Finally, theimprovement over SELF-TALK depicts the benefit of using astructured graphical representation for reasoning comparedto one that uses language models to generate additional situa-tional context sentences for conditioning.

We note, however, that the state-of-the-art performance ofthe supervised BERT and RoBERTa models is significantlyhigher, meaning there is room for improvement in developingcomparable zero-shot approaches to QA. However, one pointof interest is that the performance of training BERT with only5000 training examples (rather than the full 30k) is close(54%) to the performance of COMET - DynaGen, indicatingthat knowledge models and joint neuro-symbolic solutionsare already promising in low-data regimes.

Qualitative Analysis. In Table 1, we present top reasoningpaths from the graphs generated by COMET - DynaGen.The strength of our approach can be seen in the first example,where the correct answer, drained, is more likely to be a feel-ing associated with wanting “to go home," a post-condition inthe graph generated by COMET - DynaGen. In the originalcontext, this condition is implicit. This benefit to leveraginggraph reasoning is also seen in the second example, whereQuinn’s foolishness is linked to “[getting] hurt.” We notethat COMET - Direct, RoBERTa-large, and GPT2-345M allanswer this question incorrectly, reinforcing the importanceof explicit reasoning graphs.

In the final two examples, we present uninteresting orfailure cases. In the first, the model predicts that Alex willexperience joy after reasoning through the path that he will be

Algorithm # nodes # edges φ`a φ`amax

Argmax Decoding 10.6 26.4 50.1 49.6Beam Search - 5 43.2 156.8 49.5 49.1Beam Search - 10 83.0 316.2 50.0 49.1Top-5 sampling 32.0 111.9 49.0 49.0Top-10 sampling 59.9 223.8 49.3 49.4

Table 3: Development set accuracy for different graph con-struction techniques. The average number of nodes and edgesin the constructed graphs is presented.

“happy,” which, while correct, is merely leveraging synonymy.In the final example, we show a case where the model selectsan incorrect answer by reasoning through an incorrect path.By recognizing that “Taylor wants to celebrate” as a likelypost-condition of the context, the model selects an answerthat is incorrect. An interesting secondary failure mode in thisexample is in the second path through the inference “Taylorwants to be home.” While this path selects the correct answer,it would not be considered explanatory by humans. In general,we find these cases to be more common in multi-sentencesituations. The compositionality of the context makes it morechallenging to generate directed inferences, and the factornodes become less reliable in the graph. We observe thatperformance on multi-sentence contexts drops by ∼5%.

Graph Construction Algorithm. As the quality of the rea-soning paths is essential to our approach, we investigate theeffect of the inference generation algorithm. We evaluate thefollowing candidate generation algorithms: argmax decoding,beam search with beam size b = 5, 10 and top-k sampling(Fan, Lewis, and Dauphin 2018; Holtzman et al. 2018) with k= 5, 10. For each decoding method, we dynamically generatea graph using every candidate produced by the decoder (e.g.,argmax decoding produces one candidate, top-10 samplingproduces 10 candidates).

Our results in Table 3 show that the performance COMET- DynaGen is not dependent on the decoding strategy usedto dynamically generate the commonsense knowledge graph.This result is promising as it shows that the reasoning pro-cedure is robust to variability in the candidate generations(larger graphs will be less precise). However, it also showsthat the approach has difficulty using richer dynamically-generated commonsense knowledge representations to an-swer questions correctly. These results point to the needfor future work in developing algorithms that can aggregatelarger sets of commonsense inference paths as more expan-sive knowledge graphs are constructed using more powerfulknowledge models.

6 STORYCS StudyBaselines. As with SOCIALIQA, we report the results ofa random baseline, pretrained language models adapted tothe task, and a model that only uses φ0a to select answers(COMET - Direct). As supervised comparison models, wereport the performance of several BERT-based models fromGaonkar et al. (2020) that are state-of-the-art for the task.

Model P R F1Zero-shot CDF-weighted No Training DataRandom 20.6 20.8 20.7GPT 34.7 36.4 35.5GPT2 - 117M 30.8 31.8 31.3GPT2 - 345M 33.3 35.3 34.3GPT2 - 762M 35.5 37.4 36.4COMET - Direct 37.4 36.9 37.2COMET - DynaGen 38.9 39.3 39.1SupervisedBERT 65.6 56.9 61.0BERT + LE 63.1 61.7 62.4BERT + SS 57.9 76.4 65.9

Table 4: Precision, Recall, F1 on the STORYCS dataset. Bestmodels in different training settings are bolded

Overall Performance. Our results indicate that our zero-shotalgorithm, COMET - DynaGen, significantly outperformsother zero-shot baselines such as language models, includingmodels with twice the number of parameters. Importantly,again, we see consistent improvement from dynamically gen-erating a contextual commonsense knowledge graph, ratherthan directly evaluating the answer choices with COMET -Direct. Our full approach yields higher precision, recall, andF1, than the COMET - Direct baseline.

Qualitative Analysis. We once again see the benefit of gen-erating a reasoning graph in Table 5. COMET - DynaGen isable to select the two correct answers to “How does Danielfeel?” leveraging the path through the commonsense infer-ence that “His Dad is helpful” to predict that Daniel is trust-ing, and the path through the commonsense inference “Danielwants to try something new” to predict that Daniel is excited.However, there is still much room for improvement, as large-scale pretrained language models that are fine-tuned usingsupervised data perform considerably better on the task.

Few-shot Tuning. To evaluate the quality of our untunedthresholds from Section 4 based on the label distributionthreshold of the CDF of the model’s scores (CDF-label inTable 6), we also report the results of our approach usingdifferent strategies to set thresholds κ. First, we explore theimpact of tuning the κ thresholds on varying amounts of thedevelopment set data: 4 examples, 10 examples, 20 examples,and 20% of the development data (the same amount used forvalidation in Rashkin et al. 2018). In each of these settings,we run a study with 5 different randomly selected sets ofexamples, and report the average performance. We also reportthe performance of using the 50th percentile score of the CDFas the threshold (CDF-50). In Table 6, we observe large recallgains from these tuning strategies at the expense of precision.However, tuning using merely 10 examples achieves higherF1 than the default strategy, showing the potential of relaxingto a few-shot setting when limited examples are available.

Situation Most Contributing Paths in Graph AnswersDaniel was excited to get a remote control boat forhis birthday. He asked his dad to drive him to thelake to try it out.

His dad is helpful disgusted, angry, sad, afraid,happy, trusting 3, excited, surprised

Daniel wants to try something new disgusted, angry, sad, afraid,How does Daniel feel? happy, trusting , excited 3, surprised

Table 5: Example STORYCS context, high-scoring paths, and answers for our approach. We show which emotions are predictedthrough which path by bolding them. Correct answers are highlighted in blue 3. As in Table 1, only a subset of paths in thegraph generated by COMET - DynaGen are shown. Generated graphs for STORYCS have on average 8.8 nodes and 19.3 edges.

Model P R F1Zero-shot No Training DataCDF-label 39.5 39.5 39.5CDF-50 25.9 75.0 38.5

Few-shot TuningTuned from 4 examples 31.1 54.6 39.4Tuned from 10 examples 30.2 64.3 41.0Tuned from 20 examples 28.6 73.5 41.120% development tuning 31.2 65.1 42.2

Table 6: Development set Precision, Recall, and F1 of emo-tion prediction on the STORYCS dataset for different strate-gies for setting prediction thresholds.

7 Related WorkQuestion Answering with Knowledge Graphs Previouswork has explored integrating reasoning over static knowl-edge graphs for question answering and story understanding.In general, these approaches extract knowledge tuples fromthe static KG by linking canonicalized entities to nodes andperforming multi-hop inference along relation paths to formfull tuples that can be encoded by a downstream neural ar-chitecture (Mihaylov and Frank 2018; Bauer, Wang, andBansal 2018; Weissenborn, Kovcisk’y, and Dyer 2017; Linet al. 2019; Paul and Frank 2019). Similar to our approach ofdiscovering reasoning chains between contexts and answers,Paul and Frank (2019) extract reasoning paths in ConceptNetbetween normalized entities from the context answer candi-dates, but can only discover paths through nodes in the staticknowledge graph. Finally, there exists works that also dynam-ically construct latent knowledge graphs (Das et al. 2019;Bosselut et al. 2018), but these works presuppose a fixed setof entities that can be KG nodes and then approximate graphedges with neural transformations. In contrast, our algorithmcan generate arbitrary nodes, thereby constructing a uniquegraphical structure for any example.

Multi-hop Reading Comprehension Similar in spirit to rea-soning over knowledge graphs for question answering iswork in multi-hop reading comprehension. Many datasets forlearning to aggregate facts without graph structure have beenreleased in recent years (Weston et al. 2016; Welbl, Stene-torp, and Riedel 2018; Yang et al. 2018; Talmor and Berant2018). Approaches designed for these resources generally uselarge-scale neural networks to attend over supporting facts

across text (Zhong et al. 2019; Dhingra et al. 2018). Mostsimilar to our work are approaches that construct real-timeentity mention graphs as neural reasoning paths (Cao, Aziz,and Titov 2018; Jiang et al. 2019; Jiang and Bansal 2019;Fan et al. 2019). Our approach differs from these models inthat we generate relevant supporting information rather thanmining it from accompanying documents and conduct ourstudy in a zero-shot setting with no additional training.

Automatic Commonsense KG Construction Multi-hopreasoning over commonsense inferences requires construc-tion of knowledge resources and recent approaches haveinvestigated how to mine commonsense knowledge fromdeep learning models. Sap et al. (2019a) investigated whetherLSTM models could generate new tuples for the ATOMICknowledge graph. Similarly, Li et al. (2016) and Saito et al.(2018) explored whether neural models could be used tovalidate proposed knowledge rather than generating it. Jas-trzebski et al. (2018) built on these approaches for evaluat-ing novel commonsense knowledge mined from Wikipedia.More recent work mapped commonsense tuples to natural lan-guage with templates and used pretrained language modelsto validate them (Davison, Feldman, and Rush 2019; Petroniet al. 2019). Concurrently, other research has explored usingpretrained language models and adapting them as generativeknowledge graph constructors (Bosselut et al. 2019; Malaviyaet al. 2019). In contrast to these works that augment staticknowledge graphs, our approach focuses on constructingknowledge graphs on demand to provide context-dependentcommonsense for downstream inference.

8 ConclusionOur neuro-symbolic approach uses neural representationsof large-scale commonsense knowledge graphs (COMET)to generate contextual knowledge graphs on demand forzero-shot question answering. Our approach dynamicallyconstructs a knowledge graph of commonsense inferencesrelated to a presented context and uses it to evaluate answeroptions for a posed question. A novel inference algorithmreasons over the constructed graph to select the most likelyanswer to a question. Our approach shows promising resultsat answering questions without training on the end task ontwo datasets, SOCIALIQA and STORYCS, outperformingzero-shot pretrained language models. Finally, our analysisindicates that dynamically generating a contextualized com-monsense knowledge graph for inference performs betterthan using vanilla knowledge models (COMET - Direct) todirectly answer questions.

AcknowledgmentsWe thank Maarten Sap, Hannah Rashkin, Vered Shwartz, andChandra Bhagavatula for helpful feedback. This researchwas supported in part by NSF (IIS-1524371, IIS-1714566),DARPA under the CwC program through the ARO (W911NF-15-1- 0543), DARPA under the MCS program through NIWCPacific (N66001-19-2-4031), JD.com, and the Allen Institutefor AI (AI2).

ReferencesAmmanabrolu, P.; Cheung, W.; Broniec, W.; and Riedl, M. O.2020. Automated Storytelling via Causal, CommonsensePlot Ordering. arXiv preprint arXiv:2009.00829 .

Bauer, L.; Wang, Y.; and Bansal, M. 2018. Commonsensefor Generative Multi-Hop Question Answering Tasks. InEMNLP.

Bosselut, A.; Levy, O.; Holtzman, A.; Ennis, C.; Fox, D.; andChoi, Y. 2018. Simulating Action Dynamics with NeuralProcess Networks. In Proceedings of the 6th InternationalConference on Learning Representations.

Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Çelikyil-maz, A.; and Choi, Y. 2019. COMET: Commonsense Trans-formers for Automatic Knowledge Graph Construction. InProceedings of the 57th Annual Meeting of the Associationfor Computational Linguistics (ACL).

Cao, N. D.; Aziz, W.; and Titov, I. 2018. Question Answeringby Reasoning Across Documents with Graph ConvolutionalNetworks. In NAACL-HLT.

Chakrabarty, T.; Ghosh, D.; Muresan, S.; and Peng, N. 2020.Rˆ3: Reverse, Retrieve, and Rank for Sarcasm Generationwith Commonsense Knowledge. In Proceedings of the58th Annual Meeting of the Association for ComputationalLinguistics, 7976–7986. Online: Association for Computa-tional Linguistics. doi:10.18653/v1/2020.acl-main.711. URLhttps://www.aclweb.org/anthology/2020.acl-main.711.

Das, R.; Munkhdalai, T.; Yuan, X.; Trischler, A.; and Mc-Callum, A. 2019. Building Dynamic Knowledge Graphsfrom Text using Machine Reading Comprehension. In Pro-ceedings of the 7th International Conference on LearningRepresentations.

Davison, J.; Feldman, J.; and Rush, A. 2019. CommonsenseKnowledge Mining from Pretrained Models. In Proceedingsof the 2019 Conference on Empirical Methods in NaturalLanguage Processing and the 9th International Joint Con-ference on Natural Language Processing (EMNLP-IJCNLP),1173–1178. Hong Kong, China: Association for Compu-tational Linguistics. doi:10.18653/v1/D19-1109. URLhttps://www.aclweb.org/anthology/D19-1109.

Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.Bert: Pre-training of deep bidirectional transformers for lan-guage understanding. arXiv preprint arXiv:1810.04805 .

Dhingra, B.; Jin, Q.; Yang, Z.; Cohen, W. W.; and Salakhut-dinov, R. 2018. Neural Models for Reasoning over MultipleMentions Using Coreference. In NAACL-HLT.

Fan, A.; Gardent, C.; Braud, C.; and Bordes, A. 2019. Us-ing Local Knowledge Graph Construction to Scale Seq2SeqModels to Multi-Document Inputs. ArXiv abs/1910.08435.

Fan, A.; Lewis, M.; and Dauphin, Y. 2018. HierarchicalNeural Story Generation. In Proceedings of the 56th AnnualMeeting of the Association for Computational Linguistics(Volume 1: Long Papers), 889–898. Melbourne, Australia:Association for Computational Linguistics. doi:10.18653/v1/P18-1082. URL https://www.aclweb.org/anthology/P18-1082.

Gaonkar, R.; Kwon, H.; Bastan, M.; Balasubramanian, N.;and Chambers, N. 2020. Modeling Label Semantics forPredicting Emotional Reactions.

Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.;Bowman, S.; and Smith, N. A. 2018. Annotation Artifactsin Natural Language Inference Data. In Proceedings of the2018 Conference of the North American Chapter of the As-sociation for Computational Linguistics: Human LanguageTechnologies, Volume 2 (Short Papers), 107–112. New Or-leans, Louisiana: Association for Computational Linguistics.doi:10.18653/v1/N18-2017. URL https://www.aclweb.org/anthology/N18-2017.

Holtzman, A.; Buys, J.; Forbes, M.; Bosselut, A.; Golub,D.; and Choi, Y. 2018. Learning to Write with CooperativeDiscriminators. In Proceedings of the 56th Annual Meetingof the Association for Computational Linguistics (Volume 1:Long Papers), 1638–1649. Melbourne, Australia: Associationfor Computational Linguistics. doi:10.18653/v1/P18-1152.URL https://www.aclweb.org/anthology/P18-1152.

Jastrzebski, S.; Bahdanau, D.; Hosseini, S.; Noukhovitch,M.; Bengio, Y.; and Cheung, J. 2018. Commonsense miningas knowledge base completion? A study on the impact ofnovelty. In Proceedings of the Workshop on Generalizationin the Age of Deep Learning, 8–16. New Orleans, Louisiana:Association for Computational Linguistics. doi:10.18653/v1/W18-1002. URL https://www.aclweb.org/anthology/W18-1002.

Jiang, Y.; and Bansal, M. 2019. Self-Assembling Modu-lar Networks for Interpretable Multi-Hop Reasoning. InEMNLP.

Jiang, Y.; Joshi, N.; Chen, Y.-C.; and Bansal, M. 2019. Ex-plore, Propose, and Assemble: An Interpretable Model forMulti-Hop Reading Comprehension. In ACL.

Kearns, W. R.; Kaura, N.; Divina, M.; Vo, C. V.; Si, D.; Ward,T. M.; and Yuwen, W. 2020. A Wizard-of-Oz Interface andPersona-based Methodology for Collecting Health Counsel-ing Dialog. Extended Abstracts of the 2020 CHI Conferenceon Human Factors in Computing Systems .

Lewis, P.; Stenetorp, P.; and Riedel, S. 2020. Question andAnswer Test-Train Overlap in Open-Domain Question An-swering Datasets. ArXiv abs/2008.02637.

Li, L.; and Gauthier, J. 2017. Are Distributional Represen-tations Ready for the Real World? Evaluating Word Vectorsfor Grounded Perceptual Meaning. ArXiv abs/1705.11168.

Li, X.; Taheri, A.; Tu, L.; and Gimpel, K. 2016. Common-sense Knowledge Base Completion. In ACL, volume 1, 1445–1455.Lin, B. Y.; Chen, X.; Chen, J.; and Ren, X. 2019. KagNet:Knowledge-Aware Graph Networks for Commonsense Rea-soning. ArXiv abs/1909.02151.Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.;Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692 .Malaviya, C.; Bhagavatula, C.; Bosselut, A.; and Choi,Y. 2019. Exploiting Structural and Semantic Contextfor Commonsense Knowledge Base Completion. ArXivabs/1910.02915.Mihaylov, T.; and Frank, A. 2018. Knowledgeable Reader:Enhancing Cloze-Style Reading Comprehension with Exter-nal Commonsense Knowledge. In ACL.Paul, D.; and Frank, A. 2019. Ranking and Selecting Multi-Hop Knowledge Paths to Better Predict Human Needs. ArXivabs/1904.00676.Petroni, F.; Rocktäschel, T.; Riedel, S.; Lewis, P.; Bakhtin, A.;Wu, Y.; and Miller, A. 2019. Language Models as KnowledgeBases? In Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the 9th Inter-national Joint Conference on Natural Language Processing(EMNLP-IJCNLP), 2463–2473. Hong Kong, China: Associ-ation for Computational Linguistics. doi:10.18653/v1/D19-1250. URL https://www.aclweb.org/anthology/D19-1250.Plutchik, R. 1980. A general psychoevolutionary theory ofemotion. Theories of emotion 1(3-31): 4.Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I.2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language un-derstanding paper. pdf .Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; andSutskever, I. 2019. Language models are unsupervised multi-task learners. Unpublished manuscript.Rashkin, H.; Bosselut, A.; Sap, M.; Knight, K.; and Choi, Y.2018. Modeling Naive Psychology of Characters in SimpleCommonsense Stories. In ACL.Saito, I.; Nishida, K.; Asano, H.; and Tomita, J. 2018. Com-monsense Knowledge Base Completion and Generation. InProceedings of the 22nd Conference on Computational Natu-ral Language Learning, 141–150.Sap, M.; Le Bras, R.; Allaway, E.; Bhagavatula, C.; Lourie,N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y. 2019a.ATOMIC: An Atlas of Machine Commonsense for If-ThenReasoning. In AAAI.Sap, M.; Rashkin, H.; Chen, D.; Le Bras, R.; and Choi, Y.2019b. Social IQA: Commonsense Reasoning about SocialInteractions. ArXiv abs/1904.09728.Shwartz, V.; West, P.; Bras, R. L.; Bhagavatula, C.; and Choi,Y. 2020. Unsupervised Commonsense Question Answeringwith Self-Talk. In EMNLP.

Speer, R.; Chin, J.; and Havasi, C. 2017. ConceptNet 5.5:An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence.Talmor, A.; and Berant, J. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In Proceedingsof the 2018 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Lan-guage Technologies, Volume 1 (Long Papers), 641–651. NewOrleans, Louisiana: Association for Computational Linguis-tics. doi:10.18653/v1/N18-1059. URL https://www.aclweb.org/anthology/N18-1059.Weissenborn, D.; Kovcisk’y, T.; and Dyer, C. 2017. DynamicIntegration of Background Knowledge in Neural NLU Sys-tems. CoRR abs/1706.02596. URL http://arxiv.org/abs/1706.02596.Welbl, J.; Stenetorp, P.; and Riedel, S. 2018. ConstructingDatasets for Multi-hop Reading Comprehension Across Doc-uments. Transactions of the Association for ComputationalLinguistics 6.Weston, J.; Bordes, A.; Chopra, S.; and Mikolov, T. 2016.Towards AI-Complete Question Answering: A Set of Prereq-uisite Toy Tasks. In ICLR.Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.;Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA:A Dataset for Diverse, Explainable Multi-hop Question An-swering. In EMNLP.Zhong, V.; Xiong, C.; Keskar, N. S.; and Socher, R. 2019.Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering. In ICLR.

A Datasets and PreprocessingDatasetsDataset Statistics We report statistics of the SO-CIALIQA and STORYCS datasets in Table 7 below:

Dataset # dev # testSOCIALIQA 1952 2217STORYCS 202360 182160

Table 7: Dataset statistics for the SOCIALIQA and STO-RYCS datasets.

We use the original dataset splits proposed by the authors. Wefilter 2 and 7 examples from the SOCIALIQA developmentand test sets, respectively, that are spam.

ATOMIC and SOCIALIQA During its construction, SO-CIALIQA was seeded with ATOMIC triples during its cura-tion. We address whether this could be a potential sourceof bias that benefits the approaches based on COMET. Inour analysis, we find there is minimal opportunity for dataleakage between these resources.

First, the ATOMIC knowledge graph was designed with theidea in mind that it could be trained on using neural models totransfer learn knowledge from language. As a result, to eval-uate transfer in this setting, the knowledge graph is split intoa training, development, and test knowledge graph. Thesesplits were made adversarially, meaning no head entities inthe training knowledge graph are found in the evaluationknowledge graphs. The SOCIALIQA evaluation sets maintainthis split in their design (SOCIALIQA training set seededby ATOMIC training KG, etc.). As a result, no example inthe SOCIALIQA evaluation sets is derived from a tuple inthe ATOMIC training knowledge graph. Our COMET im-plementation is only trained on the training portion of theATOMIC knowledge graph, meaning our method does notlearn from any examples used to design the SOCIALIQAevaluation sets. In our work, we do not use any examplesfrom the SOCIALIQA training set.

Second, the SOCIALIQA authors state that crowdworkersheavily re-edited the ATOMIC triples to generate contexts,questions, and answers for each SOCIALIQA example. In anycase, to evaluate whether unintentional overlap could stillremain, we ran an analysis to recover close ATOMIC trainingtuples for each example in the SOCIALIQA development set.We removed stopwords from events, stemmed their tokens,and checked whether they could be recovered in the stemmedtokens of SOCIALIQA contexts. Among recovered ATOMICevents, we checked whether any of their associated tail enti-ties were present in the answer choices of the SOCIALIQAexample.

Using this matching scheme, we found an overlap foronly ∼1.7% of examples (34/1954 examples in the devel-opment set, largely from the fact that the stemming causescompression that makes events appear to be a subset of a SO-CIALIQA example). Furthermore, the COMET - DynaGenand COMET - Direct models would still perform better than

all baselines on the SOCIALIQA development set with theseexamples removed. Finally, we note that this level of leakagefalls far short of the 30% leakage identified in commonly-used QA datasets (Lewis, Stenetorp, and Riedel 2020).

B Additional Experimental SettingsApproximating the Marginal Distribution We approxi-mate the marginal distribution for the PMI calculation inEquation 5 using Equation 2, but set c = “PersonX”. Ev-ery training example in the ATOMIC knowledge graph onwhich COMET is trained begins with this token, so using itas the only token in the context essentially provides an outputdistribution that is only conditioned on the question q.

Generation Processing To ground the conditional distri-bution on which COMET and GPT2 (for baselines) weretrained we process the data and generations in the followingways:

• For language model baselines (i.e., the class of GPT2 mod-els), we adapt the QA task as natural language statementsto be evaluated by the language models. Question-answerpairs are automatically converted to a templated form.For example, a question such as "How does Alice feelafter?" will be replaced by the template “Alice feels" andprepended to the answer. The resulting snippet is then con-catenated to the context, and the language models scorethe answer words conditioned on the context and template.We record the perplexity of each statement and select thelower perplexity score as the answer. Table 9 provides thetemplate for each question variety.

• When converting generated inferences to contexts for an-swer scoring (Eq. 4), we add a prefix that is specific tothe inference type to the generated tokens (e.g., happy⇒Person is happy).

• We append the following prefixes to COMET-generatedinferences when using them in Equation 4 to computefactor nodes between them and answer nodes:

Relation PrefixxWant PersonX wantsxReact PersonX isxNeed PersonX needsxIntent PersonX wantsxAttr PersonX isxEffect PersonXoReact PersonX isoEffect PersonXoWant PersonX wants

Table 8: Prefixes appended to COMET-produced common-sense inferences for the evaluation step (Eq. 4)

• For the STORYCS dataset, when scoring the answer text,we use formulations of the words that make up the clas-sification label (e.g., disgust, surprise, fear, anger, trust,anticipation, sadness, joy⇒ disgusted, surprised, afraid,

Question TemplateWhat will happen to Others? The effect on others will be ___How would Others feel as a result? Others feel ___What will Others want to do next? After, others will want to ___How would you describe CHARACTER? CHARACTER is ___What will happen to CHARACTER? The effect on CHARACTER will be ___What does CHARACTER need to do before this? Before, CHARACTER needs to ___Why did CHARACTER do this? CHARACTER did this because ___How would CHARACTER feel afterwards? CHARACTER feels ___What will CHARACTER want to do next? After, CHARACTER will want to ___

Table 9: Templates used to convert question answering pairs from SOCIALIQA to a format that can be evaluated by the baselinepretrained language models: GPT, GPT2-117M, GPT2-345M, and GPT2-762M.

angry, trusting, excited, sad, happy). As question repre-sentations q to give to COMET, we use the relations fromATOMIC (Sap et al. 2019a) that correspond to reactions toevents: xReact and oReact. We compute φga (Eq. 4, 5)for each q and average them.

• For our main models and ablations, names that appear incontexts and answers are anonymized.

Rules for pruning generation sets We use the followingrules to prune the set of commonsense inferences generatedby COMET as it constructs a graph of commonsense knowl-edge:

1. Any generation that is “none" is pruned2. Any generation that is identical to a previous generation

from the same inputs, but has added punctuation is pruned(e.g., to go to the mall vs. to go to the mall.)

3. Any generation that has the phrase “PersonY" for thefollowing relations is removed: oEffect, oReact,oWant. These generations are untrustworthy as they areoften impossible to resolve with an actual person in thecontext.

4. Any generation for the following relations that doesnot have a token that is a verb is removed: xEffect,oEffect

5. In multiple candidate settings (i.e., beam search, top-ksampling), if one of the candidates is “none,” we prune allcandidates with less likely scores.

6. For the STORYCS dataset, we only generate inferencesalong the following ATOMIC relations: xReact, oReact,xEffect, oEffect, xIntent. The logic for pruningxWant, oWant, xNeed, xAttr inferences is that emo-tional reactions for these dimensions could be irrelevant tothe context. For example, the emotional reaction to gettinginto a car accident is different from needing to own a carto do this. Emotional reactions to the kept relations aremore likely to be relevant to the original context.

Prediction thresholds We set the following κ thresholdsto make positive predictions on the STORYCOMMONSENSEdataset.

Dimension CDF - labelDirect κ DynaGen κ

disgust 5.878 6.272surprise 4.790 5.452fear 6.504 6.640anger 3.773 4.093trust 8.064 8.126anticipation 3.765 4.008sadness 3.473 3.548joy 1.907 1.913

Table 10: Percentile thresholds κ for predicting an emotionfor the COMET - Direct and COMET - DynaGen models

Relation Description Example Completion:

Event: Person X puts Person X’s trust inPerson Y

oEffect The effect the event has on others be-sides Person X

is considered trustworthyis believedgains Person X’s loyalty

oReact The reaction of others besides PersonX to the event

trustedhonoredtrustworthy

oWant What others besides Person X maywant to do after the event

work with Person Xpartner with Person Xto help Person X

xAttr How Person X might be describedgiven their part in the event

faithfulhopefultrusting

xEffect The effect that the event would haveon Person X

gets relievedstays faithfulIs betrayed

xIntent The reason why X would cause theevent

to be trustinghis or her help/guidance/adviceto be friends

xNeed What Person X might need to do be-fore the event

to be friends with Person Yto have heard a lot of good things about Per-son Yto get to know Person Y

xReact The reaction that Person X wouldhave to the event

trustingsafe, not aloneunderstood

xWant What Person X may want to do afterthe event

to rely on Person Yto go into business with Person Yto make sure that their heart feeling is right

Table 11: Definitions of the relations in ATOMIC. Events in ATOMIC center around the personal situations of a central figure,Person X, with potentially more participants.