Click here to load reader

Semantic (or Thematic) Proto-Roles Drew Reisinger, Rachel Rudinger, Frank Ferraro, Craig Harman, Aaron White, Kyle Rawlins, and Benjamin Van Durme

  • View
    216

  • Download
    1

Embed Size (px)

Text of Semantic (or Thematic) Proto-Roles Drew Reisinger, Rachel Rudinger, Frank Ferraro, Craig Harman,...

PowerPoint Presentation

Semantic (or Thematic)Proto-RolesDrew Reisinger, Rachel Rudinger, Frank Ferraro, Craig Harman, Aaron White, Kyle Rawlins, and Benjamin Van Durme1Talk OutlineOverview of thematic/semantic roles

Dowty (1991)s proto-roles

New: A crowdsourced proto-role corpus

Future workIll start with a brief overview of thematic roles and their role in argument realization, and then Ill present Dowtys proto-role theory. After that bit of background, Ill move on to our contribution, a new corpus of proto-role property judgments, and present some of the results of this data collection effort.2Talk OutlineOverview of thematic/semantic roles

Dowty (1991)s proto-roles

New: A crowdsourced proto-role corpus

Future workFirst, let me briefly explain my possibly confusing use of the words semantic and thematic in the title of this talk.3Thematic roles (linguistics)Semantic roles (comp. ling.)Verbal meanings consist of information like:Who did what to who(m)?What happened to which individual?Thematic roles fill in the who (and some of the what) Agent, Patient, ThemeFillmore 1968; Gruber 1965; Jackendoff 1972, 1976My use of both of these terms is meant to allude to a rough terminological divide between linguistics on one hand and computational linguistics/NLP on the other. On this slide, I have what is essentially the lowest-common denominator of how linguists and NLP researchers think of thematic or semantic roles: the aspects of verbal (or predicative) meaning that specify information like who did what to whom.4Generalized Thematic Roles Lexical entry

lemma: /ht/category:V

role-list:AGENTTHEMEINSTRUMENTDowty 1991; Schlesinger 1995; Van Valin 1990, 1999; Croft 1998However, once we try to be more precise about what these roles are, we see that linguistics and NLP think of these roles in rather different ways. In linguistics, one common way of thinking of thematic roles, which Ill focus on today, is as the part of a verbs lexical semantic representation that determines how the verbs semantic arguments are realized in the syntax.5Generalized Thematic Roles Lexical entry

lemma: /ht/category:V

role-list:AGENTTHEMEINSTRUMENTLINKING RULESDowty 1991; Schlesinger 1995; Van Valin 1990, 1999; Croft 1998However, once we try to be more precise about what these roles are, we see that linguistics and NLP think of these roles in rather different ways. In linguistics, one common way of thinking of thematic roles, which Ill focus on today, is as the part of a verbs lexical semantic representation that determines how the verbs semantic arguments are realized in the syntax.

6Generalized Thematic Roles Lexical entry

lemma: /ht/category:V

role-list:AGENTTHEMEINSTRUMENTLINKING RULESSUBJ hit DOBJ with OBLQDowty 1991; Schlesinger 1995; Van Valin 1990, 1999; Croft 1998However, once we try to be more precise about what these roles are, we see that linguistics and NLP think of these roles in rather different ways. In linguistics, one common way of thinking of thematic roles, which Ill focus on today, is as the part of a verbs lexical semantic representation that determines how the verbs semantic arguments are realized in the syntax.

7PropBank (comp. ling.)

The Proposition Bank: An annotated corpus of semantic roles. Palmer, Gildea, and Kingsbury. Computational Linguistics 31.1 (2005): 71-106.One such corpus and annotation scheme is PropBank, which well need to know for later. PropBank consists of lots of sentences like these, in which spans of text are associated with labels of the form Arg0, Arg1, and so on. The semantics of these Arg labels is, in theory, verb-dependent, but the annotators clearly intended Arg0 to be something like Agent and Arg1 to be something like Theme or Patient, and other role annotation schemes like VerbNet and FrameNet have roles that generalize across verbs.8Role FragmentationWho did what to who(m)?What happened to which individual?

Agent, PatientDowty 1991Despite these differences in perspective and purpose, roles in both fields run into some of the same issues. One issue arises when we try to agree on an exhaustive list of roles. Obviously, we need Agent and Patient9Role FragmentationWho did what to who(m)?What happened to which individual?

Agent, Patient, Theme, BeneficiaryDowty 1991 and Theme and Beneficiary seem pretty linguistically relevant 10Role FragmentationWho did what to who(m)?What happened to which individual?

Agent, Patient, Theme, Beneficiary, Actor, Instrument, Co-Patient, ValueDowty 1991 and where would be be without such crucial roles as Co-Patient and Value 11Role FragmentationWho did what to who(m)?What happened to which individual?

Agent, Patient, Theme, Beneficiary, Actor, Instrument, Co-Patient, Value,Item, Speaker, Difference, Message,Goods, Addressee, Sender, Donor, Seller,Cognizer, Co-Theme, Experiencer, Buyer,Dowty 1991; Baker, Fillmore & Lowe 1998 (FrameNet) and then we end up with this actual, still incomplete list of roles, which I believe comes from FrameNet.12Talk OutlineOverview of thematic/semantic roles

Dowty (1991)s proto-roles

New: A crowdsourced proto-role corpus

Future work13 then along came DowtyThematic proto-roles and argument selection. David Dowty. Language. 1991.

So many roles!Dowty identified role fragmentation as a problem, among other reasons because it preempts broad generalization about the syntax-semantics interface.14Dowty (1991)for [roles to have] explicit semantic content, the meanings of all natural-language predicates must permit us to assign the argument to some official thematic role or other it cannot fall in the cracks between roles

He also brought up the problem of role specification, the fact that its really hard to come up with necessary and sufficient conditions that precisely define which arguments belong to a particular role.15Dowty (1991)This is a very strong empirical claim and as soon as we try to be precise about exactly what Agent, Patient, etc., mean, it is all to subject to difficulties and apparent counterexamplesDowty (1991)we may have have a hard time pinning down the traditional role type because role types are simply not discrete categories at allIn fact, he argues that rather than trying to tackle this role specification problem head-on, we should go back and rethink what the logical type of thematic roles actually is.17Roles = property configurationsDowty argued for the notion of: proto-Agent and proto-Patient

Verb arguments only tend to have certain basic properties, and these correlate in Agent/Patient like ways

Arguments with more Agent properties tend to be SUBJECT, those with more Patient properties, OBJECTInstead, he argues for proto-roles. Each role is associated with a set of properties that can be thought of as describing what, say, a prototypical Agent looks like, and any actual verbal argument can be more or less Agent-like based on how many of these proto-Agent properties it satisfies. In addition, he claims that you only need two proto-roles, Agent and Patient, to explain basic argument realization.18More of a bridge might be helpful here. Something like: what are the ways in which the things in the previous few slides need improvement? Some ideas: * Problems on the theoretical side with generalized roles (led to PropBank Arg0, Arg1) * Useful to enrich the "what" aspect of role labels * Maybe something about the challenges you had in the project with meg? I.e. making it easier to rapidly get dataDowtys Properties

Here are Dowtys original properties for Agent and Patient. He doesnt try to claim this list is definitive, but it is well motivated, and subsequent work doesnt stray too far from these.19Argument Selection PrincipleIn predicates with grammatical subject and object, the argument for which the predicate entails the greatest number of Proto-Agent properties will be lexicalized as the subject of the predicate; the argument having the greatest number of Proto-Patient entailments will be lexicalized as the direct object.Dowty (1991), p. 576

Argument Selection PrincipleSUBJhitDOBJwith OBLQ

HITTERHITTEEHIT-WITHSYNTAXSEMANTICSHeres a sketch of the linking rule that he proposes. We have the same event participants for the verb to hit21Argument Selection PrincipleSUBJhitDOBJwith OBLQ

HITTERHITTEEHIT-WITHSYNTAXSEMANTICScauses:+exists:+volitional:-stationary:-causes:-exists:+volitional:-stationary:+causes:-exists:-volitional:-stationary:-But now, each participant is associated with a set of proto-Agent and proto-Patient entailments. It is worth noting that no argument actually gets associated with labels proto-Agent and proto-Patient, and no rules in the grammar refer to such labels.22Argument Selection PrincipleSUBJhitDOBJwith OBLQ

HITTERHITTEEHIT-WITHSYNTAXSEMANTICSp-Agent:3p-Patient:0p-Agent:1p-Patient:1p-Agent:1p-Patient:0For the purposes of linking, all that matters are the counts of proto-Agent and Patient properties for each argument. Lets say these are the counts for to hits arguments.23Argument Selection PrincipleSUBJhitDOBJwith OBLQ

HITTERHITTEEHIT-WITHSYNTAXSEMANTICSp-Agent:3p-Patient:0p-Agent:1p-Patient:1p-Agent:1p-Patient:0The argument with the most proto-Agent properties becomes the subject, the most proto-Patient properties the direct object, and anything else is again relegated to oblique positions.24Kako (2006)Do normal people (student subjects) have stable judgments akin to Dowtys?

Experiment with simple sentences,using nonce argumentsThematic role properties of subjects and objects. Kako. Cognition 101.1 (2006): 1-42.Thats the gist of Dowtys proposal. MOTIVATE KAKO. Edward Kako made an effort to validate Dowtys proposal on the psycholinguistics side by collecting property judgments on simple transitive sentences with nonce arguments.25Kako (2006)Do normal people (student subjects) have stable judgments akin to Dowtys?

The rom found the zarg.How likely is it that the rom chose to be involved in finding?How likely is it that the rom moved?Thematic role properties of subjects and objects. Kako. Cognition 101.1 (2006): 1-42.A subject would see a sentence like the rom found the zarg and would be asked about various Dowty-inspired properties of the rom and zarg in this sentence.26Kako (2006)Do normal people (student subjects) have stable judgments akin to Dowtys?

(subj. rating obj. rating) measure of association between property and proto-AgentThematic role properties of subjects and objects. Kako. Cognition 101.1 (2006): 1-42.A subject would see a sentence like the rom found the zarg and would be asked about various Dowty-inspired properties of the rom and zarg in this sentence.27

Kakos FindingsIf we look at the average difference in 7-point Likert scale property ratings between the subjects and objects of these sentences, the general take-home is that the association between proto-role properties and grammatical function is quite consistent with Dowtys proposal.28Talk OutlineOverview of thematic/semantic roles

Dowty (1991)s proto-roles

New: A crowdsourced proto-role corpus

Future work29For details, see

Semantic proto-roles.Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, Benjamin Van Durme. Transactions of the Association for Computational Linguistics 3 (2015): 475-488.The neeglur .killedthe bogrubFor :the bogrub- How likely or unlikely is it that was/were altered or somehow changed during or by the end of the ?the bogrubkillingveryunlikelysomewhatunlikelysomewhatlikelyverylikelynot enoughinformationFirst, we ported Kakos nonce argument experiment to Amazon Mechanical Turk, the online crowdsourcing platform. A Turker would see an interface similar to this: a sentence, with the verb and one of its argument highlighted, and then a series of How likely questions about properties of the highlighted argument.31The neeglur .killedthe bogrubFor :the bogrub- How likely or unlikely is it that was/were altered or somehow changed during or by the end of the ?the bogrubkillingveryunlikelysomewhatunlikelysomewhatlikelyverylikelynot enoughinformation12345Responses were on a five-point Likert scale.32How likely or unlikely is it that

Arg caused Pred to happen?Arg chose to be involved in the Pred?Arg was/were aware of being involved in the Pred?Arg was sentient?Arg changes location during Pred?Arg existed as a physical object?Arg existed before the Pred began?Arg existed during the Pred?Arg existed after the Pred stopped?Arg changed possession during the Pred?The Arg was/were altered or somehow changed during or by the end of the Pred?Arg was stationary during the Pred?Here are the question templates we used; they correspond pretty closely with the original Dowty questions.33How likely or unlikely is it that

Arg caused Pred to happen?Arg chose to be involved in the Pred?Arg was/were aware of being involved in the Pred?Arg was sentient?Arg changes location during Pred?Arg existed as a physical object?Arg existed before the Pred began?Arg existed during the Pred?Arg existed after the Pred stopped?Arg changed possession during the Pred?The Arg was/were altered or somehow changed during or by the end of the Pred?Arg was stationary during the Pred?InstigatedIn the results, well refer to these questions by one- or two-word shorthands, like instigated34How likely or unlikely is it that

Arg caused Pred to happen?Arg chose to be involved in the Pred?Arg was/were aware of being involved in the Pred?Arg was sentient?Arg changes location during Pred?Arg existed as a physical object?Arg existed before the Pred began?Arg existed during the Pred?Arg existed after the Pred stopped?Arg changed possession during the Pred?The Arg was/were altered or somehow changed during or by the end of the Pred?Arg was stationary during the Pred?VolitionalHow likely or unlikely is it that

Arg caused Pred to happen?Arg chose to be involved in the Pred?Arg was/were aware of being involved in the Pred?Arg was sentient?Arg changes location during Pred?Arg existed as a physical object?Arg existed before the Pred began?Arg existed during the Pred?Arg existed after the Pred stopped?Arg changed possession during the Pred?The Arg was/were altered or somehow changed during or by the end of the Pred?Arg was stationary during the Pred?Moved

Kako (2006)lab setting,nonce sentencesJHU (2015)crowd sourced,nonce sentences

And here are the results of these nonce argument judgments compared to Kakos original judgments. Theyre a little bit noisier, as we might expect from Mechanical Turk, but the proto-Patient and especially the proto-Agent properties line up with grammatical function in the way that Dowty would expect.37Lets Build a Corpus!Why? Lets Build a CorpusWhy?

Extending Dowty requires broad data e.g. oblique arguments, alt. linking rules

of Naturalistic DataDowty concerned with verbal entailmentse.g. If x is a KILLER, then x is volitionally involved in the event

Our data: entailments of particular arguments in contextMechanical Turk

Emboldened by these results, we might try to extend this protocol to actual sentences, like this one.41Why?Possible to factor out argument entailmentsWhy?Possible to factor out argument entailments

Expose counterexamples to default inferencese.g. Mary accidentally killed her pet fish.

Why?Possible to factor out argument entailments

Expose counterexamples to default inferencese.g. Mary accidentally killed her pet fish.

Some morphosyntactic realizations depend on argument properties, e.g. DOM (Aissen 2003; Bossong 1991, 1998)(re-)Annotate PropBank~350 hours of annotator time

~10,000 unique arguments labeled

@ http://decomp.netIn fact, we did this. We took the sentences annotated with verb and argument spans in PropBank and used MTurk to re-annotate those argument spans with proto-role property judgments. This is a pretty substantial corpus, and were very happy to share it with you all at this address: decomp dot net. Please check it out!45

Kako (2006)Small scale,nonce sentencesJHU (2015)Large scale,real sentencesIt turns out that even though were getting judgments on real, contentful arguments rather than nonces, and even though our sentences are pretty syntactically diverse and contain a wide range of verbs, the general pattern from Kakos carefully controlled experiments still holds remarkably well.46There now exists corpus-based evidence in support of Dowtys Proto-Role hypothesis

First big empirical takeaway: corpus-based evidence for proto-roles!47Roles

Each configuration of 11 responses = one role

~10,000 arguments labeled leads to ~800 unique roles

At least 100 of these configurations appear at least 10 timesHeres the actual distribution of these roles in our data. You can see that if we really take roles to be disjoint sets of arguments, we need a lot of roles to cover the 10,000ish arguments in our data. Even if you assume that some of these roles out in the long tail are really just annotation noise, there are still over 100 roles that are pretty robustly attested. At least for the properties were interested in, it makes a lot more sense to decompose these roles into their parts than to treat them as atomic.48

Entailment Corner Cases

Entailment Corner CasesTypical killer: volitional, aware, sentient

Entailment Corner CasesAccidental killer: not volitional or aware

Entailment Corner CasesAtypical killer: not volitional, aware, sentient

Entailment Corner CasesAtypical killer: not volitional, aware, sentientEven independent existence might fail!Verbal EntailmentsThe kill example shows how argument entailments constrain verbal entailmentsVerbal EntailmentsThe kill example shows how argument entailments constrain verbal entailments

We factor out individual argument effects to estimate general property ratings for verbsArgument SelectionQuantify how well a verb conforms to Dowty (1991)s Argument Selection principle with the following score:

AgtSUBJ AgtOBJ + PatOBJ PatSUBJ

A verb is consistent with Dowty (1991) if the score is positiveArgument Selection

Argument Selection

Which verbs aredown here?Verbs with Negative Scoresaccelerateadorn anger appease assemble beef bill blow bolster brave call call calm captain catch chain clear comprise concern confirm cover crop define detect disappointdistort disturb double dust elevate embarrass employ employ exceed exclude feature feed fill flatten follow force free freeze fuel galvanize halt hamstring haul haunt hire house hurt ignore illustrate

impress include inhibit involve justify keep lag last leave limit list lock merit mimic misstate move name outnumber

outpace outstrip phone pit poll preserve protect pursue puzzle rattle recover regain remember renew repel represent review rivet

scandalizescare sense set settle shake shield shock shroud shrug sign soil stun suggest surprise survey swamp swell

take target thrust top touch trouble turn underscoreunmask unnerve vent waste wed worry wreck yield

Talk OutlineOverview of thematic/semantic roles

Dowty (1991)s proto-roles

New: A crowdsourced proto-role corpus

Future work60Future WorkMorphosyntax: verifying and extending Grimm (2011)

JHU Decompositional Semantics Initiative

Applications to MorphosyntaxGrimm (2011): Each case corresponds to a connected region of the lattice of proto-role property configurations

Can we predict case alternations? Cross-linguistic case usage? Semantics of case. Grimm. Morphology 21 (2011): 515-544.The Johns HopkinsDecompositional Semantics InitiativeSemantic Proto-Role Labeling (systems)

Nominal semantics (factored word sense, )

Verbal semantics (general entailments)

Constraints on lexical representation learning

Connections to: Common SenseAcknowledgementsDARPA LORELEI BAA-15-04 (Low resource event understanding)

NSF BCS-1344269 (Gradient Symbolic Computation)

JHU Science of Learning Institute

Benjamin Van DurmeDrew Reisinger

Kyle Rawlins

Rachel Rudinger

Frank Ferraro

Craig HarmanQuestions?http://decomp.net

Aaron White